Amazon MSK quota
Amazon MSK quota
-
Up to 90 brokers per account and 30 brokers per cluster. To request higher quota, go to the Amazon console Support Center and create a support case
. -
A minimum of 1 GiB of storage per broker.
-
A maximum of 16384 GiB of storage per broker.
-
A cluster that uses IAM access control can have up to 3000 TCP connections per broker at any given time. To increase this limit, you can adjust the
listener.name.client_iam.max.connections
or thelistener.name.client_iam_public.max.connections
configuration property using the Kafka AlterConfig API or thekafka-configs.sh
tool. It's important to note that increasing either property to a high value can result in unavailability. -
Limits on TCP connections. With connection rate bursts enabled, MSK allows 100 connections per second. The exception is the kafka.t3.small instance type, which is allowed 4 connections per second with connection rate bursts enabled. Older clusters that don't have connection rate bursts enabled will have the feature automatically enabled when the cluster is patched.
To handle retries on failed connections, you can set the
reconnect.backoff.ms
configuration parameter on the client side. For example, if you want a client to retry connections after 1 second, setreconnect.backoff.ms
to 1000. For more information, see reconnect.backoff.msin the Apache Kafka documentation. -
Up to 100 configurations per account. To request a quota adjustment, go to the Amazon console Support Center and create a support case
. -
A maximum of 50 revisions per configuration.
-
To update the configuration or the Apache Kafka version of an MSK cluster, first ensure the number of partitions per broker is under the limits described in Right-size your cluster: Number of partitions per broker.
MSK Replicator quotas
A maximum of 15 MSK Replicators per account.
-
MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the Amazon console Support Center and create a support case
if you need support for more than 750 topics per Replicator. You can monitor the number of topics being replicated using the "TopicCount" metric. -
A maximum ingress throughput of 1GB per second per MSK Replicator. To request higher quota, go to the Amazon console Support Center and create a support case
. -
MSK Replicator Record Size - A maximum of 1MB record size (message.max.bytes). To request higher quota, go to the Amazon console Support Center and create a support case
.
MSK Serverless quota
Note
If you experience any issues with quota limits, contact Amazon Support by creating a support
case
Limits are per cluster, unless otherwise stated. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Dimension | Quota | Quota violation result | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum ingress throughput |
200 MBps |
Slowdown with throttle duration in response |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum egress throughput | 400 MBps |
Slowdown with throttle duration in response |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum retention duration | Unlimited | N/A | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of client connections |
3000 | Connection close | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum connection attempts | 100 per second | Connection close | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum message size |
8 MB | Request fails with ErrorCode: INVALID_REQUEST | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum request rate | 15,000 per second | Slowdown with throttle duration in response | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum rate of topic management APIs requests rate |
2 per second | Slowdown with throttle duration in response | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum fetch bytes per request |
55 MB |
Request fails with ErrorCode: INVALID_REQUEST | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of consumer groups |
500 | JoinGroup request fails | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of partitions (leaders) | 2400 for non-compacted topics. 120 for compacted topics. To request a
quota adjustment, go to the Amazon console Support Center and create a support
case |
Request fails with ErrorCode: INVALID_REQUEST | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum rate of partition creation and deletion | 250 in 5 minutes | Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum ingress throughput per partition |
5 MBps | Slowdown with throttle duration in response | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum egress throughput per partition |
10 MBps | Slowdown with throttle duration in response | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum partition size (for compacted topics) |
250 GB | Request fails with ErrorCode: THROUGHPUT_QUOTA_EXCEEDED | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of client VPCs per serverless cluster | 5 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of serverless clusters per account | 10. To request a quota adjustment, go to the Amazon console Support
Center and create a support
case |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of MSK Replicators per account | 15 replicators | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum number of topics supported by metrics per MSK Replicator | MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the Amazon console Support Center and create a
support case |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Maximum ingress throughput per MSK Replicator | 1 GB per second per MSK Replicator. To request a quota adjustment, go
to the Amazon console Support Center and create a support
case |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MSK Replicator record size | A maximum of 1MB record size (message.max.bytes). To request higher
quota, go to the Amazon console Support Center and create a support
case |
MSK Connect quota
Up to 100 custom plugins.
-
Up to 100 worker configurations.
-
Up to 60 connect workers. If a connector is set up to have auto scaled capacity, then the maximum number of workers that the connector is set up to have is the number MSK Connect uses to calculate the quota for the account.
-
Up to 10 workers per connector.
To request higher quota for MSK Connect, go
to the Amazon console Support Center and
create a support case