Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting throttling issues in Amazon DynamoDB

In service-oriented architectures and distributed systems, limiting the rate at which API calls are processed by various service components is called throttling. This helps to smooths spikes, control for mismatches in component throughput, and allows for more predictable recoveries when there's an unexpected operational event. DynamoDB is designed for these types of architectures, and DynamoDB clients have retries built in for throttled requests. Some degree of throttling is not necessarily a problem for your application, but persistent throttling of a latency-sensitive part of your data workflow can negatively impact user experience and reduce the overall efficiency of the system.

If your read or write operations on your DynamoDB table are being throttled (or if you are getting ProvisionedThroughputExceededException when you perform an operation), consider using the following strategies to help resolve the issue.

Make sure provisioned mode tables have adequate capacity

DynamoDB reports minute-level metrics to CloudWatch. The metrics are calculated as the sum for a minute and then averaged. However, the DynamoDB rate limits are applied per second. For example, if you provisioned 60 write capacity units for your table, you can perform 3600 write units in one minute, but trying to perform more than 60 write units in any one second might result in some requests being throttled.

To resolve this issue, make sure that your table has enough capacity to serve your traffic and retry throttled requests using exponential backoff. If you are using an Amazon SDK, then this logic is implemented by default. For this reason, throttles might occur, and be retried successfully on the second or third retry, while never causing errors in your code. In order to understand throttles as they happen, each AWS-SDK flavor has a way to detect throttles. For example, in Python check the number in the returned ResponseMetadata.RetryAttempts field. If this is greater than zero, it indicates there were throttles. A great strategy upon detecting this is to log the Partition Key involved, as it may be a hot key pattern that would be hard to diagnose with CloudWatch alone.

Also, consider using auto scaling on your provisioned tables to automatically adjust provisioned capacity based on workload. You should also consider updating the scaling policy to have a higher minimum or a lower target utilization, and make sure the maximum setting isn't set too low.

Consider switching to on-demand mode

Amazon DynamoDB auto scaling uses the Application Auto Scaling service to dynamically adjust provisioned throughput capacity in response to actual traffic patterns. Application Auto Scaling initiates a scale up only when two consecutive data points for consumed capacity units exceed the configured target utilization value within a one-minute span. Application Auto Scaling automatically scales the provisioned capacity only when the consumed capacity is higher than target utilization for two consecutive minutes. Also, a scale-down event is initiated when 15 consecutive data points for consumed capacity in CloudWatch are lower than the target utilization.

After Application Auto Scaling is initiated, an UpdateTable API call is invoked to update the provisioned capacity for your DynamoDB table or index. Application Auto Scaling requires consecutive data points with higher target utilization values to scale up the provisioned capacity of the DynamoDB table. During this period, any requests that exceed the provisioned capacity of the table are throttled. Therefore, consider using On-demand mode for extremely unpredictable (bursty) traffic patterns. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

Distribute read and write operations evenly between partition keys

In DynamoDB, a partition key with low cardinality can result in many requests targeting only a few partitions and resulting in a hot partition. A hot partition can cause throttling if the partition limits of 3000 RCU and 1000 WCU per second are exceeded.

To find the most accessed and throttled items in your table, use Amazon CloudWatch Contributor Insights. Amazon CloudWatch Contributor Insights is a diagnostic tool that provides a summarized view of your DynamoDB tables traffic trends and helps you identify the most frequently accessed partition keys. With this tool, you can continuously monitor the graphs for your table’s item access patterns. A hot partition can degrade the overall performance of your table. To avoid this poor performance, distribute the read and write operations as evenly as possible by choosing a high cardinality key. For more information, see Designing partition keys to distribute your workload and Choosing the right DynamoDB partition key.

Increase your table-level read or write throughput quota

The table-level read throughput and table-level write throughput quotas apply at the account level in any Region. These quotas apply for tables with both provisioned capacity mode and on-demand capacity mode. By default, the throughput quota placed on your table is 40,000 read requests units and 40,000 write requests units. If the traffic to your table exceeds this quota, then the table might be throttled. For more information on how to prevent this from happening, see Monitoring Amazon DynamoDB for operational awareness.

To resolve this issue, use the Service Quotas console to increase the table-level read or write throughput quota for your account.