Amazon CLI retries - Amazon Command Line Interface
Available retry modesConfiguring a retry modeViewing logs of retry attempts
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

This documentation is for Version 1 of the Amazon CLI only. For documentation related to Version 2 of the Amazon CLI, see the Version 2 User Guide.

Amazon CLI retries

This topic describes how the Amazon CLI might see calls to Amazon services fail due to unexpected issues. These issues can occur on the server side or might fail due to rate limiting from the Amazon service you're attempting to call. These kinds of failures usually don’t require special handling and the call is automatically made again, often after a brief waiting period. The Amazon CLI provides many features to assist in retrying client calls to Amazon services when these kinds of errors or exceptions are experienced.

Available retry modes

The Amazon CLI has multiple modes to choose from depending on your version:

Legacy retry mode

Legacy mode is the default mode used by the Amazon CLI version 1. Legacy mode uses an older retry handler that has limited functionality which includes:

  • A default value of 4 for maximum retry attempts, making a total of 5 call attempts. This value can be overwritten through the max_attempts configuration parameter.

  • DynamoDB has a default value of 9 for maximum retry attempts, making a total of 10 call attempts. This value can be overwritten through the max_attempts configuration parameter.

  • Retry attempts for the following limited number of errors/exceptions:

    • General socket/connection errors:

      • ConnectionError

      • ConnectionClosedError

      • ReadTimeoutError

      • EndpointConnectionError

    • Service-side throttling/limit errors and exceptions:

      • Throttling

      • ThrottlingException

      • ThrottledException

      • RequestThrottledException

      • ProvisionedThroughputExceededException

  • Retry attempts on several HTTP status codes, including 429, 500, 502, 503, 504, and 509.

  • Any retry attempt will include an exponential backoff by a base factor of 2.

Standard retry mode

Standard mode is a standard set of retry rules across the Amazon SDKs with more functionality than legacy. Standard mode was created for the Amazon CLI version 2 and is backported to Amazon CLI version 1. Standard mode’s functionality includes:

  • A default value of 2 for maximum retry attempts, making a total of 3 call attempts. This value can be overwritten through the max_attempts configuration parameter.

  • Retry attempts for the following expanded list of errors/exceptions:

    • Transient errors/exceptions

      • RequestTimeout

      • RequestTimeoutException

      • PriorRequestNotComplete

      • ConnectionError

      • HTTPClientError

    • Service-side throttling/limit errors and exceptions:

      • Throttling

      • ThrottlingException

      • ThrottledException

      • RequestThrottledException

      • TooManyRequestsException

      • ProvisionedThroughputExceededException

      • TransactionInProgressException

      • RequestLimitExceeded

      • BandwidthLimitExceeded

      • LimitExceededException

      • RequestThrottled

      • SlowDown

      • EC2ThrottledException

  • Retry attempts on nondescriptive, transient error codes. Specifically, these HTTP status codes: 500, 502, 503, 504.

  • Any retry attempt will include an exponential backoff by a base factor of 2 for a maximum backoff time of 20 seconds.

Adaptive retry mode

Warning

Adaptive mode is an experimental mode and is subject to change, both in features and behavior.

Adaptive retry mode is an experimental retry mode that includes all the features of standard mode. In addition to the standard mode features, adaptive mode also introduces client-side rate limiting through the use of a token bucket and rate-limit variables that are dynamically updated with each retry attempt. This mode offers flexibility in client-side retries that adapts to the error/exception state response from an Amazon service.

With each new retry attempt, adaptive mode modifies the rate-limit variables based on the error, exception, or HTTP status code presented in the response from the Amazon service. These rate-limit variables are then used to calculate a new call rate for the client. Each exception/error or non-success HTTP response (provided in the list above) from an Amazon service updates the rate-limit variables as retries occur until success is reached, the token bucket is exhausted, or the configured maximum attempts value is reached.

Configuring a retry mode

The Amazon CLI includes a variety of both retry configurations as well as configuration methods to consider when creating your client object.

Available configuration methods

In the Amazon CLI, users can configure retries in the following ways:

  • Environment variables

  • Amazon CLI configuration file

Users can customize the following retry options:

  • Retry mode - Specifies which retry mode the Amazon CLI uses. As described previously, there are three retry modes available: legacy, standard, and adaptive. The default value for the Amazon CLI version 1 is legacy.

  • Max attempts - Specifies the value of maximum retry attempts the Amazon CLI retry handler uses, where the initial call counts toward the value that you provide. The default value is 5.

Defining a retry configuration in your environment variables

To define your retry configuration for the Amazon CLI, update your operating system's environment variables.

The retry environment variables are:

  • AWS_RETRY_MODE

  • AWS_MAX_ATTEMPTS

For more information on environment variables, see Environment variables to configure the Amazon CLI.

Viewing logs of retry attempts

The Amazon CLI uses Boto3's retry methodology and logging. You can use the --debug option on any command to receive debug logs. For more information on how to use the --debug option, see Command line options.

If you search for "retry" in your debug logs, you'll find the retry information you need. The client log entries for retry attempts depend on which retry mode you’ve enabled.

Legacy mode:

Retry messages are generated by botocore.retryhandler. You’ll see one of three messages:

  • No retry needed

  • Retry needed, action of: <action_name>

  • Reached the maximum number of retry attempts: <attempt_number>

Standard or adaptive mode:

Retry messages are generated by botocore.retries.standard. You’ll see one of three messages:

  • No retrying request

  • Retry needed, retrying request after delay of: <delay_value>

  • Retry needed but retry quota reached, not retrying request

For the full definition file of botocore retries, see _retry.json on the botocore GitHub Repository.