Use Lambda recursive loop detection to prevent infinite loops - Amazon Lambda
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Use Lambda recursive loop detection to prevent infinite loops

When you configure a Lambda function to output to the same service or resource that invokes the function, it's possible to create an infinite recursive loop. For example, a Lambda function might write a message to an Amazon Simple Queue Service (Amazon SQS) queue, which then invokes the same function. This invocation causes the function to write another message to the queue, which in turn invokes the function again.

Unintentional recursive loops can result in unexpected charges being billed to your Amazon Web Services account. Loops can also cause Lambda to scale and use all of your account's available concurrency. To help reduce the impact of unintentional loops, Lambda can detect certain types of recursive loops shortly after they occur. When Lambda detects a recursive loop, it stops your function being invoked and notifies you.

If your design intentionally uses recursive patterns, then you can request to turn off Lambda recursive loop detection. To request this change, contact Amazon Web Services Support.

Important

If your design intentionally uses a Lambda function to write data back to the same Amazon resource that invokes the function, then use caution and implement suitable guard rails to prevent unexpected charges being billed to your Amazon Web Services account. To learn more about best practices for using recursive invocation patterns, see Recursive patterns that cause run-away Lambda functions in Serverless Land.

Understanding recursive loop detection

Recursive loop detection in Lambda works by tracking events. Lambda is an event-driven compute service that runs your function code when certain events occur. For example, when an item is added to an Amazon SQS queue or Amazon Simple Notification Service (Amazon SNS) topic. Lambda passes events to your function as JSON objects, which contain information about the change in the system state. When an event causes your function to run, this is called an invocation.

To detect recursive loops, Lambda uses Amazon X-Ray tracing headers. When Amazon Web Services that support recursive loop detection send events to Lambda, those events are automatically annotated with metadata. When your Lambda function writes one of these events to another supported Amazon Web Service using a supported version of an Amazon SDK, it updates this metadata. The updated metadata includes a count of the number of times that the event has invoked the function.

Note

You don't need to enable X-Ray active tracing for this feature to work. Recursive loop detection is turned on by default for all Amazon customers. There is no charge to use the feature.

A chain of requests is a sequence of Lambda invocations caused by the same triggering event. For example, imagine that an Amazon SQS queue invokes your Lambda function. Your Lambda function then sends the processed event back to the same Amazon SQS queue, which invokes your function again. In this example, each invocation of your function falls in the same chain of requests.

If your function is invoked more than 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you. If your function is configured with multiple triggers, then invocations from other triggers aren't affected.

Note

When the maxReceiveCount setting on the source queue's redrive policy is higher than 16, Lambda recursion protection does not prevent Amazon SQS from retrying the message after a recursive loop is detected and terminated. When Lambda detects a recursive loop and drops subsequent invocations, it returns a RecursiveInvocationException to the event source mapping. This increments the receiveCount value on the message. Lambda continues to retry the message, and continues to block function invocations, until Amazon SQS determines that the maxReceiveCount is exceeded and sends the message to the configured dead-letter queue.

If you have an on-failure destination or dead-letter queue configured for your function, then Lambda also sends the event from the stopped invocation to your destination or dead-letter queue. When configuring a destination or dead-letter queue for your function, be sure not to use an Amazon SNS topic or Amazon SQS queue that your function also uses as an event trigger or event source mapping. If you send events to the same resource that invokes your function, then you can create another recursive loop.

Supported Amazon Web Services and SDKs

Lambda can detect only recursive loops that include certain supported Amazon Web Services. For recursive loops to be detected, your function must also use one of the supported Amazon SDKs.

Supported Amazon Web Services

Lambda currently detects recursive loops between your functions, Amazon SQS, and Amazon SNS. Lambda also detects loops comprised only of Lambda functions, which may invoke each other synchronously or asynchronously. The following diagrams show some examples of loops that Lambda can detect:

Diagrams of recursive loops between a Lambda function, Amazon SNS, and an Amazon SQS queue.

When another Amazon Web Service such as Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3) forms part of the loop, Lambda can't currently detect and stop it.

Because Lambda currently detects only recursive loops involving Amazon SQS and Amazon SNS, it's still possible that loops involving other Amazon Web Services can result in unintended usage of your Lambda functions.

To guard against unexpected charges being billed to your Amazon Web Services account, we recommend that you configure Amazon CloudWatch alarms to alert you to unusual usage patterns. For example, you can configure CloudWatch to notify you about spikes in Lambda function concurrency or invocations. You can also configure a billing alarm to notify you when spending in your account exceeds a threshold that you specify. Or, you can use Amazon Cost Anomaly Detection to alert you to unusual billing patterns.

Supported Amazon SDKs

For Lambda to detect recursive loops, your function must use one of the following SDK versions or higher:

Runtime Minimum required Amazon SDK version

Node.js

2.1147.0 (SDK version 2)

3.105.0 (SDK version 3)

Python

1.24.46 (boto3)

1.27.46 (botocore)

Java 8 and Java 11

1.12.200 (SDK version 1)

2.17.135 (SDK version 2)

Java 17

2.20.81

Java 21

2.21.24

.NET

3.7.293.0

Ruby

3.134.0

PHP

3.232.0

Some Lambda runtimes such as Python and Node.js include a version of the Amazon SDK. If the SDK version included in your function's runtime is lower than the minimum required, then you can add a supported version of the SDK to your function's deployment package. You can also add a supported SDK version to your function using a Lambda layer. For a list of the SDKs included with each Lambda runtime, see Lambda runtimes.

Lambda recursion detection is not supported for the Lambda Go runtime.

Recursive loop notifications

When Lambda stops a recursive loop, you receive notifications through the Amazon Health Dashboard and through email. You can also use CloudWatch metrics to monitor the number of recursive invocations that Lambda has stopped.

Amazon Health Dashboard notifications

When Lambda stops a recursive invocation, the Amazon Health Dashboard displays a notification on the Your account health page, under Open and recent issues. Note that it can take up to three hours after Lambda stops a recursive invocation before this notification is displayed. For more information about viewing account events in the Amazon Health Dashboard, see Getting started with your Amazon Health Dashboard – Your account health in the Amazon Health User Guide.

Email alerts

When Lambda first stops a recursive invocation of your function, it sends you an email alert. Lambda sends a maximum of one email every 24 hours for each function in your Amazon Web Services account. After Lambda sends an email notification, you won't receive any more emails for that function for another 24 hours, even if Lambda stops further recursive invocations of the function. Note that it can take up to three hours after Lambda stops a recursive invocation before you receive this email alert.

Lambda sends recursive loop email alerts to your Amazon Web Services account's primary account contact and alternate operations contact. For information about viewing or updating the email addresses in your account, see Updating contact information in the Amazon General Reference.

Amazon CloudWatch metrics

The CloudWatch metric RecursiveInvocationsDropped records the number of function invocations that Lambda has stopped because your function has been invoked more than 16 times in a single chain of requests. Lambda emits this metric as soon as it stops a recursive invocation. To view this metric, follow the instructions for Viewing metrics on the CloudWatch console and choose the metric RecursiveInvocationsDropped.

Responding to recursive loop detection notifications

When your function is invoked more than 16 times by the same triggering event, Lambda stops the next function invocation for that event to break the recursive loop. To prevent a reoccurrence of a recursive loop that Lambda has broken, do the following:

  • Reduce your function's available concurrency to zero, which throttles all future invocations.

  • Remove or disable the trigger or event source mapping that's invoking your function.

  • Identify and fix code defects that write events back to the Amazon resource that's invoking your function. A common source of defects occurs when you use variables to define a function's event source and target. Check that you're not using the same value for both variables.

Additionally, if the event source for your Lambda function is an Amazon SQS queue, then consider configuring a dead-letter queue on the source queue.

Note

Make sure that you configure the dead-letter queue on the source queue, not on the Lambda function. The dead-letter queue that you configure on a function is used for the function's asynchronous invocation queue, not for event source queues.

If the event source is an Amazon SNS topic, then consider adding an on-failure destination for your function.

To reduce your function's available concurrency to zero (console)
  1. Open the Functions page of the Lambda console.

  2. Choose the name of your function.

  3. Choose Throttle.

  4. In the Throttle your function dialog box, choose Confirm.

To remove a trigger or event source mapping for your function (console)
  1. Open the Functions page of the Lambda console.

  2. Choose the name of your function.

  3. Choose the Configuration tab, then choose Triggers.

  4. Under Triggers, select the trigger or event source mapping that you want to delete, then choose Delete.

  5. In the Delete triggers dialog box, choose Delete.

To disable an event source mapping for your function (Amazon CLI)
  1. To find the UUID for the event source mapping that you want to disable, run the Amazon Command Line Interface (Amazon CLI) list-event-source-mappings command.

    aws lambda list-event-source-mappings
  2. To disable the event source mapping, run the following Amazon CLI update-event-source-mapping command.

    aws lambda update-event-source-mapping --function-name MyFunction \ --uuid a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 --no-enabled