Creating and configuring an Amazon SQS event source mapping
To process Amazon SQS messages with Lambda, configure your queue with the appropriate settings, then create a Lambda event source mapping.
Configuring a queue to use with Lambda
If you don't already have an existing Amazon SQS queue, create one to serve as an event source for your Lambda function. Then configure the queue to allow sufficient time for your Lambda function to process each batch of events.
To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the configuration timeout on your function. The extra time allows Lambda to retry if your function is throttled while processing a previous batch.
By default, if Lambda encounters an error at any point while processing a batch, all
messages in that batch return to the queue. After the
visibility timeout, the messages become visible to Lambda again. You can
configure your event source mapping to use
partial batch responses to return only the failed messages back to the queue. In
addition, if your function fails to process a message multiple times, Amazon SQS can send it to a
dead-letter queue. We recommend setting the maxReceiveCount on your
source queue's
redrive policy to at least 5. This gives Lambda a few chances to retry before
sending failed messages directly to the dead-letter queue.
Setting up Lambda execution role permissions
The AWSLambdaSQSQueueExecutionRole Amazon managed policy includes the permissions that Lambda needs to read from your Amazon SQS queue. You can add this managed policy to your function's execution role.
Optionally, if you're using an encrypted queue, you also need to add the following permission to your execution role:
Creating an SQS event source mapping
Create an event source mapping to tell Lambda to send items from your queue to a Lambda function. You can create multiple event source mappings to process items from multiple queues with a single function. When Lambda invokes the target function, the event can contain multiple items, up to a configurable maximum batch size.
To configure your function to read from Amazon SQS, attach the AWSLambdaSQSQueueExecutionRole Amazon managed policy to your execution role. Then, create an SQS event source mapping from the console using the following steps.
To add permissions and create a trigger
Open the Functions page
of the Lambda console. -
Choose the name of a function.
-
Choose the Configuration tab, and then choose Permissions.
-
Under Role name, choose the link to your execution role. This link opens the role in the IAM console.
-
Choose Add permissions, and then choose Attach policies.
-
In the search field, enter
AWSLambdaSQSQueueExecutionRole. Add this policy to your execution role. This is an Amazon managed policy that contains the permissions your function needs to read from an Amazon SQS queue. For more information about this policy, see AWSLambdaSQSQueueExecutionRole in the Amazon Managed Policy Reference. -
Go back to your function in the Lambda console. Under Function overview, choose Add trigger.
-
Choose a trigger type.
-
Configure the required options, and then choose Add.
Lambda supports the following configuration options for Amazon SQS event sources:
- SQS queue
-
The Amazon SQS queue to read records from.
- Enable trigger
-
The status of the event source mapping. Enable trigger is selected by default.
- Batch size
-
The maximum number of records to send to the function in each batch. For a standard queue, this can be up to 10,000 records. For a FIFO queue, the maximum is 10. For a batch size over 10, you must also set the batch window (
MaximumBatchingWindowInSeconds) to at least 1 second.Configure your function timeout
to allow enough time to process an entire batch of items. If items take a long time to process, choose a smaller batch size. A large batch size can improve efficiency for workloads that are very fast or have a lot of overhead. If you configure reserved concurrency on your function, set a minimum of five concurrent executions to reduce the chance of throttling errors when Lambda invokes your function. Lambda passes all of the records in the batch to the function in a single call, as long as the total size of the events doesn't exceed the invocation payload size quota for synchronous invocation (6 MB). Both Lambda and Amazon SQS generate metadata for each record. This additional metadata is counted towards the total payload size and can cause the total number of records sent in a batch to be lower than your configured batch size. The metadata fields that Amazon SQS sends can be variable in length. For more information about the Amazon SQS metadata fields, see the ReceiveMessage API operation documentation in the Amazon Simple Queue Service API Reference.
- Batch window
-
The maximum amount of time to gather records before invoking the function, in seconds. This applies only to standard queues.
If you're using a batch window greater than 0 seconds, you must account for the increased processing time in your queue's visibility timeout. We recommend setting your queue's visibility timeout to six times your function timeout, plus the value of
MaximumBatchingWindowInSeconds. This allows time for your Lambda function to process each batch of events and to retry in the event of a throttling error.When messages become available, Lambda starts processing messages in batches. Lambda starts processing five batches at a time with five concurrent invocations of your function. If messages are still available, Lambda adds up to 300 more instances of your function a minute, up to a maximum of 1,000 function instances. To learn more about function scaling and concurrency, see Lambda function scaling.
To process more messages, you can optimize your Lambda function for higher throughput. For more information, see Understanding how Amazon Lambda scales with Amazon SQS standard queues
. - Maximum concurrency
-
The maximum number of concurrent functions that the event source can invoke. For more information, see Configuring maximum concurrency for Amazon SQS event sources.
- Filter criteria
-
Add filter criteria to control which events Lambda sends to your function for processing. For more information, see Lambda event filtering.