Troubleshoot invocation issues in Lambda
When you invoke a Lambda function, Lambda validates the request and checks for scaling capacity before sending the event to your function or, for asynchronous invocation, to the event queue. Invocation errors can be caused by issues with request parameters, event structure, function settings, user permissions, resource permissions, or limits.
If you invoke your function directly, you see any invocation errors in the response from Lambda. If you invoke your function asynchronously with an event source mapping or through another service, you might find errors in logs, a dead-letter queue, or a failed-event destination. Error handling options and retry behavior vary depending on how you invoke your function and on the type of error.
For a list of error types that the Invoke
operation can return, see Invoke.
Topics
- Lambda: Function times out during Init phase (Sandbox.Timedout)
- IAM: lambda:InvokeFunction not authorized
- Lambda: Couldn't find valid bootstrap (Runtime.InvalidEntrypoint)
- Lambda: Operation cannot be performed ResourceConflictException
- Lambda: Function is stuck in Pending
- Lambda: One function is using all concurrency
- General: Cannot invoke function with other accounts or services
- General: Function invocation is looping
- Lambda: Alias routing with provisioned concurrency
- Lambda: Cold starts with provisioned concurrency
- Lambda: Cold starts with new versions
- EFS: Function could not mount the EFS file system
- EFS: Function could not connect to the EFS file system
- EFS: Function could not mount the EFS file system due to timeout
- Lambda: Lambda detected an IO process that was taking too long
Lambda: Function times out during Init phase (Sandbox.Timedout)
Error: Task timed out after 3.00 seconds
When the Init phase times out, Lambda initializes the execution environment again by re-running the Init
phase when the next invoke request arrives. This is called a suppressed init. However, if your function is configured with a short timeout duration (generally around 3 seconds), the suppressed init might not complete during the allocated timeout duration, causing the Init
phase to time out again. Alternatively, the suppressed init completes but does not leave enough time for the Invoke phase to complete, causing the Invoke
phase to time out.
To reduce timeout errors, use one or more of the following strategies:
-
Increase the function timeout duration: Extend the timeout to give the
Init
andInvoke
phases time to complete successfully. -
Increase the function memory allocation: More memory also means more proportional CPU allocation, which can speed up both the
Init
andInvoke
phases. -
Optimize the function initialization code: Reduce the time needed for initialization to ensure that the the
Init
andInvoke
phase can complete within the configured timeout. -
Add error handling: Proper error handling in the function code can prevent the Lambda execution environment from failing and triggering repeated initialization attempts.
IAM: lambda:InvokeFunction not authorized
Error: User: arn:aws-cn:iam::123456789012:user/developer is not authorized to perform: lambda:InvokeFunction on resource: my-function
Your user, or the role that you assume, must have permission to invoke a function. This
requirement also applies to Lambda functions and other compute resources that invoke functions. Add the Amazon managed
policy AWSLambdaRole to your user, or add a custom policy that allows the
lambda:InvokeFunction
action on the target function.
Note
The name of the IAM action (lambda:InvokeFunction
) refers to the Invoke
Lambda API operation.
For more information, see Managing permissions in Amazon Lambda .
Lambda: Couldn't find valid bootstrap (Runtime.InvalidEntrypoint)
Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]
This error typically occurs when the root of your deployment package doesn't contain an executable file named bootstrap
. For example, if you're deploying a provided.al2023
function with a .zip file, the bootstrap
file must be at the root of the .zip file, not in a directory.
Lambda: Operation cannot be performed ResourceConflictException
Error: ResourceConflictException: The operation cannot be performed at this time. The function is currently in the following state: Pending
When you connect a function to a virtual private cloud (VPC) at the time of creation, the function enters a
Pending
state while Lambda creates elastic network interfaces. During this time, you can't invoke or
modify your function. If you connect your function to a VPC after creation, you can invoke it while the update is
pending, but you can't modify its code or configuration.
For more information, see Lambda function states .
Lambda: Function is stuck in Pending
Error:
A function is stuck in the Pending
state for several minutes.
If a function is stuck in the Pending
state for more than six minutes, call one of the following
API operations to unblock it:
Lambda cancels the pending operation and puts the function into the Failed
state. You can then
attempt another update.
Lambda: One function is using all concurrency
Issue: One function is using all of the available concurrency, causing other functions to be throttled.
To divide your Amazon account's available concurrency in an Amazon Region into pools, use reserved concurrency. Reserved concurrency ensures that a function can always scale to its assigned concurrency, and that it doesn't scale beyond its assigned concurrency.
General: Cannot invoke function with other accounts or services
Issue: You can invoke your function directly, but it doesn't run when another service or account invokes it.
You grant other services and accounts permission to invoke a function in the function's resource-based policy. If the invoker is in another account, that user must also have permission to invoke functions.
General: Function invocation is looping
Issue: Function is invoked continuously in a loop.
This typically occurs when your function manages resources in the same Amazon service that triggers it. For example, it's possible to create a function that stores an object in an Amazon Simple Storage Service (Amazon S3) bucket that's configured with a notification that invokes the function again. To stop the function from running, reduce the available concurrency to zero, which throttles all future invocations. Then, identify the code path or configuration error that caused the recursive invocation. Lambda automatically detects and stops recursive loops for some Amazon services and SDKs. For more information, see Use Lambda recursive loop detection to prevent infinite loops.
Lambda: Alias routing with provisioned concurrency
Issue: Provisioned concurrency spillover invocations during alias routing.
Lambda uses a simple probabilistic model to distribute the traffic between the two function versions. At low traffic levels, you might see a high variance between the configured and actual percentage of traffic on each version. If your function uses provisioned concurrency, you can avoid spillover invocations by configuring a higher number of provisioned concurrency instances during the time that alias routing is active.
Lambda: Cold starts with provisioned concurrency
Issue: You see cold starts after enabling provisioned concurrency.
When the number of concurrent executions on a function is less than or equal to the configured level of provisioned concurrency, there shouldn't be any cold starts. To help you confirm if provisioned concurrency is operating normally, do the following:
-
Check that provisioned concurrency is enabled on the function version or alias.
Note
Provisioned concurrency is not configurable on the unpublished version of the function ($LATEST).
-
Ensure that your triggers invoke the correct function version or alias. For example, if you're using Amazon API Gateway, check that API Gateway invokes the function version or alias with provisioned concurrency, not $LATEST. To confirm that provisioned concurrency is being used, you can check the ProvisionedConcurrencyInvocations Amazon CloudWatch metric. A non-zero value indicates that the function is processing invocations on initialized execution environments.
-
Determine whether your function concurrency exceeds the configured level of provisioned concurrency by checking the ProvisionedConcurrencySpilloverInvocations CloudWatch metric. A non-zero value indicates that all provisioned concurrency is in use and some invocation occurred with a cold start.
-
Check your invocation frequency (requests per second). Functions with provisioned concurrency have a maximum rate of 10 requests per second per provisioned concurrency. For example, a function configured with 100 provisioned concurrency can handle 1,000 requests per second. If the invocation rate exceeds 1,000 requests per second, some cold starts can occur.
Lambda: Cold starts with new versions
Issue: You see cold starts while deploying new versions of your function.
When you update a function alias, Lambda automatically shifts provisioned concurrency to the new version based on the weights configured on the alias.
Error: KMSDisabledException: Lambda was unable to decrypt the environment variables because the KMS key used is disabled. Please check the function's KMS key settings.
This error can occur if your Amazon Key Management Service (Amazon KMS) key is disabled, or if the grant that allows Lambda to use the key is revoked. If the grant is missing, configure the function to use a different key. Then, reassign the custom key to recreate the grant.
EFS: Function could not mount the EFS file system
Error: EFSMountFailureException: The function could not mount the EFS file system with access point arn:aws-cn:elasticfilesystem:us-east-2:123456789012:access-point/fsap-015cxmplb72b405fd.
The mount request to the function's file system was rejected. Check the function's permissions, and confirm that its file system and access point exist and are ready for use.
EFS: Function could not connect to the EFS file system
Error: EFSMountConnectivityException: The function couldn't connect to the Amazon EFS file system with access point arn:aws-cn:elasticfilesystem:us-east-2:123456789012:access-point/fsap-015cxmplb72b405fd. Check your network configuration and try again.
The function couldn't establish a connection to the function's file system with the NFS protocol (TCP port 2049). Check the security group and routing configuration for the VPC's subnets.
If you get these errors after updating your function's VPC configuration settings, try unmounting and remounting the file system.
EFS: Function could not mount the EFS file system due to timeout
Error: EFSMountTimeoutException: The function could not mount the EFS file system with access point {arn:aws-cn:elasticfilesystem:us-east-2:123456789012:access-point/fsap-015cxmplb72b405fd} due to mount time out.
The function could connect to the function's file system, but the mount operation timed out. Try again after a short time and consider limiting the function's concurrency to reduce load on the file system.
Lambda: Lambda detected an IO process that was taking too long
EFSIOException: This function instance was stopped because Lambda detected an IO process that was taking too long.
A previous invocation timed out and Lambda couldn't terminate the function handler. This issue can occur when an attached file system runs out of burst credits and the baseline throughput is insufficient. To increase throughput, you can increase the size of the file system or use provisioned throughput.