Design your applications on Amazon EC2 Auto Scaling to gracefully handle instance termination
This topic covers the different approaches that you can take if you have applications running on instances that ideally shouldn't terminate unexpectedly when Amazon EC2 Auto Scaling responds to a scale-in event.
For example, suppose you have an Amazon SQS queue that collects incoming messages for long-running jobs. When a new message arrives, an instance in the Auto Scaling group retrieves the message and starts processing it. Each message takes 3 hours to process. As the number of messages increase, new instances are automatically added to the Auto Scaling group. As the number of messages decrease, existing instances are automatically terminated. In this case, Amazon EC2 Auto Scaling must decide which instance to terminate. By default, it's possible that Amazon EC2 Auto Scaling might terminate an instance that is 2.9 hours into processing a 3-hour long job, rather than an instance that's currently idle. To avoid issues with unexpected terminations when using Amazon EC2 Auto Scaling, you must design your application to respond to this scenario.
You can use the following features to prevent your Auto Scaling group from terminating instances that aren't yet ready to terminate, or from terminating instances too quickly for them to complete their assigned jobs. All three of these features can be used in combination or separately.
Important
When designing your applications on Amazon EC2 Auto Scaling to gracefully handle instance termination, keep these points in mind.
-
If an instance is unhealthy, Amazon EC2 Auto Scaling will replace it regardless of which feature you use (unless you suspend the
ReplaceUnhealthyprocess). You can use a lifecycle hook to allow the application to shut down gracefully or copy any data that you need to recover before the instance is terminated. -
A termination lifecycle hook is not guaranteed to run or finish before an instance is terminated. If something fails, Amazon EC2 Auto Scaling still terminates the instance.
Instance scale-in protection
You can use instance scale-in protection in many situations where terminating instances is a critical action that should be denied by default, and only explicitly allowed for specific instances. For example, when running containerized workloads, it’s common to want to protect all instances and remove protection only for instances with no current or scheduled tasks. Services such as Amazon ECS have built integrations with instance scale-in protection into their products.
You can enable scale-in protection on the Auto Scaling group to apply scale-in protection to instances when they're created and enable it for existing instances. When an instance has no more work to do, it can toggle off protection. The instance can continue polling for new jobs and re-enable protection when there are new jobs assigned.
Applications can set protection either from a centralized control plane that manages whether an instance is terminable or not, or from the instances themselves. However, a large fleet could run into throttling issues if large numbers of instances are continuously toggling their scale-in protection.
For more information, see Use instance scale-in protection.
Custom termination policy
Like instance scale-in protection, a custom termination policy helps you prevent your Auto Scaling group from terminating specific instances.
By default, your Auto Scaling group uses a default termination policy to determine which instances it terminates first. If you want more control over which instances terminate first, you can implement your own custom termination policy using a Lambda function. Amazon EC2 Auto Scaling calls the function whenever it must decide which instance to terminate. It will only terminate an instance that's returned by the function. If the function errors, times out, or produces an empty list, Amazon EC2 Auto Scaling doesn't terminate instances.
A custom termination policy is useful if it's known when an instance is sufficiently redundant or underutilized so that it can be terminated. To support this, you need to implement your application with a control plane that monitors workload across the group. That way, if an instance is still processing jobs, the Lambda function knows not to include it.
For more information, see Create a custom termination policy with Lambda.
Termination lifecycle hooks
A termination lifecycle hook extends the life of an instance that's already selected for termination. It provides extra time to complete all messages or requests currently assigned to the instance, or to save progress and transfer the work to another instance.
For many workloads, a lifecycle hook may be enough to gracefully shut down an application on an instance that's selected for termination. This is a best-effort approach and can't be used to prevent termination if there's a failure.
To use a lifecycle hook, you need to know when an instance is selected to be terminated. You have two ways to know this:
| Option | Description | Best used for | Link to documentation |
|---|---|---|---|
| Inside the instance | The Instance Metadata Service (IMDS) is a secure endpoint
that you can poll for the status of an instance directly from
the instance. If the metadata comes back with
Terminated, then your instance is scheduled to
be terminated. |
Applications where you must perform an action on the instance before the instance is terminated. | Retrieve the target lifecycle state |
| Outside the instance | When an instance is terminating, an event notification is generated. You can create rules using Amazon EventBridge, Amazon SQS, or Amazon SNS to capture these events, and invoke a response such as with a Lambda function. | Applications that need to take action outside of the instance. | Configure a notification target |
To use a lifecycle hook, you also need to know when your instance is ready to be fully terminated. Amazon EC2 Auto Scaling will not tell Amazon EC2 to terminate the instance until it receives a CompleteLifecycleAction call or the timeout elapses, whichever happens first.
By default, an instance can continue running for one hour (heartbeat timeout) due to a termination lifecycle hook. You can configure the default timeout if one hour is not enough time to complete the lifecycle action. When a lifecycle action is actually in progress, you can extend the timeout with RecordLifecycleActionHeartbeat API calls.
For more information, see Amazon EC2 Auto Scaling lifecycle hooks.