Health checks for Auto Scaling instances - Amazon EC2 Auto Scaling
Health check typesAmazon EC2 health checksElastic Load Balancing health checksVPC Lattice health checksCustom health detection tasksUnhealthy instance replacementHow Amazon EC2 Auto Scaling minimizes downtimeHealth check considerationsAdditional information
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Health checks for Auto Scaling instances

The health status of an Auto Scaling instance indicates whether it's healthy or unhealthy. All instances in your Auto Scaling group start with a Healthy status. Instances are assumed to be healthy unless Amazon EC2 Auto Scaling receives notification that they are unhealthy. This notification can come from sources such as Amazon EC2, Elastic Load Balancing, VPC Lattice, or custom health checks. When Amazon EC2 Auto Scaling detects an unhealthy instance, it replaces it with a new instance.

Contents

Health check types

Amazon EC2 Auto Scaling can determine the health status of an instance by using one or more of the following health checks:

Health check type What it checks

Amazon EC2 status checks and scheduled events

  • Checks that the instance is running

  • Checks for underlying hardware or software issues that might impair the instance

This is the default health check type for an Auto Scaling group.

Elastic Load Balancing health checks

  • Checks whether the load balancer reports the instance as healthy, confirming whether the instance is available to handle requests

To run this health check type, you must enable it for your Auto Scaling group.

VPC Lattice health checks

  • Checks whether VPC Lattice reports the instance as healthy, confirming whether the instance is available to handle requests

To run this health check type, you must enable it for your Auto Scaling group.

Custom health checks

  • Checks for any other problems that might indicate instance health issues, according to your custom health checks

Amazon EC2 health checks

After an instance launches, it's attached to the Auto Scaling group and enters the InService state. For more information about the different lifecycle states for instances in an Auto Scaling group, see Amazon EC2 Auto Scaling instance lifecycle.

Amazon EC2 Auto Scaling periodically checks the health status of all instances within the Auto Scaling group to make sure that they're running and in good condition.

Status checks

Amazon EC2 Auto Scaling uses the results of the Amazon EC2 instance status checks and system status checks to determine the health status of an instance. If the instance is in any Amazon EC2 state other than running, or if its status for the status checks becomes impaired, Amazon EC2 Auto Scaling considers the instance to be unhealthy and replaces it. This includes when the instance has any of the following states:

  • stopping

  • stopped

  • shutting-down

  • terminated

The Amazon EC2 status checks do not require any special configuration and are always enabled. For more information, see Types of status checks in the Amazon EC2 User Guide for Linux Instances.

Important

Amazon EC2 Auto Scaling lets the status checks fail occasionally, without taking any action. When a status check fails, Amazon EC2 Auto Scaling waits a few minutes for Amazon to fix the issue. It does not immediately mark an instance Unhealthy when its status for the status checks becomes impaired.

However, if Amazon EC2 Auto Scaling detects that an instance is no longer in the running state, this situation is treated as an immediate failure. In this case, it immediately marks the instance Unhealthy and replaces it.

Scheduled events

Amazon EC2 can occasionally schedule events on your instances to be run after a particular timestamp. For more information, see Scheduled events for your instances in the Amazon EC2 User Guide for Linux Instances.

If one of your instances is affected by a scheduled event, Amazon EC2 Auto Scaling considers the instance to be unhealthy and replaces it. The instance doesn't start terminating until the date and time specified in the timestamp is reached.

Elastic Load Balancing health checks

When you enable Elastic Load Balancing health checks for your Auto Scaling group, Amazon EC2 Auto Scaling can use the results of those health checks to determine the health status of an instance.

Before you can enable Elastic Load Balancing health checks for your Auto Scaling group, you must do the following:

  • Set up an Elastic Load Balancing load balancer and configure a health check for it to use to determine if your instances are healthy.

  • Attach the load balancer to your Auto Scaling group.

After you complete the preceding actions, the following occurs:

  • Amazon EC2 Auto Scaling registers the instances in the Auto Scaling group with the load balancer.

  • After an instance finishes registering, it enters the InService state and becomes available for use with the load balancer.

By default, Amazon EC2 Auto Scaling ignores the results of the Elastic Load Balancing health checks. However, you can enable these health checks for your Auto Scaling group. After you do this, when Elastic Load Balancing reports a registered instance as Unhealthy, Amazon EC2 Auto Scaling marks the instance Unhealthy on its next periodic health check and replaces it.

If connection draining (deregistration delay) is enabled for your load balancer, Amazon EC2 Auto Scaling waits for either in-flight requests to complete or the maximum timeout to expire before it terminates unhealthy instances.

To learn how to enable Elastic Load Balancing health checks for your Auto Scaling group, see Add Elastic Load Balancing health checks to an Auto Scaling group.

Note

When you enable Elastic Load Balancing health checks for a group, Amazon EC2 Auto Scaling can terminate and replace instances that Elastic Load Balancing reports as unhealthy, but only after the load balancer is in the InService state. For more information, see Verify the attachment status of your load balancer.

VPC Lattice health checks

By default, Amazon EC2 Auto Scaling ignores the results of the VPC Lattice health checks. You can optionally enable these health checks for your Auto Scaling group. After you do this, when VPC Lattice reports a registered instance as Unhealthy, Amazon EC2 Auto Scaling marks the instance Unhealthy on its next periodic health check and replaces it. The process of registering instances and then checking their health is the same as how Elastic Load Balancing health checks work.

To learn how to enable VPC Lattice health checks for your Auto Scaling group, see Attach a VPC Lattice target group to your Auto Scaling group.

Note

When you enable VPC Lattice health checks for a group, Amazon EC2 Auto Scaling can replace instances that VPC Lattice reports as unhealthy, but only after the target group is in the InService state. For more information, see Verify the attachment status of your VPC Lattice target group.

Custom health detection tasks

You might also want to run custom health detection tasks on the instances in your Auto Scaling group and set the health status of an instance to Unhealthy if the task fails. This extends your health checks by using a combination of custom health checks, Amazon EC2 status checks, and Elastic Load Balancing health checks, if enabled.

You can send the instance health information directly to Amazon EC2 Auto Scaling using the Amazon CLI or an SDK. The following examples show how to use the Amazon CLI to configure the health status of an instance and then verify the instance's health status.

Use the following set-instance-health command to set the health status of the specified instance to Unhealthy.

aws autoscaling set-instance-health --instance-id i-1234567890abcdef0 --health-status Unhealthy

By default, this command respects the health check grace period. However, you can override this behavior and not respect the grace period by including the --no-should-respect-grace-period option.

Use the following describe-auto-scaling-groups command to verify that the instance's health status is Unhealthy.

aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names my-asg

The following is an example response that shows that the health status of the instance is Unhealthy and that the instance is terminating.

{
    "AutoScalingGroups": [
        {
            ....
            "Instances": [
                {
                    "ProtectedFromScaleIn": false,
                    "AvailabilityZone": "us-west-2a",
                    "LaunchTemplate": {
                        "LaunchTemplateName": "my-launch-template",
                        "Version": "1",
                        "LaunchTemplateId": "lt-1234567890abcdef0"
                    },
                    "InstanceId": "i-1234567890abcdef0",
                    "InstanceType": "t2.micro",
                    "HealthStatus": "Unhealthy",
                    "LifecycleState": "Terminating"
                },
                ...
            ]
        }
    ]
}

Unhealthy instance replacement

When Amazon EC2 Auto Scaling determines that an InService instance is unhealthy, it terminates the instance while it launches a new replacement instance. The new instance launches using the current settings of the Auto Scaling group and its associated launch template or launch configuration.

Amazon EC2 Auto Scaling creates a new scaling activity for terminating the unhealthy instance and then terminates it. While the instance is terminating, another scaling activity launches a new instance.

To view the reason for health check failures (console)

  1. Open the Amazon EC2 console at https://console.amazonaws.cn/ec2/, and choose Auto Scaling Groups from the navigation pane.

  2. Select the check box next to the Auto Scaling group.

    A split pane opens up in the bottom of the Auto Scaling groups page.

  3. On the Activity tab, under Activity history, the Status column shows whether your Auto Scaling group has successfully launched or terminated instances.

    If it terminated any unhealthy instances, the Cause column shows the date and time of the termination and the reason for the health check failure. For example, At 2022-05-14T20:11:53Z an instance was taken out of service in response to an ELB system health check failure.

How Amazon EC2 Auto Scaling minimizes downtime

Health check replacements require instances to terminate first, which might prevent new requests from being accepted until new instances launch.

If Amazon EC2 Auto Scaling determines that any instances are no longer running or were marked Unhealthy with the set-instance-health command, it immediately replaces them. However, if other instances are found to be unhealthy, Amazon EC2 Auto Scaling uses the following approach to recover from failures. This approach minimizes any downtime that might occur because of temporary issues or misconfigured health checks.

  • If a scaling activity is in progress and your Auto Scaling group is below its desired capacity by 10 percent or more, Amazon EC2 Auto Scaling waits for the in-progress scaling activity before replacing the unhealthy instances.

  • When scaling out, Amazon EC2 Auto Scaling waits for the instances to pass an initial health check. It also waits for the default instance warmup to finish to make sure that the new instances are ready.

  • After the instances finish warming up and the group has risen to above 90 percent of its desired capacity, Amazon EC2 Auto Scaling replaces the unhealthy instances as follows:

    • Amazon EC2 Auto Scaling only replaces up to 10 percent of the group's desired capacity at a time. It does this until all of the unhealthy instances are replaced.

    • When replacing instances, it waits for the new instances to pass an initial health check. It also waits for the default instance warmup to finish before continuing.

Note

If the size of an Auto Scaling group is small enough that the resulting value of 10 percent is less than one, Amazon EC2 Auto Scaling replaces the unhealthy instances one at a time instead. This might result in some downtime for the group.

Also, if all instances in an Auto Scaling group are reported unhealthy by Elastic Load Balancing health checks and the load balancer is in the InService state, Amazon EC2 Auto Scaling might mark fewer instances unhealthy at a time. This can result in much fewer instances replaced at a time than the 10 percent applied in other scenarios. This provides you time to fix the problem without Amazon EC2 Auto Scaling automatically terminating the entire group.

Health check considerations

This section contains considerations for Amazon EC2 Auto Scaling health checks.

  • If you need something to happen on the instance that is terminating, or on the instance that is starting up, you can use lifecycle hooks. These hooks let you perform a custom action as Amazon EC2 Auto Scaling launches or terminates instances. For more information, see Amazon EC2 Auto Scaling lifecycle hooks.

  • Amazon EC2 Auto Scaling does not provide a way of removing the Amazon EC2 status checks and scheduled events from its health checks. If you do not want instances to be replaced, we recommend that you suspend the ReplaceUnhealthy and HealthCheck process for individual Auto Scaling groups. For more information, see Suspend and resume a process for an Auto Scaling group.

  • To manually set an unhealthy instance's health status back to Healthy, you can try to use the set-instance-health command. If you get an error, this is probably because the instance is already terminating. Generally, setting an instance's health status back to Healthy with the set-instance-health command is only useful in cases where either the ReplaceUnhealthy process or the Terminate process is suspended.

  • Amazon EC2 Auto Scaling does not perform health checks on instances that are in the Standby state. For more information, see Temporarily remove instances from your Auto Scaling group.

  • When your instance is terminated, any associated Elastic IP addresses are disassociated and are not automatically associated with the new instance. You must manually associate the Elastic IP addresses with the new instance, or do it automatically with a lifecycle hook-based solution. For more information, see Elastic IP addresses in the Amazon EC2 User Guide for Linux Instances.

  • Similarly, when your instance is terminated, its attached EBS volumes are detached (or deleted depending on the volume's DeleteOnTermination attribute). You must manually attach these EBS volumes to the new instance, or do it automatically with a lifecycle hook-based solution. For more information, see Attach an Amazon EBS volume to an instance in the Amazon EC2 User Guide for Linux Instances.

Additional information

For information about troubleshooting health checks, see Troubleshoot Amazon EC2 Auto Scaling: Health checks. If your health checks fail, check this topic for troubleshooting steps. This topic will help you figure out what has gone wrong in your Auto Scaling group and give you suggestions on how to fix it.

Amazon EC2 Auto Scaling also monitors the health of instances that you launch into a warm pool using Amazon EC2, Amazon EBS, or custom health checks. For more information, see View health check status and the reason for health check failures.