Troubleshoot Amazon EC2 Auto Scaling issues

1. Do one of the following:

   • Call the delete-deployment-group command to delete the deployment group associated with the Auto Scaling group that is causing the heartbeat timeout.

   • Call the update-deployment-group command with a non-null empty list of Auto Scaling group names to detach all CodeDeploy-managed Auto Scaling lifecycle hooks.

     For example, enter the following Amazon CLI command:

     aws deploy update-deployment-group --application-name my-example-app --current-deployment-group-name my-deployment-group --auto-scaling-groups

     As another example, if you are using the CodeDeploy API with Java, call UpdateDeploymentGroup and set autoScalingGroups to new ArrayList<String>(). This sets autoScalingGroups to an empty list and removes the existing list. Do not use null, which is the default, because this leaves autoScalingGroups as-is, which is not what you want.

   Examine the output of the call. If the output contains a hooksNotCleanedUp structure with a list of Amazon EC2 Auto Scaling lifecycle hooks, there are leftover lifecycle hooks.

2. Call the describe-lifecycle-hooks command, specifying the name of the Amazon EC2 Auto Scaling group associated with the EC2 instances that failed to launch. In the output, look for any of the following:

   • Amazon EC2 Auto Scaling lifecycle hook names that correspond to the hooksNotCleanedUp structure you identified in step 1.

   • Amazon EC2 Auto Scaling lifecycle hook names that contain the name of the deployment group associated with the Auto Scaling group that's failing.

   • Amazon EC2 Auto Scaling lifecycle hook names that may have caused the heartbeat timeout for the CodeDeploy deployment.

3. If a hook falls into one of the categories listed in step 2, call the delete-lifecycle-hook command to delete it. Specify the Amazon EC2 Auto Scaling group and lifecycle hook in the call.

   Important

   Only delete hooks that are causing problems, as outlined in step 2. If you delete viable hooks, your deployments might fail, or CodeDeploy might not be able to deploy your application revisions to scaled out EC2 instances.

4. Call either the update-deployment-group or create-deployment-group command with the desired Auto Scaling group names. CodeDeploy re-installs the Auto Scaling hooks with new UUIDs.

1. Call the describe-lifecycle-hooks command, specifying the name of the Amazon EC2 Auto Scaling group associated with the deployment group in CodeDeploy. In the output, in the LifecycleHooks list, make a note of each LifecycleHookName value.

2. Call the get-deployment-group command, specifying the name of the deployment group associated with the Amazon EC2 Auto Scaling group. In the output, in the autoScalingGroups list, find each item whose name value matches the Amazon EC2 Auto Scaling group name, and then make a note of the corresponding hook value.

1. If there are lifecycle hook names in the describe-lifecycle-hooks command output that are not also in the get-deployment-group command output, then do the following:

   1. For each lifecycle hook name in the describe-lifecycle-hooks command output, call the delete-lifecycle-hook command.

   2. Call the update-deployment-group command, specifying the name of the original Amazon EC2 Auto Scaling group. CodeDeploy creates new, replacement lifecycle hooks in the Amazon EC2 Auto Scaling group and associates the lifecycle hooks with the deployment group. Automatic deployments should now resume as new instances are added to the Amazon EC2 Auto Scaling group.

2. If there are lifecycle hook names in the get-deployment-group command output that are not also in the describe-lifecycle-hooks command output, do the following:

   1. Call the update-deployment-group command, but do not specify the name of the original Amazon EC2 Auto Scaling group.

   2. Call the update-deployment-group command again, but this time specify the name of the original Amazon EC2 Auto Scaling group. CodeDeploy re-creates the missing lifecycle hooks in the Amazon EC2 Auto Scaling group. Automatic deployments should now resume as new instances are added to the Amazon EC2 Auto Scaling group.

To troubleshoot the error if there are no instances in the InService state

1. In the Amazon EC2 console, verify the Desired Capacity setting. If it is zero, set it to a positive number. Wait for the instance to be InService, which means the deployment succeeded. You have fixed the problem and can skip the remaining steps of this troubleshooting procedure. For information on setting the Desired Capacity setting, see Setting capacity limits on your Auto Scaling group in the Amazon EC2 Auto Scaling User Guide.

2. If Auto Scaling keeps attempting to launch new EC2 instances to meet the desired capacity but can never fulfill the scale out, it is usually due to a failing Auto Scaling lifecycle hook. Troubleshoot this problem as follows:

   1. To check which Auto Scaling lifecycle hook event is failing, see Verifying a scaling activity for an Auto Scaling group in the Amazon EC2 Auto Scaling User Guide.

   2. If the failing hook's name is CodeDeploy-managed-automatic-launch-deployment-hook-DEPLOYMENT_GROUP_NAME, go to CodeDeploy, find the deployment group, and find the failed deployment that was started by Auto Scaling. Then investigate why the deployment failed.

   3. If you understand why the deployment failed (for example, CloudWatch alarms were occurring) and you can fix the problem without changing the revision, then do so now.

   4. If, after investigation, you determine that CodeDeploy’s last successful revision is no longer healthy, and there are zero healthy instances in your Auto Scaling group, you are in a deployment deadlock scenario. To solve this issue, you must fix the bad CodeDeploy revision by temporarily removing CodeDeploy’s lifecycle hook from the Auto Scaling group, and then reinstalling the hook and redeploying a new (good) revision. For instructions, see:

      • To fix the deployment deadlock issue (CLI)

      • To fix the deployment deadlock issue (console)

To fix the deployment deadlock issue (CLI)

1. (Optional) Block your CI/CD pipelines that are causing the CodeDeploy error so that unexpected deployments do not occur while you’re fixing this problem.

2. Take note of your current Auto Scaling DesiredCapacity setting:

   aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name ASG_NAME

   You may need to scale back to this number at the end of this procedure.

3. Set your Auto Scaling DesiredCapacity setting to 1. This is optional if your desired capacity was greater than 1 to begin with. By decreasing it to 1, the instance will take less time to provision and deploy later, which speeds up troubleshooting. If your Auto Scaling desired capacity was originally set to 0, you must increase it to 1. This is mandatory.

   aws autoscaling set-desired-capacity --auto-scaling-group-name ASG_NAME --desired-capacity 1

   Note

   The remaining steps of this procedure assume you have set your DesiredCapacity to 1.

   At this point, Auto Scaling attempts to scale to one instance. Then, because the hook that CodeDeploy added is still present, CodeDeploy tries to deploy; the deployment fails; Auto Scaling cancels the instance; and Auto Scaling tries to re-launch an instance to reach desired capacity of one, which again fails. You are in a cancel-relaunch loop.

4. De-register the Auto Scaling group from the deployment group:

   Warning

   The following command will launch a new EC2 instance with no software on it. Before running the command, make sure that an Auto Scaling InService instance running no software is acceptable. For example, make sure the load balancer associated with the instance is not sending traffic to this host without software.

   Important

   Use the CodeDeploy command shown below to remove the hook. Do not remove the hook through the Auto Scaling service, because the removal will not be recognized by CodeDeploy.

   aws deploy update-deployment-group --application-name APPLICATION_NAME --current-deployment-group-name DEPLOYMENT_GROUP_NAME --auto-scaling-groups

   After running this command, the following occurs:

   1. CodeDeploy de-registers the Auto Scaling group from the deployment group.

   2. CodeDeploy removes the Auto Scaling lifecycle hook from the Auto Scaling group.

   3. Since the hook that was causing a failed deployment is no longer present, Auto Scaling cancels the existing EC2 instance and immediately launches a new one to scale to the desired capacity. The new instance should soon move into InService state. The new instance does not include software.

5. Wait for the EC2 instance to enter the InService state. To verify its state, use the following command:

   aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ASG_NAME --query AutoScalingGroups[0].Instances[*].LifecycleState

6. Add the hook back to the EC2 instance:

   Important

   Use the CodeDeploy command shown below to add the hook. Do not use the Auto Scaling service to add the hook, because the addition will not be recognized by CodeDeploy.

   aws deploy update-deployment-group --application-name APPLICATION_NAME --current-deployment-group-name DEPLOYMENT_GROUP_NAME --auto-scaling-groups ASG_NAME

   After running this command, the following occurs:

   1. CodeDeploy re-installs the Auto Scaling lifecycle hook to the EC2 instance

   2. CodeDeploy reregisters the Auto Scaling group with the deployment group.

7. Create a fleet-wide deployment with the Amazon S3 or GitHub revision that you know is healthy and want to use.

   For example, if the revision is a .zip file in an Amazon S3 bucket called my-revision-bucket with an object key of httpd_app.zip, enter the following command:

   aws deploy create-deployment --application-name APPLICATION_NAME --deployment-group-name DEPLOYMENT_GROUP_NAME --revision "revisionType=S3,s3Location={bucket=my-revision-bucket,bundleType=zip,key=httpd_app.zip}"

   Since there is now one InService instance in the Auto Scaling group, this deployment should work, and you should no longer see the error The deployment failed because no instances were found for your deployment group.

8. After the deployment succeeds, scale your Auto Scaling group back out to the original capacity, if you previously scaled it in:

   aws autoscaling set-desired-capacity --auto-scaling-group-name ASG_NAME --desired-capacity ORIGINAL_CAPACITY

To fix the deployment deadlock issue (console)

1. (Optional) Block your CI/CD pipelines that are causing the CodeDeploy error so that unexpected deployments do not occur while you’re fixing this problem.

2. Go to the Amazon EC2 console, and take note of your Auto Scaling Desired capacity setting. You may need to scale back to this number at the end of this procedure. For information on finding this setting, see Setting capacity limits on your Auto Scaling group.

3. Set the desired number of EC2 instances to 1:

   This is optional if your desired capacity was greater than 1 to begin with. By decreasing it to 1, the instance will take less time to provision and deploy later, which speeds up troubleshooting. If your Auto Scaling Desired capacity was originally set to 0, you must increase it to 1. This is mandatory.

   Note

   The remaining steps of this procedure assume you have set your Desired capacity to 1.

   1. Open the Amazon EC2 console at https://console.amazonaws.cn/ec2/, and choose Auto Scaling Groups from the navigation pane.

   2. Choose the appropriate Region.

   3. Go to the problematic Auto Scaling group.

   4. In Group details, choose Edit.

   5. Set Desired capacity to 1.

   6. Choose Update.

4. De-register the Auto Scaling group from the deployment group:

   Warning

   The following sub-steps will launch a new EC2 instance with no software on it. Before running the command, make sure that an Auto Scaling InService instance running no software is acceptable. For example, make sure the load balancer associated with the instance is not sending traffic to this host without software.

   1. Open the CodeDeploy console at https://console.amazonaws.cn/codedeploy/.

   2. Choose the appropriate Region.

   3. In the navigation pane, choose Applications.

   4. Choose the name of your CodeDeploy application.

   5. Choose the name of your CodeDeploy deployment group.

   6. Choose Edit.

   7. In Environment configuration, deselect Amazon EC2 Auto Scaling groups.

      Note

      The console does not allow you to save the configuration if there is no environment configuration defined. To bypass the check, temporarily add a tag of EC2 or On-premises that you know won't resolve to any hosts. To add a tag, select Amazon EC2 instances or On-premises instance, and add a tag Key of EC2 or On-premises. You can leave the tag Value empty.

   8. Choose Save changes.

      After completing these sub-steps, the following occurs:

      1. CodeDeploy de-registers the Auto Scaling group from the deployment group.

      2. CodeDeploy removes the Auto Scaling lifecycle hook from the Auto Scaling group.

      3. Since the hook that was causing a failed deployment is no longer present, Auto Scaling cancels the existing EC2 instance and immediately launches a new one to scale to the desired capacity. The new instance should soon move into InService state. The new instance does not include software.

5. Wait for the EC2 instance to enter the InService state. To verify its state:

   1. Open the Amazon EC2 console at https://console.amazonaws.cn/ec2/.

   2. In the navigation pane, choose Auto Scaling Groups.

   3. Choose your Auto Scaling group.

   4. In the content pane, choose the Instance Management tab.

   5. Under Instances, make sure that the Lifecycle column indicates InService next to the instance.

6. Re-register the Auto Scaling group with the CodeDeploy deployment group using the same method you used to remove it:

   1. Open the CodeDeploy console at https://console.amazonaws.cn/codedeploy/.

   2. Choose the appropriate Region.

   3. In the navigation pane, choose Applications.

   4. Choose the name of your CodeDeploy application.

   5. Choose the name of your CodeDeploy deployment group.

   6. Choose Edit.

   7. In Environment configuration, select Amazon EC2 Auto Scaling groups and select your Auto Scaling group from the list.

   8. Under Amazon EC2 instances or On-premises instances, find the tag you added and remove it.

   9. Deselect the check box next to Amazon EC2 instances or On-premises instances.

   10. Choose Save changes.

   This configuration re-installs the lifecycle hook into the Auto Scaling group.

7. Create a fleet-wide deployment with the Amazon S3 or GitHub revision that you know is healthy and want to use.

   For example, if the revision is a .zip file in an Amazon S3 bucket called my-revision-bucket with an object key of httpd_app.zip, do the following:

   1. In the CodeDeploy console, in the Deployment Group page, choose Create deployment.

   2. For Revision type, choose My application is stored in Amazon S3.

   3. For Revision location, choose s3://my-revision-bucket/httpd_app.zip.

   4. For Revision file type, choose .zip.

   5. Choose Create deployment.

   Since there is now one InService instance in the Auto Scaling group, this deployment should work, and you should no longer see the error The deployment failed because no instances were found for your deployment group.

8. After the deployment succeeds, scale your Auto Scaling group back out to the original capacity, if you previously scaled it in:

   1. Open the Amazon EC2 console at https://console.amazonaws.cn/ec2/, and choose Auto Scaling Groups from the navigation pane.

   2. Choose the appropriate Region.

   3. Go to your Auto Scaling group.

   4. In Group details, choose Edit.

   5. Set Desired capacity back to its original value.

   6. Choose Update.