AWSSupport-TroubleshootOpenSearchRedYellowCluster - Amazon Systems Manager Automation runbook reference
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

AWSSupport-TroubleshootOpenSearchRedYellowCluster

Description

AWSSupport-TroubleshootOpenSearchRedYellowCluster automation runbook is used to identify the cause for red or yellow cluster health status and guide you through changing the cluster back to green.

How does it work?

The runbook AWSSupport-TroubleshootOpenSearchRedYellowCluster helps you troubleshoot the cause of red or yellow cluster and provides the next steps to resolve this issue by analyzing the cluster configuration and resource utilization.

The runbook performs the following steps:

  • Calls the DescribeDomain API against the target domain to get the cluster configuration.

  • Checks if the OpenSearch Service domain is internet-based (public) or Amazon Virtual Private Cloud (VPC)-based.

  • Creates a public or Amazon VPC-based Amazon Lambda function depending on the cluster configuration. Note: The Lambda function contains the troubleshooting code that run the OpenSearch Service APIs against the cluster to determine why the cluster is in red or yellow state.

  • Deletes the Lambda function.

  • Displays the checks performed and the next recommended steps to resolve the red or yellow cluster issue.

Document type

Automation

Owner

Amazon

Platforms

Linux, macOS, Windows

Parameters

Required IAM permissions

The AutomationAssumeRole parameter requires the following actions to use the runbook successfully.

  • cloudformation:CreateStack

  • cloudformation:DescribeStacks

  • cloudformation:DescribeStackEvents

  • cloudformation:DeleteStack

  • lambda:CreateFunction

  • lambda:DeleteFunction

  • lambda:InvokeFunction

  • lambda:GetFunction

  • es:DescribeDomain

  • es:DescribeDomainConfig

  • ec2:DescribeSecurityGroups

  • ec2:DescribeSubnets

  • ec2:DescribeVpcs

  • ec2:DescribeNetworkInterfaces

  • ec2:CreateNetworkInterface

  • ec2:DeleteNetworkInterface

  • ec2:DescribeInstances

  • ec2:AttachNetworkInterface

  • cloudwatch:GetMetricData

  • iam:PassRole

The LambdaExecutionRole parameter requires the following actions to successfully use the runbook:

  • es:ESHttpGet

  • ec2:CreateNetworkInterface

  • ec2:DescribeNetworkInterfaces

  • ec2:DeleteNetworkInterface

Overview of LambdaExecutionRole policy:

The following is an example of a Lambda function's execution role (Amazon Identity and Access Management (IAM) role) that grants the function permission to access Amazon services and resources required by this runbook. For more information, see Lambda execution role.

Note

The ec2:DescribeNetworkInterfaces, ec2:CreateNetworkInterface, and ec2:DeleteNetworkInterface are only required if your OpenSearch Service cluster is Amazon VPC-based to allow the Lambda function to create and manage the Amazon VPC network interfaces. For more information, see Connecting outbound networking to resources in a Amazon VPC and Lambda execution role. 


        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "es:ESHttpGet",
                    "Resource": [
                        "arn:<partition>:es:<region>:<account-id>:domain/<domain-name>/",
                        "arn:<partition>:es:<region>:<account-id>:domain/<domain-name>/_cluster/health",
                        "arn:<partition>:es:<region>:<account-id>:domain/<domain-name>/_cat/indices",
                        "arn:<partition>:es:<region>:<account-id>:domain/<domain-name>/_cat/allocation",
                        "arn:<partition>:es:<region>:<account-id>:domain/<domain-name>/_cluster/allocation/explain"
                    ]
                },
                {
                    "Condition": {
                        "ArnLikeIfExists": {
                            "ec2:Vpc": "arn:<partition>:ec2:<region>:<account-id>:vpc/<vpc_id>"
                        }
                    },
                    "Action": [
                        "ec2:DeleteNetworkInterface",
                        "ec2:CreateNetworkInterface",
                        "ec2:DescribeNetworkInterfaces",
                        "ec2:UnassignPrivateIpAddresses",
                        "ec2:AssignPrivateIpAddresses"
                    ],
                    "Resource": "*",
                    "Effect": "Allow"
                }
                
            ]
        }

Instructions

Follow these steps to configure the automation:

  1. Navigate to the AWSSupport-TroubleshootOpenSearchRedYellowCluster in the Amazon Systems Manager console.

  2. Select Execute automation.

  3. For the input parameters enter the following:

    • AutomationAssumeRole (Optional):

      The Amazon Resource Name (ARN) of the Amazon Identity and Access Management (IAM) role that allows Systems Manager Automation to perform the actions on your behalf. If no role is specified, Systems Manager Automation uses the permissions of the user that starts this runbook.

    • LambdaExecutionRole (Required):

      The ARN of the IAM role that Lambda will use to sign requests to your Amazon OpenSearch Service cluster.

    • DomainName (Required):

      The name of the OpenSearch Service domain with red or yellow cluster health status.

    • UtilizationThreshold (Optional):

      The utilization threshold percentage used to compare the CPUUtilization and JVMMemoryPressure metrics. Default value is 80.

    Input parameters form for Amazon Systems Manager Automation with IAM roles and domain settings.

  4. If you have enabled fine-grained access control on an OpenSearch Service cluster, make sure that the LambdaExecutionRole role arn is mapped to a role with at least cluster_monitor permission.

    Cluster permissions section showing cluster_monitor permission granted.
    Backend roles interface showing an AmazonIAM role for Lambda execution and options to remove or add roles.

  5. Select Execute.

  6. The automation initiates.

  7. The automation runbook performs the following steps:

    • GetClusterConfiguration:

      Fetches the OpenSearch Service cluster configuration.

    • CreateAWSLambdaFunctionStack:

      Creates a temporary Lambda function in your account using Amazon CloudFormation. The Lambda function is used to run the OpenSearch Service APIs.

    • WaitForAWSLambdaFunctionStack:

      Waits for the CloudFormation stack to complete.

    • GetClusterMetricsFromCloudWatch:

      Gets the Amazon CloudWatch ClusterStatus, CPUUtilization, and JVMMemoryPressure OpenSearch Service cluster related metrics and its creation date.

    • RunOpenSearchAPIs:

      Uses the Lambda function to call the OpenSearch Service APIs and analyze the cluster metrics data to diagnose the cause for the red or yellow cluster status.

    • DeleteAWSLambdaFunctionStack:

      Deletes the Lambda function created by this automation in your account.

  8. After completed, review the Outputs section for the detailed results of the execution.

    • RootCause:

      Provides an overview of the identified cause for cluster health to be in red or yellow state.

    • IssueDescription:

      Provides details for why the cluster is in red or yellow state and possible steps to return the cluster to green state.

References

Systems Manager Automation

Amazon service documentation