Streaming CloudWatch Logs data to Amazon OpenSearch Service - Amazon CloudWatch Logs
PrerequisitesSubscribe a log group to OpenSearch Service
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Streaming CloudWatch Logs data to Amazon OpenSearch Service

You can configure a CloudWatch Logs log group to stream data it receives to your Amazon OpenSearch Service cluster in near real-time through a CloudWatch Logs subscription. For more information, see Real-time processing of log data with subscriptions.

Note

Streaming to OpenSearch Service is supported only for log groups in the Standard log class. For more information about log classes, see Log classes.

Depending on the amount of log data being streamed, you might want to set a function-level concurrent execution limit on the function. For more information, see Lambda function scaling.

Note

Streaming large amounts of CloudWatch Logs data to OpenSearch Service might result in high usage charges. We recommend that you create a Budget in the Amazon Billing and Cost Management console. For more information, see Managing your costs with Amazon Budgets.

Prerequisites

Before you begin, create an OpenSearch Service domain. The domain can have either public access or VPC access, but you can't then modify the type of access after the domain is created. You might want to review your OpenSearch Service domain settings later, and modify your cluster configuration based on the amount of data your cluster will be processing. For instructions to create a domain, see Creating OpenSearch Service domains.

For more information about OpenSearch Service, see the Amazon OpenSearch Service Developer Guide.

Subscribe a log group to OpenSearch Service

You can use the CloudWatch console to subscribe a log group to OpenSearch Service.

To subscribe a log group to OpenSearch Service

  1. Open the CloudWatch console at https://console.amazonaws.cn/cloudwatch/.

  2. In the navigation pane, choose Log groups.

  3. Select the name of the log group.

  4. Choose Actions, Subscription filters, Create Amazon OpenSearch Service subscription filter.

  5. Choose whether you want to stream to a cluster in this account or another account.

    • If you chose this account, select the domain you created in the previous step.

    • If you chose another account, provide the domain ARN and endpoint.

  6. For Lambda IAM Execution Role, choose the IAM role that Lambda should use when executing calls to OpenSearch.

    The IAM role you choose must fulfill these requirements:

    • It must have lambda.amazonaws.com in the trust relationship.

    • It must include the following policy:

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:es:region:account-id:domain/target-domain-name/*"
        }
    ]
}

    • If the target OpenSearch Service domain uses VPC access, the role must have the AWSLambdaVPCAccessExecutionRole policy attached. This Amazon-managed policy grants Lambda access to the customer's VPC, enabling Lambda to write to the OpenSearch endpoint in the VPC.

  7. For Log format, choose a log format.

  8. For Subscription filter pattern, type the terms or pattern to find in your log events. This ensures that you send only the data you're interested in to your OpenSearch cluster. For more information, see Creating metrics from log events using filters.

  9. (Optional) For Select log data to test, select a log stream and then choose Test pattern to verify that your search filter is returning the results you expect.

  10. Choose Start streaming.