View a markdown version of this page

Create a CloudWatch agent configuration for OpenTelemetry - Amazon CloudWatch
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Create a CloudWatch agent configuration for OpenTelemetry

With the opentelemetry section of the agent configuration file, one agent collects metrics, logs, and traces from your servers and applications and sends them to the CloudWatch OpenTelemetry Protocol (OTLP) endpoints. The agent sets the endpoints, the Region, and request signing for you. You query the metrics with PromQL.

Configure one source topic for each kind of telemetry that you collect. Each source topic lists the fields of the source and shows a focused example. The prerequisites, how to start the agent and verify data, and how to split the configuration across files in this topic apply to every source. For a summary of the opentelemetry section structure, see CloudWatch agent configuration file: OpenTelemetry section. To compare this model with Classic metrics, see Choose a configuration model for the CloudWatch agent. For common customizations and the Classic settings they replace, see Common customizations.

Note

The configuration wizard doesn't create an opentelemetry section. Create the configuration file manually.

Each signal comes from one or more sources in the collect section:

Signal Sources Destination
Metrics host_metrics (CPU, memory, disk, file system, network, processes), prometheus (Prometheus targets), otlp (applications), and database_insights CloudWatch OTLP metrics endpoint, queried with PromQL
Logs files (log files), windows_events (Windows event logs), and otlp (applications) CloudWatch Logs OTLP logs endpoint
Traces otlp (applications) X-Ray OTLP traces endpoint, viewed in Transaction Search

Prerequisites

  • Install CloudWatch agent version 1.300070.0 or later. To check the installed version on Linux, run cat /opt/aws/amazon-cloudwatch-agent/bin/CWAGENT_VERSION. On Windows Server, read C:\Program Files\Amazon\AmazonCloudWatchAgent\CWAGENT_VERSION.

  • Complete the IAM and network prerequisites. For more information, see Prerequisites.

  • If you send traces, enable Transaction Search. For more information, see Transaction Search.

Example: On-premises servers and macOS

On servers outside Amazon EC2, such as on-premises servers and Mac computers, the agent can't detect resource attributes that identify the host. Host metrics don't include @resource.host.id or @resource.host.name, so metrics from different servers can't be told apart. Set an identifier for each server with resource_attributes.

The following configuration sets host.name, collects host metrics and log files, and receives metrics, logs, and traces from applications over OTLP. The {hostname} variable names each log stream after the server.

{ "opentelemetry": { "resource_attributes": { "host.name": "server-01", "deployment.environment.name": "production" }, "collect": { "host_metrics": {}, "files": { "collect_list": [ { "file_path": "/var/log/app/*.log", "log_group_name": "/app/logs", "log_stream_name": "{hostname}" } ] }, "otlp": {} } } }

Start the agent with -m onPremise. The agent uses the credentials and Region from the AmazonCloudWatchAgent profile. For more information, see (Installing on an on-premises server) Specify IAM credentials and Amazon Region.

Start the agent and verify data

  1. Start the agent with your configuration file. The following command is for an Amazon EC2 instance running Linux.

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/path/to/config.json -s

    For other platforms, see Starting the CloudWatch agent.

    If the command doesn't return, the agent might be unable to reach the Amazon EC2 API. For more information, see Network requirements.

  2. Confirm that the agent is running:

    sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
  3. In the CloudWatch console, open Query Studio and run a PromQL query for a host metric from the server. For example, on an Amazon EC2 instance:

    {"system.memory.utilization", "@resource.host.id"="i-1234567890abcdef0"}

    On an on-premises server or Mac computer, filter on the resource_attributes that you set, such as @resource.host.name. Metrics typically appear within 1–2 minutes. A PromQL selector must include a metric name, so you can't select all metrics for an instance by label alone. To check a metric that receives data points infrequently, use last_over_time, for example last_over_time({"my_metric"}[15m]). For more information, see Running PromQL queries in Query Studio.

  4. For log files and Windows events, open the log group in the CloudWatch Logs console and confirm that the log streams exist.

How the agent stores logs

The agent sends log files and Windows events to CloudWatch Logs as OpenTelemetry log records. Each log event is a JSON object with the following fields:

  • body — The original log line. For Windows events, a JSON object that contains the event fields, such as channel, level, and provider.

  • resource.attributes — Attributes that identify the source, such as host.id, host.name, log.file.name, and any resource_attributes that you set.

  • severityText and severityNumber — For Windows events, the event level.

  • timeUnixNano and observedTimeUnixNano — The event time and the time that the agent read the event. For log files, timeUnixNano is 0 unless you set timestamp_format.

To work with the original log line in CloudWatch Logs Logs Insights, query the body field.

Split the configuration across files

You can split an opentelemetry configuration across multiple files and combine them with the append-config option. The agent merges the sources under collect from each file. For example, one file can configure host_metrics and files for all servers, and a second file can add otlp and prometheus on servers that run instrumented applications.

Each source can appear in only one file. If two files configure the same source, such as files, the agent fails to merge them with the error Different values are specified for files. Put all entries for a source in the same file. For more information about multiple configuration files, see Creating multiple CloudWatch agent configuration files.