Create a CloudWatch agent configuration for OpenTelemetry
With the opentelemetry section of the agent configuration file, one agent
collects metrics, logs, and traces from your servers and applications and sends them to the CloudWatch OpenTelemetry
Protocol (OTLP) endpoints. The agent sets the endpoints, the Region, and request signing for
you. You query the metrics with PromQL.
Configure one source topic for each kind of telemetry that you collect. Each source topic
lists the fields of the source and shows a focused example. The prerequisites, how to start the
agent and verify data, and how to split the configuration across files
in this topic apply to every source. For a summary of the opentelemetry section
structure, see CloudWatch agent configuration file: OpenTelemetry section. To compare this
model with Classic metrics, see Choose a configuration model for the CloudWatch agent. For common customizations and the Classic
settings they replace, see Common customizations.
The configuration wizard doesn't create an opentelemetry section. Create the
configuration file manually.
Each signal comes from one or more sources in the collect section:
| Signal | Sources | Destination |
|---|---|---|
| Metrics | host_metrics (CPU, memory, disk, file system, network, processes),
prometheus (Prometheus targets), otlp (applications), and
database_insights |
CloudWatch OTLP metrics endpoint, queried with PromQL |
| Logs | files (log files), windows_events (Windows event logs),
and otlp (applications) |
CloudWatch Logs OTLP logs endpoint |
| Traces | otlp (applications) |
X-Ray OTLP traces endpoint, viewed in Transaction Search |
Prerequisites
-
Install CloudWatch agent version 1.300070.0 or later. To check the installed version on Linux, run
cat /opt/aws/amazon-cloudwatch-agent/bin/CWAGENT_VERSION. On Windows Server, readC:\Program Files\Amazon\AmazonCloudWatchAgent\CWAGENT_VERSION. -
Complete the IAM and network prerequisites. For more information, see Prerequisites.
-
If you send traces, enable Transaction Search. For more information, see Transaction Search.
Example: On-premises servers and macOS
On servers outside Amazon EC2, such as on-premises servers and Mac computers, the agent can't
detect resource attributes that identify the host. Host metrics don't include
@resource.host.id or @resource.host.name, so metrics from
different servers can't be told apart. Set an identifier for each server with
resource_attributes.
The following configuration sets host.name, collects host metrics and log
files, and receives metrics, logs, and traces from applications over OTLP. The {hostname} variable names each log stream
after the server.
{ "opentelemetry": { "resource_attributes": { "host.name": "server-01", "deployment.environment.name": "production" }, "collect": { "host_metrics": {}, "files": { "collect_list": [ { "file_path": "/var/log/app/*.log", "log_group_name": "/app/logs", "log_stream_name": "{hostname}" } ] }, "otlp": {} } } }
Start the agent with -m onPremise. The agent uses the credentials and Region
from the AmazonCloudWatchAgent profile. For more information, see (Installing on an on-premises server) Specify IAM credentials and Amazon Region.
Start the agent and verify data
-
Start the agent with your configuration file. The following command is for an Amazon EC2 instance running Linux.
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/path/to/config.json-sFor other platforms, see Starting the CloudWatch agent.
If the command doesn't return, the agent might be unable to reach the Amazon EC2 API. For more information, see Network requirements.
-
Confirm that the agent is running:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status -
In the CloudWatch console, open Query Studio and run a PromQL query for a host metric from the server. For example, on an Amazon EC2 instance:
{"system.memory.utilization", "@resource.host.id"="i-1234567890abcdef0"}On an on-premises server or Mac computer, filter on the
resource_attributesthat you set, such as@resource.host.name. Metrics typically appear within 1–2 minutes. A PromQL selector must include a metric name, so you can't select all metrics for an instance by label alone. To check a metric that receives data points infrequently, uselast_over_time, for examplelast_over_time({". For more information, see Running PromQL queries in Query Studio.my_metric"}[15m]) -
For log files and Windows events, open the log group in the CloudWatch Logs console and confirm that the log streams exist.
How the agent stores logs
The agent sends log files and Windows events to CloudWatch Logs as OpenTelemetry log records. Each log event is a JSON object with the following fields:
-
body— The original log line. For Windows events, a JSON object that contains the event fields, such aschannel,level, andprovider. -
resource.attributes— Attributes that identify the source, such ashost.id,host.name,log.file.name, and anyresource_attributesthat you set. -
severityTextandseverityNumber— For Windows events, the event level. -
timeUnixNanoandobservedTimeUnixNano— The event time and the time that the agent read the event. For log files,timeUnixNanois0unless you settimestamp_format.
To work with the original log line in CloudWatch Logs Logs Insights, query the body
field.
Split the configuration across files
You can split an opentelemetry configuration across multiple files and
combine them with the append-config option. The agent merges the sources under
collect from each file. For example, one file can configure
host_metrics and files for all servers, and a second file can add
otlp and prometheus on servers that run instrumented
applications.
Each source can appear in only one file. If two files configure the same source, such as
files, the agent fails to merge them with the error Different values are
specified for files. Put all entries for a source in the same file. For more
information about multiple configuration files, see Creating multiple CloudWatch agent configuration files.