Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting the CloudWatch agent

You can use the information in this section to troubleshoot issues you might encounter with the CloudWatch agent.

When encountering issues with the CloudWatch agent, you can use the AWSSupport-TroubleshootCloudWatchAgent automation runbook. The Amazon troubleshooting tool can:

For detailed information on the Amazon troubleshooting tool, see Support Automation Workflow (SAW) Runbook - Troubleshoot CloudWatch agent.

CloudWatch agent command line parameters

To see the full list of parameters supported by the CloudWatch agent, enter the following at the command line at a computer where you have it installed:

amazon-cloudwatch-agent-ctl -help

Install the CloudWatch agent using Run Command fails

To install the CloudWatch agent using Systems Manager Run Command, the SSM Agent on the target server must be version 2.2.93.0 or later of the SSM Agent agent. If your SSM Agent isn't the correct version, you might see errors that include the following messages:

no latest version found for package AmazonCloudWatchAgent on platform linux

failed to download installation package reliably

For information about updating your SSM Agent version, see Installing and Configuring SSM Agent in the Amazon Systems Manager User Guide.

The CloudWatch agent won't start

If the CloudWatch agent fails to start, there might be an issue in your configuration. Configuration information is logged in the configuration-validation.log file. This file is located in /opt/aws/amazon-cloudwatch-agent/logs/configuration-validation.log on Linux servers and in $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\configuration-validation.log on servers running Windows Server.

Verify that the CloudWatch agent is running

You can query the CloudWatch agent to find whether it's running or stopped. You can use Amazon Systems Manager to do this remotely. You can also use the command line, but only to check the local server.

If the agent is running, the output resembles the following.

{
       "status": "running",
       "starttime": "2017-12-12T18:41:18",
       "version": "1.73.4"
}

If the agent is stopped, the "status" field displays "stopped".

The CloudWatch agent won't start, and the error mentions an Amazon EC2 Region

If the agent doesn't start and the error message mentions an Amazon EC2 Region endpoint, you might have configured the agent to need access to the Amazon EC2 endpoint without granting that access.

For example, if you specify a value for the append_dimensions parameter in the agent configuration file that depends on Amazon EC2 metadata and you use proxies, you must make sure that the server can access the endpoint for Amazon EC2. For more information about these endpoints, see Amazon Elastic Compute Cloud (Amazon EC2) in the Amazon Web Services General Reference.

The CloudWatch agent won't start on Windows Server

On Windows Server, you might see the following error:

Start-Service : Service 'Amazon CloudWatch Agent (AmazonCloudWatchAgent)' cannot be started due to the following
error: Cannot start service AmazonCloudWatchAgent on computer '.'.
At C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1:113 char:12
+     $svc | Start-Service
+            ~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
   ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

To fix this, first make sure that the server service is running. This error can be seen if the agent tries to start when the server service isn't running.

If the server service is already running, the following may be the issue. On some Windows Server installations, the CloudWatch agent takes more than 30 seconds to start. Because Windows Server, by default, allows only 30 seconds for services to start, this causes the agent to fail with an error similar to the following:

To fix this issue, increase the service timeout value. For more information, see A service does not start, and events 7000 and 7011 are logged in the Windows event log.

Where are the metrics?

If the CloudWatch agent has been running but you can't find metrics collected by it in the Amazon Web Services Management Console or the Amazon CLI, confirm that you're using the correct namespace. By default, the namespace for metrics collected by the agent is CWAgent. You can customize this namespace using the namespace field in the metrics section of the agent configuration file. If you don't see the metrics that you expect, check the configuration file to confirm the namespace being used.

When you first download the CloudWatch agent package, the agent configuration file is amazon-cloudwatch-agent.json. This file is in the directory where you ran the configuration wizard, or you might have moved it to a different directory. If you use the configuration wizard, the agent configuration file output from the wizard is named config.json. For more information about the configuration file, including the namespace field, see CloudWatch agent configuration file: Metrics section.

The CloudWatch agent takes a long time to run in a container or logs a hop limit error

When you run the CloudWatch agent as a container service and want to add Amazon EC2 metric dimensions to all metrics collected by the agent, you might see the following errors in version v1.247354.0 of the agent:

2022-06-07T03:36:11Z E! [processors.ec2tagger] ec2tagger: Unable to retrieve Instance Metadata Tags. This plugin must only be used on an EC2 instance.
2022-06-07T03:36:11Z E! [processors.ec2tagger] ec2tagger: Please increase hop limit to 2 by following this document https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html#configuring-IMDS-existing-instances.
2022-06-07T03:36:11Z E! [telegraf] Error running agent: could not initialize processor ec2tagger: EC2MetadataRequestError: failed to get EC2 instance identity document
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 401, request id: 
caused by:

You might see this error if the agent tries to get metadata from IMDSv2 inside a container without an appropriate hop limit. In versions of the agent earlier than v1.247354.0, you can experience this issue without seeing the log message.

To solve this, increase the hop limit to 2 by following the instructions in Configure the instance metadata options.

I updated my agent configuration but don't see the new metrics or logs in the CloudWatch console

If you update your CloudWatch agent configuration file, the next time that you start the agent, you need to use the fetch-config option. For example, if you stored the updated file on the local computer, enter the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -s -m ec2 -c file:configuration-file-path

The CloudWatch agent is using the wrong credentials

If the CloudWatch agent is using unexpected credentials, the agent may be resolving credentials from a different source than you expect. The agent follows a credentials provider chain that differs from the Amazon SDK in some cases. For more information, see CloudWatch agent credentials preference.

CloudWatch agent files and locations

The following table lists the files installed by and used with the CloudWatch agent, along with their locations on servers running Linux or Windows Server.

Logs generated by the CloudWatch agent

The agent generates a log while it runs. This log includes troubleshooting information. This log is the amazon-cloudwatch-agent.log file. This file is located in /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log on Linux servers and in $Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log on servers running Windows Server.

You can configure the agent to log additional details in the amazon-cloudwatch-agent.log file. In the agent configuration file, in the agent section, set the debug field to true, then reconfigure and restart the CloudWatch agent. To disable the logging of this extra information, set the debug field to false. Then, reconfigure and restart the agent. For more information, see Manually create or edit the CloudWatch agent configuration file.

In versions 1.247350.0 and later of the CloudWatch agent, you can optionally set the aws_sdk_log_level field in the agent section of the agent configuration file to one or more of the following options. Separate multiple options with the | character.

For more information about these options, see LogLevelType.

Stopping and restarting the CloudWatch agent

You can manually stop the CloudWatch agent using either Amazon Systems Manager or the command line.

To restart the agent, follow the instructions in (Optional) Modify the common configuration and named profile for CloudWatch agent.

Metrics or logs are not appearing in CloudWatch

If metrics or logs are not appearing in the CloudWatch console, follow this diagnostic flow:

For a full walkthrough of network and connectivity causes, see Why doesn't the unified CloudWatch agent push my metrics or log events to CloudWatch? on Amazon re:Post.

Troubleshoot VPC endpoint and private subnet connectivity

If the agent log shows dial tcp endpoint:443: i/o timeout or RequestError: send request failed, the agent cannot reach the Amazon service endpoints. For instances in private subnets without NAT:

Log duplicate prevention and the state file

The CloudWatch agent tracks its read position in a state file to avoid re-sending logs:

On restart, the agent resumes reading from the saved position in the state file. The agent does not re-send logs unless you delete the state file.

Keep the following behaviors in mind: