Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters - Amazon CloudWatch
Set up on Amazon EKS and Kubernetes clusters
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters

This section explains how to set up the CloudWatch agent with Prometheus monitoring in a cluster running Amazon EKS or Kubernetes. After you do this, the agent automatically scrapes and imports metrics for the following workloads running in that cluster.

  • Amazon App Mesh

  • NGINX

  • Memcached

  • Java/JMX

  • HAProxy

  • Fluent Bit

You can also configure the agent to scrape and import additional Prometheus workloads and sources.

Before following these steps to install the CloudWatch agent for Prometheus metric collection, you must have a cluster running on Amazon EKS or a Kubernetes cluster running on an Amazon EC2 instance.

VPC security group requirements

The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP.

The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP.

Topics

Install the CloudWatch agent with Prometheus metrics collection on Amazon EKS and Kubernetes clusters

This section explains how to set up the CloudWatch agent with Prometheus monitoring in a cluster running Amazon EKS or Kubernetes. After you do this, the agent automatically scrapes and imports metrics for the following workloads running in that cluster.

  • Amazon App Mesh

  • NGINX

  • Memcached

  • Java/JMX

  • HAProxy

  • Fluent Bit

You can also configure the agent to scrape and import additional Prometheus workloads and sources.

Before following these steps to install the CloudWatch agent for Prometheus metric collection, you must have a cluster running on Amazon EKS or a Kubernetes cluster running on an Amazon EC2 instance.

VPC security group requirements

The ingress rules of the security groups for the Prometheus workloads must open the Prometheus ports to the CloudWatch agent for scraping the Prometheus metrics by the private IP.

The egress rules of the security group for the CloudWatch agent must allow the CloudWatch agent to connect to the Prometheus workloads' port by private IP.

Setting up IAM roles

The first step is to set up the necessary IAM role in the cluster. There are two methods:

  • Set up an IAM role for a service account, also known as a service role. This method works for both the EC2 launch type and the Fargate launch type.

  • Add an IAM policy to the IAM role used for the cluster. This works only for the EC2 launch type.

Set up a service role (EC2 launch type and Fargate launch type)

To set up a service role, enter the following command. Replace MyCluster with the name of the cluster.

eksctl create iamserviceaccount \
 --name cwagent-prometheus \
--namespace amazon-cloudwatch \
 --cluster MyCluster \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--approve \
--override-existing-serviceaccounts

Add a policy to the cluster's IAM role (EC2 launch type only)

To set up the IAM policy in a cluster for Prometheus support

  1. Open the Amazon EC2 console at https://console.amazonaws.cn/ec2/.

  2. In the navigation pane, choose Instances.

  3. You need to find the prefix of the IAM role name for the cluster. To do this, select the check box next to the name of an instance that is in the cluster, and choose Actions, Instance Settings, Attach/Replace IAM Role. Then copy the prefix of the IAM role, such as eksctl-dev303-workshop-nodegroup.

  4. Open the IAM console at https://console.amazonaws.cn/iam/.

  5. In the navigation pane, choose Roles.

  6. Use the search box to find the prefix that you copied earlier in this procedure, and choose that role.

  7. Choose Attach policies.

  8. Use the search box to find CloudWatchAgentServerPolicy. Select the check box next to CloudWatchAgentServerPolicy, and choose Attach policy.

Installing the CloudWatch agent to collect Prometheus metrics

You must install the CloudWatch agent in the cluster to collect the metrics. How to install the agent differs for Amazon EKS clusters and Kubernetes clusters.

Delete previous versions of the CloudWatch agent with Prometheus support

If you have already installed a version of the CloudWatch agent with Prometheus support in your cluster, you must delete that version by entering the following command. This is necessary only for previous versions of the agent with Prometheus support. You do not need to delete the CloudWatch agent that enables Container Insights without Prometheus support.

kubectl delete deployment cwagent-prometheus -n amazon-cloudwatch

Installing the CloudWatch agent on Amazon EKS clusters with the EC2 launch type

To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster, follow these steps.

To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster

  1. Enter the following command to check whether the amazon-cloudwatch namespace has already been created:

    kubectl get namespace

  2. If amazon-cloudwatch is not displayed in the results, create it by entering the following command:

    kubectl create namespace amazon-cloudwatch

  3. To deploy the agent with the default configuration and have it send data to the Amazon Region that it is installed in, enter the following command:

    kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml

    To have the agent send data to a different Region instead, follow these steps:

    1. Download the YAML file for the agent by entering the following command:

      curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml

    2. Open the file with a text editor, and search for the cwagentconfig.json block of the file.

    3. Add the highlighted lines, specifying the Region that you want:

      cwagentconfig.json: |
    {
      "agent": {
        "region": "us-east-2"
      },
      "logs": { ...

    4. Save the file and deploy the agent using your updated file.

      kubectl apply -f prometheus-eks.yaml

Installing the CloudWatch agent on Amazon EKS clusters with the Fargate launch type

To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster with the Fargate launch type, follow these steps.

To install the CloudWatch agent with Prometheus support on an Amazon EKS cluster with the Fargate launch type

  1. Enter the following command to create a Fargate profile for the CloudWatch agent so that it can run inside the cluster. Replace MyCluster with the name of the cluster.

    eksctl create fargateprofile --cluster MyCluster \
--name amazon-cloudwatch \
--namespace amazon-cloudwatch

  2. To install the CloudWatch agent, enter the following command. Replace MyCluster with the name of the cluster. This name is used in the log group name that stores the log events collected by the agent, and is also used as a dimension for the metrics collected by the agent.

    Replace region with the name of the Region where you want the metrics to be sent. For example, us-west-1. 

    curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml | 
sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" | 
kubectl apply -f -

Installing the CloudWatch agent on a Kubernetes cluster

To install the CloudWatch agent with Prometheus support on a cluster running Kubernetes, enter the following command:

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-k8s.yaml | 
sed "s/{{cluster_name}}/MyCluster/;s/{{region_name}}/region/" | 
kubectl apply -f -

Replace MyCluster with the name of the cluster. This name is used in the log group name that stores the log events collected by the agent, and is also used as a dimension for the metrics collected by the agent.

Replace region with the name of the Amazon Region where you want the metrics to be sent. For example, us-west-1.

Verify that the agent is running

On both Amazon EKS and Kubernetes clusters, you can enter the following command to confirm that the agent is running.

kubectl get pod -l "app=cwagent-prometheus" -n amazon-cloudwatch

If the results include a single CloudWatch agent pod in the Running state, the agent is running and collecting Prometheus metrics. By default the CloudWatch agent collects metrics for App Mesh, NGINX, Memcached, Java/JMX, and HAProxy every minute. For more information about those metrics, see Prometheus metrics collected by the CloudWatch agent. For instructions on how to see your Prometheus metrics in CloudWatch, see Viewing your Prometheus metrics

You can also configure the CloudWatch agent to collect metrics from other Prometheus exporters. For more information, see Scraping additional Prometheus sources and importing those metrics.