Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Prometheus metrics

Prometheus is a monitoring and time series database that scrapes endpoints. It provides the ability to query, aggregate, and store collected data. You can also use it for alerting and alert aggregation. This topic explains how to set up Prometheus as either a managed or open source option. Monitoring Amazon EKS control plane metrics is a common use case.

Amazon Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that makes it easy to monitor containerized applications and infrastructure at scale. It is a fully-managed service that automatically scales the ingestion, storage, querying, and alerting of your metrics. It also integrates with Amazon security services to enable fast and secure access to your data. You can use the open-source PromQL query language to query your metrics and alert on them.

For more information about how to use the Prometheus metrics after you turn them on, see the Amazon Managed Service for Prometheus User Guide.

Turn on Prometheus metrics when creating a cluster

When you create a new cluster, you can turn on the option to send metrics to Prometheus. In the Amazon Web Services Management Console, this option is in the Configure observability step of creating a new cluster. For more information, see Creating an Amazon EKS cluster.

Prometheus discovers and collects metrics from your cluster through a pull-based model called scraping. Scrapers are set up to gather data from your cluster infrastructure and containerized applications.

When you turn on the option to send Prometheus metrics, Amazon Managed Service for Prometheus provides a fully managed agentless scraper. Use the following Advanced configuration options to customize the default scraper as needed.

Amazon Managed Service for Prometheus refers to the agentless scraper that is created alongside the cluster as an Amazon managed collector. For more information about Amazon managed collectors, see Amazon managed collectors in the Amazon Managed Service for Prometheus User Guide.

Viewing Prometheus scraper details

After creating a cluster with the Prometheus metrics option turned on, you can view your Prometheus scraper details. When viewing your cluster in the Amazon Web Services Management Console, choose the Observability tab. A table shows a list of scrapers for the cluster, including information such as the scraper ID, alias, status, and creation date.

To see more details about the scraper, choose a scraper ID link. For example, you can view the scraper configuration, Amazon Resource Name (ARN), remote write URL, and networking information. You can use the scraper ID as input to Amazon Managed Service for Prometheus API operations like DescribeScraper and DeleteScraper. You can also use the API to create more scrapers.

For more information on using the Prometheus API, see the Amazon Managed Service for Prometheus API Reference.

Deploying Prometheus using Helm

Alternatively, you can deploy Prometheus into your cluster with Helm V3. If you already have Helm installed, you can check your version with the helm version command. Helm is a package manager for Kubernetes clusters. For more information about Helm and how to install it, see Using Helm with Amazon EKS.

After you configure Helm for your Amazon EKS cluster, you can use it to deploy Prometheus with the following steps.

Viewing the control plane raw metrics

As an alternative to deploying Prometheus, the Kubernetes API server exposes a number of metrics that are represented in a Prometheus format. These metrics are useful for monitoring and analysis. They are exposed internally through a metrics endpoint that refers to the /metrics HTTP API. Like other endpoints, this endpoint is exposed on the Amazon EKS control plane. This endpoint is primarily useful for looking at a specific metric. To analyze metrics over time, we recommend deploying Prometheus.

To view the raw metrics output, use kubectl with the --raw flag. This command allows you to pass any HTTP path and returns the raw response.

kubectl get --raw /metrics

An example output is as follows.

[...]
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="127.0.0.1:21362",method="POST"} 4994
rest_client_requests_total{code="200",host="127.0.0.1:443",method="DELETE"} 1
rest_client_requests_total{code="200",host="127.0.0.1:443",method="GET"} 1.326086e+06
rest_client_requests_total{code="200",host="127.0.0.1:443",method="PUT"} 862173
rest_client_requests_total{code="404",host="127.0.0.1:443",method="GET"} 2
rest_client_requests_total{code="409",host="127.0.0.1:443",method="POST"} 3
rest_client_requests_total{code="409",host="127.0.0.1:443",method="PUT"} 8
# HELP ssh_tunnel_open_count Counter of ssh tunnel total open attempts
# TYPE ssh_tunnel_open_count counter
ssh_tunnel_open_count 0
# HELP ssh_tunnel_open_fail_count Counter of ssh tunnel failed open attempts
# TYPE ssh_tunnel_open_fail_count counter
ssh_tunnel_open_fail_count 0

This raw output returns verbatim what the API server exposes. The different metrics are listed by line, with each line including a metric name, tags, and a value.

metric_name{"tag"="value"[,...]}
            value