View a markdown version of this page

Collecting Karpenter metrics with OTel Container Insights - Amazon CloudWatch
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Collecting Karpenter metrics with OTel Container Insights

Karpenter is the recommended node autoscaler for Amazon EKS. Container Insights collects Karpenter metrics through Prometheus scraping. With Container Insights, you gain built-in visibility into node provisioning, scaling speed, and resource use. You do not need to set up Prometheus or Grafana manually.

Karpenter metrics help you answer operational questions such as:

  • Why is my pod pending?

  • How fast is Karpenter provisioning nodes?

  • Is consolidation working effectively?

Prerequisites

Karpenter metrics collection is enabled by default when the following requirements are met.

  • Amazon EKS cluster running Kubernetes 1.28 or later

  • Karpenter installed and running in the cluster (default namespace: kube-system)

  • OTel Container Insights enabled on the cluster (see Quick start)

  • The amazon-cloudwatch-observability add-on in ACTIVE status

Configuring Karpenter metrics collection

Karpenter metrics collection is enabled by default. If you already have OTel Container Insights enabled (otelContainerInsights.enabled: true) and Karpenter is running in your cluster, metrics start flowing automatically with no additional configuration.

If the Karpenter scrape target is not present in the cluster, the pipeline finds no pods and does nothing.

To disable Karpenter metrics collection, set otelContainerInsights.solutions.karpenter.enabled to false. To disable all solutions at once, set otelContainerInsights.solutions.enabled to false. The global toggle overrides individual solution settings. If solutions.enabled is false, no solutions collect metrics, even if their individual enabled flag is true.

The following Helm values show the default configuration.

otelContainerInsights: solutions: enabled: true karpenter: enabled: true

To disable Karpenter metrics using the Amazon EKS add-on, run the following Amazon CLI command. Replace cluster-name with the name of your Amazon EKS cluster.

aws eks update-addon \ --cluster-name cluster-name \ --addon-name amazon-cloudwatch-observability \ --configuration-values '{"otelContainerInsights":{"solutions":{"karpenter":{"enabled":false}}}}' \ --resolve-conflicts OVERWRITE

To disable Karpenter metrics using the Helm chart, run the following command.

helm upgrade amazon-cloudwatch-observability \ aws-observability/amazon-cloudwatch-observability \ -n amazon-cloudwatch \ --reuse-values \ --set otelContainerInsights.solutions.karpenter.enabled=false

After you enable Karpenter metrics, the CloudWatch agent finds Karpenter pods by the label app.kubernetes.io/name=karpenter. It scrapes the /metrics endpoint on the port named http-metrics (default 8080) every 30 seconds.

Karpenter metrics pipeline architecture

The Karpenter metrics pipeline uses the following data flow:

  1. The Prometheus receiver discovers Karpenter pods using Kubernetes service discovery.

  2. The receiver scrapes the /metrics endpoint on each Karpenter pod.

  3. The OpenTelemetry processor chain adds Kubernetes and cloud resource labels to each metric.

  4. The exporter sends the enriched metrics to CloudWatch through OTLP.

The Karpenter pipeline runs independently from other metric pipelines (API server, Kube State Metrics, cAdvisor). This means a failure in one pipeline does not affect the others. If the Karpenter scrape target goes down, other pipelines keep working.

Note

Karpenter metrics are cluster-scoped.

Using Karpenter node labels with Container Insights

Karpenter adds labels to each node that it creates. Container Insights adds node labels to its metrics as k8s.node.label.* attributes. This applies to all Container Insights metrics from a Karpenter node, not only to the Karpenter metrics on this page.

From version 7.0.0, Container Insights does not send the node labels that Karpenter derives from the instance type, such as the instance family and size. It also does not send the Karpenter node status labels. Use host.type for the instance type.

Labels that describe how Karpenter provisioned the node stay, for example the node pool, the capacity type, and the node class.

Viewing the Karpenter dashboard

After you turn on Karpenter metrics, you can view the dashboard in the CloudWatch console.

To view the Karpenter dashboard
  1. Open the CloudWatch console at CloudWatch console.

  2. In the navigation pane, choose Container Insights, then choose Performance monitoring.

  3. For Service, choose EKS OTEL.

  4. In the navigation pane, expand Solutions and choose Karpenter.

The dashboard displays 30 metrics organized into 8 categories: Pod Scheduling, Node Resources, Cluster State, NodePool, Cloud Provider, Disruption, Interruption, and Controller Health.

Available metrics

The following table lists the 30 Karpenter metrics that Container Insights collects.

Metric name Category Description
karpenter_pods_state Pod Scheduling Number of pods by state (phase label)
karpenter_pods_bound_duration_seconds Pod Scheduling Time for pods to reach bound state
karpenter_nodes_allocatable Node Resources Allocatable resources per node by resource type
karpenter_nodes_total_daemon_requests Node Resources Total resource requests from DaemonSet pods
karpenter_nodes_total_daemon_limits Node Resources Total resource limits from DaemonSet pods
karpenter_nodes_total_pod_requests Node Resources Total resource requests from all pods
karpenter_nodes_total_pod_limits Node Resources Total resource limits from all pods
karpenter_nodes_system_overhead Node Resources Resources reserved for system daemons
karpenter_nodes_current_lifetime_seconds Node Resources Current age of each node in seconds
karpenter_cluster_state_node_count Cluster State Number of nodes tracked in Karpenter cluster state
karpenter_cluster_state_synced Cluster State Whether cluster state is synced (1=synced, 0=unsynced)
karpenter_cluster_state_unsynced_time_seconds Cluster State Duration cluster state has been unsynced
karpenter_cluster_utilization_percent Cluster State Percentage of allocatable resources used by pod requests
karpenter_nodepools_limit NodePool Configured resource limits per NodePool
karpenter_nodepools_usage NodePool Current resource usage per NodePool
karpenter_cloudprovider_duration_seconds Cloud Provider Cloud provider API call latency by method
karpenter_cloudprovider_instance_type_cpu_cores Cloud Provider Available vCPU cores per instance type
karpenter_cloudprovider_instance_type_memory_bytes Cloud Provider Available memory per instance type
karpenter_cloudprovider_instance_type_offering_available Cloud Provider Whether an instance type offering is available
karpenter_cloudprovider_instance_type_offering_price_estimate Cloud Provider Estimated hourly price per instance type offering
karpenter_voluntary_disruption_eligible_nodes Disruption Nodes eligible for voluntary disruption by reason
karpenter_voluntary_disruption_consolidation_timeouts_total Disruption Number of consolidation evaluation timeouts
karpenter_voluntary_disruption_decision_evaluation_duration_seconds Disruption Time to evaluate disruption decisions
karpenter_nodes_created_total Node Resources Total nodes created by Karpenter
karpenter_nodes_terminated_total Node Resources Total nodes terminated by Karpenter (by reason)
karpenter_interruption_received_messages_total Interruption SQS interruption messages received (spot, rebalance, etc.)
karpenter_interruption_deleted_messages_total Interruption SQS interruption messages deleted/processed
controller_runtime_reconcile_errors_total Controller Health Total reconciliation errors across controllers
workqueue_depth Controller Health Current depth of the controller work queue
leader_election_master_status Controller Health Whether this instance is the elected leader (1=leader, 0=standby)