Collecting Karpenter metrics with OTel Container Insights
Karpenter is the recommended node autoscaler for Amazon EKS. Container Insights collects Karpenter metrics through Prometheus scraping. With Container Insights, you gain built-in visibility into node provisioning, scaling speed, and resource use. You do not need to set up Prometheus or Grafana manually.
Karpenter metrics help you answer operational questions such as:
-
Why is my pod pending?
-
How fast is Karpenter provisioning nodes?
-
Is consolidation working effectively?
Prerequisites
Karpenter metrics collection is enabled by default when the following requirements are met.
-
Amazon EKS cluster running Kubernetes 1.28 or later
-
Karpenter installed and running in the cluster (default namespace:
kube-system) -
OTel Container Insights enabled on the cluster (see Quick start)
-
The
amazon-cloudwatch-observabilityadd-on inACTIVEstatus
Configuring Karpenter metrics collection
Karpenter metrics collection is enabled by default. If you already have OTel Container
Insights enabled (otelContainerInsights.enabled: true) and Karpenter is running
in your cluster, metrics start flowing automatically with no additional configuration.
If the Karpenter scrape target is not present in the cluster, the pipeline finds no pods and does nothing.
To disable Karpenter metrics collection, set
otelContainerInsights.solutions.karpenter.enabled to false.
To disable all solutions at once, set otelContainerInsights.solutions.enabled
to false. The global toggle overrides individual solution settings. If
solutions.enabled is false, no solutions collect metrics, even if
their individual enabled flag is true.
The following Helm values show the default configuration.
otelContainerInsights: solutions: enabled: true karpenter: enabled: true
To disable Karpenter metrics using the Amazon EKS add-on, run the following Amazon CLI command.
Replace cluster-name with the name of your Amazon EKS cluster.
aws eks update-addon \ --cluster-namecluster-name\ --addon-name amazon-cloudwatch-observability \ --configuration-values '{"otelContainerInsights":{"solutions":{"karpenter":{"enabled":false}}}}' \ --resolve-conflicts OVERWRITE
To disable Karpenter metrics using the Helm chart, run the following command.
helm upgrade amazon-cloudwatch-observability \ aws-observability/amazon-cloudwatch-observability \ -n amazon-cloudwatch \ --reuse-values \ --set otelContainerInsights.solutions.karpenter.enabled=false
After you enable Karpenter metrics, the CloudWatch agent finds Karpenter pods by the label
app.kubernetes.io/name=karpenter. It scrapes the /metrics
endpoint on the port named http-metrics (default 8080) every 30
seconds.
Karpenter metrics pipeline architecture
The Karpenter metrics pipeline uses the following data flow:
-
The Prometheus receiver discovers Karpenter pods using Kubernetes service discovery.
-
The receiver scrapes the
/metricsendpoint on each Karpenter pod. -
The OpenTelemetry processor chain adds Kubernetes and cloud resource labels to each metric.
-
The exporter sends the enriched metrics to CloudWatch through OTLP.
The Karpenter pipeline runs independently from other metric pipelines (API server, Kube State Metrics, cAdvisor). This means a failure in one pipeline does not affect the others. If the Karpenter scrape target goes down, other pipelines keep working.
Karpenter metrics are cluster-scoped.
Using Karpenter node labels with Container Insights
Karpenter adds labels to each node that it creates. Container Insights adds
node labels to its metrics as k8s.node.label.* attributes. This applies to all
Container Insights metrics from a Karpenter node, not only to the
Karpenter metrics on this page.
From version 7.0.0, Container Insights does not send the node labels that
Karpenter derives from the instance type, such as the instance family and size.
It also does not send the Karpenter node status labels. Use
host.type for the instance type.
Labels that describe how Karpenter provisioned the node stay, for example the node pool, the capacity type, and the node class.
Viewing the Karpenter dashboard
After you turn on Karpenter metrics, you can view the dashboard in the CloudWatch console.
To view the Karpenter dashboard
-
Open the CloudWatch console at CloudWatch console
. -
In the navigation pane, choose Container Insights, then choose Performance monitoring.
-
For Service, choose EKS OTEL.
-
In the navigation pane, expand Solutions and choose Karpenter.
The dashboard displays 30 metrics organized into 8 categories: Pod Scheduling, Node Resources, Cluster State, NodePool, Cloud Provider, Disruption, Interruption, and Controller Health.
Available metrics
The following table lists the 30 Karpenter metrics that Container Insights collects.
| Metric name | Category | Description |
|---|---|---|
karpenter_pods_state |
Pod Scheduling | Number of pods by state (phase label) |
karpenter_pods_bound_duration_seconds |
Pod Scheduling | Time for pods to reach bound state |
karpenter_nodes_allocatable |
Node Resources | Allocatable resources per node by resource type |
karpenter_nodes_total_daemon_requests |
Node Resources | Total resource requests from DaemonSet pods |
karpenter_nodes_total_daemon_limits |
Node Resources | Total resource limits from DaemonSet pods |
karpenter_nodes_total_pod_requests |
Node Resources | Total resource requests from all pods |
karpenter_nodes_total_pod_limits |
Node Resources | Total resource limits from all pods |
karpenter_nodes_system_overhead |
Node Resources | Resources reserved for system daemons |
karpenter_nodes_current_lifetime_seconds |
Node Resources | Current age of each node in seconds |
karpenter_cluster_state_node_count |
Cluster State | Number of nodes tracked in Karpenter cluster state |
karpenter_cluster_state_synced |
Cluster State | Whether cluster state is synced (1=synced, 0=unsynced) |
karpenter_cluster_state_unsynced_time_seconds |
Cluster State | Duration cluster state has been unsynced |
karpenter_cluster_utilization_percent |
Cluster State | Percentage of allocatable resources used by pod requests |
karpenter_nodepools_limit |
NodePool | Configured resource limits per NodePool |
karpenter_nodepools_usage |
NodePool | Current resource usage per NodePool |
karpenter_cloudprovider_duration_seconds |
Cloud Provider | Cloud provider API call latency by method |
karpenter_cloudprovider_instance_type_cpu_cores |
Cloud Provider | Available vCPU cores per instance type |
karpenter_cloudprovider_instance_type_memory_bytes |
Cloud Provider | Available memory per instance type |
karpenter_cloudprovider_instance_type_offering_available |
Cloud Provider | Whether an instance type offering is available |
karpenter_cloudprovider_instance_type_offering_price_estimate |
Cloud Provider | Estimated hourly price per instance type offering |
karpenter_voluntary_disruption_eligible_nodes |
Disruption | Nodes eligible for voluntary disruption by reason |
karpenter_voluntary_disruption_consolidation_timeouts_total |
Disruption | Number of consolidation evaluation timeouts |
karpenter_voluntary_disruption_decision_evaluation_duration_seconds |
Disruption | Time to evaluate disruption decisions |
karpenter_nodes_created_total |
Node Resources | Total nodes created by Karpenter |
karpenter_nodes_terminated_total |
Node Resources | Total nodes terminated by Karpenter (by reason) |
karpenter_interruption_received_messages_total |
Interruption | SQS interruption messages received (spot, rebalance, etc.) |
karpenter_interruption_deleted_messages_total |
Interruption | SQS interruption messages deleted/processed |
controller_runtime_reconcile_errors_total |
Controller Health | Total reconciliation errors across controllers |
workqueue_depth |
Controller Health | Current depth of the controller work queue |
leader_election_master_status |
Controller Health | Whether this instance is the elected leader (1=leader, 0=standby) |