Metrics helper - Amazon EKS
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Metrics helper

The CNI metrics helper is a tool that you can use to scrape network interface and IP address information, aggregate metrics at the cluster level, and publish the metrics to Amazon CloudWatch. To learn more about the metrics helper, see cni-metrics-helper on GitHub.

When managing an Amazon EKS cluster, you may want to know how many IP addresses have been assigned and how many are available. The CNI metrics helper helps you to:

  • Track these metrics over time

  • Troubleshoot and diagnose issues related to IP assignment and reclamation

  • Provide insights for capacity planning

When a node is provisioned, the CNI plugin automatically allocates a pool of secondary IP addresses from the node’s subnet to the primary network interface (eth0). This pool of IP addresses is known as the warm pool, and its size is determined by the node’s instance type. For example, a c4.large instance can support three network interfaces and nine IP addresses per interface. The number of IP addresses available for a given pod is one less than the maximum (of ten) because one of the IP addresses is reserved for the elastic network interface itself. For more information, see IP Addresses Per Network Interface Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

As the pool of IP addresses is depleted, the plugin automatically attaches another elastic network interface to the instance and allocates another set of secondary IP addresses to that interface. This process continues until the node can no longer support additional elastic network interfaces.

The following metrics are collected for your cluster and exported to CloudWatch:

  • The maximum number of network interfaces that the cluster can support

  • The number of network interfaces have been allocated to pods

  • The number of IP addresses currently assigned to pods

  • The total and maximum numbers of IP addresses available

  • The number of ipamD errors

Prerequisites

  • An existing Amazon Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. To determine whether you already have one, or to create one, see Create an IAM OIDC provider for your cluster.

  • Version 2.7.13 or later or 1.25.35 or later of the Amazon CLI installed and configured on your computer or Amazon CloudShell. For more information, see Installing, updating, and uninstalling the Amazon CLI and Quick configuration with aws configure in the Amazon Command Line Interface User Guide.

  • The kubectl command line tool is installed on your computer or Amazon CloudShell. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.21, you can use kubectl version 1.20,1.21, or 1.22 with it. To install or upgrade kubectl, see Installing or updating kubectl.

  • If your cluster is 1.21 or later, make sure that your Amazon VPC CNI plugin for Kubernetes, kube-proxy, and CoreDNS add-ons are at the minimum versions listed in Service account tokens.

Deploy the CNI metrics helper

Create an IAM policy and role and deploy the metrics helper.

To deploy the CNI metrics helper

  1. Create an IAM policy that grants the CNI metrics helper cloudwatch:PutMetricData permissions to send metric data to CloudWatch.

    1. Copy the following contents to a file named cni-metrics-helper-policy.json.

      { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": "*" } ] }
    2. Create an IAM policy named AmazonEKSVPCCNIMetricsHelperPolicy.

      aws iam create-policy --policy-name AmazonEKSVPCCNIMetricsHelperPolicy \ --description "Grants permission to write metrics to CloudWatch" \ --policy-document file://cni-metrics-helper-policy.json
  2. Create an IAM role and attach the IAM policy to it. Create a Kubernetes service account. Annotate the Kubernetes service account with the IAM role ARN and the IAM role with the Kubernetes service account name. You can create the role using eksctl or the Amazon CLI.

    eksctl

    Run the following command to create the IAM role. Replace my-cluster with your cluster name, 111122223333 with your account ID, and region-code with the Amazon Web Services Region that your cluster is in.

    eksctl create iamserviceaccount \ --name cni-metrics-helper \ --namespace kube-system \ --cluster my-cluster \ --role-name "AmazonEKSVPCCNIMetricsHelperRole" \ --attach-policy-arn arn:aws-cn:iam::111122223333:policy/AmazonEKSVPCCNIMetricsHelperPolicy \ --approve
    Amazon CLI
    1. Determine your cluster's OIDC provider URL. Replace my-cluster with your cluster name. If the output from the command is None, review the Prerequisites.

      aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text

      The example output is as follows.

      oidc.eks.region-code.amazonaws.com.cn/id/EXAMPLED539D4633E53DE1B71EXAMPLE
    2. Create the IAM role, granting the Kubernetes service account the AssumeRoleWithWebIdentity action.

      1. Copy the following contents to a file named trust-policy.json. Replace 111122223333 with your account ID. Replace EXAMPLED539D4633E53DE1B71EXAMPLE and region-code with the values returned in the previous step.

        { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws-cn:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com.cn/id/EXAMPLED539D4633E53DE1B71EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.region-code.amazonaws.com.cn/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com", "oidc.eks.region-code.amazonaws.com.cn/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:kube-system:cni-metrics-helper" } } } ] }
      2. Create the role.

        aws iam create-role \ --role-name AmazonEKSVPCCNIMetricsHelperRole \ --assume-role-policy-document file://"trust-policy.json"
    3. Attach the IAM policy to the role. Replace 111122223333 with your account ID.

      aws iam attach-role-policy \ --policy-arn arn:aws-cn:iam::111122223333:policy/AmazonEKSVPCCNIMetricsHelperPolicy \ --role-name AmazonEKSVPCCNIMetricsHelperRole
  3. Add the recommended version of the CNI metrics helper to your cluster with the following command.

    Important

    You should only update one minor version at a time. For example, if your current minor version is 1.9 and you want to update to 1.11, you should update to 1.10 first, then update to 1.11 by changing the version number in the following command.

    The recommended and latest version work with all Amazon EKS supported Kubernetes versions.

    China (Beijing) (cn-north-1) or China (Ningxia) (cn-northwest-1)

    kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.11.2/config/master/cni-metrics-helper-cn.yaml
  4. Annotate the cni-metrics-helper Kubernetes service account created in a previous step with the ARN of the IAM role that you created previously. Replace 111122223333 with your account ID, my-cluster with your cluster name, and AmazonEKSVPCCNIMetricsHelperRole with the name of the IAM role that you created in a previous step.

    kubectl annotate serviceaccount cni-metrics-helper \ -n kube-system \ eks.amazonaws.com/role-arn=arn:aws-cn:iam::111122223333:role/AmazonEKSVPCCNIMetricsHelperRole
  5. (Optional) Configure the Amazon Security Token Service endpoint type used by your Kubernetes service account. For more information, see Configure the Amazon Security Token Service endpoint for a service account.

  6. Restart the cni-metrics-helper deployment.

    kubectl rollout restart \ deployment cni-metrics-helper \ -n kube-system

Creating a metrics dashboard

After you have deployed the CNI metrics helper, you can view the CNI metrics in the CloudWatch console. This topic helps you to create a dashboard for viewing your cluster's CNI metrics.

To create a CNI metrics dashboard

  1. Open the CloudWatch console at https://console.amazonaws.cn/cloudwatch/.

  2. In the left navigation pane, choose Metrics and then select All metrics.

  3. Under Custom Namespaces, select Kubernetes.

  4. Select CLUSTER_ID.

  5. On the Metrics tab, select the metrics you want to add to the dashboard.

  6. At the upper right of the console, select Actions, and then Add to dashboard.

  7. In the Select a dashboard section, select Create new, enter a name for your dashboard, such as EKS-CNI-metrics, and then select Create.

  8. In the Widget type section, select Number.

  9. In the Customize widget title section, enter a logical name for your dashboard title, such as EKS CNI metrics.

  10. Select Add to dashboard to finish. Now your CNI metrics are added to a dashboard that you can monitor.

    
                        EKS CNI metrics