Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Attach multiple network interfaces to Pods

By default, the Amazon VPC CNI plugin assigns one IP address to each pod. This IP address is attached to an elastic network interface that handles all incoming and outgoing traffic for the pod. To increase the bandwidth and packet per second rate performance, you can use the Multi-NIC feature of the VPC CNI to configure a multi-homed pod. A multi-homed pod is a single Kubernetes pod that uses multiple network interfaces (and multiple IP addresses). By running a multi-homed pod, you can spread its application traffic across multiple network interfaces by using concurrent connections. This is especially useful for Artificial Intelligence (AI), Machine Learning (ML), and High Performance Computing (HPC) use cases.

The following diagram shows a multi-homed pod running on a worker node with multiple network interface cards (NICs) in use.

Background

On Amazon EC2, an elastic network interface is a logical networking component in a VPC that represents a virtual network card. For many EC2 instance types, the network interfaces share a single network interface card (NIC) in hardware. This single NIC has a maximum bandwidth and packet per second rate.

If the multi-NIC feature is enabled, the VPC CNI doesn’t assign IP addresses in bulk, which it does by default. Instead, the VPC CNI assigns one IP address to a network interface on each network card on-demand when a new pod starts. This behavior reduces the rate of IP address exhaustion, which is increased by using multi-homed pods. Because the VPC CNI is assigning IP address on-demand, pods might take longer to start on instances with the multi-NIC feature enabled.

Considerations

IPv6 Considerations

Usage

After the multi-NIC feature is enabled in the VPC CNI and the aws-node pods have restarted, you can configure each workload to be multi-homed. The following example of a YAML configuration with the required annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-deployment
  namespace: ecommerce
  labels:
    app: orders
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      annotations:
         k8s.amazonaws.com/nicConfig: multi-nic-attachment
      labels:
        app: orders
    spec:
...

Frequently Asked Questions

1. What is a network interface card (NIC)?

A network interface card (NIC), also simply called a network card, is a physical device that enables network connectivity for the underlying cloud compute hardware. In modern EC2 servers, this refers to the Nitro network card. An Elastic Network Interface (ENI) is a virtual representation of this underlying network card.

Some EC2 instance types have multiple NICs for greater bandwidth and packet rate performance. For such instances, you can assign secondary ENIs to the additional network cards. For example, ENI #1 can function as the interface for the NIC attached to network card index 0, whereas ENI #2 can function as the interface for the NIC attached to a separate network card index.

2. What is a multi-homed pod?

A multi-homed pod is a single Kubernetes pod with multiple network interfaces (and by implication multiple IP addresses). Each pod network interface is associated with an Elastic Network Interface (ENI), and these ENIs are logical representations of separate NICs on the underlying worker node. With multiple network interfaces, a multi-homed pod has additional data transfer capacity, which also raises its data transfer rate.

3. Why should I use this feature?

If you need to scale network performance in your Kubernetes-based workloads, you can use the multi-NIC feature to run multi-homed pods that interface with all the underlying NICs that have an ENA device attached to it. Leveraging additional network cards raises the bandwidth capacity and packet rate performance in your applications by distributing application traffic across multiple concurrent connections. This is especially useful for Artificial Intelligence (AI), Machine Learning (ML), and High Performance Computing (HPC) use cases.

4. How do I use this feature?

5. How do I configure my workloads to use multiple NICs on a supported worker node?

To use multi-homed pods, you need to add the following annotation: k8s.amazonaws.com/nicConfig: multi-nic-attachment. This will attach an ENI from every NIC in the underlying instance to the pod (one to many mapping between a pod and the NICs).

If this annotation is missing, the VPC CNI assumes that your pod only requires 1 network interface and assigns it an IP from an ENI on any available NIC.

6. What network interface adapters are supported with this feature?

You can use any network interface adapter if you have at least one ENA attached to the underlying network card for IP traffic. For more information about ENA, see Elastic Network Adapter (ENA) in the Amazon EC2 User Guide.

Supported network device configurations:

7. Can I see if a node in my cluster has ENA support?

Yes, you can use the Amazon CLI or EC2 API to retrieve network information about an EC2 instance in your cluster. This provides details on whether or not the instance has ENA support. In the following example, replace <your-instance-id> with the EC2 instance ID of a node.

Amazon CLI example:

aws ec2 describe-instances --instance-ids <your-instance-id> --query "Reservations[].Instances[].EnaSupport"

Example output:

[ true ]

8. Can I see the different IP addresses associated with a pod?

No, not easily. However, you can use nsenter from the node to run common network tools such as ip route show and see the additional IP addresses and interfaces.

9. Can I control the number of network interfaces for my pods?

No. When your workload is configured to use multiple NICs on a supported instance, a single pod automatically has an IP address from every network card on the instance. Alternatively, single-homed pods will have one network interface attached to one NIC on the instance.

10. Can I configure my pods to use a specific NIC?

No, this isn’t supported. If a pod has the relevant annotation, then the VPC CNI automatically configures it to use every NIC with an ENA adapter on the worker node.

11. Does this feature work with the other VPC CNI networking features?

Yes, the multi-NIC feature in the VPC CNI works with both custom networking and enhanced subnet discovery. However, the multi-homed pods don’t use the custom subnets or security groups. Instead, the VPC CNI assigns IP addresses and network interfaces to the multi-homed pods with the same subnet and security group configuration as the node. For more information about custom networking, see Deploy Pods in alternate subnets with custom networking.

The multi-NIC feature in the VPC CNI doesn’t work with and can’t be combined with security groups for pods.

12. Can I use network policies with this feature?

Yes, you can use Kubernetes network policies with multi-NIC. Kubernetes network policies restrict network traffic to and from your pods. For more information about applying network policies with the VPC CNI, see Limit Pod traffic with Kubernetes network policies.

13. Is multi-NIC support enabled in EKS Auto Mode?

Multi-NIC isn’t supported for EKS Auto Mode clusters.