Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Help improve this page

To contribute to this user guide, choose the Edit this page on GitHub link that is located in the right pane of every page.

Deploy an accelerated workload

This tutorial demonstrates how Amazon EKS Auto Mode simplifies launching hardware-accelerated workloads. Amazon EKS Auto Mode streamlines operations beyond the cluster itself by automating key infrastructure components providing compute, networking, load balancing, storage, and Identity and Access Management capabilities out of the box.

Amazon EKS Auto Mode includes the drivers and device plugins required for certain instance types, such as NVIDIA and Amazon Neuron drivers, along with the components that manage Elastic Fabric Adapter (EFA) network devices. You do not have to install or update these components.

EKS Auto Mode automatically manages drivers for these accelerators:

EKS Auto Mode also manages the high-performance network devices that distributed training and inference workloads depend on:

To expose these accelerators and network devices to your workloads, the EKS Auto Mode node runtime includes the following device management components. Each runs automatically and isn’t visible as a daemon set in your cluster, and you don’t install, configure, or upgrade any of them.

For a comparison of DRA and device plugins, guidance on which to choose, and the full set of considerations for each, see Manage hardware devices on Amazon EKS.

Amazon EKS Auto Mode eliminates the toil of accelerator driver and device plugin management.

On Kubernetes version 1.36 and later, you can use Dynamic Resource Allocation (DRA) instead of device plugins for NVIDIA GPUs and EFA devices. With DRA, you select devices by attribute, allocate topology-aligned GPU and EFA devices, and share a GPU across Pods, while EKS Auto Mode scales instances to match your claims. EKS Auto Mode keeps a DRA driver and a device plugin from managing the same device type on one node. For more information, see Deploy an accelerated workload with Dynamic Resource Allocation and Using DRA with EKS Auto Mode.

You can also benefit from cost savings by scaling the cluster to zero. You can configure EKS Auto Mode to terminate instances when no workloads are running. This is useful for batch based inference workloads.

The following provides an example of how to launch accelerated workloads with Amazon EKS Auto Mode.

Prerequisites

Step 1: Deploy a GPU workload

In this example, you will create a NodePool for NVIDIA based workloads that requires 45GB GPU memory. With EKS Auto Mode, you use Kubernetes scheduling constraints to define your instance requirements.

To deploy the Amazon EKS Auto Mode NodePool and the sample workload, review the following NodePool and Pod definition and save as nodepool-gpu.yaml and pod.yaml:

nodepool-gpu.yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  disruption:
    budgets:
    - nodes: 10%
    consolidateAfter: 1h
    consolidationPolicy: WhenEmpty
  template:
    metadata: {}
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "eks.amazonaws.com/instance-family"
          operator: In
          values:
          - g6e
          - g6
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      terminationGracePeriod: 24h0m0s

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
spec:
  nodeSelector:
    eks.amazonaws.com/compute-type: auto
  restartPolicy: OnFailure
  containers:
  - name: nvidia-smi
    image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
    args:
    - "nvidia-smi"
    resources:
      requests:
        memory: "30Gi"
        cpu: "3500m"
        nvidia.com/gpu: 1
      limits:
        memory: "30Gi"
        nvidia.com/gpu: 1
  tolerations:
  - key: nvidia.com/gpu
    effect: NoSchedule
    operator: Exists

Note the eks.amazonaws.com/compute-type: auto selector requires the workload be deployed on an Amazon EKS Auto Mode node. The NodePool also sets a taint that only allows pods with tolerations for NVIDIA GPUs to be scheduled.

Apply the NodePool and workload to your cluster.

kubectl apply -f nodepool-gpu.yaml
kubectl apply -f pod.yaml

You should see the following output:

nodepool.karpenter.sh/gpu created
pod/nvidia-smi created

Wait a few seconds, and check the nodes in your cluster. You should now see a new node provisioned in your Amazon EKS Auto Mode cluster:

> kubectl get nodes

NAME        TYPE          CAPACITY    ZONE         NODE                  READY   AGE
gpu-dnknr   g6e.2xlarge   on-demand   us-west-2b   i-02315c7d7643cdee6   True    76s

Step 2: Validate

You can see Amazon EKS Auto Mode launched a g6e.2xlarge rather than a g6.2xlarge as the workload required an instance with an L40S GPU, according to the following Kubernetes scheduling constraints:

...
  nodeSelector:
    eks.amazonaws.com/instance-gpu-name: l40s
...
    requests:
        memory: "30Gi"
        cpu: "3500m"
        nvidia.com/gpu: 1
      limits:
        memory: "30Gi"
        nvidia.com/gpu: 1

Now, look at the container’s logs by running the following command:

kubectl logs nvidia-smi

Sample output:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.178.04             Driver Version: 580.178.04   CUDA Version: 13.0     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA L40S                    On  | 00000000:30:00.0 Off |                    0 |
| N/A   27C    P8              23W / 350W |      0MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

You can see that the container has detected it’s running on an instance with an NVIDIA GPU and that you’ve not had to install any device drivers, as this is managed by Amazon EKS Auto Mode.

Step 3: Clean-up

To remove all objects created, use kubectl to delete the sample deployment and NodePool so the node is terminated:

kubectl delete -f nodepool-gpu.yaml
kubectl delete -f pod.yaml

Deploy an accelerated workload with Dynamic Resource Allocation

The walkthrough in Deploy an accelerated workload requests a GPU as the nvidia.com/gpu extended resource, which the NVIDIA device plugin advertises. On Kubernetes version 1.36 and later, you can instead request devices through Dynamic Resource Allocation (DRA). DRA lets you select devices by attributes such as GPU model or memory, constrain a GPU and an EFA device to the same PCIe root for distributed training, and share one GPU across several Pods.

EKS Auto Mode includes the NVIDIA DRA driver and the EFA DRA driver (DRANET) in the node runtime, so there are no drivers to install. You do need to create the DeviceClass objects that your claims reference, which a driver’s Helm chart would otherwise create for you.

EKS Auto Mode keeps DRA and device plugins from managing the same device type on one node. EKS Auto Mode chooses the device manager for each device type when it launches a node, based on the workloads it launches the node for, and the node keeps that role for its lifetime. With dynamic capacity provisioning, as in this walkthrough, you don’t configure anything for this. Static capacity provisioning requires the device manager label on the NodePool, because a static NodePool has no pending Pods to infer the choice from. For more information, see Using DRA with EKS Auto Mode.

Prerequisites

Step 1: Create the DeviceClass

A DeviceClass names a category of devices and defines the expression that selects them. Your ResourceClaimTemplate references it by name. EKS Auto Mode does not create these objects, so create the one for NVIDIA GPUs before you deploy the workload.

cat <<EOF | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com
spec:
  selectors:
  - cel:
      expression: "device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].type == 'gpu'"
EOF

Confirm that it exists:

kubectl get deviceclass

An example output is as follows.

NAME             AGE
gpu.nvidia.com   5s

The name and selector match the DeviceClass that the upstream NVIDIA DRA driver Helm chart installs, so the same claims work on clusters where you manage the driver yourself. For the EFA DeviceClass, the optional extendedResourceName field, and more information, see Create the NVIDIA GPU DeviceClass and DeviceClass objects on EKS Auto Mode.

Step 2: Deploy a GPU workload with DRA

In this example, you create a NodePool for NVIDIA based workloads, a ResourceClaimTemplate that requests one GPU, and a Pod that references the template.

The NodePool doesn’t set a device manager label. EKS Auto Mode sees that the pending Pod claims a GPU through DRA and launches a node that uses the NVIDIA DRA driver, and it labels the node accordingly. You confirm that label in Step 3.

To deploy the NodePool, ResourceClaimTemplate, and sample workload, review the following definitions and save them as nodepool-gpu-dra.yaml and pod-dra.yaml:

nodepool-gpu-dra.yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-dra
spec:
  disruption:
    budgets:
    - nodes: 10%
    consolidateAfter: 1h
    consolidationPolicy: WhenEmpty
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "eks.amazonaws.com/instance-family"
          operator: In
          values:
          - g6e
          - g6
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      terminationGracePeriod: 5m

pod-dra.yaml

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          count: 1
---
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi-dra
spec:
  nodeSelector:
    eks.amazonaws.com/compute-type: auto
  restartPolicy: OnFailure
  containers:
  - name: nvidia-smi
    image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
    args:
    - "nvidia-smi"
    resources:
      claims:
      - name: gpu
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: single-gpu
  tolerations:
  - key: nvidia.com/gpu
    effect: NoSchedule
    operator: Exists

Note that the Pod requests the GPU through resources.claims and resourceClaims, rather than through the nvidia.com/gpu extended resource used in Deploy an accelerated workload. The NodePool also sets a taint that only allows Pods with tolerations for NVIDIA GPUs to be scheduled.

Apply the NodePool and workload to your cluster.

kubectl apply -f nodepool-gpu-dra.yaml
kubectl apply -f pod-dra.yaml

You should see the following output:

nodepool.karpenter.sh/gpu-dra created
resourceclaimtemplate.resource.k8s.io/single-gpu created
pod/nvidia-smi-dra created

Wait a few seconds, and check the nodes in your cluster. EKS Auto Mode launches an instance whose GPUs satisfy the claim, enables the NVIDIA DRA driver on that node, and disables the NVIDIA device plugin on it.

Step 3: Validate

Confirm that the node is a DRA node for NVIDIA GPUs by checking the device manager label:

kubectl get nodes -L eks.amazonaws.com/device-manager-nvidia

An example output is as follows.

NAME                  STATUS   ROLES    AGE   VERSION   DEVICE-MANAGER-NVIDIA
i-02315c7d7643cdee6   Ready    <none>   76s   v1.36.0   dra

Look at the container’s logs by running the following command:

kubectl logs nvidia-smi-dra

The output shows the GPU that the DRA driver allocated to the container, and confirms that you did not have to install any device drivers or a DRA driver, as this is managed by Amazon EKS Auto Mode.

You can also confirm that the claim was allocated. Kubernetes generates a ResourceClaim for each Pod that uses a ResourceClaimTemplate, named after the Pod and the claim name in the Pod’s resourceClaims list. For the preceding Pod, that is nvidia-smi-dra-gpu- followed by a random suffix.

kubectl get resourceclaims

An example output is as follows.

NAME                       STATE                AGE
nvidia-smi-dra-gpu-h4gff   allocated,reserved   12s

Step 4: Clean-up

To remove all objects created, use kubectl to delete the sample workload and NodePool so the node is terminated:

kubectl delete -f pod-dra.yaml
kubectl delete -f nodepool-gpu-dra.yaml

You can leave the gpu.nvidia.com DeviceClass in place for future workloads. To remove it as well, run the following command:

kubectl delete deviceclass gpu.nvidia.com

For topology-aligned GPU and EFA allocation, device sharing, and the full set of considerations, see Manage hardware devices on Amazon EKS.

Example NodePools Reference

Create an NVIDIA NodePool

The following NodePool defines:

Example NodePool with GPU instance family and consolidation

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  disruption:
    budgets:
    - nodes: 10%
    consolidateAfter: 1h
    consolidationPolicy: WhenEmpty
  template:
    metadata: {}
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: "eks.amazonaws.com/instance-family"
          operator: In
          values:
          - g6e
          - g6
      terminationGracePeriod: 24h0m0s

Instead of setting the eks.amazonaws.com/instance-gpu-name you might use eks.amazonaws.com/instance-family to specify the instance family. For other well-known labels which influence scheduling review, see EKS Auto Mode Supported Labels.

If you have specific storage requirements you can tune the nodes ephemeral storage iops, size and throughput by creating your own NodeClass to reference in the NodePool. Learn more about the configurable NodeClass options.

Example storage configuration for NodeClass

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: gpu
spec:
  ephemeralStorage:
    iops: 3000
    size: 80Gi
    throughput: 125

Define an Amazon Trainium and Amazon Inferentia NodePool

The following NodePool has an eks.amazonaws.com/instance-category set that says, only launch instances of Inferentia and Trainium family:

        - key: "eks.amazonaws.com/instance-category"
          operator: In
          values:
            - inf
            - trn

Create a static NodePool with P instances from a Capacity Block

Capacity Blocks for ML reserve GPU accelerated instances for a defined window, placed close together in an Amazon EC2 UltraCluster for low-latency networking. Because you pay for the block for its whole duration whether or not the instances are running, pair a Capacity Block with a static capacity node pool rather than a dynamic one. A static node pool maintains a fixed node count through spec.replicas instead of scaling in response to pending Pods, so your reserved instances come up and stay up for the life of the block.

This combination suits distributed training runs, pre-warmed inference fleets, and any workload where you want the reserved GPUs available before the Pods arrive.

The following example reserves eight p5.48xlarge nodes from a Capacity Block.

Example NodeClass referencing a Capacity Block

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: gpu-capacity-block
spec:
  role: AmazonEKSNodeRole
  subnetSelectorTerms:
    - tags:
        kubernetes.io/cluster/my-cluster: "owned"
  securityGroupSelectorTerms:
    - tags:
        kubernetes.io/cluster/my-cluster: "owned"
  capacityReservationSelectorTerms:
    - id: cr-56fac701cc1951b03
         

Replace cr-56fac701cc1951b03 with your Capacity Block reservation ID, which you can find in the Amazon EC2 console. Unlike open On-Demand Capacity Reservations, EKS Auto Mode never uses a Capacity Block automatically — you must select it with capacityReservationSelectorTerms.

Example static NodePool using the Capacity Block

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-capacity-block
spec:
  replicas: 8
  limits:
    nodes: 10
  template:
    metadata:
      labels:
        eks.amazonaws.com/device-manager-nvidia: dra
        eks.amazonaws.com/device-manager-efa: dra
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: gpu-capacity-block
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["reserved"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["p5.48xlarge"]
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      terminationGracePeriod: 5m

Note the following about this configuration:

What happens when the Capacity Block ends

A Capacity Block has a fixed end time, and Amazon EC2 terminates the instances before it is reached: 30 minutes before the end time for most instance types, or 60 minutes before for UltraServer instance types. EKS Auto Mode begins draining the nodes in the block 10 minutes before EC2 starts terminating them, which gives your workloads a 10-minute window to checkpoint and shut down.

Those nodes are drained rather than terminated abruptly. EKS Auto Mode cordons each node and evicts its Pods, respecting PodDisruptionBudgets and the karpenter.sh/do-not-disrupt annotation until the node’s terminationGracePeriod is reached. This applies to static and dynamic capacity node pools alike. Scaling a static capacity node pool down with kubectl scale drains the removed nodes the same way, though scale operations bypass NodePool disruption budgets.

Because the drain has to finish inside that 10-minute window, note the following:

This early drain is specific to Capacity Blocks. On-Demand Capacity Reservations do not get it, because their expiry semantics are softer: when an instance leaves an ODCR, EKS Auto Mode relabels the node from reserved to on-demand and the Pods keep running.

For more information, see Reservation expiration behavior, Control deployment of workloads into Capacity Reservations with EKS Auto Mode, and Find and purchase Capacity Blocks in the Amazon EC2 User Guide.

Related resources

This tutorial covers launching accelerated workloads on EKS Auto Mode. To go deeper on managing the devices themselves, see the following topics.

Device management:

Compute configuration:

AI/ML on Amazon EKS: