Simplify compute management with Amazon Fargate
Important
Amazon Fargate with Amazon EKS isn’t available in Amazon GovCloud (US-East) and Amazon GovCloud (US-West).
This topic discusses using Amazon EKS to run Kubernetes
Pods on Amazon Fargate. Fargate is a technology that provides on-demand, right-sized compute capacity for containers
You can control which Pods start on Fargate and how they run with Fargate profiles. Fargate profiles are defined as part of your Amazon EKS cluster. Amazon EKS integrates Kubernetes with Fargate by using controllers that are built by Amazon using the upstream, extensible model provided by Kubernetes. These controllers run as part of the Amazon EKS managed Kubernetes control plane and are responsible for scheduling native Kubernetes Pods onto Fargate. The Fargate controllers include a new scheduler that runs alongside the default Kubernetes scheduler in addition to several mutating and validating admission controllers. When you start a Pod that meets the criteria for running on Fargate, the Fargate controllers that are running in the cluster recognize, update, and schedule the Pod onto Fargate.
This topic describes the different components of Pods that run on Fargate, and calls out special considerations for using Fargate with Amazon EKS.
Amazon Fargate considerations
Here are some things to consider about using Fargate on Amazon EKS.
-
Each Pod that runs on Fargate has its own isolation boundary. They don’t share the underlying kernel, CPU resources, memory resources, or elastic network interface with another Pod.
-
Network Load Balancers and Application Load Balancers (ALBs) can be used with Fargate with IP targets only. For more information, see Create a network load balancer and Route application and HTTP traffic with Application Load Balancers.
-
Fargate exposed services only run on target type IP mode, and not on node IP mode. The recommended way to check the connectivity from a service running on a managed node and a service running on Fargate is to connect via service name.
-
Pods must match a Fargate profile at the time that they’re scheduled to run on Fargate. Pods that don’t match a Fargate profile might be stuck as
Pending
. If a matching Fargate profile exists, you can delete pending Pods that you have created to reschedule them onto Fargate. -
Daemonsets aren’t supported on Fargate. If your application requires a daemon, reconfigure that daemon to run as a sidecar container in your Pods.
-
Privileged containers aren’t supported on Fargate.
-
Pods running on Fargate can’t specify
HostPort
orHostNetwork
in the Pod manifest. -
The default
nofile
andnproc
soft limit is 1024 and the hard limit is 65535 for Fargate Pods. -
GPUs aren’t currently available on Fargate.
-
Pods that run on Fargate are only supported on private subnets (with NAT gateway access to Amazon services, but not a direct route to an Internet Gateway), so your cluster’s VPC must have private subnets available. For clusters without outbound internet access, see Deploy private clusters with limited internet access.
-
You can use the Adjust pod resources with Vertical Pod Autoscaler to set the initial correct size of CPU and memory for your Fargate Pods, and then use the Scale pod deployments with Horizontal Pod Autoscaler to scale those Pods. If you want the Vertical Pod Autoscaler to automatically re-deploy Pods to Fargate with larger CPU and memory combinations, set the mode for the Vertical Pod Autoscaler to either
Auto
orRecreate
to ensure correct functionality. For more information, see the Vertical Pod Autoscalerdocumentation on GitHub. -
DNS resolution and DNS hostnames must be enabled for your VPC. For more information, see Viewing and updating DNS support for your VPC.
-
Amazon EKS Fargate adds defense-in-depth for Kubernetes applications by isolating each Pod within a Virtual Machine (VM). This VM boundary prevents access to host-based resources used by other Pods in the event of a container escape, which is a common method of attacking containerized applications and gain access to resources outside of the container.
Using Amazon EKS doesn’t change your responsibilities under the shared responsibility model. You should carefully consider the configuration of cluster security and governance controls. The safest way to isolate an application is always to run it in a separate cluster.
-
Fargate profiles support specifying subnets from VPC secondary CIDR blocks. You might want to specify a secondary CIDR block. This is because there’s a limited number of IP addresses available in a subnet. As a result, there’s also a limited number of Pods that can be created in the cluster. By using different subnets for Pods, you can increase the number of available IP addresses. For more information, see Adding IPv4 CIDR blocks to a VPC.
-
The Amazon EC2 instance metadata service (IMDS) isn’t available to Pods that are deployed to Fargate nodes. If you have Pods that are deployed to Fargate that need IAM credentials, assign them to your Pods using IAM roles for service accounts. If your Pods need access to other information available through IMDS, then you must hard code this information into your Pod spec. This includes the Amazon Region or Availability Zone that a Pod is deployed to.
-
You can’t deploy Fargate Pods to Amazon Outposts, Amazon Wavelength, or Amazon Local Zones.
-
Amazon EKS must periodically patch Fargate Pods to keep them secure. We attempt the updates in a way that reduces impact, but there are times when Pods must be deleted if they aren’t successfully evicted. There are some actions you can take to minimize disruption. For more information, see Set actions for Amazon Fargate OS patching events.
-
The Amazon VPC CNI plugin for Amazon EKS
is installed on Fargate nodes. You can’t use Alternate CNI plugins for Amazon EKS clusters with Fargate nodes. -
A Pod running on Fargate automatically mounts an Amazon EFS file system, without needing manual driver installation steps. You can’t use dynamic persistent volume provisioning with Fargate nodes, but you can use static provisioning.
-
Amazon EKS doesn’t support Fargate Spot.
-
You can’t mount Amazon EBS volumes to Fargate Pods.
-
You can run the Amazon EBS CSI controller on Fargate nodes, but the Amazon EBS CSI node DaemonSet can only run on Amazon EC2 instances.
-
After a Kubernetes Job
is marked Completed
orFailed
, the Pods that the Job creates normally continue to exist. This behavior allows you to view your logs and results, but with Fargate you will incur costs if you don’t clean up the Job afterwards.To automatically delete the related Pods after a Job completes or fails, you can specify a time period using the time-to-live (TTL) controller. The following example shows specifying
.spec.ttlSecondsAfterFinished
in your Job manifest.apiVersion: batch/v1 kind: Job metadata: name: busybox spec: template: spec: containers: - name: busybox image: busybox command: ["/bin/sh", "-c", "sleep 10"] restartPolicy: Never ttlSecondsAfterFinished: 60 # <-- TTL controller
Fargate Comparison Table
Criteria | Amazon Fargate |
---|---|
Can be deployed to Amazon Outposts |
No |
Can be deployed to an Amazon Local Zone |
No |
Can run containers that require Windows |
No |
Can run containers that require Linux |
Yes |
Can run workloads that require the Inferentia chip |
No |
Can run workloads that require a GPU |
No |
Can run workloads that require Arm processors |
No |
Can run Amazon
Bottlerocket |
No |
Pods share a kernel runtime environment with other Pods |
No – Each Pod has a dedicated kernel |
Pods share CPU, memory, storage, and network resources with other Pods. |
No – Each Pod has dedicated resources and can be sized independently to maximize resource utilization. |
Pods can use more hardware and memory than requested in Pod specs |
No – The Pod can be re-deployed using a larger vCPU and memory configuration though. |
Must deploy and manage Amazon EC2 instances |
No |
Must secure, maintain, and patch the operating system of Amazon EC2 instances |
No |
Can provide bootstrap arguments at deployment of a node, such as extra kubelet |
No |
Can assign IP addresses to Pods from a different CIDR block than the IP address assigned to the node. |
No |
Can SSH into node |
No – There’s no node host operating system to SSH to. |
Can deploy your own custom AMI to nodes |
No |
Can deploy your own custom CNI to nodes |
No |
Must update node AMI on your own |
No |
Must update node Kubernetes version on your own |
No – You don’t manage nodes. |
Can use Amazon EBS storage with Pods |
No |
Can use Amazon EFS storage with Pods |
|
Can use Amazon FSx for Lustre storage with Pods |
No |
Can use Network Load Balancer for services |
Yes, when using the Create a network load balancer |
Pods can run in a public subnet |
No |
Can assign different VPC security groups to individual Pods |
Yes |
Can run Kubernetes DaemonSets |
No |
Support |
No |
Amazon Region availability |
|
Can run containers on Amazon EC2 dedicated hosts |
No |
Pricing |
Cost of an individual Fargate memory and CPU configuration. Each Pod has its own cost. For more information, see Amazon Fargate pricing |