Network load balancing on Amazon EKS
When you create a Kubernetes Service
of type LoadBalancer
, an
AWS Network Load Balancer (NLB) or Classic Load Balancer (CLB) is provisioned that
load balances network traffic. To learn
more about the differences between the two types of load balancers, see Elastic Load Balancing
features
Network traffic is load balanced at L4 of the OSI model. To load balance application
traffic at L7, you deploy a Kubernetes Ingress
, which provisions an AWS
Application Load Balancer. For more information, see Application load balancing on Amazon EKS. To learn more about the
differences between the two types of load balancing, see Elastic Load Balancing
features
In Amazon EKS, you can load balance network traffic to an NLB (instance or IP target) or a CLB
(instance target only). For more information about
NLB target types, see Target type
Prerequisites
Before you can load balance network traffic to an application, you must meet the following requirements.
-
Have an existing cluster. If you don't have an existing cluster, see Getting started with Amazon EKS. If you're load balancing to IP targets, the cluster must be 1.18 or later. To update an existing cluster, see Updating a cluster.
-
If you're load balancing to IP targets, you must have the AWS Load Balancer Controller provisioned on your cluster. For more information, see AWS Load Balancer Controller.
-
At least one subnet. In the case of multiple tagged subnets found in an Availability Zone, the controller chooses the first subnet in lexicographical order by the subnet IDs.
-
If you're using the AWS Load Balancer controller version
v2.1.1
or earlier, subnets must be tagged as follows. If using version 2.1.2 or later, this tag is optional. You might want to tag a subnet if you have multiple clusters running in the same VPC, or multiple AWS services sharing subnets in a VPC, and want more control over where load balancers are provisioned per cluster. If you explicitly specify subnet IDs as an annotation on a Service object, then Kubernetes and the AWS load balancer controller use those subnets directly to create the load balancer. Subnet tagging is not required if you choose to use this method for provisioning load balancers and you can skip the following private and public subnet tagging requirements. Replace
(including<cluster-name>
) with your cluster name.<>
-
Key –
kubernetes.io/cluster/
<cluster-name>
-
Value –
shared
orowned
-
-
Subnet tagging – Your public and private subnets must meet the following requirements, unless you explicitly specify subnet IDs as an annotation on a Service or Ingress object. If you provision load balancers by explicitly specifying subnet IDs as an annotation on a Service or Ingress object, then Kubernetes and the AWS load balancer controller use those subnets directly to create the load balancer and the following tags are not required.
-
Private subnets – Must be tagged as follows so that Kubernetes and the AWS load balancer controller know that the subnets can be used for internal load balancers. If you use
eksctl
or an Amazon EKS AWS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS AWS AWS CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.-
Key –
kubernetes.io/role/internal-elb
-
Value –
1
-
-
Public subnets – Must be tagged as follows so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (in lexicographical order by subnet ID). If you use
eksctl
or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS AWS CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.-
Key –
kubernetes.io/role/elb
-
Value –
1
-
If the subnet role tags are not explicitly added, the Kubernetes service controller examines the route table of your cluster VPC subnets to determine if the subnet is private or public. We recommend that you do not rely on this behavior, and instead explicitly add the private or public role tags. The AWS load balancer controller does not examine route tables, and requires the private and public tags to be present for successful auto discovery.
-
Considerations
-
Use of the UDP protocol is supported with the load balancer on Amazon EKS clusters with the following platform versions. For more information, see Amazon EKS platform versions.
Amazon EKS version Platform version 1.19 .1 1.18 .1 1.17 .2 1.16 .3 1.15 .4 -
You can only use NLB IP targets with the Amazon EKS VPC CNI plugin. You can use NLB instance targets with the Amazon EKS VPC CNI plugin or alternate compatible CNI plugins.
-
You can only use IP targets with NLB. You can't use IP targets with CLBs.
-
The configuration of your load balancer is controlled by annotations that are added to the manifest for your service. If you want to add tags to the load balancer when (or after) it's created, add the following annotation in your service specification. For more information, see Other ELB annotations
in the Kubernetes documentation. service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags
-
If you're using Amazon EKS 1.16 or later, you can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. Replace the
(including<example-values>
) with the Allocation IDs of your Elastic IP addresses. The number of Allocation IDs must match the number of subnets used for the load balancer.<>
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-
<xxxxxxxxxxxxxxxxx>
,eipalloc-<yyyyyyyyyyyyyyyyy>
-
For each NLB that you create on 1.16 or later clusters, Amazon EKS adds one inbound rule to the node's security group for client traffic and one rule for each load balancer subnet in the VPC for health checks. On 1.15 clusters, one rule is added for each CIDR block assigned to the VPC. Deployment of a service of type
LoadBalancer
can fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group. For more information, see Security groups in Amazon VPC quotas in the Amazon VPC User Guide. Consider the following options to minimize the chances of exceeding the maximum number of rules for a security group.-
Request an increase in your rules per security group quota. For more information, see Requesting a quota increase in the Service Quotas User Guide.
-
Use Load balancer – IP targets, rather than instance targets. With IP targets, rules can potentially be shared for the same target ports. Load balancer subnets can be manually specified with an annotation. For more information, see Annotations
on GitHub. -
Use an Ingress, instead of a Service of type
LoadBalancer
to send traffic to your service. The AWS Application Load Balancer (ALB) requires fewer rules than NLBs. An ALB can also be shared across multiple Ingresses. For more information, see Application load balancing on Amazon EKS. -
Deploy your clusters to multiple accounts.
-
-
If your pods run on Windows, In an Amazon EKS cluster a single service with a load balancer can support up to 64 backend pods. Each pod has its own unique IP address. This is a limitation of the Windows OS on the Amazon EC2 nodes.
Load balancer – Instance targets
NLB or CLBs with instance targets are created by the Kubernetes in-tree load balancing
controller. The in-tree controller is included with Kubernetes, so you don't need
to
deploy it to your cluster. You can use NLB instance targets with pods deployed to
nodes,
but not to Fargate. To load balance network traffic across pods deployed to Fargate,
you must
use IP targets. By default, external (public)
Classic Load Balancers are created when you deploy a Kubernetes service of type
LoadBalancer
. To deploy a Network Load Balancer instead, apply the following annotation
to your service:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Do not edit this annotation after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation.
To deploy a load balancer to a private subnet, your service specification must have the following annotation:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
For internal load balancers, your Amazon EKS cluster must be configured to use at least one private subnet in your VPC. Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.
For an example service manifest that specifies a load balancer, see Type LoadBalancer
Load balancer – IP targets
NLBs with IP targets are created by the AWS Load Balancer Controller (you cannot use
CLBs with IP targets). You can use NLB IP targets with pods deployed to Amazon EC2
nodes or
Fargate. Your Kubernetes service must be created as type LoadBalancer
. For
more information, see Type LoadBalancer
To create a load balancer that uses IP targets, add the following annotation to a service manifest and deploy your service.
service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
Do not edit this annotation after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation. You can only use NLB IP targets with clusters running at least Amazon EKS version 1.18. To upgrade your current version, see Updating a cluster.
To deploy a sample application
-
Deploy a sample application.
-
Save the following contents to a file named
file on your computer.sample-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: sample-app spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: public.ecr.aws/nginx/nginx:1.19.6 ports: - name: http containerPort: 80
-
Apply the manifest to the cluster.
kubectl apply -f
sample-deployment
.yaml
-
-
Create a service of type
LoadBalancer
with an annotation to create an NLB with IP targets.-
Save the following contents to a file named
file on your computer.sample-service.yaml
apiVersion: v1 kind: Service metadata: name: sample-service annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip spec: ports: - port: 80 targetPort: 80 protocol: TCP type: LoadBalancer selector: app: nginx
-
Apply the manifest to the cluster.
kubectl apply -f
sample-service.yaml
-
-
Verify that the service was deployed.
kubectl get svc sample-service
Output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE sample-service LoadBalancer 10.100.240.137
k8s-default-samplese-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com
80:32400/TCP 16h -
Open the Amazon EC2 AWS Management Console
. Select Target Groups (under Load Balancing) in the left panel. In the Name column, select the target group's name that matches the name in the EXTERNAL-IP
column of the output in the previous step. For example, you'd select the target group named rk8s-default-samplese-xxxxxxxxxx
if your output were the same as the output above. The Target type isIP
because that was specified in the sample service deployment manifest. -
Select the Target group and then select the Targets tab. Under Registered targets, you should see three IP addresses of the three replicas deployed in a previous step. Wait until the status of all targets is healthy before continuing. It may take several minutes before all targets are
healthy
. The targets may have anunhealthy
state before changing tohealthy
. -
Send traffic to the service replacing the example value with the value returned in a previous step for the
EXTERNAL-IP
.curl
<k8s-default-samplese-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com>
Output
<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> ...
-
When you're finished with the sample deployment and service, remove them.
kubectl delete service sample-service kubectl delete deployment sample-app