Network load balancing on Amazon EKS
Network traffic is load balanced at L4
of the OSI model. To load balance
application traffic at L7
, you deploy a Kubernetes ingress
, which
provisions an Amazon Application Load Balancer. For more information, see Application load balancing on Amazon EKS. To learn more about the differences between the two types
of load balancing, see Elastic
Load Balancing features
When you create a Kubernetes Service
of type LoadBalancer
, the
Amazon cloud provider load balancer controller creates Amazon Classic Load
Balancers by default, but can also create Amazon Network Load
Balancers. This controller is only receiving critical bug fixes in the future.
For more information about using the Amazon cloud provider load balancer , see Amazon cloud provider load balancer controller
We recommend that you use version 2.5.4
or
later of the Amazon Load Balancer Controller instead of the Amazon
cloud provider load balancer controller. The Amazon Load Balancer Controller creates Amazon Network Load Balancers, but doesn't create
Amazon Classic Load Balancers. The remainder of this topic is about using the Amazon Load Balancer
Controller.
An Amazon Network Load Balancer can load balance network traffic to Pods deployed to Amazon EC2
IP and instance targets or to Amazon Fargate IP targets. For more information, see Amazon Load
Balancer Controller
Prerequisites
Before you can load balance network traffic using the Amazon Load Balancer Controller, you must meet the following requirements.
-
Have an existing cluster. If you don't have an existing cluster, see Getting started with Amazon EKS. If you need to update the version of an existing cluster, see Updating an Amazon EKS cluster Kubernetes version.
-
Have the Amazon Load Balancer Controller deployed on your cluster. For more information, see Installing the Amazon Load Balancer Controller add-on. We recommend version
2.5.4
or later. -
At least one subnet. If multiple tagged subnets are found in an Availability Zone, the controller chooses the first subnet whose subnet ID comes first lexicographically. The subnet must have at least eight available IP addresses.
-
If you're using the Amazon Load Balancer Controller version
2.1.1
or earlier, subnets must be tagged as follows. If using version2.1.2
or later, this tag is optional. You might want to tag a subnet if you have multiple clusters running in the same VPC, or multiple Amazon services sharing subnets in a VPC, and want more control over where load balancers are provisioned for each cluster. If you explicitly specify subnet IDs as an annotation on a service object, then Kubernetes and the Amazon Load Balancer Controller use those subnets directly to create the load balancer. Subnet tagging isn't required if you choose to use this method for provisioning load balancers and you can skip the following private and public subnet tagging requirements. Replace
with your cluster name.my-cluster
-
Key –
kubernetes.io/cluster/
my-cluster
-
Value –
shared
orowned
-
-
Your public and private subnets must meet the following requirements, unless you explicitly specify subnet IDs as an annotation on a service or ingress object. If you provision load balancers by explicitly specifying subnet IDs as an annotation on a service or ingress object, then Kubernetes and the Amazon Load Balancer Controller use those subnets directly to create the load balancer and the following tags aren't required.
-
Private subnets – Must be tagged in the following format. This is so that Kubernetes and the Amazon Load Balancer Controller know that the subnets can be used for internal load balancers. If you use
eksctl
or an Amazon EKS Amazon Amazon CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS Amazon Amazon CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.-
Key –
kubernetes.io/role/internal-elb
-
Value –
1
-
-
Public subnets – Must be tagged in the following format. This is so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (based on the lexicographical order of the subnet IDs). If you use
eksctl
or an Amazon EKS Amazon CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS Amazon CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.-
Key –
kubernetes.io/role/elb
-
Value –
1
-
If the subnet role tags aren't explicitly added, the Kubernetes service controller examines the route table of your cluster VPC subnets to determine if the subnet is private or public. We recommend that you don't rely on this behavior, and instead explicitly add the private or public role tags. The Amazon Load Balancer Controller doesn't examine route tables, and requires the private and public tags to be present for successful auto discovery.
-
Considerations
-
The configuration of your load balancer is controlled by annotations that are added to the manifest for your service. Service annotations are different when using the Amazon Load Balancer Controller than they are when using the Amazon cloud provider load balancer controller. Make sure to review the annotations
for the Amazon Load Balancer Controller before deploying services. -
When using the Amazon VPC CNI plugin for Kubernetes, the Amazon Load Balancer Controller can load balance to Amazon EC2 IP or instance targets and Fargate IP targets. When using Alternate compatible CNI plugins, the controller can only load balance to instance targets. For more information about Network Load Balancer target types, see Target type in the User Guide for Network Load Balancers
-
If you want to add tags to the load balancer when or after it's created, add the following annotation in your service specification. For more information, see Amazon Resource Tags
in the Amazon Load Balancer Controller documentation. service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags
-
You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. Replace the
with theexample values
Allocation IDs
of your Elastic IP addresses. The number ofAllocation IDs
must match the number of subnets that are used for the load balancer. For more information, see the Amazon Load Balancer Controllerdocumentation. service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-
xxxxxxxxxxxxxxxxx
,eipalloc-yyyyyyyyyyyyyyyyy
-
Amazon EKS adds one inbound rule to the node's security group for client traffic and one rule for each load balancer subnet in the VPC for health checks for each Network Load Balancer that you create. Deployment of a service of type
LoadBalancer
can fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group. For more information, see Security groups in Amazon VPC quotas in the Amazon VPC User Guide. Consider the following options to minimize the chances of exceeding the maximum number of rules for a security group:-
Request an increase in your rules per security group quota. For more information, see Requesting a quota increase in the Service Quotas User Guide.
-
Use IP targets, rather than instance targets. With IP targets, you can share rules for the same target ports. You can manually specify load balancer subnets with an annotation. For more information, see Annotations
on GitHub. -
Use an ingress, instead of a service of type
LoadBalancer
, to send traffic to your service. The Amazon Application Load Balancer requires fewer rules than Network Load Balancers. You can share an ALB across multiple ingresses. For more information, see Application load balancing on Amazon EKS. You can't share a Network Load Balancer across multiple services. -
Deploy your clusters to multiple accounts.
-
-
If your Pods run on Windows in an Amazon EKS cluster, a single service with a load balancer can support up to 1024 back-end Pods. Each Pod has its own unique IP address.
-
We recommend only creating new Network Load Balancers with the Amazon Load Balancer Controller. Attempting to replace existing Network Load Balancers created with the Amazon cloud provider load balancer controller can result in multiple Network Load Balancers that might cause application downtime.
Create a network load balancer
You can create a network load balancer with IP or instance targets.
(Optional) Deploy a sample application
Prerequisites
-
At least one public or private subnet in your cluster VPC.
-
Have the Amazon Load Balancer Controller deployed on your cluster. For more information, see Installing the Amazon Load Balancer Controller add-on. We recommend version
2.5.4
or later.
To deploy a sample application
-
If you're deploying to Fargate, make sure you have an available private subnet in your VPC and create a Fargate profile. If you're not deploying to Fargate, skip this step. You can create the profile by running the following command or in the Amazon Web Services Management Console using the same values for
name
andnamespace
that are in the command. Replace the
with your own.example values
eksctl create fargateprofile \ --cluster
\my-cluster
\ --regionregion-code
\ --namenlb-sample-app
--namespace nlb-sample-app
-
Deploy a sample application.
-
Create a namespace for the application.
kubectl create namespace
nlb-sample-app
-
Save the following contents to a file named
file on your computer.sample-deployment
.yamlapiVersion: apps/v1 kind: Deployment metadata: name:
nlb-sample-app
namespace:nlb-sample-app
spec: replicas:3
selector: matchLabels: app:nginx
template: metadata: labels: app:nginx
spec: containers: - name:nginx
image:public.ecr.aws/nginx/nginx:1.23
ports: - name:tcp
containerPort:80
-
Apply the manifest to the cluster.
kubectl apply -f
sample-deployment
.yaml
-
-
Create a service with an internet-facing Network Load Balancer that load balances to IP targets.
-
Save the following contents to a file named
file on your computer. If you're deploying to Fargate nodes, remove thesample-service
.yamlservice.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
line.apiVersion: v1 kind: Service metadata: name:
nlb-sample-service
namespace:nlb-sample-app
annotations: service.beta.kubernetes.io/aws-load-balancer-type: external service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing spec: ports: - port:80
targetPort:80
protocol:TCP
type: LoadBalancer selector: app:nginx
-
Apply the manifest to the cluster.
kubectl apply -f
sample-service
.yaml
-
-
Verify that the service was deployed.
kubectl get svc
nlb-sample-service
-nnlb-sample-app
An example output is as follows.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
sample-service
LoadBalancer10.100.240.137
k8s-nlbsampl
-nlbsampl
-xxxxxxxxxx
-xxxxxxxxxxxxxxxx
.elb.region-code
.amazonaws.com.cn80
:32400
/TCP
16hNote
The values for
and10.100.240.137
-xxxxxxxxxx
xxxxxxxxxxxxxxxx
will be different than the example output (they will be unique to your load balancer) andcn-north-1
may be different for you, depending on which Amazon Web Services Region that your cluster is in. -
Open the Amazon EC2 Amazon Web Services Management Console
. Select Target Groups (under Load Balancing) in the left navigation pane. In the Name column, select the target group's name where the value in the Load balancer column matches a portion of the name in the EXTERNAL-IP
column of the output in the previous step. For example, you'd select the target group namedk8s-default-samplese-
if your output were the same as the previous output. The Target type isxxxxxxxxxx
IP
because that was specified in the sample service manifest. -
Select the Target group and then select the Targets tab. Under Registered targets, you should see three IP addresses of the three replicas deployed in a previous step. Wait until the status of all targets is healthy before continuing. It might take several minutes before all targets are
healthy
. The targets might be in anunhealthy
state before changing to ahealthy
state. -
Send traffic to the service replacing
andxxxxxxxxxx-xxxxxxxxxxxxxxxx
with the values returned in the output for a previous step forcn-north-1
EXTERNAL-IP
. If you deployed to a private subnet, then you'll need to view the page from a device within your VPC, such as a bastion host. For more information, see Linux Bastion Hosts on Amazon. curl k8s-default-samplese-
xxxxxxxxxx-xxxxxxxxxxxxxxxx
.elb.region-code
.amazonaws.com.cnAn example output is as follows.
<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> [...]
-
When you're finished with the sample deployment, service, and namespace, remove them.
kubectl delete namespace
nlb-sample-app