Network load balancing on Amazon EKS - Amazon EKS
Load balancer – Instance targetsLoad balancer – IP targets
AWS services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with AWS services in China.

Network load balancing on Amazon EKS

When you create a Kubernetes Service of type LoadBalancer, an AWS Network Load Balancer (NLB) or Classic Load Balancer (CLB) is provisioned that load balances network traffic. To learn more about the differences between the two types of load balancers, see Elastic Load Balancing features on the AWS website. For more information about a Kubernetes Service, see Service in the Kubernetes documentation. NLBs can be used with pods deployed to nodes or to AWS Fargate IP targets. You can deploy an AWS load balancer to a public or private subnet.

Network traffic is load balanced at L4 of the OSI model. To load balance application traffic at L7, you deploy a Kubernetes Ingress, which provisions an AWS Application Load Balancer. For more information, see Application load balancing on Amazon EKS. To learn more about the differences between the two types of load balancing, see Elastic Load Balancing features on the AWS website.

In Amazon EKS, you can load balance network traffic to an NLB (instance or IP target) or a CLB (instance target only). For more information about NLB target types, see Target type in the User Guide for Network Load Balancers. When you load balance network traffic to instance targets, the Kubernetes in-tree controller creates the NLB or CLB. To load balance network traffic to IP target types, the AWS Load Balancer Controller creates an NLB. For more information, see AWS load balancer controller on GitHub.

Prerequisites

Before you can load balance network traffic to an application, you must meet the following requirements.

  • Have an existing cluster. If you don't have an existing cluster, see Getting started with Amazon EKS. If you're load balancing to IP targets, the cluster must be 1.18 or later. To update an existing cluster, see Updating a cluster.

  • If you're load balancing to IP targets, you must have the AWS Load Balancer Controller provisioned on your cluster. For more information, see AWS Load Balancer Controller.

  • At least one subnet. In the case of multiple tagged subnets found in an Availability Zone, the controller chooses the first subnet in lexicographical order by the subnet IDs.

  • If you're using the AWS Load Balancer controller version v2.1.1 or earlier, subnets must be tagged as follows. If using version 2.1.2 or later, this tag is optional. You might want to tag a subnet if you have multiple clusters running in the same VPC, or multiple AWS services sharing subnets in a VPC, and want more control over where load balancers are provisioned per cluster. If you explicitly specify subnet IDs as an annotation on a Service object, then Kubernetes and the AWS load balancer controller use those subnets directly to create the load balancer. Subnet tagging is not required if you choose to use this method for provisioning load balancers and you can skip the following private and public subnet tagging requirements. Replace <cluster-name> (including <>) with your cluster name.

    • Key – kubernetes.io/cluster/<cluster-name>

    • Value – shared or owned

  • Subnet tagging – Your public and private subnets must meet the following requirements, unless you explicitly specify subnet IDs as an annotation on a Service or Ingress object. If you provision load balancers by explicitly specifying subnet IDs as an annotation on a Service or Ingress object, then Kubernetes and the AWS load balancer controller use those subnets directly to create the load balancer and the following tags are not required.

    • Private subnets – Must be tagged as follows so that Kubernetes and the AWS load balancer controller know that the subnets can be used for internal load balancers. If you use eksctl or an Amazon EKS AWS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS AWS AWS CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.

      • Key – kubernetes.io/role/internal-elb

      • Value – 1

    • Public subnets – Must be tagged as follows so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (in lexicographical order by subnet ID). If you use eksctl or an Amazon EKS AWS CloudFormation template to create your VPC after March 26, 2020, then the subnets are tagged appropriately when they're created. For more information about the Amazon EKS AWS CloudFormation VPC templates, see Creating a VPC for your Amazon EKS cluster.

      • Key – kubernetes.io/role/elb

      • Value – 1

    If the subnet role tags are not explicitly added, the Kubernetes service controller examines the route table of your cluster VPC subnets to determine if the subnet is private or public. We recommend that you do not rely on this behavior, and instead explicitly add the private or public role tags. The AWS load balancer controller does not examine route tables, and requires the private and public tags to be present for successful auto discovery.

Considerations

  • Use of the UDP protocol is supported with the load balancer on Amazon EKS clusters with the following platform versions. For more information, see Amazon EKS platform versions.

    Amazon EKS version Platform version
    1.19 .1
    1.18 .1
    1.17 .2
    1.16 .3
    1.15 .4

  • You can only use NLB IP targets with the Amazon EKS VPC CNI plugin. You can use NLB instance targets with the Amazon EKS VPC CNI plugin or alternate compatible CNI plugins.

  • You can only use IP targets with NLB. You can't use IP targets with CLBs.

  • The configuration of your load balancer is controlled by annotations that are added to the manifest for your service. If you want to add tags to the load balancer when (or after) it's created, add the following annotation in your service specification. For more information, see Other ELB annotations in the Kubernetes documentation. 

    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags

  • If you're using Amazon EKS 1.16 or later, you can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. Replace the <example-values> (including <>) with the Allocation IDs of your Elastic IP addresses. The number of Allocation IDs must match the number of subnets used for the load balancer. 

    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-<xxxxxxxxxxxxxxxxx>,eipalloc-<yyyyyyyyyyyyyyyyy>

  • For each NLB that you create on 1.16 or later clusters, Amazon EKS adds one inbound rule to the node's security group for client traffic and one rule for each load balancer subnet in the VPC for health checks. On 1.15 clusters, one rule is added for each CIDR block assigned to the VPC. Deployment of a service of type LoadBalancer can fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group. For more information, see Security groups in Amazon VPC quotas in the Amazon VPC User Guide. Consider the following options to minimize the chances of exceeding the maximum number of rules for a security group.

    • Request an increase in your rules per security group quota. For more information, see Requesting a quota increase in the Service Quotas User Guide.

    • Use Load balancer – IP targets, rather than instance targets. With IP targets, rules can potentially be shared for the same target ports. Load balancer subnets can be manually specified with an annotation. For more information, see Annotations on GitHub.

    • Use an Ingress, instead of a Service of type LoadBalancer to send traffic to your service. The AWS Application Load Balancer (ALB) requires fewer rules than NLBs. An ALB can also be shared across multiple Ingresses. For more information, see Application load balancing on Amazon EKS.

    • Deploy your clusters to multiple accounts.

  • If your pods run on Windows, In an Amazon EKS cluster a single service with a load balancer can support up to 64 backend pods. Each pod has its own unique IP address. This is a limitation of the Windows OS on the Amazon EC2 nodes.

Load balancer – Instance targets

NLB or CLBs with instance targets are created by the Kubernetes in-tree load balancing controller. The in-tree controller is included with Kubernetes, so you don't need to deploy it to your cluster. You can use NLB instance targets with pods deployed to nodes, but not to Fargate. To load balance network traffic across pods deployed to Fargate, you must use IP targets. By default, external (public) Classic Load Balancers are created when you deploy a Kubernetes service of type LoadBalancer. To deploy a Network Load Balancer instead, apply the following annotation to your service: 

service.beta.kubernetes.io/aws-load-balancer-type: nlb
Important

Do not edit this annotation after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation.

To deploy a load balancer to a private subnet, your service specification must have the following annotation: 

service.beta.kubernetes.io/aws-load-balancer-internal: "true"

For internal load balancers, your Amazon EKS cluster must be configured to use at least one private subnet in your VPC. Kubernetes examines the route table for your subnets to identify whether they are public or private. Public subnets have a route directly to the internet using an internet gateway, but private subnets do not.

For an example service manifest that specifies a load balancer, see Type LoadBalancer in the Kubernetes documentation. For more information about using Network Load Balancer with Kubernetes, see Network Load Balancer support on AWS in the Kubernetes documentation.

Load balancer – IP targets

NLBs with IP targets are created by the AWS Load Balancer Controller (you cannot use CLBs with IP targets). You can use NLB IP targets with pods deployed to Amazon EC2 nodes or Fargate. Your Kubernetes service must be created as type LoadBalancer. For more information, see Type LoadBalancer in the Kubernetes documentation.

To create a load balancer that uses IP targets, add the following annotation to a service manifest and deploy your service. 

service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
Important

Do not edit this annotation after creating your service. If you need to modify it, delete the service object and create it again with the desired value for this annotation. You can only use NLB IP targets with clusters running at least Amazon EKS version 1.18. To upgrade your current version, see Updating a cluster.

To deploy a sample application

  1. Deploy a sample application.

    1. Save the following contents to a file named sample-deployment.yaml file on your computer. 

      apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: public.ecr.aws/nginx/nginx:1.19.6
          ports:
            - name: http
              containerPort: 80

    2. Apply the manifest to the cluster.

      kubectl apply -f sample-deployment.yaml

  2. Create a service of type LoadBalancer with an annotation to create an NLB with IP targets.

    1. Save the following contents to a file named sample-service.yaml file on your computer. 

      apiVersion: v1
kind: Service
metadata:
  name: sample-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: LoadBalancer
  selector:
    app: nginx

    2. Apply the manifest to the cluster.

      kubectl apply -f sample-service.yaml

  3. Verify that the service was deployed.

    kubectl get svc sample-service

    Output

    NAME            TYPE           CLUSTER-IP       EXTERNAL-IP                                                                    PORT(S)        AGE
sample-service  LoadBalancer   10.100.240.137   k8s-default-samplese-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com   80:32400/TCP   16h

  4. Open the Amazon EC2 AWS Management Console. Select Target Groups (under Load Balancing) in the left panel. In the Name column, select the target group's name that matches the name in the EXTERNAL-IP column of the output in the previous step. For example, you'd select the target group named r k8s-default-samplese-xxxxxxxxxx if your output were the same as the output above. The Target type is IP because that was specified in the sample service deployment manifest.

  5. Select the Target group and then select the Targets tab. Under Registered targets, you should see three IP addresses of the three replicas deployed in a previous step. Wait until the status of all targets is healthy before continuing. It may take several minutes before all targets are healthy. The targets may have an unhealthy state before changing to healthy.

  6. Send traffic to the service replacing the example value with the value returned in a previous step for the EXTERNAL-IP. 

    curl <k8s-default-samplese-xxxxxxxxxx-xxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com>

    Output

    <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

  7. When you're finished with the sample deployment and service, remove them. 

    kubectl delete service sample-service
kubectl delete deployment sample-app