Updating a cluster

To update an existing cluster

1. Compare the Kubernetes version of your cluster control plane to the Kubernetes version of your nodes.

   • Get the Kubernetes version of your cluster control plane with the following command.

     kubectl version --short

   • Get the Kubernetes version of your nodes with the following command. This command returns all self-managed and managed Amazon EC2 and Fargate nodes. Each Fargate pod is listed as its own node.

     kubectl get nodes

   The Kubernetes minor version of the managed and Fargate nodes in your cluster must be the same as the version of your control plane's current version before you update your control plane to a new Kubernetes version. For example, if your control plane is running version 1.18 and any of your nodes are running version 1.17, update your nodes to version 1.18 before updating your control plane's Kubernetes version to 1.19. We also recommend that you update your self-managed nodes to the same version as your control plane before updating the control plane. For more information see Updating a managed node group and Self-managed node updates. To update the version of a Fargate node, delete the pod that is represented by the node and redeploy the pod after you update your control plane.

2. The pod security policy admission controller is enabled by default on Amazon EKS clusters. Before updating your cluster, ensure that the proper pod security policies are in place before you update to avoid any issues. You can check for the default policy with the following command:

   kubectl get psp eks.privileged

   If you receive the following error, see default pod security policy before proceeding.

   Error from server (NotFound): podsecuritypolicies.extensions "eks.privileged" not found

3. If you originally deployed your cluster on Kubernetes 1.17 or earlier, then you may need to remove a discontinued term from your CoreDNS manifest.

   1. Check to see if your CoreDNS manifest has a line that only has the word upstream.

      kubectl get configmap coredns -n kube-system -o jsonpath='{$.data.Corefile}' | grep upstream

      If no output is returned, your manifest doesn't have the line and you can skip to the next step to update your cluster. If the word upstream is returned, then you need to remove the line.

   2. Edit the configmap, removing the line near the top of the file that only has the word upstream. Don't change anything else in the file. After the line is removed, save the changes.

      kubectl edit configmap coredns -n kube-system -o yaml

4. Update your cluster using eksctl, the AWS Management Console, or the AWS CLI.

   Important

   • Because Amazon EKS runs a highly available control plane, you can update only one minor version at a time. See Kubernetes Version and Version Skew Support Policy for the rationale behind this requirement. Therefore, if your current version is 1.17 and you want to update to 1.19, then you must first update your cluster to 1.18 and then update it from 1.18 to 1.19.

   • Make sure that the kubelet on your managed and Fargate nodes are at the same Kubernetes version as your control plane before you update. We also recommend that your self-managed nodes are at the same version as the control plane, though they can be up to one version behind the control plane's current version.

   • Updating a cluster from 1.16 to 1.17 will fail if you have any AWS Fargate pods that have a kubelet minor version earlier than 1.16. Before updating your cluster from 1.16 to 1.17, you need to recycle your Fargate pods so that their kubelet is 1.16 before attempting to update the cluster to 1.17.

   • You may need to update some of your deployed resources before you can update to 1.16. For more information, see Kubernetes 1.16 update prerequisites.

   • Updating your cluster to a newer version may overwrite custom configurations.

   eksctl

   This procedure requires eksctl version 0.44.0 or later. You can check your version with the following command:

   eksctl version

   For more information about installing or updating eksctl, see Installing or upgrading eksctl.

   Update your Amazon EKS control plane's Kubernetes version one minor version later than its current version with the following command. Replace <my-cluster> (including <>) with your cluster name.

   eksctl upgrade cluster --name <my-cluster> --approve

   The update takes several minutes to complete.

   AWS Management Console

   1. Open the Amazon EKS console at https://console.amazonaws.cn/eks/home#/clusters.

   2. Choose the name of the cluster to update and choose Update cluster version.

   3. For Kubernetes version, select the version to update your cluster to and choose Update.

   4. For Cluster name, type the name of your cluster and choose Confirm.

      The update takes several minutes to complete.

   AWS CLI

   1. Update your cluster with the following AWS CLI command. Replace the <example-values> (including <>) with your own.

      aws eks update-cluster-version \
       --region <region-code> \
       --name <my-cluster> \
       --kubernetes-version <1.19>

      Output:

      {
          "update": {
              "id": "<b5f0ba18-9a87-4450-b5a0-825e6e84496f>",
              "status": "InProgress",
              "type": "VersionUpdate",
              "params": [
                  {
                      "type": "Version",
                      "value": "1.19"
                  },
                  {
                      "type": "PlatformVersion",
                      "value": "eks.1"
                  }
              ],
      ...
              "errors": []
          }
      }

   2. Monitor the status of your cluster update with the following command. Use the cluster name and update ID that the previous command returned. Your update is complete when the status appears as Successful. The update takes several minutes to complete.

      aws eks describe-update \
        --region <region-code> \
        --name <my-cluster> \
        --update-id <b5f0ba18-9a87-4450-b5a0-825e6e84496f>

      Output:

      {
          "update": {
              "id": "b5f0ba18-9a87-4450-b5a0-825e6e84496f",
              "status": "Successful",
              "type": "VersionUpdate",
              "params": [
                  {
                      "type": "Version",
                      "value": "1.19"
                  },
                  {
                      "type": "PlatformVersion",
                      "value": "eks.1"
                  }
              ],
      ...
              "errors": []
          }
      }

5. Patch the kube-proxy daemonset to use the image that corresponds to your cluster's Region and current Kubernetes version (in this example, 1.19.6).

   Kubernetes version	1.19	1.18	1.17	1.16	1.15
   KubeProxy	1.19.6	1.18.8	1.17.9	1.16.13	1.15.11

   1. First, retrieve your current kube-proxy image:

      kubectl get daemonset kube-proxy --namespace kube-system -o=jsonpath='{$.spec.template.spec.containers[:1].image}'

      Example output

      602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.18.8-eksbuild.1

   2. Update kube-proxy to the recommended version by replacing 602401143452, us-west-2, and and com with the values from your output and replace 1.19.6 with your cluster's recommended kube-proxy version. If you're deploying a version that is earlier than 1.19.6, then replace eksbuild.2 with eksbuild.1.

      kubectl set image daemonset.apps/kube-proxy \
        -n kube-system \
        kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.19.6-eksbuild.2

   3. (Optional) If you're using x86 and Arm nodes in the same cluster and your cluster was deployed before August 17,2020. Then, edit your kube-proxy manifest to include a node selector for multiple hardware architectures with the following command. This is a one-time operation. After you've added the selector to your manifest, you don't need to do it each time you update. If your cluster was deployed on or after August 17, 2020, then kube-proxy is already multi-architecture capable.

      kubectl edit -n kube-system daemonset/kube-proxy

      Add the following node selector to the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub. This enables Kubernetes to pull the correct hardware image based on the node's hardware architecture.

      - key: "beta.kubernetes.io/arch"
                          operator: In
                          values:
                            - amd64
                            - arm64

   4. (Optional) If your cluster was oriniginally created with Kubernetes v1.14 or later, then you can skip this step because kube-proxy already includes this Affinity Rule. If you originally created an Amazon EKS cluster with Kubernetes version 1.13 or earilier and intend to use Fargate nodes, then edit your kube-proxy manifest to include a NodeAffinity rule to prevent kube-proxy pods from sheduling on Fargate nodes. This is a one-time edit. Once you've added the Affinity Rule to your manifest, you don't need to do it each time you upgrade your cluster. Edit your kube-proxy daemonset.

      kubectl edit -n kube-system daemonset/kube-proxy

      Add the following Affinity Rule to the Daemonset spec section of the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub.

      - key: eks.amazonaws.com/compute-type
        operator: NotIn
        values:
        - fargate

6. Check your cluster's DNS provider. Clusters that were created with Kubernetes version 1.10 shipped with kube-dns as the default DNS and service discovery provider. If you have updated a 1.10 cluster to a newer version and you want to use CoreDNS for DNS and service discovery, then you must install CoreDNS and remove kube-dns.

   To check if your cluster is already running CoreDNS, use the following command.

   kubectl get pod -n kube-system -l k8s-app=kube-dns

   If the output shows coredns in the pod names, you're already running CoreDNS in your cluster. If not, see Installing or upgrading CoreDNS to install CoreDNS on your cluster, update it to the recommended version, return here, and skip steps 7-8.

7. Check the current version of your cluster's coredns deployment.

   kubectl describe deployment coredns --namespace kube-system | grep Image | cut -d "/" -f 3

   Output:

   coredns:v<1.6.6>

   The recommended coredns versions for the corresponding Kubernetes versions are as follows:

   Kubernetes version	1.19	1.18	1.17	1.16	1.15
   CoreDNS	1.8.0	1.7.0	1.6.6	1.6.6	1.6.6

8. If your current coredns version is 1.5.0 or later, but earlier than the recommended version, then skip this step. If your current version is earlier than 1.5.0, then you need to modify the config map for coredns to use the forward plug-in, rather than the proxy plug-in.

   1. Open the configmap with the following command.

      kubectl edit configmap coredns -n kube-system

   2. Replace proxy in the following line with forward. Save the file and exit the editor.

      proxy . /etc/resolv.conf

9. Retrieve your current coredns image:

   kubectl get deployment coredns --namespace kube-system -o=jsonpath='{$.spec.template.spec.containers[:1].image}'

10. Update coredns to the recommended version by replacing 602401143452, us-west-2, and amazonaws.com in the following command with your output from the previous command. Replace 1.8.0 with the version recommended for your cluster's Kubernetes version and then run the modified command.

    kubectl set image --namespace kube-system deployment.apps/coredns \
                coredns=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.0-eksbuild.1

11. (Optional) If you're using x86 and Arm nodes in the same cluster and your cluster was deployed before August 17,2020, then edit your coredns manifest to include a node selector for multiple hardware architectures with the following command. This is a one-time operation. After you've added the selector to your manifest, you don't need to do it each time you update. If you cluster was deployed on or after August 17, 2020, then coredns is already multi-architecture capable.

    kubectl edit -n kube-system deployment/coredns

    Add the following node selector to the file in the editor and then save the file. For an example of where to include this text in the editor, see the CNI manifest file on GitHub.

    - key: "beta.kubernetes.io/arch"
                        operator: In
                        values:
                          - amd64
                          - arm64

12. Check the version of your cluster's Amazon VPC CNI Plugin for Kubernetes. Use the following command to print your cluster's CNI version.

    kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

    The output is as follows:

    amazon-k8s-cni:<1.6.3>

    If your CNI version is earlier than 1.7, then use the appropriate command below to update your CNI version to the latest 1.7 patch version. You can view the latest patch version on GitHub.

    Important

    Any changes you've made to the plugin's default settings on your cluster can be overwritten with default settings when applying the new version of the manifest. To prevent loss of your custom settings, download the manifest, change the default settings as necessary, and then apply the modified manifest to your cluster.

    • China (Beijing) (cn-north-1) or China (Ningxia) (cn-northwest-1)

      kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.7/config/v1.7/aws-k8s-cni-cn.yaml

    • For all other Regions

      • Download the manifest file.

        curl -o aws-k8s-cni.yaml https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.7/config/v1.7/aws-k8s-cni.yaml

      • Replace <region-code> in the following command with the Region that your cluster is in. Then, run the modified command to replace the Region code in the file (currently us-west-2).

        sed -i -e 's/us-west-2/<region-code>/' aws-k8s-cni.yaml

      • Apply the modified manifest file to your cluster.

        kubectl apply -f aws-k8s-cni.yaml

13. (Optional) If you deployed the Kubernetes Cluster Autoscaler to your cluster before updating the cluster, update the Cluster Autoscaler to the latest version that matches the Kubernetes major and minor version that you updated to.

    1. Open the Cluster Autoscaler releases page in a web browser and find the latest Cluster Autoscaler version that matches your cluster's Kubernetes major and minor version. For example, if your cluster's Kubernetes version is 1.19 find the latest Cluster Autoscaler release that begins with 1.19. Record the semantic version number (<1.19.n>) for that release to use in the next step.

    2. Set the Cluster Autoscaler image tag to the version that you recorded in the previous step with the following command. If necessary, replace 1.19.n with your own value.

       kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.19.n

14. (Clusters with GPU nodes only) If your cluster has node groups with GPU support (for example, p3.2xlarge), you must update the NVIDIA device plugin for Kubernetes DaemonSet on your cluster with the following command.

    kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.8.0/nvidia-device-plugin.yml

15. After your cluster update is complete, update your nodes to the same Kubernetes version of your updated cluster. For more information, see Self-managed node updates or Updating a managed node group. Any new pods launched on Fargate will have a kubelet version that matches your cluster version. Existing Fargate pods won't be changed.