Troubleshooting local clusters for Amazon EKS on Amazon Outposts

Local clusters are created through the Amazon EKS API, but are run in an asynchronous manner. This means that requests to the Amazon EKS API return immediately for local clusters. However, these requests might succeed, fail fast because of input validation errors, or fail and have descriptive validation errors. This behavior is similar to the Kubernetes API.

Local clusters don't transition to a FAILED status. Amazon EKS attempts to reconcile the cluster state with the user-requested desired state in a continuous manner. As a result, a local cluster might remain in the CREATING state for an extended period of time until the underlying issue is resolved.

Local cluster issues can be discovered using the describe-cluster Amazon EKS Amazon CLI command. Local cluster issues are surfaced by the cluster.health field of the describe-cluster command's response. The message contained in this field includes an error code, descriptive message, and related resource IDs. This information is available through the Amazon EKS API and Amazon CLI only. In the following example, replace my-cluster with the name of your local cluster.

aws eks describe-cluster --name my-cluster --query 'cluster.health'

An example of the output is as follows.

{
    "issues": [
        {
            "code": "ConfigurationConflict",
            "message": "The instance type 'm5.large' is not supported in Outpost 'my-outpost-arn'.",
            "resourceIds": [
                "my-cluster-arn"
            ]
        }
    ]
}

If the problem is beyond repair, you might need to delete the local cluster and create a new one. For example, trying to provision a cluster with an instance type that's not available on your Outpost. The following table includes common health related errors.

The following table lists errors from other Amazon Web Services that are presented in the health field of the describe-cluster response.

Local clusters require different permissions and policies than Amazon EKS clusters that are hosted in the cloud. When a cluster fails to create and produces an InvalidPermissions error, double check that the cluster role that you're using has the AmazonEKSLocalOutpostClusterPolicy managed policy attached to it. All other API calls require the same set of permissions as Amazon EKS clusters in the cloud.

The amount of time it takes to create a local cluster varies depending on several factors. These factors include your network configuration, Outpost configuration, and the cluster's configuration. In general, a local cluster is created and changes to the ACTIVE status within 15–20 minutes. If a local cluster remains in the CREATING state, you can call describe-cluster for information about the cause in the cluster.health output field.

The most common issues are the following:

VPC and subnet DNS configuration is misconfigured or missing

Review Amazon EKS local cluster VPC and subnet requirements and considerations.

When an Outpost gets disconnected from the Amazon Web Services Region that it's associated with, the Kubernetes cluster likely will continue working normally. However, if the cluster doesn't work properly, follow the troubleshooting steps in Preparing for network disconnects. If you encounter other issues, contact Amazon Web Services Support. Amazon Web Services Support can guide you on downloading and running a log collection tool. That way, you can collect logs from your Kubernetes cluster control plane instances and send them to Amazon Web Services Support support for further investigation.

When the Amazon EKS control plane instances aren't reachable through Amazon Systems Manager (Systems Manager), Amazon EKS displays the following error for your cluster.

Amazon EKS control plane instances are not reachable through SSM. Please verify your SSM and network configuration, and reference the EKS on Outposts troubleshooting documentation.

To resolve this issue, make sure that your VPC and subnets meet the requirements in Amazon EKS local cluster VPC and subnet requirements and considerations and that you completed the steps in Setting up Session Manager in the Amazon Systems Manager User Guide.