Troubleshoot Amazon EKS Connector issues
This topic covers some of the common errors that you might encounter while using the Amazon EKS Connector, including instructions on how to resolve them and workarounds.
Basic troubleshooting
This section describes steps to diagnose Amazon EKS Connector issues.
Check Amazon EKS Connector status
To check the Amazon EKS Connector status, type:
kubectl get pods -n eks-connector
Inspect Amazon EKS Connector logs
The Amazon EKS Connector Pod consists of three containers. To retrieve full logs for all of these containers so that you can inspect them, run the following commands:
-
connector-init
kubectl logs eks-connector-0 --container connector-init -n eks-connector kubectl logs eks-connector-1 --container connector-init -n eks-connector
-
connector-proxy
kubectl logs eks-connector-0 --container connector-proxy -n eks-connector kubectl logs eks-connector-1 --container connector-proxy -n eks-connector
-
connector-agent
kubectl exec eks-connector-0 --container connector-agent -n eks-connector -- cat /var/log/amazon/ssm/amazon-ssm-agent.log kubectl exec eks-connector-1 --container connector-agent -n eks-connector -- cat /var/log/amazon/ssm/amazon-ssm-agent.log
Get the effective cluster name
Amazon EKS clusters are uniquely identified by clusterName
within a single Amazon account and Amazon Region. If you have multiple connected clusters in Amazon EKS, you can confirm which Amazon EKS cluster that the current Kubernetes cluster is registered to. To do this, enter the following to find out the clusterName
of the current cluster.
kubectl exec eks-connector-0 --container connector-agent -n eks-connector \ -- cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep -m1 -oE "eks_c:[a-zA-Z0-9_-]+" | sed -E "s/^.*eks_c:([a-zA-Z0-9_-]+)_[a-zA-Z0-9]+.*$/\1/" kubectl exec eks-connector-1 --container connector-agent -n eks-connector \ -- cat /var/log/amazon/ssm/amazon-ssm-agent.log | grep -m1 -oE "eks_c:[a-zA-Z0-9_-]+" | sed -E "s/^.*eks_c:([a-zA-Z0-9_-]+)_[a-zA-Z0-9]+.*$/\1/"
Miscellaneous commands
The following commands are useful to retrieve information that you need to troubleshoot issues.
-
Use the following command to gather images that’s used by Pods in Amazon EKS Connector.
kubectl get pods -n eks-connector -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n'
-
Use the following command to determine the node names that Amazon EKS Connector is running on.
kubectl get pods -n eks-connector -o jsonpath="{.items[*].spec.nodeName}" | tr -s '[[:space:]]' '\n'
-
Run the following command to get your Kubernetes client and server versions.
kubectl version
-
Run the following command to get information about your nodes.
kubectl get nodes -o wide --show-labels
Helm issue: 403 Forbidden
If you received the following error when running helm install commands:
Error: INSTALLATION FAILED: unexpected status from HEAD request to https://public.ecr.aws/v2/eks-connector/eks-connector-chart/manifests/0.0.6: 403 Forbidden
You can run the following line to fix it:
docker logout public.ecr.aws
Console error: the cluster is stuck in the Pending state
If the cluster gets stuck in the Pending
state on the Amazon EKS console after you’re registered it, it might be because the Amazon EKS Connector didn’t successfully connect the cluster to Amazon yet. For a registered cluster, the Pending
state means that the connection isn’t successfully established. To resolve this issue, make sure that you have applied the manifest to the target Kubernetes cluster. If you applied it to the cluster, but the cluster is still in the Pending
state, then the eks-connector
statefulset might be unhealthy. To troubleshoot this issue, see Amazon EKS connector Pods are crash loopingin this topic.
Console error: User system:serviceaccount:eks-connector:eks-connector can’t impersonate resource users in API group at cluster scope
The Amazon EKS Connector uses Kubernetes
user impersonationeks-connector
service account must be granted permission to impersonate the corresponding Kubernetes user with an IAM ARN as its Kubernetes user name. In the following examples, the IAM ARN is mapped to a Kubernetes user.
-
IAM user
john
from Amazon account111122223333
is mapped to a Kubernetes user. IAM best practices recommend that you grant permissions to roles instead of users.arn:aws-cn:iam::111122223333:user/john
-
IAM role
admin
from Amazon account111122223333
is mapped to a Kubernetes user:arn:aws-cn:iam::111122223333:role/admin
The result is an IAM role ARN, instead of the Amazon STS session ARN.
For instructions on how to configure the ClusterRole
and ClusterRoleBinding
to grant the eks-connector
service account privilege to impersonate the mapped user, see Grant access to view Kubernetes cluster resources on an Amazon EKS console. Make sure that in the template, %IAM_ARN%
is replaced with the IAM ARN of the Amazon Web Services Management Console IAM principal.
Console error: […] is forbidden: User […] cannot list resource […] in API group at the cluster scope
Consider the following problem. The Amazon EKS Connector has successfully impersonated the requesting Amazon Web Services Management Console IAM principal in the target Kubernetes cluster. However, the impersonated principal doesn’t have RBAC permission for Kubernetes API operations.
To resolve this issue, there are two methods to give permissions to additional users. If you previously installed eks-connector via helm chart, you can easily grant users access by running the following command. Replace the userARN1
and userARN2
with a list of the ARNs of the IAM roles to give access to view the Kubernetes resources:
helm upgrade eks-connector oci://public.ecr.aws/eks-connector/eks-connector-chart \ --reuse-values \ --set 'authentication.allowedUserARNs={userARN1,userARN2}'
Or, as the cluster administrator, grant the appropriate level of RBAC privileges to individual Kubernetes users. For more information and examples, see Grant access to view Kubernetes cluster resources on an Amazon EKS console.
Console error: Amazon EKS can’t communicate with your Kubernetes cluster API server. The cluster must be in an ACTIVE state for successful connection. Try again in few minutes.
If the Amazon EKS service can’t communicate with the Amazon EKS connector in the target cluster, it might be because of one of the following reasons:
-
The Amazon EKS Connector in the target cluster is unhealthy.
-
Poor connectivity or an interrupted connection between the target cluster and the Amazon Region.
To resolve this problem, check the Amazon EKS Connector logs. If you don’t see an error for the Amazon EKS Connector, retry the connection after a few minutes. If you regularly experience high latency or intermittent connectivity for the target cluster, consider re-registering the cluster to an Amazon Region that’s located closer to you.
Amazon EKS connector Pods are crash looping
There are many reasons that can cause an Amazon EKS connector Pod to enter the CrashLoopBackOff
status. This issue likely involves the connector-init
container. Check the status of the Amazon EKS connector Pod.
kubectl get pods -n eks-connector
An example output is as follows.
NAME READY STATUS RESTARTS AGE eks-connector-0 0/2 Init:CrashLoopBackOff 1 7s
If your output is similar to the previous output, see Inspect Amazon EKS Connector logs to troubleshoot the issue.
Failed to initiate eks-connector: InvalidActivation
When you start the Amazon EKS Connector for the first time, it registers an activationId
and activationCode
with Amazon Web Services. The registration might fail, which can cause the connector-init
container to crash with an error similar to the following error.
F1116 20:30:47.261469 1 init.go:43] failed to initiate eks-connector: InvalidActivation:
To troubleshoot this issue, consider the following causes and recommended fixes:
-
Registration might have failed because the
activationId
andactivationCode
aren’t in your manifest file. If this is the case, make sure that they are the correct values that were returned from theRegisterCluster
API operation, and that theactivationCode
is in the manifest file. TheactivationCode
is added to Kubernetes secrets, so it must bebase64
encoded. For more information, see Step 1: Registering the cluster. -
Registration might have failed because your activation expired. This is because, for security reasons, you must activate the Amazon EKS Connector within three days after registering the cluster. To resolve this issue, make sure that the Amazon EKS Connector manifest is applied to the target Kubernetes cluster before the expiry date and time. To confirm your activation expiry date, call the
DescribeCluster
API operation.aws eks describe-cluster --name my-cluster
In the following example response, the expiry date and time is recorded as
2021-11-12T22:28:51.101000-08:00
.{ "cluster": { "name": "my-cluster", "arn": "arn:aws-cn:eks:region:111122223333:cluster/my-cluster", "createdAt": "2021-11-09T22:28:51.449000-08:00", "status": "FAILED", "tags": { }, "connectorConfig": { "activationId": "00000000-0000-0000-0000-000000000000", "activationExpiry": "2021-11-12T22:28:51.101000-08:00", "provider": "OTHER", "roleArn": "arn:aws-cn:iam::111122223333:role/my-connector-role" } } }
If the
activationExpiry
passed, deregister the cluster and register it again. Doing this generates a new activation.
Cluster node is missing outbound connectivity
To work properly, the Amazon EKS Connector requires outbound connectivity to several Amazon endpoints. You can’t connect a private cluster without outbound connectivity to a target Amazon Region. To resolve this issue, you must add the necessary outbound connectivity. For information about connector requirements, see Amazon EKS Connector considerations.
Amazon EKS connector Pods are in ImagePullBackOff
state
If you run the get pods
command and Pods are in the ImagePullBackOff
state, they can’t work properly. If the Amazon EKS Connector Pods are in the ImagePullBackOff
state, they can’t work properly. Check the status of your Amazon EKS Connector Pods.
kubectl get pods -n eks-connector
An example output is as follows.
NAME READY STATUS RESTARTS AGE eks-connector-0 0/2 Init:ImagePullBackOff 0 4s
The default Amazon EKS Connector manifest file references images from the Amazon ECR Public Gallery