Amazon EKS — Troubleshooting Decision Tree

Use this systematic approach to identify and resolve EKS issues quickly. Start by identifying the affected layer, then follow the appropriate branch.

---

Layer Identification

Problem Reported
│
├── Can't access cluster?          → Cluster Access Branch
├── Pod not running?               → Pod Status Branch
├── Pod running but not reachable? → Networking Branch
├── Node unhealthy?                → Node Health Branch
└── Slow performance?              → Performance Branch

---

Cluster Access Branch

Can't access cluster
│
├── "Unauthorized" error?
│   ├── Check: aws sts get-caller-identity
│   ├── Is this identity in access entries / aws-auth?
│   │   ├── No  → Add access entry or aws-auth mapping
│   │   └── Yes → Check role bindings (RBAC)
│   └── Using SSO? → Verify SSO role ARN matches
│
├── "Unable to connect" error?
│   ├── Check: Is endpoint public or private?
│   │   ├── Private only → Access from VPC (bastion/VPN)
│   │   └── Public → Check CIDR restrictions
│   ├── Check: Network/VPN connectivity
│   └── Check: aws eks describe-cluster --query 'cluster.status'
│
└── "exec plugin" error?
    └── Update AWS CLI and regenerate kubeconfig

---

Pod Status Branch

Pod Not Running
│
├── Status: Pending
│   ├── "Insufficient cpu/memory" → Scale up nodes or reduce requests
│   ├── "node(s) had taint" → Add toleration
│   ├── "didn't match node selector" → Fix nodeSelector / affinity
│   ├── "persistentvolumeclaim not found" → Check PVC status
│   └── No events → Check Cluster Autoscaler / Karpenter logs
│
├── Status: ImagePullBackOff
│   ├── Image exists? → Verify image:tag in registry
│   ├── ECR? → Check node IAM role for ecr permissions
│   ├── Private registry? → Check imagePullSecrets
│   └── Cross-account? → Add ECR repo policy
│
├── Status: CrashLoopBackOff
│   ├── Exit code 137 → OOMKilled → Increase memory limit
│   ├── Exit code 1 → App error → Check logs (kubectl logs --previous)
│   ├── Exit code 0 → App exiting → Fix CMD / entrypoint
│   └── Liveness probe? → Adjust probe settings
│
├── Status: ContainerCreating (stuck)
│   ├── "failed to assign IP" → VPC CNI IP exhaustion
│   ├── "failed to mount" → Check PV / storage driver
│   └── Secret/ConfigMap not found → Create missing resource
│
└── Status: Terminating (stuck)
    ├── Finalizers? → Remove finalizer (carefully)
    └── Force delete: kubectl delete pod <name> --grace-period=0 --force

---

Networking Branch

Networking Issue
│
├── DNS not resolving?
│   ├── CoreDNS running? → kubectl get pods -n kube-system -l k8s-app=kube-dns
│   ├── CoreDNS healthy? → Check logs for errors
│   ├── Service exists? → kubectl get svc -A | grep <service>
│   └── External DNS slow? → Set ndots: 2 in pod spec
│
├── Pod-to-pod broken?
│   ├── Same node? → Likely CNI issue
│   ├── Cross-node? → Check security groups (allow all within cluster SG)
│   ├── Cross-namespace? → Check NetworkPolicy
│   └── aws-node healthy? → kubectl get ds aws-node -n kube-system
│
├── Service unreachable?
│   ├── Endpoints exist? → kubectl get endpoints <svc>
│   ├── Port correct? → Compare svc port vs container port
│   ├── Selector matches? → Compare svc selector vs pod labels
│   └── kube-proxy running? → kubectl get ds kube-proxy -n kube-system
│
└── ALB/NLB issues?
    ├── LB created? → Check Ingress/Service address
    ├── Controller running? → Check aws-load-balancer-controller logs
    ├── 502 error? → Target group health checks failing
    ├── Targets empty? → Wrong target-type annotation
    └── SSL error? → Verify ACM certificate ARN

---

kubectl Diagnostic Cheat Sheet

# ── Cluster Health ──
kubectl cluster-info
kubectl get componentstatuses
kubectl get nodes -o wide
kubectl top nodes

# ── Pod Diagnostics ──
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
kubectl describe pod <name> -n <ns>
kubectl logs <name> -n <ns> --previous --tail=100
kubectl top pods -n <ns> --sort-by=memory

# ── Networking ──
kubectl get svc -A
kubectl get endpoints <svc> -n <ns>
kubectl get ingress -A
kubectl exec -it <pod> -- nslookup kubernetes.default

# ── Events ──
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get events -n <ns> --field-selector type=Warning

# ── Resources ──
kubectl describe nodes | grep -A5 "Allocated resources"
kubectl get pvc -A
kubectl get networkpolicy -A

---

AWS CLI Diagnostic Commands

# ── Cluster ──
aws eks describe-cluster --name my-cluster --query 'cluster.{Status:status,Version:version,Endpoint:endpoint}'
aws eks list-addons --cluster-name my-cluster

# ── Node Groups ──
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name my-nodes --query 'nodegroup.{Status:status,Health:health}'

# ── Networking ──
aws ec2 describe-subnets --filters "Name=tag:kubernetes.io/cluster/my-cluster,Values=*" --query 'Subnets[*].{ID:SubnetId,AZ:AvailabilityZone,FreeIPs:AvailableIpAddressCount}'

# ── IAM ──
aws sts get-caller-identity
aws eks list-access-entries --cluster-name my-cluster

---

CloudWatch Logs Insights Queries

Control Plane — Authenticator Errors

fields @timestamp, @message
| filter @logStream like /authenticator/
| filter @message like /access denied|unauthorized/
| sort @timestamp desc
| limit 50

Control Plane — API Server Errors

fields @timestamp, @message
| filter @logStream like /kube-apiserver-audit/
| filter responseStatus.code >= 400
| sort @timestamp desc
| limit 50

Control Plane — Scheduler Failures

fields @timestamp, @message
| filter @logStream like /scheduler/
| filter @message like /FailedScheduling/
| sort @timestamp desc
| limit 50

---

Support Case Checklist

Before opening an AWS Support case, gather:

• Cluster name and region
• aws eks describe-cluster output
• kubectl get nodes -o wide output
• kubectl get pods -A output (focus on kube-system)
• kubectl get events -A --sort-by=.lastTimestamp (last 50)
• CloudWatch control plane logs (if enabled)
• Timeline — when did the issue start?
• Recent changes (upgrades, deployments, config changes)
• VPC/subnet configuration
• Node group or Karpenter configuration

---