Amazon EKS — Pod & Workload Failures

Pods fail for many reasons — image pull errors, application crashes, insufficient resources, or scheduling constraints. Each failure mode has specific symptoms and fixes.

---

ImagePullBackOff

Symptom: Pod stuck in ImagePullBackOff or ErrImagePull.

kubectl describe pod <pod-name> -n <namespace>
# Look for Events section

Cause	Error Message	Fix
Wrong image name	repository does not exist	Fix image path in deployment
ECR auth expired	no basic auth credentials	Verify node IAM role has ECR permissions
Cross-account ECR	denied: User	Add ECR repository policy or imagePullSecret
Private repo no secret	unauthorized: authentication required	Create imagePullSecret
Image tag doesn’t exist	manifest unknown	Verify tag exists in ECR

Fix — ECR Permissions on Node IAM Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken"
      ],
      "Resource": "*"
    }
  ]
}

Fix — Cross-Account imagePullSecret

kubectl create secret docker-registry ecr-cross-account \
  --docker-server=987654321098.dkr.ecr.us-east-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region us-east-1) \
  -n production

---

CrashLoopBackOff

Symptom: Pod starts, crashes, restarts repeatedly with increasing backoff.

# Check current state
kubectl get pod <pod-name> -n <namespace>

# Check logs from current (crashing) container
kubectl logs <pod-name> -n <namespace>

# Check logs from previous crash
kubectl logs <pod-name> -n <namespace> --previous

# Check container exit code
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Last State"

Exit Code	Meaning	Common Cause
0	Success (but pod shouldn’t exit)	App exits immediately; wrong CMD/entrypoint
1	Application error	Config missing, dependency unavailable
137	OOMKilled (SIGKILL)	Memory limit too low
139	Segfault	Application bug
143	SIGTERM (graceful)	Liveness probe failing

Fix — OOMKilled (Exit 137)

# Confirm OOMKill
kubectl describe pod <pod-name> | grep -A3 "Reason: OOMKilled"

Increase memory limit:

resources:
  requests:
    memory: 512Mi
  limits:
    memory: 512Mi    # Increase this

Fix — Application Crash (Exit 1)

# Debug interactively
kubectl run debug --image=<same-image> --restart=Never -it -- /bin/sh

# Check environment variables
kubectl exec <pod-name> -- env

# Check if config exists
kubectl describe configmap <config-name>
kubectl describe secret <secret-name>

---

Pending Pods

Symptom: Pod stuck in Pending indefinitely.

kubectl describe pod <pod-name> | grep -A10 Events

Event Message	Cause	Fix
Insufficient cpu	Not enough CPU on any node	Add nodes or reduce requests
Insufficient memory	Not enough memory on any node	Add nodes or reduce requests
0/3 nodes are available: 3 node(s) had taint	Taint / toleration mismatch	Add toleration or remove taint
0/3 nodes are available: 3 node(s) didn't match	Node affinity mismatch	Fix affinity rules or label nodes
persistentvolumeclaim not found	PVC doesn’t exist or is pending	Check PVC status

Fix — No Resources Available

# Check node capacity
kubectl describe nodes | grep -A5 "Allocated resources"

# Check if Cluster Autoscaler or Karpenter is running
kubectl get pods -n kube-system | grep -E "autoscaler|karpenter"

Fix — Taint Mismatch

# List node taints
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

# Add toleration to pod spec

---

Fargate Pod Scheduling Failures

Fargate pods fail if no Fargate Profile matches:

# Check Fargate profiles
aws eks list-fargate-profiles --cluster-name my-cluster

# Check if namespace and labels match a profile
aws eks describe-fargate-profile \
  --cluster-name my-cluster \
  --fargate-profile-name <profile-name> \
  --query 'fargateProfile.selectors'

Common issues:

• Pod namespace doesn’t match any Fargate Profile selector
• Pod labels don’t match profile selectors
• Fargate Profile has 0 selectors

---

Debugging Commands Cheat Sheet

# Pod status overview
kubectl get pods -n <namespace> -o wide

# Detailed pod info
kubectl describe pod <pod-name> -n <namespace>

# Container logs
kubectl logs <pod-name> -n <namespace> -c <container>

# Previous logs (after crash)
kubectl logs <pod-name> -n <namespace> --previous

# Exec into running container
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

# Ephemeral debug container
kubectl debug -it <pod-name> --image=busybox:latest --target=<container>

# Events sorted by time
kubectl get events -n <namespace> --sort-by=.lastTimestamp

---