Amazon EKS — Troubleshooting Hands-On Lab

3 minute read

In this lab you’ll intentionally break things and fix them — the best way to build troubleshooting muscle memory.

Prerequisites:

  • Running EKS cluster
  • kubectl and aws CLI configured
  • A test namespace

Step 1 — Create a Test Namespace

kubectl create namespace troubleshooting-lab

Step 2 — Debug ImagePullBackOff

Deploy a pod with a wrong image name:

# broken-image.yaml
apiVersion: v1
kind: Pod
metadata:
  name: broken-image
  namespace: troubleshooting-lab
spec:
  containers:
    - name: app
      image: nginx:does-not-exist-12345
      ports:
        - containerPort: 80

kubectl apply -f broken-image.yaml

# Watch the status
kubectl get pod broken-image -n troubleshooting-lab -w
# STATUS: ImagePullBackOff

# Diagnose
kubectl describe pod broken-image -n troubleshooting-lab | grep -A5 Events

# Fix — update the image
kubectl set image pod/broken-image app=nginx:latest -n troubleshooting-lab

# Verify
kubectl get pod broken-image -n troubleshooting-lab
# STATUS: Running

Step 3 — Debug CrashLoopBackOff

Deploy a pod that crashes immediately:

# crashing-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: crashing-pod
  namespace: troubleshooting-lab
spec:
  containers:
    - name: app
      image: busybox:latest
      command: ["sh", "-c", "echo 'starting'; exit 1"]

kubectl apply -f crashing-pod.yaml

# Watch
kubectl get pod crashing-pod -n troubleshooting-lab -w
# STATUS: CrashLoopBackOff

# Check logs
kubectl logs crashing-pod -n troubleshooting-lab
# Output: starting

# Check exit code
kubectl describe pod crashing-pod -n troubleshooting-lab | grep -A5 "Last State"
# Exit Code: 1

# Fix — replace with a working command
kubectl delete pod crashing-pod -n troubleshooting-lab

# fixed-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: crashing-pod
  namespace: troubleshooting-lab
spec:
  containers:
    - name: app
      image: busybox:latest
      command: ["sh", "-c", "echo 'running'; sleep 3600"]

kubectl apply -f fixed-pod.yaml
kubectl get pod crashing-pod -n troubleshooting-lab
# STATUS: Running

Step 4 — Debug Pending Pod (Resource Starvation)

Deploy a pod requesting impossible resources:

# greedy-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: greedy-pod
  namespace: troubleshooting-lab
spec:
  containers:
    - name: app
      image: nginx:latest
      resources:
        requests:
          cpu: "100"
          memory: "1000Gi"

kubectl apply -f greedy-pod.yaml

# Watch
kubectl get pod greedy-pod -n troubleshooting-lab
# STATUS: Pending

# Diagnose
kubectl describe pod greedy-pod -n troubleshooting-lab | grep -A5 Events
# 0/X nodes are available: Insufficient cpu, Insufficient memory

# Fix — reduce to reasonable values
kubectl delete pod greedy-pod -n troubleshooting-lab

# reasonable-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: greedy-pod
  namespace: troubleshooting-lab
spec:
  containers:
    - name: app
      image: nginx:latest
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"

kubectl apply -f reasonable-pod.yaml
kubectl get pod greedy-pod -n troubleshooting-lab
# STATUS: Running

Step 5 — Debug DNS Resolution

# Deploy a debug pod
kubectl run dns-debug --image=busybox:latest --restart=Never -n troubleshooting-lab -- sleep 3600

# Test DNS resolution
kubectl exec dns-debug -n troubleshooting-lab -- nslookup kubernetes.default
# Should resolve to cluster IP

# Test external DNS
kubectl exec dns-debug -n troubleshooting-lab -- nslookup google.com
# Should resolve

# Check CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

Step 6 — Debug Service Connectivity

# Create a service with wrong selector
kubectl create deployment web --image=nginx:latest -n troubleshooting-lab
kubectl expose deployment web --port=80 --target-port=80 -n troubleshooting-lab

# Verify endpoints exist
kubectl get endpoints web -n troubleshooting-lab
# Should show pod IPs

# Now create a broken service
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: web-broken
  namespace: troubleshooting-lab
spec:
  selector:
    app: wrong-label
  ports:
    - port: 80
      targetPort: 80
EOF

# Check endpoints — should be empty
kubectl get endpoints web-broken -n troubleshooting-lab
# ENDPOINTS: <none>

# Diagnose
kubectl get pods -n troubleshooting-lab --show-labels
# Find the correct label

# Fix the selector
kubectl patch svc web-broken -n troubleshooting-lab -p '{"spec":{"selector":{"app":"web"}}}'

# Verify endpoints now exist
kubectl get endpoints web-broken -n troubleshooting-lab

Step 7 — Use Ephemeral Debug Containers

# Attach a debug container to a running pod
kubectl debug -it crashing-pod -n troubleshooting-lab --image=busybox:latest --target=app

# Inside the debug container you can:
# - Inspect the filesystem
# - Check environment variables
# - Test network connectivity
# - View process info

Cleanup

kubectl delete namespace troubleshooting-lab

Key Takeaways

Skill Commands
Check pod status kubectl describe pod, kubectl logs --previous
Check events kubectl get events --sort-by=.lastTimestamp
Debug networking kubectl exec -- nslookup, kubectl get endpoints
Debug node issues SSM session + journalctl -u kubelet
Quick debug pod kubectl run netshoot --image=nicolaka/netshoot -it --rm

Updated:

You may also enjoy