Amazon EKS — Troubleshooting — Practice Exercises

3 minute read

Chapter 14 — Practice Exercises

Exercise 1: CrashLoopBackOff

Question: A pod is in CrashLoopBackOff. kubectl logs shows nothing. What are your next steps?

Show Solution 
# Step 1: Check previous container logs
$ kubectl logs <pod> --previous
# This shows logs from the crashed container

# Step 2: If still no logs, check events
$ kubectl describe pod <pod>
# Look at Events, State, and Last State sections

# Step 3: Check if it's OOMKilled
$ kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}'
# If "reason: OOMKilled" → increase memory limit

# Step 4: Check the container command
$ kubectl get pod <pod> -o jsonpath='{.spec.containers[0].command}'
# Wrong entrypoint? Missing script?

# Step 5: Debug with ephemeral container
$ kubectl debug -it <pod> --image=busybox --target=mycontainer
# Inspect filesystem, env vars, etc.

If logs are truly empty, the process likely crashes before writing anything (segfault, wrong binary, missing libraries).

Exercise 2: Pods Stuck Pending

Question: 5 pods are Pending. kubectl describe shows: “0/3 nodes are available: 3 Insufficient memory.” What do you do?

Show Solution 
# Step 1: Check current node resources
$ kubectl top nodes
NAME       CPU(cores)   MEMORY(bytes)
node-1     1800m/2000m  3.5Gi/4Gi     ← 87% memory used
node-2     1500m/2000m  3.2Gi/4Gi     ← 80% memory used
node-3     1900m/2000m  3.8Gi/4Gi     ← 95% memory used

# Step 2: Check what's consuming memory
$ kubectl top pods -A --sort-by=memory | head -10

# Step 3: Immediate fix — increase node count
$ eksctl scale nodegroup --cluster demo --name ng-default --nodes 5

# Step 4: Long-term fixes:
# a. Right-size pod requests (VPA recommendations)
# b. Add Cluster Autoscaler or Karpenter
# c. Use larger instance types
# d. Review if any pods have excessive memory requests

Exercise 3: DNS Failure

Question: Pods can’t resolve external DNS names (google.com) but can resolve cluster DNS (kubernetes.default). What’s the issue?

Show Solution 
# Step 1: Verify the problem
$ kubectl exec -it test-pod -- nslookup kubernetes.default
# Works ✅

$ kubectl exec -it test-pod -- nslookup google.com
# Fails ❌

# Step 2: Check CoreDNS upstream configuration
$ kubectl get configmap coredns -n kube-system -o yaml
# Look for "forward" directive — should forward to VPC DNS

# Step 3: Check NAT Gateway (for private subnets)
# If nodes are in private subnets, they need a NAT Gateway
# for outbound internet (which includes external DNS)

# Step 4: Check VPC DNS settings
# VPC → DHCP Options → DNS should be AmazonProvidedDNS

# Most common cause: Missing or misconfigured NAT Gateway
# in private subnets. Without outbound internet,
# CoreDNS can't forward queries to upstream resolvers.

Exercise 4: EBS Volume Won’t Attach

Question: A StatefulSet pod is stuck with “Multi-Attach error for volume”. What happened?

Show Solution

EBS volumes are single-attach — only one node at a time.

# Step 1: Find where the volume is currently attached
$ kubectl describe pv <pv-name>
# Check which node has it

# Step 2: Why is it still attached?
# a. Previous pod on different node didn't terminate cleanly
# b. Node was terminated without draining

# Step 3: Fix options:
# a. Wait — Kubernetes will force-detach after ~6 minutes
# b. Force detach via AWS CLI:
$ aws ec2 detach-volume --volume-id vol-xxx --force

# c. Delete the stuck VolumeAttachment:
$ kubectl get volumeattachments
$ kubectl delete volumeattachment <name>

# Step 4: Prevention:
# - Always drain nodes before terminating
# - Use PodDisruptionBudgets
# - For multi-attach needs, use EFS instead of EBS

Exercise 5: Access Denied

Question: A new team member can kubectl get pods but gets “Forbidden” on kubectl get nodes. How do you fix it?

Show Solution 
# Step 1: Check current RBAC
$ kubectl auth can-i get nodes --as=<user>
no

# Step 2: Check aws-auth ConfigMap
$ kubectl get configmap aws-auth -n kube-system -o yaml
# Verify user's IAM role/user is mapped to a K8s group

# Step 3: Check what group they're in
# If they're mapped to a group, check the ClusterRoleBinding:
$ kubectl get clusterrolebindings | grep <group>

# Step 4: Create or update RBAC
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-reader-binding
subjects:
  - kind: Group
    name: dev-team
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

# Step 5: Apply
$ kubectl apply -f node-reader-rbac.yaml

What’s Next?

Build real projects: Chapter 15 — Practice Projects.

You may also enjoy