CKA — Troubleshooting Lab

10 minute read

Lab Overview

This lab simulates broken cluster scenarios similar to what you’ll face on the CKA exam. Each task presents a broken component — your job is to diagnose the issue and fix it.

Troubleshooting is worth 30% of the CKA exam. This lab is the most important practice you can do.

Lab Rules

  • Read the task description carefully
  • Follow the same debugging flow you’d use on the exam: observe → diagnose → fix → verify
  • Time yourself — in the real exam, you’ll have roughly 5–8 minutes per question
  • Try to solve each task on your own before looking at the answer

Prerequisites

  • A working Kubernetes cluster (kubeadm, minikube, or kind)
  • kubectl access with admin privileges
  • SSH access to nodes (for node and control plane tasks)

Task 1: Fix a CrashLoopBackOff Pod

Setup

Deploy a Pod with a broken command:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crash-app
  labels:
    app: crash-app
spec:
  containers:
  - name: app
    image: nginx:1.25
    command: ["/bin/start-app"]
EOF

The Problem

kubectl get pods

NAME        READY   STATUS             RESTARTS      AGE
crash-app   0/1     CrashLoopBackOff   3 (20s ago)   1m

Your Task

Diagnose why the Pod is crashing and fix it so it transitions to Running.

Show Answer 
# Step 1: Check the logs
kubectl logs crash-app
# Output: exec /bin/start-app: no such file or directory

kubectl logs crash-app --previous
# Same output — the command doesn't exist in the nginx image

# Step 2: The fix — remove the wrong command so nginx uses its default entrypoint

# Export the Pod spec
kubectl get pod crash-app -o yaml > crash-app.yaml

# Delete the broken Pod
kubectl delete pod crash-app

# Edit the YAML — remove the "command" line
# The file should look like:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crash-app
  labels:
    app: crash-app
spec:
  containers:
  - name: app
    image: nginx:1.25
EOF

# Step 3: Verify
kubectl get pod crash-app

Expected final state:

NAME        READY   STATUS    RESTARTS   AGE
crash-app   1/1     Running   0          10s

Task 2: Fix an ImagePullBackOff Pod

Setup

Deploy a Pod with a wrong image name:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pull-app
  labels:
    app: pull-app
spec:
  containers:
  - name: app
    image: ngingx:1.25
EOF

The Problem

kubectl get pods

NAME       READY   STATUS             RESTARTS   AGE
pull-app   0/1     ImagePullBackOff   0          1m

Your Task

Diagnose the issue and fix the Pod so it runs successfully.

Show Answer 
# Step 1: Describe the Pod to see the error
kubectl describe pod pull-app

# Events show:
#   Warning  Failed   kubelet  Failed to pull image "ngingx:1.25":
#   rpc error: ... manifest unknown ...

# The image name is "ngingx" — should be "nginx"

# Step 2: Fix — you can't edit the image of an existing Pod directly
# Export, fix, delete, and recreate

kubectl get pod pull-app -o yaml > pull-app.yaml
kubectl delete pod pull-app

# Edit the YAML: change "ngingx" to "nginx"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pull-app
  labels:
    app: pull-app
spec:
  containers:
  - name: app
    image: nginx:1.25
EOF

# Step 3: Verify
kubectl get pod pull-app

Expected final state:

NAME       READY   STATUS    RESTARTS   AGE
pull-app   1/1     Running   0          15s

Task 3: Troubleshoot a Pending Pod

Setup

Deploy a Pod with resource requests that exceed available resources:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pending-app
  labels:
    app: pending-app
spec:
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        cpu: "100"
        memory: "1Ti"
  nodeSelector:
    gpu: "true"
EOF

The Problem

kubectl get pods

NAME          READY   STATUS    RESTARTS   AGE
pending-app   0/1     Pending   0          5m

Your Task

Diagnose why the Pod won’t schedule, and fix it.

Show Answer 
# Step 1: Check why it's pending
kubectl describe pod pending-app

# Events show:
#   Warning  FailedScheduling  default-scheduler
#   0/3 nodes are available: 3 Insufficient cpu, 3 Insufficient memory,
#   3 node(s) didn't match Pod's node selector

# Two problems:
# 1. CPU request of "100" (100 cores) and memory of "1Ti" (1 tebibyte) is way too high
# 2. nodeSelector gpu=true doesn't match any node

# Step 2: Fix — reduce resources to something reasonable and remove/fix nodeSelector

kubectl delete pod pending-app

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pending-app
  labels:
    app: pending-app
spec:
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
EOF

# Step 3: Verify
kubectl get pod pending-app

Expected final state:

NAME          READY   STATUS    RESTARTS   AGE
pending-app   1/1     Running   0          10s

Task 4: Fix a NotReady Node

Setup

Note: This task requires SSH access to a worker node. If you’re using minikube, you can simulate this with minikube ssh.

Simulate a node failure by stopping the kubelet:

# SSH into the worker node
ssh node01

# Stop kubelet (simulates failure)
sudo systemctl stop kubelet

# Exit back to the control plane
exit

The Problem

kubectl get nodes

NAME           STATUS     ROLES           AGE   VERSION
controlplane   Ready      control-plane   10d   v1.29.0
node01         NotReady   <none>          10d   v1.29.0

Your Task

Diagnose why the node is NotReady and bring it back to Ready status.

Show Answer 
# Step 1: Check the node
kubectl describe node node01
# Conditions show Ready=False, and the message mentions "Kubelet stopped posting node status"

# Step 2: SSH to the node
ssh node01

# Step 3: Check kubelet status
systemctl status kubelet
# Output: inactive (dead) — kubelet is stopped

# Step 4: Check kubelet logs for any underlying issues
journalctl -u kubelet | tail -20

# Step 5: Start and enable kubelet
sudo systemctl start kubelet
sudo systemctl enable kubelet

# Step 6: Verify kubelet is running
systemctl status kubelet
# Output: active (running)

# Step 7: Exit and verify from the control plane
exit
kubectl get nodes

Expected final state (may take 30-60 seconds):

NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   10d   v1.29.0
node01         Ready    <none>          10d   v1.29.0

Task 5: Fix a Broken Control Plane Component

Setup

Warning: This task modifies a control plane manifest. Be ready to fix it quickly!

Introduce a typo in the kube-scheduler manifest:

# On the control plane node:
# Backup the scheduler manifest first!
sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/kube-scheduler.yaml.bak

# Introduce a typo in the kubeconfig path
sudo sed -i 's|--kubeconfig=/etc/kubernetes/scheduler.conf|--kubeconfig=/etc/kubernetes/schedulerr.conf|' \
  /etc/kubernetes/manifests/kube-scheduler.yaml

The Problem

After 30 seconds, try creating a Deployment:

kubectl create deployment test-deploy --image=nginx --replicas=3
kubectl get pods

NAME                          READY   STATUS    RESTARTS   AGE
test-deploy-6d9f4b7c8-xxxxx  0/1     Pending   0          2m
test-deploy-6d9f4b7c8-yyyyy  0/1     Pending   0          2m
test-deploy-6d9f4b7c8-zzzzz  0/1     Pending   0          2m

All Pods are Pending and not being scheduled.

Your Task

Diagnose why Pods are stuck in Pending and fix the issue.

Show Answer 
# Step 1: Check why Pods are pending
kubectl describe pod test-deploy-6d9f4b7c8-xxxxx
# Events section is empty — no scheduler is attempting to schedule!

# Step 2: Check the scheduler
kubectl get pods -n kube-system | grep scheduler
# The scheduler Pod is either missing or in CrashLoopBackOff

kubectl logs -n kube-system kube-scheduler-controlplane
# Error: "failed to read kubeconfig" or similar

# Step 3: Check the scheduler manifest
cat /etc/kubernetes/manifests/kube-scheduler.yaml
# Notice: --kubeconfig=/etc/kubernetes/schedulerr.conf (extra 'r')

# Step 4: Fix the typo
sudo vi /etc/kubernetes/manifests/kube-scheduler.yaml
# Change: schedulerr.conf → scheduler.conf

# Or restore from backup:
sudo cp /tmp/kube-scheduler.yaml.bak /etc/kubernetes/manifests/kube-scheduler.yaml

# Step 5: Wait 15-30 seconds for the kubelet to restart the scheduler

# Step 6: Verify
kubectl get pods -n kube-system | grep scheduler
# Should be Running

kubectl get pods
# test-deploy Pods should now be Running (or at least Scheduling)

Expected final state:

NAME                          READY   STATUS    RESTARTS   AGE
test-deploy-6d9f4b7c8-xxxxx  1/1     Running   0          5m
test-deploy-6d9f4b7c8-yyyyy  1/1     Running   0          5m
test-deploy-6d9f4b7c8-zzzzz  1/1     Running   0          5m

Task 6: Fix DNS Resolution

Setup

Scale down CoreDNS to simulate a DNS failure:

kubectl scale deployment -n kube-system coredns --replicas=0

The Problem

# Try to resolve a Service name from a Pod
kubectl run dnstest --image=busybox:1.36 --restart=Never --rm -it -- nslookup kubernetes.default

;; connection timed out; no servers could be reached

DNS resolution fails for all Pods in the cluster.

Your Task

Diagnose the DNS failure and restore DNS functionality.

Show Answer 
# Step 1: Check CoreDNS Pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# No resources found — CoreDNS has 0 replicas!

# Step 2: Check the CoreDNS Deployment
kubectl get deployment -n kube-system coredns
# DESIRED: 0, CURRENT: 0

# Step 3: Scale CoreDNS back up
kubectl scale deployment -n kube-system coredns --replicas=2

# Step 4: Wait for CoreDNS Pods to be ready
kubectl get pods -n kube-system -l k8s-app=kube-dns -w
# Wait until both Pods show READY 1/1

# Step 5: Verify DNS works
kubectl run dnstest --image=busybox:1.36 --restart=Never --rm -it -- nslookup kubernetes.default

Expected final output:

Server:    10.96.0.10
Address:   10.96.0.10#53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

Task 7: Fix NetworkPolicy Issue

Setup

Create a namespace with a default-deny NetworkPolicy:

# Create namespace and Pods
kubectl create namespace np-lab

kubectl run web --image=nginx:1.25 -n np-lab --labels="app=web" --port=80
kubectl run client --image=busybox:1.36 -n np-lab --labels="app=client" --command -- sleep 3600

# Wait for Pods to be ready
kubectl wait --for=condition=ready pod/web -n np-lab --timeout=60s
kubectl wait --for=condition=ready pod/client -n np-lab --timeout=60s

# Apply a default-deny ingress policy
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: np-lab
spec:
  podSelector: {}
  policyTypes:
  - Ingress
EOF

The Problem

# Try to reach the web Pod from the client Pod
WEB_IP=$(kubectl get pod web -n np-lab -o jsonpath='{.status.podIP}')
kubectl exec -n np-lab client -- wget -qO- --timeout=3 http://$WEB_IP

wget: download timed out
command terminated with exit code 1

The client Pod can’t reach the web Pod because of the default-deny NetworkPolicy.

Your Task

Create a NetworkPolicy that allows the client Pod (label app=client) to access the web Pod (label app=web) on port 80, while keeping the default-deny for all other traffic.

Show Answer 
# Step 1: Check the existing NetworkPolicies
kubectl get networkpolicies -n np-lab
kubectl describe networkpolicy default-deny-ingress -n np-lab
# This policy denies all ingress traffic to all Pods in the namespace

# Step 2: Create an allow policy for client → web on port 80
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-client-to-web
  namespace: np-lab
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: client
    ports:
    - protocol: TCP
      port: 80
EOF

# Step 3: Verify connectivity
WEB_IP=$(kubectl get pod web -n np-lab -o jsonpath='{.status.podIP}')
kubectl exec -n np-lab client -- wget -qO- --timeout=5 http://$WEB_IP

Expected final output:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

The client can now reach the web Pod on port 80, while all other traffic remains denied by the default-deny policy.

Cleanup

Remove all resources created during this lab:

# Delete individual Pods
kubectl delete pod crash-app pull-app pending-app --ignore-not-found

# Delete the test Deployment
kubectl delete deployment test-deploy --ignore-not-found

# Delete the NetworkPolicy lab namespace (removes everything in it)
kubectl delete namespace np-lab --ignore-not-found

# Restore CoreDNS replicas (if you changed them)
kubectl scale deployment -n kube-system coredns --replicas=2

# Restore the scheduler manifest (if you backed it up)
# sudo cp /tmp/kube-scheduler.yaml.bak /etc/kubernetes/manifests/kube-scheduler.yaml

echo "Cleanup complete!"

Lab Summary

Task Scenario Key Skill
1 CrashLoopBackOff Pod kubectl logs --previous, fix command
2 ImagePullBackOff Pod kubectl describe pod, fix image name
3 Pending Pod kubectl describe pod, fix resources/nodeSelector
4 NotReady Node systemctl status/start kubelet
5 Broken Scheduler Check static Pod manifest, fix typo
6 DNS Failure Check CoreDNS, scale up
7 NetworkPolicy Blocking Create allow NetworkPolicy

Pro Tip: Run through this entire lab under timed conditions. Set a timer for 45 minutes and try to complete all 7 tasks. On the real CKA exam, you’ll face similar scenarios with the same time pressure.

What’s Next

Congratulations — you’ve completed the Troubleshooting chapter! This is the single most important chapter for the CKA exam (30% of the total score).

Key takeaways:

  • Always follow the systematic debugging flow: observe → diagnose → fix → verify
  • Know your tools: kubectl, journalctl, crictl
  • Pod issues are the most common — master the status meanings
  • Node issues require SSH and systemctl/journalctl
  • Control plane issues involve static Pod manifests in /etc/kubernetes/manifests/
  • Networking issues require understanding DNS, Services, and NetworkPolicies

Back to Troubleshooting Hub →

Updated:

You may also enjoy