CKA — Troubleshoot Nodes

CKA Exam Domain

Domain	Weight
Troubleshooting	30%

When Pods are stuck in Pending or workloads are being evicted, the problem might not be the Pod — it might be the node. The CKA exam will test your ability to SSH into a node, diagnose what’s wrong with the kubelet or container runtime, and bring the node back to Ready.

NotReady Node Diagnosis

Step 1: Identify the Problem

kubectl get nodes

NAME           STATUS     ROLES           AGE   VERSION
controlplane   Ready      control-plane   10d   v1.29.0
node01         NotReady   <none>          10d   v1.29.0
node02         Ready      <none>          10d   v1.29.0

Step 2: Describe the Node

kubectl describe node node01

Look at the Conditions section — this tells you exactly what’s wrong:

Conditions:
  Type                 Status    Reason                       Message
  ----                 ------    ------                       -------
  MemoryPressure       False     KubeletHasSufficientMemory   kubelet has sufficient memory
  DiskPressure         False     KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False     KubeletHasSufficientPID      kubelet has sufficient PID
  Ready                False     KubeletNotReady              container runtime not ready

Exam Tip: If a node shows NotReady, the first thing to check is the Conditions in kubectl describe node. This immediately points you in the right direction — kubelet, runtime, memory, disk, or PID.

Node Conditions

Condition	True (Unhealthy)	False (Healthy)
Ready	Node is healthy, kubelet running	Node has a problem — kubelet down or misconfigured
MemoryPressure	Node is running low on memory	Sufficient memory available
DiskPressure	Node is running low on disk space	Sufficient disk space
PIDPressure	Too many processes running on the node	Sufficient PIDs available
NetworkUnavailable	Node network is not correctly configured	Network is configured properly

Important: Ready=True is healthy. For all other conditions, True means there’s a problem.

kubelet Issues

The kubelet is the agent on every node that manages Pod lifecycle. If the kubelet is down, the node goes NotReady.

Diagnosing kubelet

# SSH to the node (or use the terminal provided in the exam)
ssh node01

# Check kubelet status
systemctl status kubelet

# Watch kubelet logs in real-time
journalctl -u kubelet -f

# Check recent kubelet logs
journalctl -u kubelet --since "5 minutes ago"

# Check kubelet logs with more context
journalctl -u kubelet --no-pager | tail -50

Common kubelet Failures

kubelet is stopped:

# Check status
systemctl status kubelet
# Output: inactive (dead)

# Fix: start and enable kubelet
systemctl start kubelet
systemctl enable kubelet

kubelet config is wrong:

# Check kubelet logs for the specific error
journalctl -u kubelet | grep -i "error"

# Common errors:
# "failed to load kubelet config file" → wrong --config path
# "unable to load client CA file" → certificate path wrong
# "node not found" → wrong hostname or registration issue

kubelet is running but node is still NotReady:

# Check if the container runtime is running
systemctl status containerd

# Check kubelet logs for runtime errors
journalctl -u kubelet | grep -i "container runtime"

Quick Fix Pattern

# The exam fix pattern for kubelet:
systemctl status kubelet          # See if it's running
journalctl -u kubelet | tail -20  # See why it failed
# Fix the config/certificate/path issue
systemctl daemon-reload           # If you changed the systemd unit file
systemctl restart kubelet         # Restart after fixing
systemctl enable kubelet          # Ensure it starts on boot

Disk Pressure

When a node runs out of disk space, kubelet evicts Pods and the node may become NotReady.

Diagnosing Disk Pressure

# Check disk usage
df -h

# Check kubelet's directory specifically
df -h /var/lib/kubelet

# Check container images consuming space
crictl images

# Check large log files
du -sh /var/log/*

Fixing Disk Pressure

# Remove unused container images
crictl rmi --prune

# Clean up old container logs
find /var/log/containers -name "*.log" -mtime +7 -delete

# Remove unused Pods/containers
crictl rm $(crictl ps -a -q --state exited)

Exam Tip: On the CKA exam, disk pressure is usually caused by large log files or too many container images. Check /var/lib/kubelet and container images first.

Memory Pressure

Diagnosing Memory Pressure

# Check available memory
free -h

# See what's consuming memory
top

# Check which Pods are using the most memory on this node
kubectl top pods --sort-by=memory -A

Fixing Memory Pressure

• Identify and delete memory-heavy Pods
• Increase memory limits on Pods that are misbehaving
• Add more nodes to distribute the workload

PID Pressure

PID pressure means too many processes are running on the node, exhausting the kernel’s PID limit.

Diagnosing PID Pressure

# Check total number of processes
ps aux | wc -l

# Check the PID limit
cat /proc/sys/kernel/pid_max

# Find which processes are consuming the most
ps aux --sort=-%mem | head -20

This is less common on the exam but good to know.

Container Runtime Issues

If the container runtime (containerd) is down, no containers can run, and the node goes NotReady.

Diagnosing Container Runtime

# Check containerd status
systemctl status containerd

# Check containerd logs
journalctl -u containerd -f

# Check runtime info via crictl
crictl info

# List running containers
crictl ps

# List all containers (including stopped)
crictl ps -a

Fixing Container Runtime

# Restart containerd
systemctl restart containerd

# If containerd won't start, check the config
cat /etc/containerd/config.toml

# Check for socket issues
ls -la /run/containerd/containerd.sock

Getting Container Logs via crictl

# When kubectl doesn't work (API server or kubelet down), crictl is your friend
crictl ps                         # Find the container
crictl logs <container-id>        # Get logs
crictl inspect <container-id>     # Get detailed info

Exam Tip: When kubectl can’t reach the node (kubelet down, API server down), SSH to the node and use crictl to inspect containers directly. This is a critical CKA skill.

Node Events

Check events specifically related to nodes:

# Events for a specific node
kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=node01

# All node events
kubectl get events --field-selector involvedObject.kind=Node --sort-by=.metadata.creationTimestamp

Practice Questions

Q1. Node node01 is in NotReady state. Investigate and fix the issue.

Show Answer

# Step 1: Check node status
kubectl describe node node01
# Look at Conditions to identify the problem

# Step 2: SSH to the node
ssh node01

# Step 3: Check kubelet
systemctl status kubelet
# If inactive/dead:
systemctl start kubelet
systemctl enable kubelet

# If running but erroring, check logs:
journalctl -u kubelet | tail -30

# Step 4: Check container runtime
systemctl status containerd

# Step 5: Verify the fix
kubectl get nodes
# node01 should now be Ready

Q2. The kubelet on node02 is running but the node keeps going NotReady. The kubelet logs show certificate-related errors. Fix it.

Show Answer

# Step 1: SSH to the node
ssh node02

# Step 2: Check kubelet logs for the exact error
journalctl -u kubelet | grep -i "cert\|certificate\|tls"

# Step 3: Common certificate issues:
# - Wrong CA path in kubelet config
# - Expired certificate
# - Mismatched certificate and key

# Check kubelet config for certificate paths
cat /var/lib/kubelet/config.yaml | grep -i cert

# Check if certificates exist and are valid
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

# Step 4: If the config has a wrong path, fix it
vi /var/lib/kubelet/config.yaml

# Step 5: Restart kubelet
systemctl restart kubelet

Q3. node01 shows DiskPressure=True. Diagnose and resolve the issue.

Show Answer

# Step 1: SSH to the node
ssh node01

# Step 2: Check disk usage
df -h
# Identify which filesystem is full

# Step 3: Find what's consuming space
du -sh /var/lib/kubelet/*
du -sh /var/log/*
du -sh /var/lib/containerd/*

# Step 4: Clean up
# Remove unused images
crictl rmi --prune

# Remove old container logs if taking too much space
find /var/log -name "*.log" -size +100M -exec truncate -s 0 {} \;

# Step 5: Verify
df -h
# Disk usage should be lower now

Q4. The containerd runtime on node01 is not running. Fix it so the node becomes Ready.

Show Answer

# Step 1: SSH to the node
ssh node01

# Step 2: Check containerd status
systemctl status containerd
# Likely shows inactive (dead) or failed

# Step 3: Check why it failed
journalctl -u containerd | tail -20

# Step 4: Start containerd
systemctl start containerd
systemctl enable containerd

# Step 5: After containerd is up, kubelet should recover
# If not, restart kubelet too
systemctl restart kubelet

# Step 6: Verify from the control plane
exit  # back to control plane
kubectl get nodes
# node01 should become Ready within ~40 seconds

Q5. Pods on node02 are being evicted. The node conditions show MemoryPressure=True. What should you do?

Show Answer

# Step 1: Check which Pods are using the most memory
kubectl top pods --sort-by=memory -A | head -10

# Step 2: SSH to the node and check memory
ssh node02
free -h
top  # press M to sort by memory

# Step 3: Options to resolve:
# a) Delete or scale down memory-heavy Pods
# b) Increase memory limits and redeploy
# c) Cordon the node and drain Pods to other nodes
kubectl cordon node02
kubectl drain node02 --ignore-daemonsets --delete-emptydir-data

# Step 4: After addressing the root cause, uncordon
kubectl uncordon node02

What’s Next

Sometimes the issue isn’t with Pods or worker nodes — it’s with the control plane itself. Next, we’ll learn how to troubleshoot the API server, scheduler, controller manager, and etcd.

Next: Troubleshoot Control Plane →