Troubleshooting Kubernetes

Things will break. Pods crash, Services stop responding, deployments fail. This chapter gives you a systematic troubleshooting toolkit to diagnose and fix the most common Kubernetes issues quickly.

The Troubleshooting Framework

When something goes wrong, follow this systematic approach:

1. Identify the Problem
   ↓
2. Check Pod Status
   ↓
3. Check Events
   ↓
4. Check Logs
   ↓
5. Check Configuration
   ↓
6. Check Networking
   ↓
7. Check Resources

Step 1: Quick Health Check

# Cluster health
kubectl cluster-info
kubectl get nodes
kubectl get componentstatuses    # May be deprecated in newer versions

# All resources in a namespace
kubectl get all -n <namespace>

# Pods not in Running state
kubectl get pods -A --field-selector 'status.phase!=Running'

Step 2: Pod Troubleshooting

Common Pod States and Fixes

Status	Meaning	Likely Cause	Fix
Pending	Not scheduled yet	No node has enough resources	Check resource requests, node capacity
ImagePullBackOff	Can’t pull container image	Wrong image name, missing registry credentials	Verify image name, check imagePullSecrets
CrashLoopBackOff	Container keeps crashing and restarting	Application error, wrong command, missing config	Check logs, fix application
ErrImagePull	Failed to pull image	Registry auth, image not found	Fix image reference, add pull secrets
CreateContainerConfigError	Can’t create container config	Missing ConfigMap or Secret	Create the referenced ConfigMap/Secret
OOMKilled	Out of memory	Memory limit too low	Increase memory limits
Evicted	Pod removed by kubelet	Node under resource pressure	Check node resources, set proper requests

Debugging a Pending Pod

kubectl describe pod <pod-name>
# Look at Events section:
# FailedScheduling: Insufficient cpu
# FailedScheduling: Insufficient memory
# FailedScheduling: 0/3 nodes are available: 3 node(s) had taint...

# Check node resources
kubectl describe node <node-name> | grep -A 10 "Allocated resources"
kubectl top nodes

Debugging ImagePullBackOff

kubectl describe pod <pod-name>
# Events: Failed to pull image "my-registry/my-app:1.0": ...

# Verify the image exists
docker pull my-registry/my-app:1.0

# Check if imagePullSecrets are configured
kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

# Verify the secret exists and is correct
kubectl get secret my-registry-secret -o yaml

Debugging CrashLoopBackOff

# Check logs from the CURRENT (crashing) container
kubectl logs <pod-name>

# Check logs from the PREVIOUS crashed container
kubectl logs <pod-name> --previous

# If the container exits too quickly, check events
kubectl describe pod <pod-name>

# Try to run the container interactively
kubectl run debug --image=<image> --restart=Never -it -- /bin/sh

Debugging OOMKilled

kubectl describe pod <pod-name>
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137

# Check resource usage vs limits
kubectl top pod <pod-name>

# Fix: Increase memory limits
kubectl set resources deployment/<name> -c=<container> --limits=memory=512Mi

Step 3: Service Troubleshooting

Service Not Reachable

# 1. Check the Service exists
kubectl get svc <service-name>

# 2. Check Endpoints — are Pods registered?
kubectl get endpoints <service-name>

# If endpoints are empty, the selector doesn't match any Pods
# 3. Compare Service selector with Pod labels
kubectl get svc <service-name> -o jsonpath='{.spec.selector}'
kubectl get pods --show-labels

# 4. Test from inside the cluster
kubectl run test --image=busybox --rm -it --restart=Never -- \
  wget -qO- --timeout=5 http://<service-name>:<port>

# 5. Check kube-proxy
kubectl get pods -n kube-system | grep kube-proxy
kubectl logs -n kube-system <kube-proxy-pod>

Common Service Issues

Problem	Symptom	Fix
Wrong selector	Endpoints empty	Match Service selector to Pod labels
Wrong port	Connection refused	Verify port and targetPort
Pod not Ready	Endpoints missing	Fix readiness probe
DNS not working	Can’t resolve service name	Check CoreDNS pods

Step 4: Deployment Troubleshooting

Deployment Won’t Roll Out

# Check rollout status
kubectl rollout status deployment/<name>

# Describe for events
kubectl describe deployment/<name>

# Check ReplicaSet
kubectl get rs -l app=<name>

# Common fix: Check the pod template
kubectl get deployment/<name> -o yaml | grep -A 20 "template:"

Deployment Stuck in Progressing

# Check if it exceeded progressDeadlineSeconds (default 600s)
kubectl describe deployment/<name>
# Condition: Progressing = False, reason: ProgressDeadlineExceeded

# Rollback
kubectl rollout undo deployment/<name>

Step 5: Node Troubleshooting

# Check node conditions
kubectl describe node <node-name>

# Look for conditions
# MemoryPressure: True → Node running out of memory
# DiskPressure: True → Node running out of disk
# PIDPressure: True → Too many processes
# Ready: False → Node is unhealthy

# Check node resources
kubectl top node <node-name>

# Check kubelet logs (on the node)
journalctl -u kubelet -f

# Cordon a problematic node (prevent new scheduling)
kubectl cordon <node-name>

# Drain a node (move all Pods to other nodes)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Step 6: DNS Troubleshooting

# Check CoreDNS is running
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Test DNS resolution from a Pod
kubectl run dnsutils --image=tutum/dnsutils --rm -it --restart=Never -- \
  nslookup kubernetes.default.svc.cluster.local

# Test DNS to a specific Service
kubectl run dnsutils --image=tutum/dnsutils --rm -it --restart=Never -- \
  nslookup <service-name>.<namespace>.svc.cluster.local

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Check CoreDNS ConfigMap
kubectl get configmap coredns -n kube-system -o yaml

Step 7: Network Troubleshooting

# Test Pod-to-Pod connectivity
kubectl exec <pod-a> -- ping <pod-b-ip>

# Test Pod-to-Service connectivity
kubectl exec <pod-a> -- wget -qO- --timeout=3 http://<service>:<port>

# Check Network Policies
kubectl get networkpolicies -A

# Describe a Network Policy
kubectl describe netpol <name>

# Temporary debug Pod with network tools
kubectl run netdebug --image=nicolaka/netshoot --rm -it --restart=Never -- /bin/bash
# Inside the Pod:
# curl, ping, dig, nslookup, traceroute, tcpdump, etc.

Debugging Toolkit Commands

Essential Commands

# Describe everything about a resource
kubectl describe <resource> <name>

# Get resource as YAML
kubectl get <resource> <name> -o yaml

# Check events
kubectl get events --sort-by='.lastTimestamp'
kubectl get events --field-selector type=Warning

# Resource diff (compare desired vs live)
kubectl diff -f my-manifest.yaml

# API resources available
kubectl api-resources

Exec into a Running Container

# Open a shell
kubectl exec -it <pod-name> -- /bin/bash
kubectl exec -it <pod-name> -- /bin/sh  # If bash isn't available

# Run a command
kubectl exec <pod-name> -- env
kubectl exec <pod-name> -- cat /etc/config/app.conf

# Multi-container Pod
kubectl exec -it <pod-name> -c <container-name> -- /bin/sh

Ephemeral Debug Containers

For minimal images that don’t have shells or debug tools:

# Attach a debug container to a running Pod (K8s 1.25+)
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Create a copy of the Pod with a debug container
kubectl debug <pod-name> -it --image=busybox --copy-to=debug-pod

# Debug a node directly
kubectl debug node/<node-name> -it --image=busybox

Quick Reference: Troubleshooting Decision Tree

Pod not Running?
├── Pending → Check scheduling (resources, taints, affinity)
├── ImagePullBackOff → Check image name and registry credentials
├── CrashLoopBackOff → Check logs (kubectl logs --previous)
├── OOMKilled → Increase memory limits
└── CreateContainerConfigError → Check referenced ConfigMaps/Secrets

Service not working?
├── Endpoints empty → Fix label selector
├── Connection refused → Fix port/targetPort
├── DNS not resolving → Check CoreDNS
└── Timeout → Check Network Policies

Deployment stuck?
├── ProgressDeadlineExceeded → Rollback
├── ReplicaSet not scaling → Check Pod template
└── Pods not Ready → Fix readiness probe

Summary

• Always start with kubectl describe — events tell you what went wrong
• Check logs with kubectl logs and kubectl logs --previous
• Use the right tools — netshoot for networking, ephemeral containers for minimal images
• Follow the framework: Pod → Events → Logs → Config → Network → Resources
• kubectl debug is your best friend for containers without shells

What’s Next?

Let’s wrap up the Foundation tutorial with Best Practices — patterns and anti-patterns for running Kubernetes in production.

Next: Best Practices →