CKA — Troubleshoot Networking

9 minute read

CKA Exam Domain

Domain Weight
Troubleshooting 30%

Networking issues are some of the trickiest to debug in Kubernetes. DNS resolution failures, Services not routing traffic, NetworkPolicies silently blocking connections, and CNI plugin problems all fall under this section. The CKA exam loves networking questions because they test deep understanding.

DNS Resolution Failures

CoreDNS is the default DNS provider in Kubernetes. When it’s broken, Pods can’t resolve Service names, and everything falls apart.

Test DNS Resolution

# Quick DNS test from a Pod
kubectl run dnstest --image=busybox:1.36 --restart=Never --rm -it -- nslookup kubernetes.default

# Or use an existing Pod
kubectl exec -it <pod> -- nslookup kubernetes.default

Expected output:

Server:    10.96.0.10
Address:   10.96.0.10#53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

If DNS fails:

;; connection timed out; no servers could be reached

Check CoreDNS

# Check if CoreDNS Pods are running
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Check the CoreDNS Service
kubectl get svc -n kube-system kube-dns

# Check CoreDNS ConfigMap
kubectl get configmap -n kube-system coredns -o yaml

Common DNS Issues

CoreDNS Pods not running:

# Check why
kubectl describe pods -n kube-system -l k8s-app=kube-dns

# If they are scaled to 0
kubectl scale deployment -n kube-system coredns --replicas=2

Forward loop detected:

If the CoreDNS logs show Loop ... detected, the upstream DNS is pointing back to CoreDNS itself:

# Check the CoreDNS ConfigMap
kubectl edit configmap coredns -n kube-system

# BROKEN — forward points to node's resolv.conf which has 127.0.0.1
forward . /etc/resolv.conf

# FIXED — forward to an external DNS server
forward . 8.8.8.8 8.8.4.4

Pod’s resolv.conf is wrong:

# Check what DNS server the Pod is using
kubectl exec -it <pod> -- cat /etc/resolv.conf

Expected:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local

Service Not Reachable

When a Service exists but traffic doesn’t reach the backend Pods.

Debugging Flow

# Step 1: Does the Service exist?
kubectl get svc <svc-name>

# Step 2: Does it have Endpoints?
kubectl get endpoints <svc-name>

# Step 3: Describe the Service for details
kubectl describe svc <svc-name>

Common Issues

No Endpoints (empty):

kubectl get endpoints my-svc
# NAME     ENDPOINTS   AGE
# my-svc   <none>      5m

This means the Service selector doesn’t match any Pod labels:

# Check what selector the Service uses
kubectl describe svc my-svc | grep Selector

# Check what labels the Pods have
kubectl get pods --show-labels

# Fix: update the Service selector or the Pod labels to match
kubectl edit svc my-svc

Wrong target port:

# BROKEN — Service targets port 8080 but the container listens on 80
spec:
  ports:
  - port: 80
    targetPort: 8080  # wrong! container listens on 80

# FIXED
spec:
  ports:
  - port: 80
    targetPort: 80

Verify connectivity:

# Test from within the cluster
kubectl run curltest --image=curlimages/curl --restart=Never --rm -it -- \
  curl -s http://my-svc.default.svc.cluster.local:80

NetworkPolicy Blocking Traffic

NetworkPolicies are like firewalls for Pods. They can silently block traffic without any error messages — the connection just times out.

Key Rules to Remember

  • No NetworkPolicy = all traffic allowed (default)
  • Any NetworkPolicy selecting a Pod = default deny for the selected direction (ingress/egress)
  • NetworkPolicies are additive — you allow specific traffic, everything else is denied
  • DNS requires port 53 egress! — If you create an egress policy, you must explicitly allow DNS

Debugging NetworkPolicies

# List all NetworkPolicies in a namespace
kubectl get networkpolicies -n <namespace>

# Describe a specific policy
kubectl describe networkpolicy <name> -n <namespace>

# Check if a NetworkPolicy is selecting your Pod
kubectl get networkpolicies -n <namespace> -o yaml | grep -A 5 "podSelector"

Common NetworkPolicy Issues

Default deny blocking everything:

# This policy denies ALL ingress traffic to Pods with label app=web
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress

Fix — allow traffic from specific sources:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 80

DNS blocked by egress policy:

# If you add an egress policy, DNS won't work unless you allow it!
# Add this to your egress rules:
  egress:
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Exam Tip: If a Pod suddenly can’t resolve DNS after you add a NetworkPolicy with egress rules, you forgot to allow port 53 to CoreDNS. This is a very common CKA trap.

kube-proxy Issues

kube-proxy manages iptables/ipvs rules that route Service traffic to Pods. If kube-proxy is broken, Services won’t work even though Pods are running fine.

Debugging kube-proxy

# Check kube-proxy Pods (runs as a DaemonSet)
kubectl get pods -n kube-system -l k8s-app=kube-proxy

# Check kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy

# Check kube-proxy ConfigMap
kubectl get configmap -n kube-system kube-proxy -o yaml

# Check iptables rules on a node (SSH first)
iptables -t nat -L KUBE-SERVICES | head -20

Common kube-proxy Issues

  • kube-proxy Pods not running — Check the DaemonSet: kubectl get ds -n kube-system kube-proxy
  • Wrong cluster CIDR — kube-proxy config doesn’t match actual cluster CIDR
  • Wrong mode — Check if kube-proxy is in iptables or ipvs mode

CNI Plugin Problems

The CNI (Container Network Interface) plugin provides Pod networking. If it’s broken, Pods can’t get IP addresses and are stuck in ContainerCreating.

Symptoms

kubectl get pods
# NAME     READY   STATUS              RESTARTS   AGE
# my-app   0/1     ContainerCreating   0          5m

kubectl describe pod my-app
# Events:
#   Warning  FailedCreatePodSandBox  ...  networkPlugin cni failed to set up pod network

Debugging CNI

# Check CNI config on the node
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-flannel.conflist  # or whatever CNI you use

# Check CNI plugin Pods
kubectl get pods -n kube-system | grep -E "flannel|calico|weave|cilium"

# Check CNI plugin logs
kubectl logs -n kube-system <cni-pod-name>

# Check CNI binaries
ls /opt/cni/bin/

Common CNI Issues

  • CNI plugin not installed — No CNI config in /etc/cni/net.d/
  • CNI plugin Pods crashing — Check logs for the CNI DaemonSet
  • Mismatched Pod CIDR — The CNI plugin’s CIDR doesn’t match the cluster’s --pod-network-cidr

Network Debugging from a Pod

When you need to test networking from within the cluster, spin up a debugging Pod with network tools:

# Create a debugging Pod with networking tools
kubectl run nettools --image=nicolaka/netshoot --command -- sleep 3600

# Wait for it to be ready
kubectl wait --for=condition=ready pod/nettools --timeout=60s

# DNS resolution test
kubectl exec -it nettools -- nslookup my-svc.default.svc.cluster.local

# HTTP connectivity test
kubectl exec -it nettools -- curl -s http://my-svc:80

# Ping a Pod IP
kubectl exec -it nettools -- ping -c 3 <pod-ip>

# TCP port check
kubectl exec -it nettools -- nc -zv <service-ip> <port>

# Trace route
kubectl exec -it nettools -- traceroute <pod-ip>

# Check DNS configuration
kubectl exec -it nettools -- cat /etc/resolv.conf

# Clean up
kubectl delete pod nettools

Exam Tip: nicolaka/netshoot is the best debugging image — it has curl, nslookup, dig, ping, traceroute, tcpdump, netcat, and more. If it’s not available, use busybox:1.36 which has basic tools like nslookup, ping, and wget.

Practice Questions

Q1. Pods can’t resolve Service names. DNS lookups time out. Diagnose and fix.

Show Answer 
# Step 1: Check if CoreDNS Pods are running
kubectl get pods -n kube-system -l k8s-app=kube-dns
# If no Pods or they're crashing:

# Step 2: Check the CoreDNS Deployment
kubectl get deployment -n kube-system coredns
kubectl describe deployment -n kube-system coredns

# Step 3: If Pods are not running, scale up
kubectl scale deployment -n kube-system coredns --replicas=2

# Step 4: If Pods are crashing, check logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Step 5: If there's a loop detected, fix the ConfigMap
kubectl edit configmap coredns -n kube-system
# Change forward . /etc/resolv.conf to forward . 8.8.8.8

# Step 6: Verify
kubectl run dnstest --image=busybox:1.36 --restart=Never --rm -it -- nslookup kubernetes.default

Q2. A Service named web-svc in the default namespace has no Endpoints. The backend Deployment web has 3 running Pods. Fix it.

Show Answer 
# Step 1: Check the Service selector
kubectl describe svc web-svc | grep Selector
# Output: Selector: app=webapp

# Step 2: Check Pod labels
kubectl get pods -l app=webapp
# No resources found — the labels don't match!

kubectl get pods --show-labels
# Output shows labels like app=web (not app=webapp)

# Step 3: Fix the Service selector to match the Pod labels
kubectl edit svc web-svc
# Change selector from app: webapp to app: web

# Step 4: Verify
kubectl get endpoints web-svc
# Should now show the Pod IPs

Q3. After applying a NetworkPolicy, Pods in the production namespace can’t communicate with each other. Fix it.

Show Answer 
# Step 1: Check existing NetworkPolicies
kubectl get networkpolicies -n production
kubectl describe networkpolicy -n production

# Step 2: If there's a default-deny policy and you need to allow traffic:
# Create an allow policy for the specific traffic needed

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-internal
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector: {}
  egress:
  - to:
    - podSelector: {}
  - to:    # Allow DNS
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
EOF

Q4. A NodePort Service is not reachable from outside the cluster though the Pods are running fine. Investigate.

Show Answer 
# Step 1: Check the Service
kubectl get svc <svc-name>
kubectl describe svc <svc-name>

# Step 2: Verify Endpoints exist
kubectl get endpoints <svc-name>

# Step 3: Check kube-proxy
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy

# Step 4: Check iptables rules on the node
ssh <node>
iptables -t nat -L KUBE-NODEPORTS

# Step 5: Common fixes:
# - Restart kube-proxy if crashed
# - Check if the NodePort is in the allowed range (30000-32767)
# - Check if the targetPort matches the container port
# - Check firewall rules on the node

Q5. Pods are stuck in ContainerCreating and the events show CNI errors. Fix it.

Show Answer 
# Step 1: Check the Pod events
kubectl describe pod <pod-name>
# Look for: "networkPlugin cni failed..."

# Step 2: Check if CNI plugin Pods are running
kubectl get pods -n kube-system | grep -E "flannel|calico|weave|cilium"

# Step 3: If no CNI Pods exist, install the CNI plugin
# For Flannel:
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

# For Calico:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

# Step 4: Check CNI config on the node
ssh <node>
ls /etc/cni/net.d/
ls /opt/cni/bin/

# Step 5: Verify
kubectl get pods -n kube-system  # CNI Pods should be Running
kubectl get pods                  # Stuck Pods should transition to Running

What’s Next

You’ve learned how to troubleshoot the four key areas: Pods, nodes, control plane, and networking. Next, let’s consolidate all the key debugging commands into a handy reference.

Next: Key Debugging Commands →

Updated:

You may also enjoy