Amazon EKS — Networking Issues

Networking problems in EKS usually stem from VPC CNI misconfigurations, security group rules, DNS failures, or IP address exhaustion. Here’s how to diagnose and fix each.

---

Pod-to-Pod Connectivity Failures

Symptom

Pods can’t communicate with other pods:

# Test from one pod to another
kubectl exec -it pod-a -- wget -qO- http://pod-b-ip:8080 --timeout=5

Common Causes

Cause	Diagnosis	Fix
Security group blocking	Check node SG inbound rules	Allow all traffic within the cluster SG
Network policy blocking	kubectl get networkpolicies -A	Review policies, add explicit allow rules
Pod not running	kubectl get pods -o wide	Fix the target pod first
VPC CNI issue	Check aws-node DaemonSet logs	Restart aws-node pods

Fix — Security Group

Ensure the cluster security group allows all traffic within itself:

# Get cluster security group
CLUSTER_SG=$(aws eks describe-cluster --name my-cluster --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId' --output text)

# Allow all traffic within the SG
aws ec2 authorize-security-group-ingress \
  --group-id $CLUSTER_SG \
  --source-group $CLUSTER_SG \
  --protocol -1

---

DNS Resolution Issues

Symptom

kubectl exec -it debug-pod -- nslookup my-service.production.svc.cluster.local
# server can't find my-service: NXDOMAIN
# or timeout

Diagnosis

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Verify the Service exists
kubectl get svc my-service -n production

# Test with explicit CoreDNS IP
kubectl exec -it debug-pod -- nslookup my-service.production.svc.cluster.local 10.100.0.10

Common Fixes

Issue	Fix
CoreDNS pods not running	kubectl rollout restart deployment coredns -n kube-system
CoreDNS OOMKilled	Increase memory limit in CoreDNS deployment
ndots issue (slow external DNS)	Set dnsPolicy or add ndots: 2 in pod spec
Custom DNS not working	Check CoreDNS Corefile ConfigMap

Fix — ndots Optimization

Default ndots:5 causes 5+ DNS lookups per external domain:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

---

ALB/NLB Not Routing Traffic

Symptom

ALB Ingress created but returns 502/503 or no target registration.

# Check Ingress status
kubectl describe ingress my-ingress -n production

# Check AWS Load Balancer Controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller --tail=50

Symptom	Cause	Fix
Ingress has no ADDRESS	Controller not installed or failing	Install/check AWS LB Controller
502 Bad Gateway	Target group has no healthy targets	Check target type (ip vs instance), pod health
503 Service Unavailable	No healthy pods behind service	Check pod readiness probes
Target group empty	Wrong target-type annotation	Use alb.ingress.kubernetes.io/target-type: ip
Certificate error	Wrong ACM certificate ARN	Verify certificate-arn annotation

Fix — Target Type Mismatch

# For pods — use IP target type
annotations:
  alb.ingress.kubernetes.io/target-type: ip

# For NodePort — use instance target type
annotations:
  alb.ingress.kubernetes.io/target-type: instance

---

VPC CNI IP Exhaustion

Symptom

Pods stuck in ContainerCreating with event: failed to assign an IP address to container

# Check available IPs per node
kubectl get nodes -o custom-columns='NODE:.metadata.name,IPs:.status.addresses[*].address'

# Check VPC CNI logs
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "IP"

# Check subnet IP availability
aws ec2 describe-subnets --subnet-ids subnet-abc123 \
  --query 'Subnets[*].{SubnetId:SubnetId,AvailableIPs:AvailableIpAddressCount,CIDR:CidrBlock}'

Fixes

Approach	How
Enable prefix delegation	kubectl set env daemonset/aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
Use larger subnets	/19 or /18 instead of /24
Add secondary CIDR to VPC	Associate 100.64.0.0/16 range
Use custom networking	Separate pod CIDR from node CIDR

---

Network Policy Blocking

# List all network policies
kubectl get networkpolicies -A

# Describe specific policy
kubectl describe networkpolicy <name> -n <namespace>

If a NetworkPolicy exists with no matching egress/ingress rule, traffic is blocked:

# Allow all egress for debugging
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress-debug
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - {}

Remember: Once any NetworkPolicy selects a pod, all non-matching traffic is denied.

---

Quick Debugging Toolkit

# Deploy a debug pod with networking tools
kubectl run netshoot --image=nicolaka/netshoot --restart=Never -it --rm -- /bin/bash

# Inside netshoot:
nslookup kubernetes.default
curl -v http://my-service.production.svc:80
traceroute 10.0.1.100
tcpdump -i eth0 port 80

---