Amazon EKS — Node Issues

Node problems prevent pods from scheduling and running. Common issues include NotReady status, join failures, IAM misconfigurations, and disk pressure.

---

NotReady Nodes

Diagnosis

# List node status
kubectl get nodes

# Detailed conditions
kubectl describe node <node-name> | grep -A15 Conditions

Condition	Status	Meaning
Ready	False	kubelet is unhealthy
MemoryPressure	True	Node running out of memory
DiskPressure	True	Node running out of disk
PIDPressure	True	Too many processes
NetworkUnavailable	True	Network not configured (CNI issue)

Common Causes

Cause	Diagnosis	Fix
kubelet crashed	SSM into node, check systemctl status kubelet	Restart kubelet or replace node
VPC CNI failure	Check aws-node pod on that node	Restart aws-node DaemonSet pod
Resource exhaustion	kubectl top node <name>	Drain and replace node
EC2 instance issue	AWS Console → EC2 → Status Checks	Terminate instance (ASG replaces it)

Fix — SSM Into Node

# Find instance ID
INSTANCE_ID=$(kubectl get node <node-name> -o jsonpath='{.spec.providerID}' | cut -d'/' -f5)

# Start SSM session
aws ssm start-session --target $INSTANCE_ID

# On the node:
systemctl status kubelet
journalctl -u kubelet --since "10 minutes ago" --no-pager | tail -50

---

Node Join Failures

Nodes are in EC2 but don’t appear in kubectl get nodes.

Diagnosis

# Check node group health
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name my-nodes \
  --query 'nodegroup.health'

# Check EC2 instances
aws ec2 describe-instances \
  --filters "Name=tag:eks:cluster-name,Values=my-cluster" \
  --query 'Reservations[*].Instances[*].{ID:InstanceId,State:State.Name}'

Common Causes

Cause	Symptom	Fix
Wrong IAM role	Node can’t call EKS API	Check node group IAM role has AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly
Security group blocking	Can’t reach API server	Ensure node SG allows egress to API server SG on 443
No internet/NAT	Can’t pull container images	Add NAT Gateway or VPC endpoints
Wrong subnet	Not in EKS VPC	Verify subnet is in the cluster VPC
Bootstrap script failure	Self-managed AMI issue	Check /var/log/cloud-init-output.log via SSM

Fix — Node IAM Role

# Verify required policies
aws iam list-attached-role-policies --role-name EKSNodeRole

# Must have:
# - AmazonEKSWorkerNodePolicy
# - AmazonEKS_CNI_Policy (or IRSA for VPC CNI)
# - AmazonEC2ContainerRegistryReadOnly

---

Disk Pressure

Diagnosis

# Check disk usage via SSM
aws ssm start-session --target <instance-id>

# On node:
df -h
du -sh /var/log/*
du -sh /var/lib/containerd/*
crictl images | wc -l

Common Causes

Cause	Fix
Container images piling up	crictl rmi --prune
Logs filling disk	Configure log rotation, Fluent Bit
Large emptyDir volumes	Set sizeLimit on emptyDir
Small root volume	Increase launch template volume size

Fix — Increase Root Volume

resource "aws_eks_node_group" "main" {
  # ...
  launch_template {
    id      = aws_launch_template.node.id
    version = aws_launch_template.node.latest_version
  }
}

resource "aws_launch_template" "node" {
  name_prefix = "eks-node-"

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size = 100    # Increase from default 20GB
      volume_type = "gp3"
      encrypted   = true
    }
  }
}

---

kubelet Log Analysis

# Via SSM on the node
journalctl -u kubelet --since "30 minutes ago" --no-pager | tail -100

# Common error patterns:
# "failed to get sandbox image" → Can't pull pause container → ECR/internet connectivity
# "Failed to create pod sandbox" → VPC CNI issue → Check aws-node
# "node has DiskPressure" → Disk full → Clean up or resize
# "Unable to register node" → IAM/API server issue → Check IAM role and SG

---

EC2 Instance Status Checks

aws ec2 describe-instance-status --instance-ids <id> \
  --query 'InstanceStatuses[*].{System:SystemStatus.Status,Instance:InstanceStatus.Status}'

Status	Meaning	Action
ok / ok	Healthy	Issue is Kubernetes-level
impaired / ok	Hardware issue	Terminate (ASG replaces)
ok / impaired	OS issue	Reboot or terminate

---

Force-Replace a Stuck Node

# Cordon (no new pods)
kubectl cordon <node-name>

# Drain (evict pods)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force --timeout=120s

# Terminate instance (ASG/Karpenter will replace)
INSTANCE_ID=$(kubectl get node <node-name> -o jsonpath='{.spec.providerID}' | cut -d'/' -f5)
aws ec2 terminate-instances --instance-ids $INSTANCE_ID

---