Amazon EKS — Node Group Upgrade

After the control plane is upgraded, node groups must be updated to match. Managed node groups support rolling updates; self-managed and Karpenter nodes use different strategies.

---

Managed Node Group Rolling Update

Trigger Update

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-node-group

Or target a specific AMI release version:

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-node-group \
  --release-version 1.30.0-20240625

eksctl

eksctl upgrade nodegroup \
  --cluster my-cluster \
  --name my-node-group \
  --kubernetes-version 1.30

---

Rolling Update Process

1. EKS creates new nodes with the updated AMI
2. New nodes join the cluster and become Ready
3. EKS cordons old nodes (prevents new pods from scheduling)
4. EKS drains old nodes (evicts pods respecting PDBs)
5. Pods reschedule on new nodes
6. Old nodes are terminated

---

Update Configuration

Control how aggressive the rolling update is:

aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name my-node-group \
  --update-config '{
    "maxUnavailable": 1
  }'

Setting	Description	Trade-off
maxUnavailable: 1	One node at a time	Slow but safe
maxUnavailable: 3	Three nodes at a time	Faster but needs capacity
maxUnavailablePercentage: 33	33% of nodes at once	Scales with node count

Terraform

resource "aws_eks_node_group" "main" {
  cluster_name    = module.eks.cluster_name
  node_group_name = "main"
  # ...

  update_config {
    max_unavailable = 1
  }

  # Triggers rolling update when AMI changes
  release_version = nonsensitive(data.aws_ssm_parameter.eks_ami.value)
}

data "aws_ssm_parameter" "eks_ami" {
  name = "/aws/service/eks/optimized-ami/${module.eks.cluster_version}/amazon-linux-2/recommended/release_version"
}

---

Force Update

If a rolling update gets stuck (e.g., PDBs blocking drain):

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name my-node-group \
  --force

Warning: Force update ignores PDBs — pods are evicted immediately. Use only when stuck.

---

Self-Managed Node Upgrade

Self-managed nodes don’t support rolling updates natively. Options:

Option 1 — Update Launch Template + Instance Refresh

# Update launch template with new AMI
aws ec2 create-launch-template-version \
  --launch-template-id lt-12345 \
  --source-version 1 \
  --launch-template-data '{"ImageId": "ami-newversion"}'

# Trigger instance refresh
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name my-asg \
  --preferences '{"MinHealthyPercentage": 90}'

Option 2 — Create New ASG, Migrate, Delete Old

1. Create a new ASG with the updated AMI
2. Cordon old nodes: kubectl cordon <node>
3. Drain old nodes: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
4. Scale down old ASG

---

Karpenter Node Upgrade (Drift)

Karpenter detects configuration drift and replaces nodes automatically:

1. Update EC2NodeClass with new AMI:

spec:
  amiSelectorTerms:
    - alias: al2023@latest    # Points to new AMI

1. Karpenter detects nodes running old AMI → marks them for disruption
2. Workloads are drained and rescheduled
3. New nodes launched with updated AMI

To force immediate replacement:

# Delete the node claim
kubectl delete nodeclaim <name>

---

Fargate Pod Upgrade

Fargate pods run on AWS-managed infrastructure. To upgrade:

# Delete pods — they'll be recreated on updated infrastructure
kubectl rollout restart deployment -n production

---

Verifying Node Upgrade

# Check node versions
kubectl get nodes -o wide

# All nodes should show the target version
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}'

# Check for any nodes still on old version
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion'

---

Best Practices

1. Set maxUnavailable based on capacity — ensure enough headroom for pods during drain
2. Always configure PDBs — protect critical workloads during rolling updates
3. Upgrade node groups immediately after control plane — minimize version skew
4. Use Karpenter drift for automatic node rotation — simplest approach
5. Monitor node count during upgrade — ensure new nodes are joining before old ones are removed

---