Amazon EKS — Cluster Upgrades Hands-On Lab

2 minute read

In this lab, you will run through the pre-upgrade checks, upgrade the control plane, upgrade add-ons, and upgrade node groups on an EKS cluster.

Prerequisites

  • Running EKS cluster (version N-1, e.g., 1.29)
  • AWS CLI, kubectl, eksctl configured
  • pluto installed (brew install FairwindsOps/tap/pluto)

Step 1 — Pre-Upgrade Assessment

Check Current Version

CLUSTER_NAME="my-cluster"
CURRENT_VERSION=$(aws eks describe-cluster --name $CLUSTER_NAME --query "cluster.version" --output text)
echo "Current version: $CURRENT_VERSION"

Check EKS Upgrade Insights

aws eks list-insights --cluster-name $CLUSTER_NAME --output table

Scan for Deprecated APIs

pluto detect-all-in-cluster

pluto detect-helm --output wide

Check Add-on Compatibility

TARGET_VERSION="1.30"

for addon in vpc-cni coredns kube-proxy aws-ebs-csi-driver; do
  echo "=== $addon ==="
  CURRENT=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon --query "addon.addonVersion" --output text 2>/dev/null)
  echo "Current: $CURRENT"
  
  LATEST=$(aws eks describe-addon-versions --addon-name $addon --kubernetes-version $TARGET_VERSION \
    --query "addons[0].addonVersions[0].addonVersion" --output text)
  echo "Latest for $TARGET_VERSION: $LATEST"
  echo ""
done

Verify PDBs and Node Health

echo "=== PodDisruptionBudgets ==="
kubectl get pdb --all-namespaces

echo ""
echo "=== Node Status ==="
kubectl get nodes -o wide

echo ""
echo "=== Unhealthy Pods ==="
kubectl get pods --all-namespaces --field-selector status.phase!=Running,status.phase!=Succeeded

Step 2 — Upgrade Control Plane

echo "Upgrading control plane to $TARGET_VERSION..."
aws eks update-cluster-version \
  --name $CLUSTER_NAME \
  --kubernetes-version $TARGET_VERSION

# Get update ID
UPDATE_ID=$(aws eks list-updates --name $CLUSTER_NAME --query "updateIds[0]" --output text)

# Monitor progress
while true; do
  STATUS=$(aws eks describe-update --name $CLUSTER_NAME --update-id $UPDATE_ID --query "update.status" --output text)
  echo "Status: $STATUS ($(date))"
  if [[ "$STATUS" == "Successful" ]] || [[ "$STATUS" == "Failed" ]]; then
    break
  fi
  sleep 30
done

Verify:

aws eks describe-cluster --name $CLUSTER_NAME --query "cluster.{Version:version,Status:status}"

Step 3 — Upgrade Add-ons

# kube-proxy first
KUBE_PROXY_VERSION=$(aws eks describe-addon-versions --addon-name kube-proxy --kubernetes-version $TARGET_VERSION \
  --query "addons[0].addonVersions[0].addonVersion" --output text)

aws eks update-addon \
  --cluster-name $CLUSTER_NAME \
  --addon-name kube-proxy \
  --addon-version $KUBE_PROXY_VERSION \
  --resolve-conflicts OVERWRITE

echo "Upgrading kube-proxy to $KUBE_PROXY_VERSION"

# VPC CNI
VPC_CNI_VERSION=$(aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version $TARGET_VERSION \
  --query "addons[0].addonVersions[0].addonVersion" --output text)

aws eks update-addon \
  --cluster-name $CLUSTER_NAME \
  --addon-name vpc-cni \
  --addon-version $VPC_CNI_VERSION \
  --resolve-conflicts PRESERVE

echo "Upgrading vpc-cni to $VPC_CNI_VERSION"

# CoreDNS
COREDNS_VERSION=$(aws eks describe-addon-versions --addon-name coredns --kubernetes-version $TARGET_VERSION \
  --query "addons[0].addonVersions[0].addonVersion" --output text)

aws eks update-addon \
  --cluster-name $CLUSTER_NAME \
  --addon-name coredns \
  --addon-version $COREDNS_VERSION \
  --resolve-conflicts OVERWRITE

echo "Upgrading coredns to $COREDNS_VERSION"

Wait for all add-ons:

for addon in kube-proxy vpc-cni coredns; do
  echo "Waiting for $addon..."
  while true; do
    STATUS=$(aws eks describe-addon --cluster-name $CLUSTER_NAME --addon-name $addon --query "addon.status" --output text)
    echo "$addon: $STATUS"
    if [[ "$STATUS" == "ACTIVE" ]]; then break; fi
    sleep 10
  done
done

Step 4 — Upgrade Node Groups

NODE_GROUPS=$(aws eks list-nodegroups --cluster-name $CLUSTER_NAME --query "nodegroups[]" --output text)

for ng in $NODE_GROUPS; do
  echo "Upgrading node group: $ng"
  aws eks update-nodegroup-version \
    --cluster-name $CLUSTER_NAME \
    --nodegroup-name $ng
done

Monitor:

for ng in $NODE_GROUPS; do
  echo "=== $ng ==="
  aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name $ng \
    --query "nodegroup.{Status:status,Version:version,ReleaseVersion:releaseVersion}"
done

Watch nodes:

watch kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,STATUS:.status.conditions[-1].type,AGE:.metadata.creationTimestamp'

Step 5 — Post-Upgrade Validation

echo "=== Cluster Version ==="
kubectl version --short

echo ""
echo "=== Node Versions ==="
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion'

echo ""
echo "=== System Pods ==="
kubectl get pods -n kube-system

echo ""
echo "=== DNS Test ==="
kubectl run dns-test --image=busybox:latest --restart=Never --rm -it -- nslookup kubernetes.default

echo ""
echo "=== All Pods Health ==="
kubectl get pods --all-namespaces --field-selector status.phase!=Running,status.phase!=Succeeded | head -20

Cleanup

No extra cleanup needed — you upgraded the existing cluster in place.

Updated:

You may also enjoy