Amazon EKS — Blue-Green Cluster Upgrade

For high-risk upgrades or clusters requiring zero downtime, create an entirely new cluster at the target version and migrate workloads. This provides a full rollback path — just switch traffic back.

---

When to Use Blue-Green

Scenario	In-Place	Blue-Green
Minor version bump (N → N+1)	✅	Optional
Skipping versions	❌	✅ Required
Major infra changes (VPC redesign)	❌	✅ Required
Strict SLA (99.99%+)	Risky	✅ Recommended
Small team, simple workloads	✅ Simpler	Overhead

---

Architecture

                    ┌──────────────┐
                    │   Route 53   │
                    │  (DNS CNAME) │
                    └──────┬───────┘
                           │
                    ┌──────┴───────┐
                    │     ALB      │
                    │ (shared or   │
                    │  weighted)   │
                    └──────┬───────┘
               ┌───────────┴───────────┐
               ▼                       ▼
        ┌──────────┐           ┌──────────┐
        │  Blue    │           │  Green   │
        │  EKS     │           │  EKS     │
        │  v1.29   │           │  v1.30   │
        └──────────┘           └──────────┘

---

Step-by-Step Process

Step 1 — Provision Green Cluster

module "eks_green" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "my-cluster-green"
  cluster_version = "1.30"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    main = {
      instance_types = ["m6i.xlarge"]
      min_size       = 3
      max_size       = 10
      desired_size   = 3
    }
  }
}

Step 2 — Deploy Workloads to Green

# Export from blue
kubectl --context blue get deployments -o yaml -n production > deployments.yaml
kubectl --context blue get services -o yaml -n production > services.yaml
kubectl --context blue get configmaps -o yaml -n production > configmaps.yaml

# Apply to green
kubectl --context green apply -f deployments.yaml
kubectl --context green apply -f services.yaml
kubectl --context green apply -f configmaps.yaml

Better approach — use GitOps:

# ArgoCD — add green cluster as destination
argocd cluster add green-cluster-context

# Update ArgoCD apps to deploy to both clusters
# Green cluster gets the same manifests as blue

Step 3 — Validate Green Cluster

# Run smoke tests against green
kubectl --context green get pods --all-namespaces
kubectl --context green run test --image=busybox --restart=Never --rm -it -- wget -qO- http://my-app.production.svc

# Run integration tests
./run-tests.sh --target green-alb.example.com

Step 4 — Shift Traffic (Weighted Routing)

# Route 53 weighted record — shift 10% to green
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456 \
  --change-batch '{
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "blue",
          "Weight": 90,
          "TTL": 60,
          "ResourceRecords": [{"Value": "blue-alb.example.com"}]
        }
      },
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "green",
          "Weight": 10,
          "TTL": 60,
          "ResourceRecords": [{"Value": "green-alb.example.com"}]
        }
      }
    ]
  }'

Gradually increase green weight: 10% → 25% → 50% → 100%.

Step 5 — Decommission Blue

# Once green is handling 100% traffic and stable
# Remove blue cluster
eksctl delete cluster --name my-cluster-blue

# Or Terraform
terraform destroy -target=module.eks_blue

---

Shared Resources

Some resources can be shared between blue and green clusters:

Resource	Shared?	Notes
VPC	✅ Yes	Both clusters use same subnets
RDS / ElastiCache	✅ Yes	Stateful services shared
ECR	✅ Yes	Same container images
S3	✅ Yes	Same buckets
Route 53 zones	✅ Yes	Weighted routing between clusters
ALB	❌ No	Each cluster has its own ALB
IAM roles	Varies	IRSA roles are cluster-specific (OIDC)

---

Rollback Plan

If green cluster has issues:

1. Shift Route 53 weight back to blue (100%)
2. Fix issues on green
3. Retry migration

# Instant rollback — all traffic to blue
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456 \
  --change-batch '{
    "Changes": [
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "blue",
          "Weight": 100,
          "TTL": 60,
          "ResourceRecords": [{"Value": "blue-alb.example.com"}]
        }
      },
      {
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "app.example.com",
          "Type": "CNAME",
          "SetIdentifier": "green",
          "Weight": 0,
          "TTL": 60,
          "ResourceRecords": [{"Value": "green-alb.example.com"}]
        }
      }
    ]
  }'

---

Best Practices

1. Use GitOps — same manifests deploy to both clusters consistently
2. Share the VPC — avoids VPC peering complexity and database connectivity issues
3. Start with canary traffic (10%) — validate before full cutover
4. Keep blue cluster running for 24-48 hours after full switchover
5. Automate with a runbook — blue-green upgrades have many steps

---