Amazon EKS — Production Reliability Checklist

Reliability ensures your workloads survive failures, scale under load, and recover quickly. Validate each item.

---

High Availability

• Nodes spread across 3+ AZs — topologySpreadConstraints or pod anti-affinity
• 2+ replicas for all critical workloads — never run a single replica in production
• Pod Disruption Budgets (PDB) — for all Deployments and StatefulSets
• No single AZ bottleneck — node groups or Karpenter configured for multi-AZ

Example topology spread:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app

Scaling

• Horizontal Pod Autoscaler (HPA) — configured for CPU/memory or custom metrics
• Cluster Autoscaler or Karpenter — for node scaling
• Resource requests set on all containers — scheduler and autoscaler need these
• Resource limits set for memory — prevent OOMKill from affecting other pods
• VPA in Off mode — for right-sizing recommendations
• Load tested — verify scaling behavior under realistic traffic

Health Checks

• Liveness probes — restart containers that are deadlocked
• Readiness probes — remove unhealthy pods from Service endpoints
• Startup probes — for slow-starting applications (prevent premature kills)
• Probe timeouts are generous — avoid false positives under load

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Resilience

• Graceful shutdown — preStop hook + terminationGracePeriodSeconds
• Connection draining — ALB/NLB deregistration delay configured
• Retry and circuit-breaking — in application or service mesh
• Priority classes — for critical system components

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]
terminationGracePeriodSeconds: 30

Disaster Recovery

• EBS snapshots scheduled — for stateful workloads (Velero)
• Etcd backups — EKS manages this, but verify via AWS Support
• Cross-region strategy defined — active-passive or active-active
• Recovery runbook documented — tested quarterly
• RTO/RPO defined — per workload tier

Upgrades

• Upgrade plan documented — control plane → add-ons → nodes
• Deprecated API detection — pluto or Kubent run in CI
• Upgrade tested in staging first — identical version path
• Extended support evaluated — if extended support is needed, budget for it

---

Validation Commands

# Check PDBs exist
kubectl get pdb -A

# Check HPA status
kubectl get hpa -A

# Check topology spread
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.nodeName}{"\n"}{end}' -n production

# Check resource requests
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}: cpu={.spec.containers[0].resources.requests.cpu} mem={.spec.containers[0].resources.requests.memory}{"\n"}{end}'

# Check probe configuration
kubectl get deployment -n production -o jsonpath='{range .items[*]}{.metadata.name}: liveness={.spec.template.spec.containers[0].livenessProbe.httpGet.path} readiness={.spec.template.spec.containers[0].readinessProbe.httpGet.path}{"\n"}{end}'

---