Amazon EKS — Day-2 Operations Checklist

2 minute read

Day-2 covers everything after the cluster is running — observability, cost management, CI/CD, and operational processes.

Observability

Logging

  • Container logs collected — Fluent Bit or Fluentd DaemonSet shipping to CloudWatch / OpenSearch
  • Control plane logs enabled — all 5 log types to CloudWatch
  • Log retention policy — 30-90 days in CloudWatch, archive to S3
  • Structured logging — JSON format for easy parsing and querying
  • Log-based alerts — CloudWatch Alarms or Prometheus alerting on error patterns

Metrics

  • Prometheus or CloudWatch Container Insights — cluster and pod metrics
  • Grafana dashboards — for cluster health, pod health, node utilization
  • Custom metrics — application-specific metrics exposed via ServiceMonitor
  • Kube-state-metrics installed — for Kubernetes object metrics (deployments, pods, etc.)
  • Node exporter installed — for OS-level node metrics

Tracing (if applicable)

  • AWS X-Ray or OpenTelemetry — distributed tracing for microservices
  • ADOT Collector deployed — as DaemonSet or sidecar

Alerting

  • PagerDuty / Opsgenie / SNS integration — for on-call notification
  • Alert rules defined:
    • Pod CrashLoopBackOff > 5 minutes
    • Node NotReady > 5 minutes
    • PVC usage > 80%
    • High pod restart count
    • HPA at max replicas
    • Certificate expiring within 14 days

Cost Management

  • Cost allocation tags — Environment, Team, Service on all resources
  • Kubecost or OpenCost — per-namespace and per-workload cost visibility
  • Spot instances — for non-critical workloads (with Node Termination Handler)
  • Reserved Instances / Savings Plans — for baseline On-Demand capacity
  • Karpenter consolidation — enabled for automatic bin-packing
  • AWS Budgets — alerts at 80% and 100% of monthly budget
  • Monthly cost review — compare actual vs budget, identify waste

CI/CD & Deployment

  • GitOps — ArgoCD or Flux for declarative deployments
  • CI pipeline — build, test, scan, push to ECR
  • Progressive delivery — canary or blue/green for critical services
  • Rollback strategy — documented and tested
  • Environment promotion — dev → staging → production with approval gates
  • Secret management — External Secrets Operator or Sealed Secrets (never plain Secrets in Git)

Operational Processes

Runbooks

  • Cluster upgrade runbook — step-by-step with rollback
  • Incident response runbook — escalation paths and initial triage
  • Node replacement runbook — cordon, drain, terminate
  • Disaster recovery runbook — tested quarterly

On-Call

  • On-call rotation defined — with clear escalation
  • Runbook links in alerts — every alert includes a link to the relevant runbook
  • Post-incident review process — blameless retrospectives

Compliance

  • CIS EKS Benchmark — run with kube-bench
  • Audit logging — Kubernetes audit logs preserved
  • Access reviews — quarterly review of who has cluster access
  • Namespace isolation — teams can only access their own namespaces

Validation Script

#!/bin/bash
echo "=== EKS Day-2 Readiness Check ==="

echo "--- Logging ---"
kubectl get ds -n amazon-cloudwatch  2>/dev/null || echo "⚠ No CloudWatch agent/Fluent Bit DaemonSet"
kubectl get ds -n logging 2>/dev/null || echo "⚠ No logging DaemonSet in logging namespace"

echo "--- Metrics ---"
kubectl get pods -n monitoring -l app=prometheus 2>/dev/null || echo "⚠ No Prometheus"
kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana 2>/dev/null || echo "⚠ No Grafana"

echo "--- Cost ---"
kubectl get pods -n kubecost 2>/dev/null || echo "⚠ No Kubecost"

echo "--- GitOps ---"
kubectl get pods -n argocd 2>/dev/null || echo "⚠ No ArgoCD"
kubectl get pods -n flux-system 2>/dev/null || echo "⚠ No Flux"

echo "--- PDBs ---"
PDB_COUNT=$(kubectl get pdb -A --no-headers 2>/dev/null | wc -l)
echo "PDB count: $PDB_COUNT"

echo "--- HPA ---"
HPA_COUNT=$(kubectl get hpa -A --no-headers 2>/dev/null | wc -l)
echo "HPA count: $HPA_COUNT"

echo "=== Check Complete ==="

Updated:

You may also enjoy