Amazon EKS — Alerting

Alerting

Monitoring without alerting is just watching things fail. Set up alerts to get notified before users do.

---

Alert Sources

Source	Tool	Best For
CloudWatch Metrics	CloudWatch Alarms	AWS-native metrics
Prometheus Metrics	AlertManager	Custom K8s metrics
Logs	CloudWatch Log Metric Filters	Error patterns

---

CloudWatch Alarms

$ aws cloudwatch put-metric-alarm \
    --alarm-name "high-pod-cpu" \
    --metric-name pod_cpu_utilization \
    --namespace ContainerInsights \
    --dimensions Name=ClusterName,Value=demo \
    --statistic Average \
    --period 300 \
    --threshold 80 \
    --comparison-operator GreaterThanThreshold \
    --evaluation-periods 2 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:eks-alerts

---

Prometheus AlertManager

# Prometheus alert rule
groups:
  - name: eks-alerts
    rules:
      - alert: HighPodCPU
        expr: rate(container_cpu_usage_seconds_total[5m]) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Pod CPU > 80% for 5 minutes"

      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod is crash-looping"

---

Essential Alerts

Alert	Condition	Severity
Node NotReady	Node status NotReady > 5 min	Critical
Pod CrashLoopBackOff	Restart rate > 0	Critical
High CPU	Pod CPU > 80% for 10 min	Warning
PVC near full	Disk usage > 85%	Warning
HPA at max	HPA replicas = maxReplicas	Warning

---

Common Mistakes

Mistake	Impact
Too many alerts	Alert fatigue → ignored alerts
No severity levels	Everything treated as critical
No runbooks	Alerting without remediation steps

---

What’s Next?

Practice: Observability Exercises.