Amazon EKS — Observability — Practice Exercises

3 minute read

Chapter 12 — Practice Exercises

Exercise 1: Monitoring Stack

Question: Design the observability stack for a 50-node EKS cluster. Include metrics, logging, tracing, and alerting. Justify open-source vs managed for each.

Show Solution
Layer Tool Why
Metrics Amazon Managed Prometheus (AMP) No Prometheus scaling headaches, 150-day retention
Dashboards Amazon Managed Grafana (AMG) SSO integration, managed
Logging Fluent Bit → CloudWatch Logs AWS-native, log insights for queries
Tracing ADOT → X-Ray AWS-native, easy setup
Alerting Prometheus AlertManager + SNS Flexible rules + PagerDuty/Slack integration

For a 50-node cluster, self-managing Prometheus would require dedicated storage, HA, and maintenance. Managed services are worth the cost.

Exercise 2: Log Architecture

Question: Your cluster generates 50 GB/day of logs. CloudWatch Logs costs $0.50/GB ingestion. That’s $750/month. How would you reduce costs while keeping logs searchable?

Show Solution

Tiered logging architecture:

  1. Filter at source — Only log WARN and ERROR in production: 
    [FILTER]
 Name    grep
 Match   *
 Regex   level (WARN|ERROR)

    Reduces volume by ~80% → 10 GB/day

  2. Hot tier — CloudWatch Logs (7-day retention): 
    10 GB/day × $0.50/GB × 30 = $150/month
  3. Warm tier — S3 (90-day retention): 
    50 GB/day × $0.023/GB × 30 = $34.50/month
Use S3 Select or Athena for querying
  4. Total: ~$185/month (vs $750) — 75% savings

Exercise 3: Alert Design

Question: Write 5 critical alerts for a production EKS cluster that a SRE team should have on day one.

Show Solution 
groups:
  - name: critical-eks-alerts
    rules:
      # 1. Node down
      - alert: NodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 5m
        labels: { severity: critical }
        annotations: { summary: "Node not ready for 5 minutes" }

      # 2. Pod crash loop
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 3
        labels: { severity: critical }
        annotations: { summary: "Pod restarted 3+ times in 15 min" }

      # 3. Persistent volume almost full
      - alert: PVCAlmostFull
        expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
        for: 5m
        labels: { severity: warning }
        annotations: { summary: "PVC > 85% full" }

      # 4. HPA maxed out
      - alert: HPAMaxedOut
        expr: kube_horizontalpodautoscaler_status_current_replicas == kube_horizontalpodautoscaler_spec_max_replicas
        for: 15m
        labels: { severity: warning }
        annotations: { summary: "HPA at max replicas for 15 min" }

      # 5. API server latency
      - alert: APIServerHighLatency
        expr: histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels: { severity: critical }
        annotations: { summary: "API server P99 latency > 1 second" }

Exercise 4: CloudWatch vs Prometheus

Question: When would you choose CloudWatch Container Insights over Prometheus + Grafana?

Show Solution

Choose CloudWatch Container Insights when:

  • Small team (< 5 engineers) — no time to manage Prometheus
  • Already invested in AWS monitoring tools
  • Need quick setup (minutes vs hours)
  • < 20 nodes (cost is reasonable)
  • Want AWS-native integration (IAM, SNS, Lambda)

Choose Prometheus + Grafana when:

  • Large cluster (50+ nodes) — CloudWatch costs grow fast
  • Need custom metrics and queries (PromQL is more powerful)
  • Multi-cloud (Prometheus works everywhere)
  • Want rich dashboards (Grafana has more options)
  • Already have Kubernetes expertise

Exercise 5: Debug Slow Application

Question: Users report slow page loads. You have Prometheus metrics, CloudWatch logs, and X-Ray traces. Describe your investigation steps.

Show Solution 
Step 1: Check X-Ray traces for the slow endpoint
→ Find which service adds the most latency
→ Example: Order Service takes 2 seconds

Step 2: Check Prometheus metrics for that service
→ CPU/memory utilization
→ Request rate and error rate
→ HPA status (is it scaling?)

Step 3: Check CloudWatch logs for errors
→ Filter: level=ERROR, service=order-service
→ Look for timeout errors, connection failures

Step 4: Drill down
→ If high CPU → check for inefficient queries
→ If high memory → check for memory leaks
→ If errors → check downstream dependencies
→ If timeouts → check database/cache performance

Step 5: Verify fix
→ Deploy fix → check traces → confirm latency reduced

What’s Next?

Deploy and ship: Chapter 13 — Deployments & CI/CD.

You may also enjoy