Amazon EKS — Project 3 — Monitoring Stack

1 minute read

Project 3 — Production Monitoring Stack

Deploy a complete observability stack: Prometheus, Grafana, Fluent Bit, and alerting.

Architecture

Monitoring Stack
┌────────────────────────────────────────────────────┐
│                                                    │
│  Cluster                                           │
│  ┌──────────┐  ┌──────────┐  ┌──────────────┐    │
│  │ App Pods │  │ System   │  │ kube-state-  │    │
│  │ /metrics │  │ Pods     │  │ metrics      │    │
│  └────┬─────┘  └────┬─────┘  └──────┬───────┘    │
│       │              │               │             │
│       └──────────────┼───────────────┘             │
│                      │ scrape                      │
│              ┌───────▼────────┐                    │
│              │  Prometheus    │                    │
│              │  (metrics DB)  │                    │
│              └───┬────────┬──┘                    │
│                  │        │                        │
│          ┌───────▼──┐  ┌──▼──────────┐            │
│          │ Grafana  │  │ AlertManager│            │
│          │ (dashboards)│ (notifications)│           │
│          └──────────┘  └─────┬───────┘            │
│                              │                     │
│                        ┌─────▼─────┐               │
│                        │ Slack/    │               │
│                        │ PagerDuty │               │
│                        └───────────┘               │
│                                                    │
│  Fluent Bit (DaemonSet) ──▶ CloudWatch Logs       │
│                                                    │
└────────────────────────────────────────────────────┘

Step 1: Install Prometheus Stack

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

$ helm install monitoring prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace \
    --set grafana.adminPassword=secure-password \
    --set prometheus.prometheusSpec.retention=30d \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

Step 2: Configure Alerting

# custom-alerts.yaml
additionalPrometheusRules:
  - name: eks-custom-alerts
    groups:
      - name: pod-alerts
        rules:
          - alert: HighErrorRate
            expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
            for: 5m
            labels:
              severity: critical
            annotations:
              summary: "Error rate > 5%"

Step 3: Install Fluent Bit

$ helm repo add fluent https://fluent.github.io/helm-charts
$ helm install fluent-bit fluent/fluent-bit \
    --namespace monitoring \
    --set config.outputs="[OUTPUT]\n    Name cloudwatch_logs\n    Match *\n    region us-east-1\n    log_group_name /eks/demo\n    auto_create_group true"

Step 4: Import Grafana Dashboards

Access Grafana:

$ kubectl port-forward svc/monitoring-grafana -n monitoring 3000:80

Import these dashboard IDs:

  • 315 — Kubernetes cluster overview
  • 1860 — Node Exporter (node metrics)
  • 6336 — Kubernetes pods

What You Practiced

  • Prometheus metrics collection and retention
  • Grafana dashboards for cluster visibility
  • AlertManager for incident notification
  • Fluent Bit for centralized logging
  • Persistent storage for metrics data

What’s Next?

Make it resilient: Project 4 — HA Cluster.

You may also enjoy