Amazon EKS — Prometheus and Grafana

Prometheus is the de-facto standard for Kubernetes metrics. Grafana provides dashboards. Together they give you deep visibility into cluster and application performance — free and open-source.

---

kube-prometheus-stack

The kube-prometheus-stack Helm chart installs everything in one shot:

• Prometheus server
• Alertmanager
• Grafana
• Node exporter
• kube-state-metrics
• Pre-built dashboards and alerts

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword="admin123" \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=gp3 \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set prometheus.prometheusSpec.retention=15d

Verify

kubectl get pods -n monitoring
kubectl get svc -n monitoring

Access Grafana

kubectl port-forward svc/kube-prometheus-grafana 3000:80 -n monitoring
# Open http://localhost:3000 — admin / admin123

---

ServiceMonitor and PodMonitor

Prometheus discovers scrape targets via CRDs:

ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: my-app-pods
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: my-app
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Key: The labels.release must match the Helm release name — otherwise Prometheus won’t pick up the monitor.

---

Grafana Dashboards for EKS

Import these community dashboards:

Dashboard ID	Name	Description
315	Kubernetes cluster monitoring	Cluster-wide view
6417	Kubernetes Pods	Pod-level CPU, memory, network
13770	Node Exporter Full	Detailed node metrics
14584	CoreDNS	DNS query performance
15760	Kubernetes / Networking	Network traffic per namespace

Import via Grafana UI: Dashboards → Import → Enter ID.

---

Alertmanager

Configure alerts for EKS-specific scenarios:

PrometheusRule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: eks-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: eks.rules
      rules:
        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 5 > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

        - alert: HighMemoryUsage
          expr: container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Container {{ $labels.container }} in {{ $labels.pod }} at 90%+ memory"

        - alert: NodeNotReady
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} is NotReady"

Alertmanager Config (Slack)

# values.yaml for Helm
alertmanager:
  config:
    route:
      receiver: 'slack'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    receivers:
      - name: 'slack'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/T00000/B00000/XXXX'
            channel: '#eks-alerts'
            title: '{{ .GroupLabels.alertname }}'
            text: '{{ .CommonAnnotations.summary }}'

---

Storage for Prometheus

EBS (Default)

Good for single-cluster, short-term retention:

prometheus:
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          resources:
            requests:
              storage: 100Gi
    retention: 30d

Thanos for Long-Term Storage

For multi-cluster or long-term retention, add Thanos sidecar:

prometheus:
  prometheusSpec:
    thanos:
      objectStorageConfig:
        existingSecret:
          name: thanos-objstore
          key: objstore.yml

Thanos uploads blocks to S3 for unlimited retention at low cost.

---

Useful PromQL Queries

# Cluster CPU utilization
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum(machine_cpu_cores) * 100

# Memory utilization by namespace
sum(container_memory_working_set_bytes{container!=""}) by (namespace) / 1024 / 1024 / 1024

# Pod restart rate (last hour)
sum(increase(kube_pod_container_status_restarts_total[1h])) by (namespace, pod)

# Top 10 CPU-consuming pods
topk(10, sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace, pod))

# Persistent volume usage
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100

---

Best Practices

1. Use persistent storage for Prometheus — emptyDir loses data on pod restart
2. Set retention period based on needs — 15-30 days for most; use Thanos for longer
3. Use ServiceMonitor/PodMonitor — don’t manually configure scrape targets
4. Import community dashboards — don’t build from scratch
5. Configure alerting from day one — Prometheus without alerts is just a metrics database
6. Right-size Prometheus resources — large clusters need significant CPU/memory for Prometheus

---