Amazon EKS — Observability Hands-On Lab

In this lab, you’ll set up a complete observability stack: Prometheus + Grafana for metrics, Fluent Bit for logging, and create alerts for a sample application.

---

Prerequisites

• Running EKS cluster with at least 2 nodes
• kubectl, helm, aws CLI configured
• IRSA permissions for CloudWatch

---

Step 1 — Enable EKS Control Plane Logging

aws eks update-cluster-config \
  --name my-cluster \
  --logging '{
    "clusterLogging": [{
      "types": ["api", "audit", "authenticator"],
      "enabled": true
    }]
  }'

---

Step 2 — Install Prometheus and Grafana

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword="LabPass123" \
  --set prometheus.prometheusSpec.retention=7d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=gp2 \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=20Gi

Verify:

kubectl get pods -n monitoring
# All pods should be Running

---

Step 3 — Deploy a Sample Application

# sample-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: default
  labels:
    app: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80"
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 200m
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  selector:
    app: web-app
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f sample-app.yaml

---

Step 4 — Access Grafana Dashboard

kubectl port-forward svc/kube-prometheus-grafana 3000:80 -n monitoring

Open http://localhost:3000.

Login: admin / LabPass123

Import dashboards:

1. Dashboards → Import → ID: 315 → Load → Select Prometheus data source → Import
2. Dashboards → Import → ID: 6417 → Load → Select Prometheus data source → Import

---

Step 5 — Create a PrometheusRule Alert

# alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: lab-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: lab.rules
      rules:
        - alert: HighPodMemory
          expr: |
            sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)
            / sum(container_spec_memory_limit_bytes{container!=""}) by (namespace, pod)
            > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod memory usage above 80%"

        - alert: PodRestartRate
          expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Pod is restarting"

kubectl apply -f alerts.yaml

# Check alerts in Prometheus
kubectl port-forward svc/kube-prometheus-kube-prome-prometheus 9090:9090 -n monitoring
# Open http://localhost:9090/alerts

---

Step 6 — Install Fluent Bit for Logging

eksctl create iamserviceaccount \
  --cluster my-cluster \
  --namespace kube-system \
  --name fluent-bit-sa \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
  --approve

helm repo add eks https://aws.github.io/eks-charts

helm install aws-for-fluent-bit eks/aws-for-fluent-bit \
  --namespace kube-system \
  --set cloudWatchLogs.enabled=true \
  --set cloudWatchLogs.region=us-east-1 \
  --set cloudWatchLogs.logGroupName="/eks/my-cluster/lab" \
  --set cloudWatchLogs.autoCreateGroup=true \
  --set serviceAccount.create=false \
  --set serviceAccount.name=fluent-bit-sa

Verify logs are flowing:

# Generate some logs
kubectl exec deploy/web-app -- curl -s localhost > /dev/null

# Check CloudWatch
aws logs describe-log-streams \
  --log-group-name /eks/my-cluster/lab \
  --order-by LastEventTime \
  --descending \
  --limit 5

---

Step 7 — Query Metrics

Run PromQL queries in Grafana’s Explore view:

# CPU usage by pod
sum(rate(container_cpu_usage_seconds_total{namespace="default"}[5m])) by (pod)

# Memory usage by pod
sum(container_memory_working_set_bytes{namespace="default"}) by (pod) / 1024 / 1024

# Network traffic
sum(rate(container_network_receive_bytes_total{namespace="default"}[5m])) by (pod) / 1024

---

Cleanup

kubectl delete -f sample-app.yaml
kubectl delete -f alerts.yaml
helm uninstall kube-prometheus -n monitoring
helm uninstall aws-for-fluent-bit -n kube-system
kubectl delete namespace monitoring
eksctl delete iamserviceaccount --cluster my-cluster --namespace kube-system --name fluent-bit-sa
aws logs delete-log-group --log-group-name /eks/my-cluster/lab

---