How to Monitor Kubernetes Clusters: Prometheus + Grafana Step-by-Step

Posted: April 20, 2026 | Last updated: April 20, 2026 | 5 minute read

How to Monitor Kubernetes with Prometheus and Grafana

Monitoring a Kubernetes cluster means tracking three layers: the cluster itself (nodes, pods, resource usage), the applications running on it (request latency, error rates, throughput), and the infrastructure underneath (disk, network, CPU). Prometheus collects the metrics. Grafana visualises them. Alertmanager sends alerts when things go wrong. This guide sets up all three from scratch.

Architecture

┌─────────────────────────────────────────────────┐
│                  Kubernetes Cluster              │
│                                                  │
│  ┌──────────┐   ┌──────────┐   ┌──────────┐    │
│  │ App Pod  │   │ App Pod  │   │ App Pod  │    │
│  │ /metrics │   │ /metrics │   │ /metrics │    │
│  └────┬─────┘   └────┬─────┘   └────┬─────┘    │
│       │              │              │           │
│       └──────────────┼──────────────┘           │
│                      │                           │
│              ┌───────▼────────┐                  │
│              │   Prometheus   │ ← scrapes /metrics│
│              │   (TSDB)      │                   │
│              └───────┬────────┘                  │
│                      │                           │
│         ┌────────────┼────────────┐              │
│         │            │            │              │
│  ┌──────▼──────┐ ┌───▼────┐ ┌────▼─────┐       │
│  │  Grafana    │ │ Alert  │ │ PagerDuty│       │
│  │ (Dashboards)│ │Manager │ │ Slack    │       │
│  └─────────────┘ └────────┘ └──────────┘       │
└─────────────────────────────────────────────────┘

Step 1: Install with Helm (kube-prometheus-stack)

The easiest way to get Prometheus, Grafana, Alertmanager, and pre-built dashboards:

# Add the Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create namespace
kubectl create namespace monitoring

# Install the full stack
helm install kube-prometheus prometheus-community/kube-prometheus-stack   --namespace monitoring   --set grafana.adminPassword=changeme   --set prometheus.prometheusSpec.retention=30d   --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

This installs:

Prometheus — metrics collection and storage
Grafana — dashboards and visualisation
Alertmanager — alert routing and notification
Node Exporter — host-level metrics (CPU, memory, disk, network)
kube-state-metrics — Kubernetes object metrics (Deployments, Pods, Nodes)
Pre-built dashboards — 20+ Grafana dashboards for cluster and workload monitoring

Step 2: Access Grafana

# Port-forward Grafana
kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80

# Open http://localhost:3000
# Login: admin / changeme

Pre-built dashboards are already available:

Kubernetes / Compute Resources / Cluster — cluster-wide CPU, memory, network
Kubernetes / Compute Resources / Namespace (Pods) — per-pod resource usage
Node Exporter / Nodes — host-level metrics per node

Step 3: Understand Key Metrics

Cluster Health

# Number of ready nodes
sum(kube_node_status_condition{condition="Ready",status="true"})

# Total CPU capacity vs. usage
sum(machine_cpu_cores)                          # Total capacity
sum(rate(container_cpu_usage_seconds_total[5m])) # Current usage

# Memory usage percentage
sum(container_memory_working_set_bytes) / sum(machine_memory_bytes) * 100

Pod Health

# Pods not in Running state
sum by (namespace, pod) (kube_pod_status_phase{phase!="Running",phase!="Succeeded"})

# Container restarts (last hour)
sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total[1h]))

# OOMKilled containers
sum by (namespace, pod) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"})

Application Metrics

If your application exposes a /metrics endpoint (e.g., using the Prometheus client library):

# Request rate (requests per second)
sum(rate(http_requests_total[5m])) by (service)

# Error rate (5xx responses)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

# 99th percentile latency
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Step 4: Create Custom Dashboards

RED Method Dashboard (Rate, Errors, Duration)

In Grafana, create a new dashboard with three panels:

Panel 1: Request Rate

sum(rate(http_requests_total[5m])) by (service)

Panel 2: Error Rate (%)

sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
* 100

Panel 3: Latency (p99)

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Step 5: Configure Alerts

Alert Rules

# alert-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: application
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate on {{ $labels.service }}"
            description: "Error rate is {{ $value | humanizePercentage }} for {{ $labels.service }}"

        - alert: HighLatency
          expr: |
            histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
            > 1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High latency on {{ $labels.service }}"
            description: "p99 latency is {{ $value }}s for {{ $labels.service }}"

    - name: kubernetes
      rules:
        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

        - alert: NodeNotReady
          expr: kube_node_status_condition{condition="Ready",status="true"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} is not ready"

kubectl apply -f alert-rules.yaml

Alertmanager Config (Slack)

# alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-kube-prometheus-alertmanager
  namespace: monitoring
stringData:
  alertmanager.yaml: |
    global:
      slack_api_url: 'https://hooks.slack.com/services/T00/B00/xxx'
    route:
      receiver: 'slack-critical'
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - match:
            severity: critical
          receiver: 'slack-critical'
        - match:
            severity: warning
          receiver: 'slack-warning'
    receivers:
      - name: 'slack-critical'
        slack_configs:
          - channel: '#alerts-critical'
            title: '{{ .GroupLabels.alertname }}'
            text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
            send_resolved: true
      - name: 'slack-warning'
        slack_configs:
          - channel: '#alerts-warning'
            send_resolved: true

Step 6: Add Application Metrics

Instrument Your App (Node.js Example)

const client = require('prom-client');

// Default metrics (CPU, memory, event loop)
client.collectDefaultMetrics();

// Custom metrics
const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'path', 'status'],
});

const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'path'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

// Middleware
app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer({ method: req.method, path: req.path });
  res.on('finish', () => {
    httpRequestsTotal.inc({ method: req.method, path: req.path, status: res.statusCode });
    end();
  });
  next();
});

// Expose /metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

ServiceMonitor (Prometheus Auto-Discovery)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-api
  namespace: production
  labels:
    release: kube-prometheus
spec:
  selector:
    matchLabels:
      app: my-api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

Prometheus will automatically discover and scrape any service matching the selector.

Essential PromQL Cheat Sheet

Query	Description
`up`	1 if target is reachable, 0 if down
`rate(metric[5m])`	Per-second rate of increase over 5 minutes
`increase(metric[1h])`	Total increase over 1 hour
`sum by (label) (metric)`	Aggregate by label
`histogram_quantile(0.99, ...)`	99th percentile from histogram
`absent(metric)`	Returns 1 if metric does not exist (useful for alerts)
`predict_linear(metric[1h], 3600)`	Predict value 1 hour from now
`topk(5, metric)`	Top 5 values
`count by (namespace) (kube_pod_info)`	Count pods per namespace

Conclusion

Kubernetes monitoring with Prometheus and Grafana gives you full visibility into cluster health, application performance, and infrastructure metrics. The kube-prometheus-stack Helm chart gets you running in 5 minutes with pre-built dashboards and alert rules. From there, add ServiceMonitors for your applications, write PromQL-based alert rules, and connect Alertmanager to Slack or PagerDuty. The investment is small; the payoff — catching issues before your users do — is enormous.

Related posts:

Share on

Twitter Facebook LinkedIn

Sushil Verma