Monitoring and Logging

5 minute read

Running applications in Kubernetes is only half the battle. You need observability — the ability to understand what’s happening inside your cluster. This chapter covers monitoring (metrics), logging (events and output), and alerting to keep your applications healthy and performant.

The Three Pillars of Observability

Pillar What It Captures Tools
Metrics Numeric measurements over time (CPU, memory, request rate) Prometheus, Grafana, Datadog
Logs Event records from applications and system components EFK/ELK stack, Loki, CloudWatch
Traces Request flow across microservices Jaeger, Zipkin, OpenTelemetry

Built-in Monitoring: kubectl and Metrics Server

Metrics Server

The Metrics Server collects resource metrics (CPU, memory) from kubelets and exposes them through the Kubernetes API.

# Install on Minikube
minikube addons enable metrics-server

# Verify
kubectl get pods -n kube-system | grep metrics-server

# Wait a minute for metrics to collect, then:
kubectl top nodes
kubectl top pods
kubectl top pods --all-namespaces --sort-by=memory

kubectl Monitoring Commands

# Cluster overview
kubectl cluster-info
kubectl get nodes
kubectl get componentstatuses     # Deprecated but useful

# Resource usage
kubectl top nodes
kubectl top pods -n default

# Events (recent cluster activity)
kubectl get events --sort-by='.lastTimestamp'
kubectl get events -n kube-system

# Pod health
kubectl get pods -o wide
kubectl describe pod <pod-name>

Prometheus — Metrics Collection

Prometheus is the de facto standard for Kubernetes monitoring. It scrapes metrics from targets at regular intervals and stores them as time series data.

Architecture

┌─────────────┐    scrape     ┌──────────────┐
│ Application │◄──────────────│  Prometheus   │
│ /metrics    │               │              │
└─────────────┘               │  ┌────────┐  │
                              │  │ TSDB   │  │
┌─────────────┐    scrape     │  └────────┘  │
│ Node        │◄──────────────│              │
│ Exporter    │               └──────┬───────┘
└─────────────┘                      │
                                     │ query
┌─────────────┐    alert       ┌─────▼───────┐
│ AlertManager│◄───────────────│  Grafana    │
│             │                │ (dashboard) │
└─────────────┘                └─────────────┘

Install Prometheus Stack with Helm

The kube-prometheus-stack chart installs Prometheus, Grafana, Alertmanager, and exporters in one command:

# Add the Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install the full monitoring stack
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=admin123

# Check the Pods
kubectl get pods -n monitoring

This installs:

  • Prometheus — Metrics collection and storage
  • Grafana — Dashboards and visualization
  • Alertmanager — Alert routing and notification
  • Node Exporter — Hardware and OS metrics
  • kube-state-metrics — Kubernetes object state metrics

Access Grafana Dashboard

# Port-forward Grafana
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# Visit http://localhost:3000
# Username: admin
# Password: admin123 (or what you set)

Grafana comes with pre-built Kubernetes dashboards:

  • Kubernetes / Compute Resources / Cluster — Cluster-wide CPU, memory, network
  • Kubernetes / Compute Resources / Namespace — Per-namespace breakdown
  • Kubernetes / Compute Resources / Pod — Individual Pod metrics

Exposing Application Metrics

Your applications can expose metrics that Prometheus scrapes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"    # Tell Prometheus to scrape this Pod
        prometheus.io/port: "8080"       # Metrics port
        prometheus.io/path: "/metrics"   # Metrics endpoint
    spec:
      containers:
      - name: app
        image: my-app:1.0
        ports:
        - containerPort: 8080

PromQL — Querying Metrics

Prometheus uses PromQL (Prometheus Query Language) to query metrics:

# Total CPU usage by Pod
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)

# Memory usage by namespace
sum(container_memory_usage_bytes) by (namespace)

# HTTP request rate
rate(http_requests_total[5m])

# 95th percentile response time
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Pod restart count
kube_pod_container_status_restarts_total

Logging

kubectl Logs

The simplest way to view logs:

# View Pod logs
kubectl logs <pod-name>

# Follow logs in real-time
kubectl logs -f <pod-name>

# View logs for a specific container
kubectl logs <pod-name> -c <container-name>

# View previous container's logs (after a crash)
kubectl logs <pod-name> --previous

# View logs with timestamps
kubectl logs <pod-name> --timestamps

# Last 100 lines
kubectl logs <pod-name> --tail=100

# Logs from the last hour
kubectl logs <pod-name> --since=1h

# Logs from all Pods in a Deployment
kubectl logs deployment/my-app

# Logs from all Pods with a label
kubectl logs -l app=nginx --all-containers

Centralized Logging with EFK Stack

For production, you need centralized logging. The EFK stack (Elasticsearch, Fluentd/Fluent Bit, Kibana) is the most popular approach:

┌─────────┐   ┌─────────┐   ┌─────────┐
│  Node 1 │   │  Node 2 │   │  Node 3 │
│┌───────┐│   │┌───────┐│   │┌───────┐│
││Fluent  ││   ││Fluent  ││   ││Fluent  ││
││Bit     ││   ││Bit     ││   ││Bit     ││
│└───┬────┘│   │└───┬────┘│   │└───┬────┘│
└────┼─────┘   └────┼─────┘   └────┼─────┘
     │              │              │
     └──────────────┼──────────────┘
                    ▼
            ┌──────────────┐
            │Elasticsearch │
            │  (storage)   │
            └──────┬───────┘
                   ▼
            ┌──────────────┐
            │   Kibana      │
            │ (dashboards)  │
            └──────────────┘

Components:

  • Fluent Bit / Fluentd — DaemonSet on each node, collects container logs
  • Elasticsearch — Stores and indexes logs
  • Kibana — Web UI for searching and visualizing logs

Alternative: Loki + Grafana

A lightweight alternative to EFK:

# Install Loki stack with Helm
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --set grafana.enabled=true \
  --set loki.persistence.enabled=true

Loki is designed to be cost-effective — it indexes only metadata (labels), not the full log content, making it much cheaper to run than Elasticsearch.

Alerting

Prometheus Alert Rules

Define conditions that trigger alerts:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: monitoring
spec:
  groups:
  - name: application
    rules:
    - alert: HighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High error rate detected"
        description: "Error rate is above 10% for 5 minutes"

    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod is crash-looping"

    - alert: HighMemoryUsage
      expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Container memory usage above 90%"

Alertmanager Configuration

Route alerts to notification channels:

# alertmanager-config.yaml
route:
  receiver: 'slack-notifications'
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty'

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#k8s-alerts'
    send_resolved: true

- name: 'pagerduty'
  pagerduty_configs:
  - service_key: '<your-key>'

Monitoring Checklist for Production

Category What to Monitor
Cluster Node CPU/memory, disk usage, Pod count, API server latency
Nodes CPU, memory, disk pressure, network I/O
Pods Restart count, resource usage vs. limits, OOMKills
Applications Request rate, error rate, latency (RED method)
Storage PV usage, IOPS, throughput
Networking DNS resolution time, Service endpoint health

The RED Method (for Services)

Metric Description
Rate Requests per second
Error rate Percentage of failed requests
Duration Time per request (latency)

The USE Method (for Resources)

Metric Description
Utilization % of resource being used
Saturation How much work is queued
Errors Count of error events

Summary

  • Metrics Server provides basic kubectl top functionality
  • Prometheus + Grafana is the standard monitoring stack for K8s
  • EFK/Loki provides centralized logging
  • Alertmanager routes alerts to notification channels
  • Use RED method for services, USE method for resources
  • Start with kube-prometheus-stack Helm chart for a complete setup

What’s Next?

Things go wrong in production. In the next chapter, we’ll learn systematic Troubleshooting techniques for diagnosing and fixing common Kubernetes issues.

Next: Troubleshooting →

Updated:

You may also enjoy