KCNA — Prometheus and Monitoring
Prometheus and Monitoring
Prometheus is the de facto standard for monitoring in the Kubernetes ecosystem. It’s a CNCF Graduated project and a core topic on the KCNA exam.
What Is Prometheus?
Prometheus is an open-source systems monitoring and alerting toolkit. Originally built at SoundCloud in 2012, it became the second CNCF project to graduate (after Kubernetes).
Key characteristics:
- Pull-based — Prometheus scrapes metrics from targets
- Time-series database — stores metrics with timestamps
- Powerful query language — PromQL for querying and alerting
- Service discovery — automatically finds targets in Kubernetes
- No distributed storage — single-server by default
Architecture
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Application │ │ Kubernetes │ │ Node │
│ /metrics │ │ Objects │ │ Exporter │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
│ scrape │ scrape │ scrape
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ Prometheus Server │
│ ┌─────────────┐ ┌──────────┐ ┌────────────────┐ │
│ │ Retrieval │ │ TSDB │ │ HTTP Server │ │
│ │ (scraper) │→ │ (storage)│← │ (PromQL API) │ │
│ └─────────────┘ └──────────┘ └────────────────┘ │
│ │
│ ┌──────────────────────┐ │
│ │ Service Discovery │ │
│ │ (K8s, DNS, file) │ │
│ └──────────────────────┘ │
└──────────────────┬──────────────────────────────────┘
│
┌──────────┼──────────┐
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Alertmanager │ │ Grafana │
│ (alerts) │ │ (dashboards) │
└──────────────┘ └──────────────┘
Components
| Component | Purpose |
|---|---|
| Prometheus Server | Scrapes, stores, and queries metrics |
| Service Discovery | Automatically finds targets (pods, services, nodes) |
| TSDB | Time-series database for metric storage |
| Alertmanager | Routes and manages alerts (deduplication, grouping, silencing) |
| Exporters | Expose metrics from third-party systems (Node Exporter, MySQL Exporter) |
| Pushgateway | For short-lived jobs that can’t be scraped (push instead of pull) |
Pull vs Push
Prometheus uses a pull model — it actively scrapes metrics from targets at regular intervals.
| Model | How It Works | Used By |
|---|---|---|
| Pull | Monitoring system fetches metrics from targets | Prometheus |
| Push | Applications send metrics to the monitoring system | Datadog, StatsD, Graphite |
Pull model advantages:
- Prometheus controls scrape frequency
- Easier to detect target failures (if scrape fails, the target is likely down)
- No client-side configuration needed for metric delivery
Metrics Format
Prometheus metrics are exposed as plain text at /metrics endpoints:
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET", status="200"} 1234
http_requests_total{method="POST", status="201"} 567
http_requests_total{method="GET", status="500"} 12
# HELP memory_usage_bytes Current memory usage
# TYPE memory_usage_bytes gauge
memory_usage_bytes{pod="web-abc123"} 104857600
Metric Types
| Type | Description | Example |
|---|---|---|
| Counter | Monotonically increasing value | http_requests_total, errors_total |
| Gauge | Value that goes up and down | temperature, memory_usage_bytes |
| Histogram | Observations counted in configurable buckets | request_duration_seconds_bucket |
| Summary | Pre-calculated quantiles over a sliding window | request_latency{quantile="0.99"} |
PromQL Basics
PromQL (Prometheus Query Language) is used to query metrics.
| Query | Description |
|---|---|
http_requests_total |
All HTTP request counters |
http_requests_total{status="500"} |
Only 500 error requests |
rate(http_requests_total[5m]) |
Per-second rate over 5 minutes |
sum(rate(http_requests_total[5m])) |
Total request rate across all instances |
histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m])) |
p99 latency |
You don’t need to write complex PromQL for the KCNA exam, but understand what rate(), sum(), and label filtering do.
Grafana
Grafana is the standard visualization tool paired with Prometheus.
| Feature | Description |
|---|---|
| Dashboards | Visual panels for metrics — graphs, gauges, tables |
| Data Sources | Connects to Prometheus, Loki, Elasticsearch, and many more |
| Alerting | Alert rules based on Grafana queries |
| Community Dashboards | Pre-built dashboards for Kubernetes, Node Exporter, etc. |
The standard combination: Prometheus + Grafana for monitoring and visualization.
Prometheus in Kubernetes
kube-prometheus-stack
The most common way to deploy Prometheus in Kubernetes is via the kube-prometheus-stack Helm chart, which includes:
| Component | Purpose |
|---|---|
| Prometheus Operator | Manages Prometheus instances via CRDs |
| Prometheus | Metric collection and storage |
| Alertmanager | Alert routing and management |
| Grafana | Dashboards and visualization |
| Node Exporter | Host-level metrics (CPU, memory, disk) |
| kube-state-metrics | Kubernetes object state metrics |
ServiceMonitor
Prometheus Operator uses ServiceMonitor CRDs to configure what to scrape:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: web-app
spec:
selector:
matchLabels:
app: web-app
endpoints:
- port: metrics
interval: 15s
KCNA Exam Tips
- Prometheus is pull-based — it scrapes targets, not the other way around
- Know the four metric types: Counter, Gauge, Histogram, Summary
- The Prometheus + Grafana stack is the standard for K8s monitoring
- Alertmanager handles alert routing, deduplication, and silencing
- Pushgateway is for short-lived jobs that can’t be scraped
- Prometheus is a CNCF Graduated project (2nd after Kubernetes)
What’s Next?
Explore logging in Kubernetes — how to collect, aggregate, and analyze logs from containers and clusters.
