HashiCorp Vault — Prometheus & Grafana

Set up Prometheus to scrape Vault metrics and build Grafana dashboards for operational visibility.

---

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: 'http'
    # If authentication is required:
    # bearer_token: 'hvs.your-metrics-token'
    static_configs:
      - targets:
          - 'vault-1.internal:8200'
          - 'vault-2.internal:8200'
          - 'vault-3.internal:8200'
        labels:
          cluster: 'vault-prod'
    scrape_interval: 30s

Kubernetes ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vault
  namespace: vault
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
  endpoints:
    - port: http
      path: /v1/sys/metrics
      params:
        format: ["prometheus"]
      interval: 30s

---

Essential PromQL Queries

Request Rate

# Requests per second
rate(vault_core_handle_request_count[5m])

# By mount type
sum by (mount_type) (rate(vault_core_handle_request_count[5m]))

Request Latency

# P50 latency
histogram_quantile(0.5, rate(vault_core_handle_request_bucket[5m]))

# P99 latency
histogram_quantile(0.99, rate(vault_core_handle_request_bucket[5m]))

Token Activity

# Token creation rate
rate(vault_token_create_count[5m])

# Active token count
vault_token_count

Raft Health

# Raft commit time (should be low)
vault_raft_commitTime

# Time since last leader contact (follower health)
vault_raft_leader_lastContact

# Applied index difference (replication lag)
vault_raft_applied_index

Seal Status

# 0 = unsealed, 1 = sealed
vault_core_unsealed

---

Grafana Dashboard

Import or create a dashboard with these panels:

Row 1 — Cluster Status

Panel	Query	Type
Seal Status	vault_core_unsealed	Stat
HA Mode	vault_core_active	Stat
Uptime	vault_uptime	Stat
Token Count	vault_token_count	Stat

Row 2 — Request Performance

Panel	Query	Type
Request Rate	rate(vault_core_handle_request_count[5m])	Graph
P50 Latency	histogram_quantile(0.5, ...)	Graph
P99 Latency	histogram_quantile(0.99, ...)	Graph
Error Rate	rate(vault_core_handle_request_error_count[5m])	Graph

Row 3 — Raft Storage

Panel	Query	Type
Commit Time	vault_raft_commitTime	Graph
Leader Contact	vault_raft_leader_lastContact	Graph
FSM Apply Time	vault_raft_fsm_apply	Graph
Log Size	vault_raft_applied_index	Stat

Row 4 — Auth & Secrets

Panel	Query	Type
Login Rate	rate(vault_core_handle_login_request_count[5m])	Graph
Secret Read Rate	rate(vault_secret_kv_count[5m])	Graph
Lease Count	vault_expire_num_leases	Stat
Cert Issue Rate	rate(vault_pki_sign_count[5m])	Graph

---

Pre-Built Dashboards

Import community dashboards from Grafana.com:

Dashboard ID	Name	Description
12904	Vault Overview	General cluster monitoring
10991	Vault Telemetry	Detailed metrics

Grafana → Dashboards → Import → Enter ID → Load

---

Docker Compose Setup

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  grafana-data:

---

Quick Reference

Task	How
Scrape Vault metrics	GET /v1/sys/metrics?format=prometheus
Request rate	rate(vault_core_handle_request_count[5m])
P99 latency	histogram_quantile(0.99, rate(..._bucket[5m]))
Raft health	vault_raft_leader_lastContact
Import dashboard	Grafana → Import → Dashboard ID

---

Next: Health Checks & Alerting