HashiCorp Vault — Health Checks & Alerting

Monitor Vault health with automated checks and set up alerts for critical conditions.

---

Health Check Endpoint

# Default health check
curl -s "$VAULT_ADDR/v1/sys/health" | jq

# Response:
{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "performance_standby": false,
  "replication_performance_mode": "disabled",
  "replication_dr_mode": "disabled",
  "server_time_utc": 1705300000,
  "version": "1.15.4",
  "cluster_name": "vault-prod",
  "cluster_id": "abc-123-def"
}

HTTP Status Codes

Code	Meaning
200	Active, initialized, unsealed
429	Standby node
472	DR secondary
473	Performance standby
501	Not initialized
503	Sealed

---

Prometheus Alerting Rules

# vault-alerts.yml
groups:
  - name: vault
    rules:
      # Vault is sealed
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault is sealed"
          description: "Vault node has been sealed for more than 1 minute."

      # Vault is not initialized
      - alert: VaultNotInitialized
        expr: vault_core_active == 0 and vault_core_unsealed == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Vault is not initialized"

      # High request latency
      - alert: VaultHighLatency
        expr: histogram_quantile(0.99, rate(vault_core_handle_request_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Vault P99 latency > 1s"
          description: "P99 request latency has exceeded 1 second."

      # No leader elected
      - alert: VaultNoLeader
        expr: vault_raft_leader_lastContact > 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Vault Raft has no leader contact"

      # High number of pending leases
      - alert: VaultHighLeaseCount
        expr: vault_expire_num_leases > 10000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Vault has too many active leases"

      # Audit log failures
      - alert: VaultAuditFailure
        expr: vault_audit_log_request_failure > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault audit device is failing"
          description: "Audit logs are failing. Vault will block requests if all audit devices fail."

      # Token creation spike
      - alert: VaultTokenCreationSpike
        expr: rate(vault_token_create_count[5m]) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Unusual token creation rate"

---

Alertmanager Configuration

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: 'vault-team'
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: 'vault-pager'

receivers:
  - name: 'vault-team'
    slack_configs:
      - channel: '#vault-alerts'
        api_url: 'https://hooks.slack.com/services/...'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

  - name: 'vault-pager'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

---

Health Check Script

#!/bin/bash
# vault-health.sh

VAULT_NODES=("https://vault-1:8200" "https://vault-2:8200" "https://vault-3:8200")
HEALTHY=0
TOTAL=${#VAULT_NODES[@]}

for node in "${VAULT_NODES[@]}"; do
    HTTP_CODE=$(curl -sk -o /dev/null -w "%{http_code}" "${node}/v1/sys/health?standbyok=true" 2>/dev/null)
    
    case $HTTP_CODE in
        200|473)
            echo "✅ $node: healthy"
            ((HEALTHY++))
            ;;
        429)
            echo "✅ $node: standby (healthy)"
            ((HEALTHY++))
            ;;
        503)
            echo "🔒 $node: SEALED"
            ;;
        *)
            echo "❌ $node: UNREACHABLE ($HTTP_CODE)"
            ;;
    esac
done

echo ""
echo "Healthy: ${HEALTHY}/${TOTAL}"

if [ $HEALTHY -lt 2 ]; then
    echo "⚠️  WARNING: Quorum at risk!"
    exit 1
fi

---

Kubernetes Probes

containers:
  - name: vault
    livenessProbe:
      httpGet:
        path: /v1/sys/health?standbyok=true&sealedcode=200
        port: 8200
        scheme: HTTPS
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /v1/sys/health?standbyok=true
        port: 8200
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 5

---

Quick Reference

Alert	Condition
Vault Sealed	vault_core_unsealed == 0
High Latency	P99 > 1 second for 5 min
No Leader	leader_lastContact > 10s
Audit Failure	audit_log_request_failure > 0
High Leases	num_leases > 10000

---

Next: Practice Exercises