HashiCorp Vault — Operational Procedures

2 minute read

Day-to-day operational runbooks for managing Vault in production — from routine maintenance to incident response.

Daily Operations

Health Check

#!/bin/bash
# daily-health-check.sh

VAULT_ADDR="https://vault.example.com:8200"

# Check seal status
STATUS=$(curl -s "$VAULT_ADDR/v1/sys/health" -o /dev/null -w "%{http_code}")

case $STATUS in
  200) echo "✅ Active, initialized, unsealed" ;;
  429) echo "✅ Standby node" ;;
  472) echo "⚠️ DR secondary" ;;
  473) echo "⚠️ Performance standby" ;;
  501) echo "❌ Not initialized" ;;
  503) echo "❌ SEALED — immediate action required!" ;;
  *)   echo "❌ Unexpected status: $STATUS" ;;
esac

# Check Raft peers
vault operator raft list-peers

# Check lease count (high numbers may indicate a leak)
vault read sys/metrics | grep 'vault.expire.num_leases'

Rotate Credentials

Rotate Vault’s Database Root Password

# Rotate the root credentials Vault uses to connect to the database
vault write -force database/rotate-root/my-postgres-db

# Vault changes the password and only Vault knows the new one
# This prevents anyone from bypassing Vault to access the DB directly

Rotate Encryption Keys

# Rotate the Transit encryption key
vault write -f transit/keys/my-key/rotate

# Set minimum decryption version (removes old key versions)
vault write transit/keys/my-key min_decryption_version=5

# Rewrap existing ciphertexts to latest key version
vault write transit/rewrap/my-key \
    ciphertext="vault:v1:abc..."

Certificate Rotation

# Check certificate expiry
openssl x509 -in /opt/vault/tls/vault-cert.pem -noout -enddate

# If using Vault's PKI for its own certificates:
vault write pki_int/issue/vault-server \
    common_name="vault.example.com" \
    ttl=8760h

# Reload TLS without restart
vault operator reload
# Or send SIGHUP
kill -HUP $(pgrep vault)

Lease Management

# List active leases
vault list sys/leases/lookup/database/creds/readonly/

# Revoke a specific lease
vault lease revoke database/creds/readonly/abc123

# Revoke all leases for a path
vault lease revoke -prefix database/creds/readonly/

# Force revoke (even if backend errors)
vault lease revoke -force database/creds/readonly/abc123

Token Management

# Find and revoke orphaned tokens
vault list auth/token/accessors

# Lookup a token by accessor
vault token lookup -accessor <accessor>

# Revoke by accessor (when you don't have the token)
vault token revoke -accessor <accessor>

# Tidy token store (clean expired tokens)
vault write auth/token/tidy

Emergency Procedures

Seal Vault (Emergency)

# If compromise is suspected — seal immediately
vault operator seal

# This locks ALL secrets until unseal keys are provided
# All active connections are terminated

Generate Emergency Root Token

# Initialize the process
vault operator generate-root -init
# Returns: Nonce and OTP

# Each key holder provides their unseal key
vault operator generate-root \
    -nonce=<nonce>
# Enter unseal key when prompted

# After threshold is met, decode the root token
vault operator generate-root \
    -decode=<encoded-token> \
    -otp=<otp>

# Use the root token, then REVOKE it when done
vault token revoke <root-token>

Remove a Failed Raft Peer

# List peers
vault operator raft list-peers

# Remove the failed node
vault operator raft remove-peer <node-id>

Monitoring Runbook

Key Alerts

Alert Threshold Action
Vault sealed vault_core_unsealed == 0 Unseal or investigate
High latency p99 > 500ms Check storage backend
Lease explosion num_leases > 100000 Investigate lease leaks
Certificate expiring < 30 days Rotate certificates
Audit log failure audit_log_response_failure > 0 Fix audit device
Token creation spike > 2x normal Check for misuse

Scheduled Maintenance

Task Frequency Command/Action
Raft snapshot Every 6h vault operator raft snapshot save
Certificate check Weekly openssl x509 -enddate
Lease count review Daily vault read sys/metrics
Audit log rotation Daily logrotate
Key rotation (Transit) Monthly vault write -f transit/keys/KEY/rotate
Database root rotation Quarterly vault write -force database/rotate-root/NAME
Version upgrade check Monthly Check release notes

Next: Practice Exercises

You may also enjoy