HashiCorp Vault — Audit & Monitoring Exercises

2 minute read

Practice audit logging, telemetry, and monitoring with these exercises.

Exercise 1 — Enable Audit Logging

Task: Enable audit logging and verify requests are recorded.

  1. Enable a file-based audit device
  2. Perform several operations (write a secret, read it, list secrets)
  3. Examine the audit log entries
  4. Identify the HMAC’d values
Show Answer 
# Enable file audit
vault audit enable file file_path=/tmp/vault-audit.log

# Perform operations
vault kv put secret/audit-test username=admin password=s3cret
vault kv get secret/audit-test
vault kv list secret/

# Examine logs
cat /tmp/vault-audit.log | jq '.'

# Count by operation
cat /tmp/vault-audit.log | \
    jq -r 'select(.type=="request") | .request.operation' | \
    sort | uniq -c

# Find HMAC'd values
cat /tmp/vault-audit.log | \
    jq 'select(.type=="response" and .request.path=="secret/data/audit-test") | .response.data.data'
# Values will be hmac-sha256:...

# Verify an HMAC
vault write sys/audit-hash/file input="s3cret"
# Compare output with the HMAC in the audit log

Exercise 2 — Audit Log Analysis

Task: Write jq queries to analyze audit logs.

  1. Find all unique client IP addresses
  2. Count operations per user/token
  3. Find all failed requests
  4. List all paths accessed in the last hour
Show Answer 
# Unique client IPs
cat /tmp/vault-audit.log | \
    jq -r 'select(.type=="request") | .request.remote_address' | \
    sort -u

# Operations per display_name
cat /tmp/vault-audit.log | \
    jq -r 'select(.type=="request") | .auth.display_name' | \
    sort | uniq -c | sort -rn

# Failed requests (errors in response)
cat /tmp/vault-audit.log | \
    jq 'select(.type=="response" and .response.data.error != null) |
    {time: .time, path: .request.path, error: .response.data.error}'

# All paths accessed
cat /tmp/vault-audit.log | \
    jq -r 'select(.type=="request") | .request.path' | \
    sort -u

Exercise 3 — Enable Telemetry

Task: Configure Vault telemetry and query the Prometheus metrics endpoint.

  1. Add telemetry configuration to vault.hcl
  2. Restart Vault
  3. Query the metrics endpoint
  4. Find the request count and latency metrics
Show Answer 
# Add to vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

# Restart vault with new config
# Then query metrics
curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
    "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | head -50

# Find specific metrics
curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
    "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | \
    grep "vault_core_handle_request"

# Check token count
curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
    "$VAULT_ADDR/v1/sys/metrics?format=prometheus" | \
    grep "vault_token_count"

Exercise 4 — Health Check Script

Task: Write a health check script that monitors multiple Vault nodes.

Show Answer 
#!/bin/bash
# vault-monitor.sh

check_vault() {
    local addr=$1
    local code=$(curl -sk -o /dev/null -w "%{http_code}" \
        "${addr}/v1/sys/health?standbyok=true" 2>/dev/null || echo "000")
    
    case $code in
        200) echo "ACTIVE" ;;
        429) echo "STANDBY" ;;
        473) echo "PERF_STANDBY" ;;
        503) echo "SEALED" ;;
        000) echo "UNREACHABLE" ;;
        *)   echo "UNKNOWN($code)" ;;
    esac
}

# Check all nodes
NODES=("http://127.0.0.1:8200")
for node in "${NODES[@]}"; do
    STATUS=$(check_vault "$node")
    echo "$node → $STATUS"
done

Exercise 5 — Alerting Rules

Task: Write Prometheus alerting rules for critical Vault conditions.

Show Answer 
groups:
  - name: vault-critical
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node is sealed"

      - alert: VaultAuditBackendDown
        expr: vault_audit_log_request_failure > 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Audit backend failing — Vault may stop processing requests"

      - alert: VaultLeaderLost
        expr: vault_raft_leader_lastContact > 5
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Raft cluster has lost leader contact"

      - alert: VaultHighRequestLatency
        expr: histogram_quantile(0.99, rate(vault_core_handle_request_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Vault P99 latency exceeds 500ms"

      - alert: VaultExpiredLeases
        expr: vault_expire_num_leases > 50000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Too many active leases — check for lease exhaustion"

What’s Next?

With monitoring in place, let’s explore Advanced Features — namespaces, control groups, response wrapping, and more.

Next Chapter: Advanced Features

You may also enjoy