HashiCorp Vault — HA Exercises

3 minute read

Practice high availability concepts with these hands-on exercises.

Exercise 1 — 3-Node Raft Cluster

Task: Set up a 3-node Vault cluster using Raft storage on a single machine (different ports).

Show Answer 
# Create data directories
mkdir -p /tmp/vault-{1,2,3}/data

# Node 1 config
cat > /tmp/vault-1/config.hcl <<EOF
storage "raft" {
  path    = "/tmp/vault-1/data"
  node_id = "node1"
}
listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = true
}
api_addr     = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"
disable_mlock = true
EOF

# Node 2 config
cat > /tmp/vault-2/config.hcl <<EOF
storage "raft" {
  path    = "/tmp/vault-2/data"
  node_id = "node2"
}
listener "tcp" {
  address     = "127.0.0.1:8210"
  tls_disable = true
}
api_addr     = "http://127.0.0.1:8210"
cluster_addr = "http://127.0.0.1:8211"
disable_mlock = true
EOF

# Node 3 config
cat > /tmp/vault-3/config.hcl <<EOF
storage "raft" {
  path    = "/tmp/vault-3/data"
  node_id = "node3"
}
listener "tcp" {
  address     = "127.0.0.1:8220"
  tls_disable = true
}
api_addr     = "http://127.0.0.1:8220"
cluster_addr = "http://127.0.0.1:8221"
disable_mlock = true
EOF

# Start all 3 nodes
vault server -config=/tmp/vault-1/config.hcl &
vault server -config=/tmp/vault-2/config.hcl &
vault server -config=/tmp/vault-3/config.hcl &

# Initialize node 1
export VAULT_ADDR=http://127.0.0.1:8200
vault operator init -key-shares=1 -key-threshold=1 -format=json > /tmp/init.json
UNSEAL_KEY=$(jq -r '.unseal_keys_b64[0]' /tmp/init.json)
ROOT_TOKEN=$(jq -r '.root_token' /tmp/init.json)

vault operator unseal $UNSEAL_KEY
vault login $ROOT_TOKEN

# Join node 2
VAULT_ADDR=http://127.0.0.1:8210 vault operator raft join http://127.0.0.1:8200
VAULT_ADDR=http://127.0.0.1:8210 vault operator unseal $UNSEAL_KEY

# Join node 3
VAULT_ADDR=http://127.0.0.1:8220 vault operator raft join http://127.0.0.1:8200
VAULT_ADDR=http://127.0.0.1:8220 vault operator unseal $UNSEAL_KEY

# Verify cluster
vault operator raft list-peers

Exercise 2 — Leader Failover

Task: Using the 3-node cluster from Exercise 1, simulate a leader failure.

  1. Identify the current leader
  2. Kill the leader process
  3. Verify a new leader is elected
  4. Confirm data is still accessible
Show Answer 
# Identify leader
vault operator raft list-peers
# Node that shows "leader" state

# Kill the leader (if node1 is leader on port 8200)
kill $(pgrep -f "vault-1/config")

# Wait a moment for election
sleep 5

# Check from another node
VAULT_ADDR=http://127.0.0.1:8210 vault status
# Should show: standby = false (this is the new leader)

VAULT_ADDR=http://127.0.0.1:8210 vault operator raft list-peers
# node1 should be gone or show as failed

# Data is still accessible
VAULT_ADDR=http://127.0.0.1:8210 vault kv get secret/test

Exercise 3 — Raft Snapshots

Task: Create a backup, make changes, then restore the backup.

  1. Write a secret
  2. Take a Raft snapshot
  3. Write another secret
  4. Restore the snapshot
  5. Verify the second secret is gone
Show Answer 
# Write initial data
vault kv put secret/backup-test value="before-snapshot"

# Take snapshot
vault operator raft snapshot save /tmp/vault-backup.snap

# Write more data
vault kv put secret/after-snapshot value="this-will-be-lost"

# Verify both exist
vault kv get secret/backup-test      # ✅ exists
vault kv get secret/after-snapshot   # ✅ exists

# Restore snapshot
vault operator raft snapshot restore /tmp/vault-backup.snap

# Check data
vault kv get secret/backup-test      # ✅ exists (was in snapshot)
vault kv get secret/after-snapshot   # ❌ gone (written after snapshot)

Exercise 4 — Health Checks

Task: Write a script that checks all nodes in a Vault cluster and reports their status.

Show Answer 
#!/bin/bash
# vault-health-check.sh

NODES=("http://127.0.0.1:8200" "http://127.0.0.1:8210" "http://127.0.0.1:8220")

echo "=== Vault Cluster Health ==="
for node in "${NODES[@]}"; do
    STATUS=$(curl -s -o /dev/null -w "%{http_code}" "${node}/v1/sys/health" 2>/dev/null)
    
    case $STATUS in
        200) echo "$node: ACTIVE (leader)" ;;
        429) echo "$node: STANDBY (healthy)" ;;
        472) echo "$node: DR SECONDARY" ;;
        473) echo "$node: PERFORMANCE STANDBY" ;;
        501) echo "$node: NOT INITIALIZED" ;;
        503) echo "$node: SEALED" ;;
        *)   echo "$node: UNREACHABLE ($STATUS)" ;;
    esac
done

Exercise 5 — Auto-Unseal Configuration

Task: Write a Vault config that uses AWS KMS for auto-unseal, and explain the migration process.

Show Answer 
# vault.hcl with AWS KMS auto-unseal

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/tls/cert.pem"
  tls_key_file  = "/etc/vault/tls/key.pem"
}

api_addr     = "https://vault-1.internal:8200"
cluster_addr = "https://vault-1.internal:8201"

Migration from Shamir to auto-unseal:

# 1. Stop Vault
systemctl stop vault

# 2. Add the seal stanza to vault.hcl (as above)

# 3. Start with -migrate flag
vault server -config=/etc/vault/vault.hcl -migrate

# 4. Provide Shamir unseal keys when prompted (threshold number)
# Vault re-encrypts the master key with KMS

# 5. Restart normally
systemctl restart vault
# Vault auto-unseals — no keys needed! ✅

What’s Next?

With HA covered, let’s look at Audit & Monitoring — audit logs, telemetry, and alerting.

Next Chapter: Audit & Monitoring

You may also enjoy