HashiCorp Vault — Disaster Recovery

Disaster Recovery (DR) replication creates a hot standby cluster that can take over if the primary cluster fails completely.

---

DR vs Performance Replication

Feature	Performance Replication	DR Replication
Secondary serves traffic	✅ Read-only	❌ Standby (no traffic)
Use case	Read scaling, multi-region	Disaster recovery
Failover	Manual or automated	Promote secondary
Data replicated	Configurable (filtered)	Everything
Tokens replicated	❌ No (local)	✅ Yes
Leases replicated	❌ No (local)	✅ Yes

---

DR Architecture

Normal operation:
┌────────────────────┐         ┌────────────────────┐
│   Primary Cluster  │────────▶│   DR Secondary     │
│   (Active)         │replicate│   (Hot standby)    │
│                    │         │   NOT serving       │
│   Serves all       │         │   traffic           │
│   traffic          │         │                     │
└────────────────────┘         └────────────────────┘

After disaster:
┌────────────────────┐         ┌────────────────────┐
│   Primary Cluster  │   ✖     │   DR Secondary     │
│   (DOWN)           │         │   → Promoted to     │
│                    │         │     Primary          │
│                    │         │   NOW serving        │
│                    │         │   all traffic        │
└────────────────────┘         └────────────────────┘

---

Enable DR Replication

Primary Cluster

# Enable DR replication on primary
vault write -f sys/replication/dr/primary/enable

# Generate activation token for DR secondary
vault write sys/replication/dr/primary/secondary-token \
    id="dr-secondary"
# Returns activation token

DR Secondary

# Activate DR secondary
vault write sys/replication/dr/secondary/enable \
    token="eyJhb..."

---

DR Failover (Promote Secondary)

When the primary is down:

# Generate a DR operation token (requires unseal keys)
vault operator generate-root -dr-token -init
# Returns: nonce and OTP

# Each key holder provides their unseal key
vault operator generate-root -dr-token \
    -nonce=<nonce> <unseal-key-1>
vault operator generate-root -dr-token \
    -nonce=<nonce> <unseal-key-2>
vault operator generate-root -dr-token \
    -nonce=<nonce> <unseal-key-3>
# After threshold reached, returns encoded DR operation token

# Decode the token
vault operator generate-root -dr-token \
    -decode=<encoded-token> -otp=<otp>
# Returns: DR operation token

# Promote the DR secondary to primary
vault write sys/replication/dr/secondary/promote \
    dr_operation_token="<dr-token>"

---

Raft Snapshot for DR (OSS)

For Vault OSS (no enterprise replication), use Raft snapshots:

# Regular snapshot backup (run on primary)
vault operator raft snapshot save /backups/vault-$(date +%Y%m%d).snap

# Store snapshots off-site (S3, GCS, etc.)
aws s3 cp /backups/vault-$(date +%Y%m%d).snap s3://vault-backups/

# Restore on a new cluster
vault operator raft snapshot restore /backups/vault-20240115.snap

Automated Backup Script

#!/bin/bash
# vault-dr-backup.sh

set -e
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="/backups/vault"
S3_BUCKET="s3://vault-backups"

mkdir -p "$BACKUP_DIR"

# Create snapshot
vault operator raft snapshot save "${BACKUP_DIR}/vault-${DATE}.snap"

# Upload to S3
aws s3 cp "${BACKUP_DIR}/vault-${DATE}.snap" "${S3_BUCKET}/"

# Clean local copies older than 7 days
find "$BACKUP_DIR" -name "vault-*.snap" -mtime +7 -delete

echo "Backup complete: vault-${DATE}.snap"

---

Recovery Checklist

Step	Action
1	Confirm primary is truly down (not just network partition)
2	Generate DR operation token (requires key holders)
3	Promote DR secondary
4	Update DNS/load balancer to point to new primary
5	Verify new primary is healthy
6	Plan to add new DR secondary

---

Quick Reference

Command	Description
vault write -f sys/replication/dr/primary/enable	Enable DR on primary
vault write sys/replication/dr/primary/secondary-token id=NAME	Generate DR token
vault write sys/replication/dr/secondary/enable token=TOKEN	Activate DR secondary
vault write sys/replication/dr/secondary/promote ...	Promote in disaster
vault operator raft snapshot save FILE	Backup (OSS alternative)

Note: DR Replication is a Vault Enterprise feature. OSS users should use Raft snapshots.

---

Next: Practice Exercises