HashiCorp Vault — Backup & Recovery

2 minute read

Regular backups and tested recovery procedures are critical for Vault — losing the storage backend means losing all secrets.

Raft Snapshots

Take a Snapshot

# Manual snapshot
vault operator raft snapshot save vault-backup-$(date +%Y%m%d-%H%M%S).snap

# Verify the snapshot
ls -lh vault-backup-*.snap

Automate Snapshots

#!/bin/bash
# /opt/vault/scripts/snapshot.sh
set -euo pipefail

BACKUP_DIR="/opt/vault/backups"
RETENTION_DAYS=30
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
SNAPSHOT_FILE="${BACKUP_DIR}/vault-${TIMESTAMP}.snap"

# Take snapshot
export VAULT_ADDR="https://127.0.0.1:8200"
export VAULT_TOKEN="$(cat /opt/vault/.snapshot-token)"

vault operator raft snapshot save "$SNAPSHOT_FILE"

# Upload to S3 (optional)
aws s3 cp "$SNAPSHOT_FILE" s3://vault-backups/daily/

# Clean old local backups
find "$BACKUP_DIR" -name "*.snap" -mtime +${RETENTION_DAYS} -delete

echo "Snapshot saved: $SNAPSHOT_FILE"

Cron job:

# Every 6 hours
0 */6 * * * /opt/vault/scripts/snapshot.sh >> /var/log/vault/snapshot.log 2>&1

Restore from Snapshot

# Restore (replaces all data in the cluster)
vault operator raft snapshot restore vault-backup-20240115-103000.snap

# Force restore (if cluster state has diverged)
vault operator raft snapshot restore -force vault-backup-20240115-103000.snap

Warning: Restoring a snapshot replaces ALL current data. All changes since the snapshot are lost.

Consul Storage Backup

If using Consul as the storage backend:

# Consul snapshot
consul snapshot save consul-vault-$(date +%Y%m%d).snap

# Restore
consul snapshot restore consul-vault-20240115.snap

Backup Encryption Keys

Unseal Keys

# CRITICAL: Store unseal keys securely
# Options:
# 1. Shamir's Secret Sharing — distribute to key holders
# 2. PGP encryption — encrypt each share to a key holder's PGP key

vault operator init \
    -key-shares=5 \
    -key-threshold=3 \
    -pgp-keys="keybase:alice,keybase:bob,keybase:carol,keybase:dave,keybase:eve"

Recovery Keys (Auto-Unseal)

# Recovery keys are generated during auto-unseal init
# Store them securely — they can generate a root token
vault operator init -recovery-shares=5 -recovery-threshold=3

Disaster Recovery Plan

Step Action Owner
1 Detect failure (monitoring alert) On-call
2 Assess scope (single node vs cluster) On-call
3 If single node: restart, auto-unseal recovers On-call
4 If cluster: check Raft leader election On-call
5 If data loss: restore from latest snapshot Platform team
6 Verify restored state Platform team
7 Rotate credentials if breach suspected Security team
8 Post-mortem All

Recovery Procedures

Lost Quorum (Raft)

# If majority of nodes are permanently lost
# On the surviving node:
vault operator raft list-peers

# Remove dead peers
vault operator raft remove-peer <node-id>

# If single node remains, it becomes the new leader

Generate Emergency Root Token

# Requires unseal key holders
vault operator generate-root -init

# Each key holder provides their key
vault operator generate-root -nonce=<nonce>
# Provide unseal key when prompted

# After threshold is met, decode the root token
vault operator generate-root -decode=<encoded-token> -otp=<otp>

Backup Strategy Summary

Component Backup Method Frequency Retention
Vault data Raft snapshots Every 6h 30 days
Config files Git/SCM Every change Forever
TLS certificates Secure vault/HSM On rotation Until expiry + 1 year
Unseal keys PGP-encrypted, distributed One-time Forever
Recovery keys PGP-encrypted, distributed One-time Forever

Next: Upgrade Strategies

You may also enjoy