Runbook: Backup Strategy

Last Updated

2026-01-26

Owner

evanusmodestus

Review Frequency

Quarterly

Purpose

Protect all critical data using a defense-in-depth backup strategy that survives:

  • Hardware failure (SSD death)

  • Ransomware (offline copies)

  • Fire/flood (offsite storage)

  • Long-term bit rot (archival media)

The 3-2-1 Rule (Extended)

Rule Meaning Implementation Recovery Time

3 copies

At least 3 copies of data

Hot + Warm + Cold

N/A

2 storage types

Different media types

SSD + HDD/NAS + Optical

N/A

1 offsite

Geographic separation

LUKS USB #2 offsite

Hours to days

+Archival

1000+ year durability

M-Disc in fireproof safe

Days

Architecture Diagram

Backup Strategy Tiers

Tier 1: HOT (Primary)

Location

  • Workstation SSD (~/.secrets/, ~/.ssh/)

  • Git repositories (encrypted .age files)

What Lives Here

Data Path

age master key

~/.secrets/.metadata/keys/master.age.key

SSH keys

~/.ssh/id_*

Encrypted secrets

~/.secrets/environments/

Documents

~/.secrets/documents/

Access

Instant - this is your working copy.

Commands

# View secrets
dsec show d000 dev/network

# Edit secrets
dsec edit d000 dev/network

# View encrypted document
~/.secrets/bin/view-document <file.age>

Tier 2: WARM (Automated)

Location

  • Synology NAS-01 (RAID storage)

  • Borg backup repository

What Lives Here

Data NAS Path Backup Method

ISE configs

/ise_backups

netapi ise backup --upload-nas

WLC configs

/wlc_backups

netapi wlc backup --upload-nas

pfSense configs

/firewall_backups

netapi pfsense backup --upload-nas

IOS switch configs

/switch_backups

netapi ios backup --all --upload-nas

KVM VM definitions

/kvm_backups

netapi kvm backup --all --upload-nas

Keycloak realms

/Backups/keycloak

netapi keycloak backup --upload-nas

Workstation (Borg)

/Backups/borg

borg create

Schedule

  • Daily: Infrastructure backups via netapi

  • Weekly: Borg workstation backup

Commands

# Load secrets
dsource d000 dev/network

# Run all infrastructure backups
netapi ise backup --upload-nas
netapi wlc backup --upload-nas
netapi pfsense backup --upload-nas
netapi ios backup --all --upload-nas
netapi kvm backup --all --upload-nas

# Keycloak needs identity secrets
dsource d000 dev/identity
netapi keycloak backup --upload-nas

# Verify backups
dsource d000 dev/network
netapi synology backup-status --detailed

# Borg backup (workstation)
borg create --stats --progress \
    ssh://nas-01/volume1/Backups/borg::$(hostname)-$(date +%Y-%m-%d) \
    ~/.secrets ~/.ssh ~/atelier

Tier 3: COLD (Offline)

Location

  • LUKS USB #1: Home safe

  • LUKS USB #2: Offsite (quarterly rotation)

What Lives Here

CRITICAL: These are the recovery keys for everything else.

Data Why Critical

master.age.key

Decrypts all .age files - without this, everything is unrecoverable

SSH private keys

Access to all systems

GPG secret keys

Signing, encryption

LUKS headers

Recovery if header corrupted

gocryptfs.conf

Vault master keys

Schedule

  • Monthly: Sync to LUKS USB #1

  • Quarterly: Rotate USB #1 to offsite, bring USB #2 home

Commands

# Mount LUKS drive
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup

# Sync critical files
rsync -av ~/.secrets/.metadata/keys/ /mnt/backup/keys/
rsync -av ~/.ssh/id_* /mnt/backup/ssh/
gpg --export-secret-keys > /mnt/backup/gpg/secret-keys.asc

# Backup LUKS headers (for all encrypted volumes)
sudo cryptsetup luksHeaderBackup /dev/nvme0n1p2 \
    --header-backup-file /mnt/backup/luks/workstation-header.img

# Unmount
sudo umount /mnt/backup
sudo cryptsetup luksClose backup-usb

LUKS USB Setup (One-Time)

# Create encrypted USB
sudo cryptsetup luksFormat /dev/sdX1

# Add backup key slot (in case you forget passphrase)
sudo cryptsetup luksAddKey /dev/sdX1

# Open and format
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mkfs.ext4 /dev/mapper/backup-usb

Tier 4: ARCHIVAL (M-Disc)

Location

  • M-Disc optical media

  • Fireproof safe or safe deposit box

What Lives Here

Only the absolute essentials that NEVER change:

  • Root CA private key (if you control it)

  • age master key

  • Recovery passphrases (printed)

Schedule

  • Annual: Burn new M-Disc

  • Verify: Test readability before storing

Why M-Disc

  • Durability: 1000+ year lifespan (vs ~5 years for regular DVD)

  • Offline: Immune to ransomware

  • Disaster-proof: Survives fire, flood, EMP

Commands

# Create ISO with critical files
mkdir /tmp/mdisc-backup
cp ~/.secrets/.metadata/keys/master.age.key /tmp/mdisc-backup/
cp ~/.ssh/id_ed25519 /tmp/mdisc-backup/
genisoimage -o /tmp/backup-$(date +%Y).iso /tmp/mdisc-backup/

# Burn to M-Disc (requires M-Disc compatible drive)
wodim -v /tmp/backup-$(date +%Y).iso

# Verify readability
mount /dev/sr0 /mnt/cdrom
diff ~/.secrets/.metadata/keys/master.age.key /mnt/cdrom/master.age.key
umount /mnt/cdrom

# Secure delete temp files
shred -n 10 -z -u /tmp/mdisc-backup/*
rm -rf /tmp/mdisc-backup /tmp/backup-*.iso

Recovery Procedures

Priority Order

  1. age key - Without this, nothing else can be decrypted

  2. SSH keys - Access to systems

  3. dsec secrets - Credentials

  4. Infrastructure - ISE, WLC, etc.

Scenario: Lost Workstation

# 1. Mount LUKS backup USB
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup

# 2. Restore age key (FIRST!)
mkdir -p ~/.secrets/.metadata/keys
cp /mnt/backup/keys/master.age.key ~/.secrets/.metadata/keys/
chmod 600 ~/.secrets/.metadata/keys/master.age.key

# 3. Restore SSH keys
cp -r /mnt/backup/ssh/* ~/.ssh/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_*

# 4. Clone secrets repo
git clone <secrets-repo> ~/.secrets

# 5. Verify
dsec show d000 dev/network

See Disaster Recovery for full procedures.

Verification Schedule

Frequency Action Verification

Weekly

Check NAS backup dates

netapi synology backup-status

Monthly

Sync to LUKS USB #1

Mount and verify file dates

Quarterly

Rotate LUKS USB offsite

Test decrypt on both USBs

Annually

Full recovery drill

Restore to test VM

Quick Reference

Backup Commands

# Infrastructure (daily)
dsource d000 dev/network
netapi ise backup --upload-nas
netapi wlc backup --upload-nas
netapi pfsense backup --upload-nas
netapi ios backup --all --upload-nas
netapi kvm backup --all --upload-nas

# Verify
netapi synology backup-status --detailed

Recovery Commands

# From LUKS USB
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup
cp /mnt/backup/keys/master.age.key ~/.secrets/.metadata/keys/

Related Pages