Runbook: Backup Strategy

Last Updated

2026-02-08

Owner

evanusmodestus

Review Frequency

Quarterly

Purpose

Protect all critical data using a defense-in-depth backup strategy that survives:

  • Hardware failure (SSD death)

  • Ransomware (offline copies)

  • Fire/flood (offsite storage)

  • Long-term bit rot (archival media)

The 3-2-1 Rule (Extended)

Rule Meaning Implementation Recovery Time

3 copies

At least 3 copies of data

Hot + Warm + Cold

N/A

2 storage types

Different media types

SSD + HDD/NAS + Optical

N/A

1 offsite

Geographic separation

LUKS USB #2 offsite

Hours to days

+Archival

1000+ year durability

M-Disc in fireproof safe

Days

Architecture Diagram

Backup Strategy Tiers

Tier 1: HOT (Primary)

Location

  • Workstation SSD (~/.secrets/, ~/.ssh/)

  • Git repositories (encrypted .age files)

What Lives Here

Data Path

age master key

~/.secrets/.metadata/keys/master.age.key

SSH keys

~/.ssh/id_*

Encrypted secrets

~/.secrets/environments/

Documents

~/.secrets/documents/

Access

Instant - this is your working copy.

Commands

# View secrets
dsec show d000 dev/network

# Edit secrets
dsec edit d000 dev/network

# View encrypted document
~/.secrets/bin/view-document <file.age>

Tier 2: WARM (Automated)

Location

  • Synology NAS-01 (RAID storage)

  • Borg backup repository

What Lives Here

Data NAS Path Backup Method

ISE configs

/ise_backups

netapi ise backup --upload-nas

WLC configs

/wlc_backups

netapi wlc backup --upload-nas

VyOS configs

/firewall_backups

netapi vyos backup --upload-nas

IOS switch configs

/switch_backups

netapi ios backup --all --upload-nas

KVM VM definitions

/kvm_backups

netapi kvm backup --all --upload-nas

Keycloak realms

/Backups/keycloak

netapi keycloak backup --upload-nas

Workstation (Borg)

/Backups/borg

borg create

Schedule

  • Daily: Infrastructure backups via netapi

  • Weekly: Borg workstation backup

Infrastructure Backup Commands

Step 1: Load network credentials
dsource d000 dev/network
Step 2: Run infrastructure backups
netapi ise backup --upload-nas
netapi wlc backup --upload-nas
netapi vyos backup --upload-nas
netapi ios backup --all --upload-nas
netapi kvm backup --all --upload-nas
Step 3: Backup Keycloak (requires identity secrets)
dsource d000 dev/identity
netapi keycloak backup --upload-nas
Step 4: Verify all backups
dsource d000 dev/network
netapi synology backup-status --detailed

Borg Workstation Backup

Step 1: Mount Synology NFS share
sudo mount -t nfs nas-01.inside.domusdigitalis.dev:/volume1/borg_backups /mnt/synology
Step 2: Load storage credentials and run backup
dsource d000 dev/storage
sudo -E BORG_PASSPHRASE="$BORG_PASSPHRASE" ~/.local/bin/borg-backup-synology.sh
dsunsource
Step 3: Unmount NFS share
sudo umount /mnt/synology

If mount.nfs: Protocol not supported error occurs after kernel update, run sudo modprobe nfs or reboot into new kernel. See Borg Backup for full details.

Tier 3: COLD (Offline)

Location

  • Seagate Primary (seagate-crypt): Daily local encrypted mirror

  • Seagate Backup (seagate-backup): Weekly sync from primary

  • LUKS USB #1: Home safe

  • LUKS USB #2: Offsite (quarterly rotation)

Seagate USB Drives

Drive LUKS Mapper Mount Point

Seagate Primary

seagate-crypt

/mnt/seagate

Seagate Backup

seagate-backup

/mnt/seagate-backup

Directory structure on each drive:

Seagate Backup Structure
Figure 1. Seagate Backup Structure

Seagate Commands

Daily: Backup to Primary

Step 1: Get LUKS passphrase and find device
gopass show -c v3/domains/d000/storage/seagate-primary
lsblk -o NAME,SIZE,MODEL | grep -i seagate
Step 2: Mount (device path varies - check lsblk output)
seagate-primary-mount /dev/sdX1
Example (if Seagate is sda)
seagate-primary-mount /dev/sda1
Step 3: Run backup
seagate-primary-backup
Step 4: Unmount
seagate-primary-umount

Weekly: Sync Primary to Backup

Step 1: Get LUKS passphrases
gopass show -c v3/domains/d000/storage/seagate-primary
gopass show -c v3/domains/d000/storage/seagate-secondary
Step 2: Find devices
lsblk -o NAME,SIZE,MODEL | grep -i seagate
Step 3: Mount both drives (device paths vary)
seagate-primary-mount /dev/sdX1
seagate-backup-mount /dev/sdY1
Step 4: Sync each directory
rsync -av --delete /mnt/seagate/secrets/ /mnt/seagate-backup/secrets/
rsync -av --delete /mnt/seagate/configs/ /mnt/seagate-backup/configs/
rsync -av --delete /mnt/seagate/backups/ /mnt/seagate-backup/backups/
rsync -av --delete /mnt/seagate/storage/ /mnt/seagate-backup/storage/
Step 5: Verify sizes match
echo "PRIMARY:" && du -sh /mnt/seagate/
echo "BACKUP:" && du -sh /mnt/seagate-backup/
Step 6: Unmount both drives
seagate-backup-umount
seagate-primary-umount

What Lives Here

CRITICAL: These are the recovery keys for everything else.

Data Why Critical

master.age.key

Decrypts all .age files - without this, everything is unrecoverable

SSH private keys

Access to all systems

GPG secret keys

Signing, encryption

LUKS headers

Recovery if header corrupted

gocryptfs.conf

Vault master keys

Schedule

  • Monthly: Sync to LUKS USB #1

  • Quarterly: Rotate USB #1 to offsite, bring USB #2 home

LUKS USB Commands

Monthly Sync to USB

Step 1: Mount LUKS drive
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup
Step 2: Sync critical files
rsync -av ~/.secrets/.metadata/keys/ /mnt/backup/keys/
rsync -av ~/.ssh/id_* /mnt/backup/ssh/
gpg --export-secret-keys > /mnt/backup/gpg/secret-keys.asc
Step 3: Backup LUKS headers
sudo cryptsetup luksHeaderBackup /dev/nvme0n1p2 \
    --header-backup-file /mnt/backup/luks/workstation-header.img
Step 4: Unmount
sudo umount /mnt/backup
sudo cryptsetup luksClose backup-usb

LUKS USB Setup (One-Time)

Create encrypted USB
sudo cryptsetup luksFormat /dev/sdX1
Add backup key slot
sudo cryptsetup luksAddKey /dev/sdX1
Open and format filesystem
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mkfs.ext4 /dev/mapper/backup-usb

Tier 4: ARCHIVAL (M-Disc)

Location

  • M-Disc optical media

  • Fireproof safe or safe deposit box

What Lives Here

Disc contents (age-encrypted tarballs):

File Contents

P0-CRITICAL-*.tar.age

GPG, SSH, pass, secrets, age keys (RESTORE FIRST)

FULL-BACKUP-*.tar.age

Complete backup (~/.secrets, ~/.config, ~/atelier, etc.)

RECOVERY-README.txt

Decryption and restore instructions

SHA256SUMS.txt

Integrity verification checksums

Schedule

  • Annual: Burn new M-Disc

  • Quarterly: Verify disc readability and checksums

  • Before storing: Test readability immediately after burn

Why M-Disc

  • Durability: 1000+ year lifespan (vs ~5 years for regular DVD)

  • Offline: Immune to ransomware

  • Disaster-proof: Survives fire, flood, EMP

M-Disc Commands

Annual Burn Procedure

Step 1: Create staging directory
mkdir /tmp/mdisc-backup
Step 2: Create P0-CRITICAL archive (keys and secrets)
tar -cvf /tmp/p0-critical.tar \
    ~/.secrets/.metadata/keys/ \
    ~/.ssh/ \
    ~/.gnupg/ \
    ~/.age/ \
    ~/.passage/ \
    ~/.arcanum/ \
    ~/.password-store/ \
    ~/.pki/ \
    ~/.local/share/gopass/ \
    ~/.config/gopass/ 2>&1 | awk 'END {print NR " files archived"}'
age -e -R ~/.age/recipients/self.txt \
    -o /tmp/mdisc-backup/P0-CRITICAL-$(date +%Y-%m-%d).tar.age \
    /tmp/p0-critical.tar
Step 3: Create FULL-BACKUP archive
tar -cvf /tmp/full-backup.tar \
    ~/.secrets \
    ~/.gnupg \
    ~/.age \
    ~/.passage \
    ~/.arcanum \
    ~/.password-store \
    ~/.ssh \
    ~/.pki \
    ~/.config \
    ~/.local \
    ~/.ansible \
    ~/.claude \
    ~/.mozilla \
    ~/.vscode-oss \
    ~/.vim \
    ~/.tmux \
    ~/.terraform.d \
    ~/.gemini \
    ~/.w3m \
    ~/atelier \
    ~/bin \
    ~/Documents \
    ~/Pictures \
    ~/.zshrc.bak.* \
    ~/.zsh_history \
    ~/.bash_history \
    ~/.viminfo \
    ~/.claude.json \
    ~/.npmrc 2>&1 | awk 'END {print NR " files archived"}'
Excludes reinstallable (~25G saved): ~/.cache, ~/.npm, ~/.rustup, ~/.cargo, ~/.pyenv, ~/.ollama, ~/.zoom 
age -e -R ~/.age/recipients/self.txt \
    -o /tmp/mdisc-backup/FULL-BACKUP-$(date +%Y-%m-%d).tar.age \
    /tmp/full-backup.tar
Step 4: Create RECOVERY-README.txt
cat > /tmp/mdisc-backup/RECOVERY-README.txt << 'EOF'
COLD STORAGE ARCHIVE - RECOVERY

CONTENTS:
- P0-CRITICAL-*.tar.age   Keys and secrets (RESTORE FIRST)
- FULL-BACKUP-*.tar.age   Complete backup

DECRYPT:
  age -d -i ~/.age/identities/personal.key FILE.age > FILE.tar
  tar -xvf FILE.tar

OR PIPE DIRECTLY:
  age -d -i ~/.age/identities/personal.key FILE.age | tar -xvf -

RESTORE ORDER:
  1. Restore P0-CRITICAL first (you need keys to decrypt other things)
  2. Restore FULL-BACKUP
  3. Run: chown -R $USER:$USER ~

IF YOU LOST YOUR AGE KEY:
  These files are UNRECOVERABLE. Store age key in multiple locations.
EOF
Step 5: Generate checksums
cd /tmp/mdisc-backup && sha256sum *.age > SHA256SUMS.txt
Step 6: Create ISO and burn
genisoimage -o /tmp/backup-$(date +%Y-%m-%d).iso /tmp/mdisc-backup/
wodim -v /tmp/backup-$(date +%Y-%m-%d).iso
Step 7: Verify immediately after burn
sudo mount /dev/sr0 /mnt/cdrom
cd /mnt/cdrom && sha256sum -c SHA256SUMS.txt
sudo umount /mnt/cdrom
Step 8: Secure delete temp files
shred -vzn 3 /tmp/p0-critical.tar /tmp/full-backup.tar
rm -rf /tmp/mdisc-backup /tmp/backup-*.iso /tmp/*.tar

Quarterly Verification

Use this procedure to verify M-Disc integrity periodically.

Step 1: Mount disc
sudo mount /dev/sr0 /mnt/cdrom
Step 2: List contents
ls -la /mnt/cdrom/
Step 3: Read recovery instructions
cat /mnt/cdrom/RECOVERY-README.txt
Step 4: Verify checksums (takes time for large files)
cd /mnt/cdrom && sha256sum -c SHA256SUMS.txt
Step 5: Test decryption (list contents only, don’t extract)
age -d -i ~/.age/identities/personal.key /mnt/cdrom/P0-CRITICAL-*.tar.age | tar -tvf - | head -20
Step 6: Unmount
sudo umount /mnt/cdrom

Recovery Procedures

Priority Order

  1. age key - Without this, nothing else can be decrypted

  2. SSH keys - Access to systems

  3. dsec secrets - Credentials

  4. Infrastructure - ISE, WLC, etc.

Scenario: Lost Workstation

Step 1: Mount LUKS backup USB
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup
Step 2: Restore age key (CRITICAL - do this first)
mkdir -p ~/.secrets/.metadata/keys
cp /mnt/backup/keys/master.age.key ~/.secrets/.metadata/keys/
chmod 600 ~/.secrets/.metadata/keys/master.age.key
Step 3: Restore SSH keys
cp -r /mnt/backup/ssh/* ~/.ssh/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_*
Step 4: Clone secrets repo
git clone <secrets-repo> ~/.secrets
Step 5: Verify recovery
dsec show d000 dev/network

See Disaster Recovery for full procedures.

Verification Schedule

Frequency Action Verification

Weekly

Check NAS backup dates

netapi synology backup-status

Monthly

Sync to LUKS USB #1

Mount and verify file dates

Quarterly

Rotate LUKS USB offsite

Test decrypt on both USBs

Annually

Full recovery drill

Restore to test VM

Quick Reference

Daily Infrastructure Backup

Load credentials and run backups
dsource d000 dev/network
netapi ise backup --upload-nas
netapi wlc backup --upload-nas
netapi vyos backup --upload-nas
netapi ios backup --all --upload-nas
netapi kvm backup --all --upload-nas
Verify backups completed
netapi synology backup-status --detailed

Emergency Recovery

Restore age key from LUKS USB
sudo cryptsetup luksOpen /dev/sdX1 backup-usb
sudo mount /dev/mapper/backup-usb /mnt/backup
cp /mnt/backup/keys/master.age.key ~/.secrets/.metadata/keys/

Related Pages