Runbook: Backup All Infrastructure

Last Updated: 2026-01-24
Owner: evanusmodestus
Frequency: Daily (automated) / Weekly (verified)

Purpose

Backup all critical infrastructure to Synology NAS with verification.

Prerequisites

dsec secrets loaded for d000 dev/network
NAS accessible at 10.50.1.50
netapi CLI installed and configured

Quick Backup (All Systems)

# Load secrets
dsource d000 dev/network

# Run all backups
netapi ise backup --upload-nas
netapi wlc backup --upload-nas
netapi pfsense backup --upload-nas
netapi ios backup --all --upload-nas
netapi kvm backup --all --upload-nas

# Keycloak needs identity secrets
dsource d000 dev/identity
netapi keycloak backup --upload-nas

# Verify
dsource d000 dev/network
netapi synology backup-status --detailed

Detailed Procedure

Step 1: Load Network Secrets

dsource d000 dev/network

Step 2: Backup ISE

netapi ise backup --upload-nas

Expected output:

Backup initiated on ise-01.inside.domusdigitalis.dev
Uploading to NAS...
✓ Uploaded to /ise_backups

Step 3: Backup WLC

netapi wlc backup --upload-nas

Step 4: Backup pfSense

netapi pfsense backup --upload-nas

Step 5: Backup IOS Switches

netapi ios backup --all --upload-nas

Step 6: Backup KVM VM Definitions

netapi kvm backup --all --upload-nas

Step 7: Backup Keycloak

dsource d000 dev/identity
netapi keycloak backup --upload-nas

Verification

Check backup status on NAS:

dsource d000 dev/network
netapi synology backup-status --detailed

Expected output:

                                    Backup Status
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━┳━━━━━━━━┓
┃ System   ┃ Folder            ┃ Files ┃ Devices ┃     Latest ┃ Age ┃ Status ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━╇━━━━━━━━┩
│ ISE      │ /ise_backups      │     7 │       1 │ 2026-01-24 │  0m │  ✓ OK  │
│ WLC      │ /wlc_backups      │     4 │       1 │ 2026-01-24 │  0m │  ✓ OK  │
│ pfSense  │ /firewall_backups │     3 │       1 │ 2026-01-24 │  0m │  ✓ OK  │
│ Switches │ /switch_backups   │     3 │       2 │ 2026-01-24 │  0m │  ✓ OK  │
│ KVM VMs  │ /kvm_backups      │    16 │       8 │ 2026-01-24 │  0m │  ✓ OK  │
│ Keycloak │ /Backups/keycloak │     2 │       2 │ 2026-01-24 │  0m │  ✓ OK  │
└──────────┴───────────────────┴───────┴─────────┴────────────┴─────┴────────┘

✓ All 6 backup sets current

Troubleshooting

ISE Backup Fails

Symptom: Connection timeout or Authentication failed

Cause: ISE not reachable or credentials invalid

Resolution:

# Check connectivity
ping $ISE_PAN_IP

# Verify credentials
echo $ISE_API_USER
echo $ISE_API_PASS | head -c3  # Show first 3 chars only

NAS Upload Fails (Error 408)

Symptom: Error 408 when uploading

Cause: Target folder doesn’t exist