Vault Backup to NAS

Automated backup of Vault data from vault-01 to NAS-01 via rsync over SSH.

Overview

Component Value

Source

vault-01:/opt/vault/data

Destination

nas-01:/volume1/vault_backups/

Method

tar + rsync over SSH

Schedule

systemd timer

Retention

30 days

Prerequisites

Synology User Setup

Create dedicated backup user in Synology DSM:

  1. Control Panel → User & Group → Create

  2. Username: hashi-backups

  3. Description: "HashiCorp Vault backups"

  4. User Group: users

  5. Application Permissions: Terminal & SNMP → Allow

User shell must be /bin/sh, not /sbin/nologin. 
# Verify on NAS
grep hashi-backups /etc/passwd
# Expected: hashi-backups:x:1034:100:..:/bin/sh

# Fix if nologin
sudo sed -i 's|hashi-backups:/sbin/nologin|hashi-backups:/bin/sh|' /etc/passwd

Shared Folder Permissions

In DSM Control Panel → Shared Folder:

  • Folder: vault_backups

  • Permissions: hashi-backups → Read/Write

SSH Key Setup (vault-01)

# Generate key (no passphrase for automation)
sudo ssh-keygen -t ed25519 -C "vault-01-vault-backup" -N "" -f /root/.ssh/id_ed25519

# Add NAS host key
sudo ssh-keyscan nas-01.inside.domusdigitalis.dev | sudo tee -a /root/.ssh/known_hosts

# Copy public key to NAS
sudo ssh-copy-id -i /root/.ssh/id_ed25519.pub hashi-backups@nas-01.inside.domusdigitalis.dev

# Test
sudo ssh hashi-backups@nas-01.inside.domusdigitalis.dev "echo connected"

Service File

/etc/systemd/system/vault-backup.service
[Unit]
Description=Vault Backup to NAS
After=network.target vault.service

[Service]
Type=oneshot
User=root
Environment=HOME=/root
Environment=VAULT_ADDR=http://127.0.0.1:8200

# Backup data directory (file storage backend)
ExecStart=/bin/bash -c 'TIMESTAMP=$(date +%%Y%%m%%d-%%H%%M%%S) && tar -czf /tmp/vault-backup-$TIMESTAMP.tar.gz -C /opt/vault data && rsync -avz /tmp/vault-backup-$TIMESTAMP.tar.gz hashi-backups@nas-01.inside.domusdigitalis.dev:/volume1/vault_backups/ && rm /tmp/vault-backup-$TIMESTAMP.tar.gz'

# Cleanup old backups (keep 30 days)
ExecStartPost=/bin/bash -c 'ssh -i /root/.ssh/id_ed25519 -o BatchMode=yes hashi-backups@nas-01.inside.domusdigitalis.dev "find /volume1/vault_backups -maxdepth 1 -name vault-backup-*.tar.gz -mtime +30 -delete"'

Timer File

/etc/systemd/system/vault-backup.timer
[Unit]
Description=Daily Vault Backup

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Installation

# Create service and timer files
sudo vi /etc/systemd/system/vault-backup.service
sudo vi /etc/systemd/system/vault-backup.timer

# Reload and enable
sudo systemctl daemon-reload
sudo systemctl enable --now vault-backup.timer

# Verify timer
systemctl list-timers vault-backup.timer

Manual Execution

sudo systemctl start vault-backup.service && sudo systemctl status vault-backup.service

Expected output:

Active: inactive (dead) - SUCCESS
Process: ExecStart ... status=0/SUCCESS
Process: ExecStartPost ... status=0/SUCCESS

Troubleshooting

View Logs

journalctl -xeu vault-backup.service --no-pager | tail -50

Common Issues

Error Cause Fix

Host key verification failed

NAS not in known_hosts

ssh-keyscan nas-01…​ | tee -a /root/.ssh/known_hosts

Permission denied, please try again

User shell is /sbin/nologin

sed -i 's|nologin|/bin/sh|' /etc/passwd on NAS

Permission denied (#recycle)

find descends into recycle bin

Add -maxdepth 1 to find command

SSH key not found (systemd)

HOME not set in service

Add Environment=HOME=/root

SELinux (RHEL/Rocky)

Problem: rsync runs in rsync_t SELinux domain, which cannot execute ssh_exec_t by default.

Symptoms:

rsync: [sender] Failed to exec ssh: Permission denied (13)
rsync error: error in IPC code (code 14)

Diagnosis:

# Check for AVC denials
sudo ausearch -m avc --start today | grep rsync

# Example denial
# avc: denied { execute_no_trans } for comm="rsync" path="/usr/bin/ssh"
# scontext=system_u:system_r:rsync_t:s0 tcontext=system_u:object_r:ssh_exec_t:s0

Fix (proper SELinux policy - do NOT run unconfined):

# Step 1: Set rsync_t domain to permissive (captures all denials)
sudo semanage permissive -a rsync_t

# Step 2: Run the service (will succeed, logs all would-be denials)
sudo systemctl start vault-backup.service

# Step 3: Generate comprehensive policy from all denials
sudo ausearch -m avc --start today | grep rsync | audit2allow -M vault-backup

# Step 4: Review what it allows
cat vault-backup.te

# Step 5: Install policy module
sudo semodule -i vault-backup.pp

# Step 6: Remove permissive mode
sudo semanage permissive -d rsync_t

# Step 7: Test in enforcing mode
sudo systemctl start vault-backup.service && systemctl status vault-backup.service

Expected policy permissions (vault-backup.te):

allow rsync_t ssh_exec_t:file { execute_no_trans map };
allow rsync_t ssh_home_t:dir search;
allow rsync_t ssh_home_t:file { getattr open read };
allow rsync_t systemd_conf_t:file { getattr open read };
allow rsync_t initrc_tmp_t:file open;

Alternative (quick but less comprehensive):

# Enable rsync_client boolean (may not cover all cases)
sudo setsebool -P rsync_client on
The permissive domain approach captures ALL required permissions in one pass, avoiding denial whack-a-mole.

Backup Strategy by Storage Backend

Backend Backup Method

file (current)

tar -czf backup.tar.gz -C /opt/vault data

raft (integrated HA)

vault operator raft snapshot save snapshot.snap

consul

Consul snapshot

Restore

# Stop Vault
sudo systemctl stop vault

# Extract backup
sudo tar -xzf vault-backup-YYYYMMDD-HHMMSS.tar.gz -C /opt/vault

# Start Vault
sudo systemctl start vault
vault status

Verification

Check Backups on NAS

ssh hashi-backups@nas-01.inside.domusdigitalis.dev "ls -lah /volume1/vault_backups/"

List with sizes and dates (awk formatting):

ssh hashi-backups@nas-01.inside.domusdigitalis.dev \
  "ls -lh /volume1/vault_backups/*.tar.gz" | awk '{print $5, $6, $7, $9}'
Example Output
45M Feb 19 02:00 vault-backup-20260219-020000.tar.gz
44M Feb 18 02:00 vault-backup-20260218-020000.tar.gz
awk '{print $5, $6, $7, $9}' = print size, month, day, filename (fields 5, 6, 7, 9).

Count Backups by Age

# Count backups older than 7 days
ssh hashi-backups@nas-01.inside.domusdigitalis.dev \
  "find /volume1/vault_backups -maxdepth 1 -name '*.tar.gz' -mtime +7 | wc -l"

# List backups from last 7 days only
ssh hashi-backups@nas-01.inside.domusdigitalis.dev \
  "find /volume1/vault_backups -maxdepth 1 -name '*.tar.gz' -mtime -7 -exec ls -lh {} \;" | \
  awk '{print $5, $9}'

Check Timer Status

systemctl status vault-backup.timer

Extract next run time with awk:

systemctl list-timers vault-backup.timer | awk 'NR==2 {print "Next:", $1, $2}'

View Last Backup Log

journalctl -u vault-backup.service -n 20 --no-pager | awk '/rsync|tar|SUCCESS|FAIL/'
awk '/pattern/' = print lines matching pattern. Multiple patterns with | (OR).