Phase 13: Maintenance & Automation

Ongoing operational tasks that keep the P16g healthy after initial deployment. These are not one-time setup items — they require periodic attention or automation.

Borg Backup Automation

Currently: Borg backup is manual (borg create from shell). Automate with a systemd timer.

Create systemd service

# ~/.config/systemd/user/borg-backup.service
cat << 'EOF'
[Unit]
Description=Borg backup to NAS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=%h/bin/borg-backup.sh
EOF

Create systemd timer

# ~/.config/systemd/user/borg-backup.timer
cat << 'EOF'
[Unit]
Description=Run Borg backup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable

systemctl --user enable --now borg-backup.timer

# Verify timer registered
systemctl --user list-timers | grep borg

Check Status

Check	Status
`borg-backup.sh` script in `~/bin/` and working	[ ]
systemd user service created	[ ]
systemd user timer created and enabled	[ ]
Timer fires correctly (`systemctl --user list-timers`)	[ ]

borg-backup.sh script in ~/bin/ and working

[ ]

systemd user service created

[ ]

systemd user timer created and enabled

[ ]

Timer fires correctly (systemctl --user list-timers)

[ ]

TLP Battery Management

ThinkPad-specific power management. Set charge thresholds to extend battery lifespan.

# Verify TLP is installed and running
sudo tlp-stat -b

Configure charge thresholds

# /etc/tlp.conf — set charge thresholds
# Start charging below 40%, stop at 80%
sudo sed -i 's/^#\?START_CHARGE_THRESH_BAT0=.*/START_CHARGE_THRESH_BAT0=40/' /etc/tlp.conf
sudo sed -i 's/^#\?STOP_CHARGE_THRESH_BAT0=.*/STOP_CHARGE_THRESH_BAT0=80/' /etc/tlp.conf

# Apply changes
sudo tlp start

# Verify thresholds active
sudo tlp-stat -b | grep -E 'charge_control|threshold'

Check Status

Check	Status
TLP installed and running	[ ]
Charge thresholds set (40/80)	[ ]
Thresholds verified with `tlp-stat -b`	[ ]

TLP installed and running

[ ]

Charge thresholds set (40/80)

[ ]

Thresholds verified with tlp-stat -b

[ ]

Vault Certificate Renewal

EAP-TLS certificates from Vault PKI have a finite TTL. Renewal workflow:

# Check current cert expiration
openssl x509 -in /etc/ssl/certs/modestus-p16g-eaptls.pem -noout -enddate

# When expired: re-issue from Vault (see Phase 8b for full workflow)
# 1. Load Vault credentials
# 2. vault write pki_int/issue/domus-client common_name="modestus-p16g.inside.domusdigitalis.dev"
# 3. Extract cert + key with jq
# 4. Install to /etc/ssl/certs and /etc/ssl/private
# 5. Restart NetworkManager

Check	Status
Current cert expiration date known	[ ]
Renewal workflow documented (xref Phase 8b)	[ ]
Calendar reminder set for renewal	[ ]

Check

Status

Current cert expiration date known

[ ]

Renewal workflow documented (xref Phase 8b)

[ ]

Calendar reminder set for renewal

[ ]

Vault SSH Certificate Renewal

SSH certificates issued by Vault have an 8-hour TTL and must be renewed per session.

# Check cert validity
ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep -E 'Valid|Principal'

Vault SSH cert for machine-to-machine auth is currently blocked by VLAN anti-pivot segmentation. Using authorized_keys + YubiKey FIDO2 as workaround.

Pacman Hook Validation

The ESP kernel sync hook was written in Phase 4 but has not been tested with a real kernel update.

# Trigger on next kernel update
sudo pacman -Syu

# After kernel update: verify kernels synced to ESP
ls -la /boot/efi/vmlinuz-linux /boot/efi/initramfs-linux.img

Check	Status
Pacman hook fires on kernel update	[ ]
Kernels synced to ESP after update	[ ]
Both default and LTS kernels present	[ ]

Check

Status

Pacman hook fires on kernel update

[ ]

Kernels synced to ESP after update

[ ]

Both default and LTS kernels present

[ ]

ssh-agent Persistence

Currently: SSH keys require manual ssh-add + gopass passphrase every session. Three options under evaluation:

Option Pros Cons

Option	Pros	Cons
`keychain`	Simple, well-tested, re-uses agents across sessions	Extra package, shell init overhead
systemd user unit (`ssh-agent.service`)	Native, no extra packages, clean lifecycle	Requires `SSH_AUTH_SOCK` export in shell
gpg-agent with SSH support	Already running for GPG/YubiKey, single agent	Configuration complexity, pinentry interactions

keychain

Simple, well-tested, re-uses agents across sessions

Extra package, shell init overhead

systemd user unit (ssh-agent.service)

Native, no extra packages, clean lifecycle

Requires SSH_AUTH_SOCK export in shell

gpg-agent with SSH support

Already running for GPG/YubiKey, single agent

Configuration complexity, pinentry interactions

Check	Status
Strategy chosen	[ ]
Implemented and tested	[ ]
Keys persist across terminal sessions	[ ]

Check

Status

Strategy chosen

[ ]

Implemented and tested

[ ]

Keys persist across terminal sessions

[ ]

Wired EAP-TLS

Deferred until a wired connection (cable + switch port) is available.

# When ready: create wired 802.1X connection (see Phase 8b for cert workflow)
# nmcli connection add type ethernet con-name "Domus-Wired-EAP-TLS" \
#   ifname enp134s0 \
#   802-1x.eap tls \
#   802-1x.identity "modestus-p16g.inside.domusdigitalis.dev" \
#   802-1x.ca-cert /etc/ssl/certs/domus-ca-chain.pem \
#   802-1x.client-cert /etc/ssl/certs/modestus-p16g-eaptls.pem \
#   802-1x.private-key /etc/ssl/private/modestus-p16g-eaptls.key \
#   802-1x.private-key-password-flags 4

Check	Status
Physical cable connected to managed switch port	[ ]
Wired EAP-TLS connection created	[ ]
ISE authentication verified with netapi	[ ]

Check

Status

Physical cable connected to managed switch port

[ ]

Wired EAP-TLS connection created

[ ]

ISE authentication verified with netapi

[ ]

Btrfs Scrub Schedule

Periodic scrub verifies filesystem integrity by reading all data and checking checksums.

# Create systemd timer for monthly scrub
sudo tee /etc/systemd/system/btrfs-scrub@.timer << 'EOF'
[Unit]
Description=Monthly btrfs scrub on %f

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo tee /etc/systemd/system/btrfs-scrub@.service << 'EOF'
[Unit]
Description=btrfs scrub on %f

[Service]
Type=oneshot
ExecStart=/usr/bin/btrfs scrub start -B %f
IOSchedulingClass=idle
CPUSchedulingPolicy=idle
EOF

# Enable for root and home
sudo systemctl enable btrfs-scrub@-.timer
sudo systemctl enable btrfs-scrub@home.timer

# Verify timers
systemctl list-timers | grep btrfs

Kernel Update Procedure

# 1. Update packages (kernel included)
sudo pacman -Syu

# 2. Verify ESP sync hook ran
ls -la /boot/efi/vmlinuz-linux /boot/efi/initramfs-linux.img

# 3. Reboot into new kernel
sudo reboot

# 4. Verify new kernel loaded
uname -r

# 5. If new kernel breaks: reboot, select LTS from systemd-boot menu
# Press any key during 3-second boot timeout, then select "Arch Linux LTS"

AppArmor Profile Maintenance

After pacman -Syu updates the apparmor package, modified browser profiles may be overwritten with .pacnew files.

# Check for .pacnew files after apparmor update
find /etc/apparmor.d/ -name "*.pacnew" | head

# If .pacnew found: diff against our version
diff /etc/apparmor.d/firefox /etc/apparmor.d/firefox.pacnew

# Re-apply our customizations if overwritten:
# 1. Remove flags=(unconfined)
# 2. Add allow-all baseline
# 3. Add flags=(attach_disconnected)
# 4. Reload: sudo apparmor_parser -r /etc/apparmor.d/firefox

Rollback Procedures

Btrfs Snapshot Rollback

# List available snapshots
sudo btrfs subvolume list /.snapshots

# Rollback to a snapshot (from live USB or LTS kernel):
# 1. Mount the btrfs volume
# 2. Rename current @ to @broken
# 3. Snapshot the target to new @
sudo mount /dev/mapper/cryptroot /mnt
sudo mv /mnt/@ /mnt/@broken
sudo btrfs subvolume snapshot /mnt/@snapshots/<snapshot-name> /mnt/@
sudo umount /mnt
sudo reboot

Kernel Rollback

If mainline kernel breaks, boot into LTS:

Reboot
Press any key during systemd-boot 3-second timeout
Select "Arch Linux LTS"
Report the issue, wait for kernel fix in next pacman -Syu

Check	Status
Borg backup automation deployed	[ ]
Borg timer fires correctly	[ ]
TLP charge thresholds set (40/80)	[ ]
Vault EAP-TLS cert expiration tracked	[ ]
Vault SSH cert renewal documented	[ ]
ESP sync hook validated on real update	[ ]
ssh-agent persistence implemented	[ ]
Btrfs scrub timer enabled (monthly)	[ ]
Kernel update procedure tested	[ ]
AppArmor .pacnew handling documented	[ ]
Rollback procedure tested (snapshot or kernel)	[ ]

Check

Status

Borg backup automation deployed

[ ]

Borg timer fires correctly

[ ]

TLP charge thresholds set (40/80)

[ ]

Vault EAP-TLS cert expiration tracked

[ ]

Vault SSH cert renewal documented

[ ]

ESP sync hook validated on real update

[ ]

ssh-agent persistence implemented

[ ]

Btrfs scrub timer enabled (monthly)

[ ]

Kernel update procedure tested

[ ]

AppArmor .pacnew handling documented

[ ]

Rollback procedure tested (snapshot or kernel)

[ ]