Phase 13: Maintenance & Automation

Ongoing operational tasks that keep the P16g healthy after initial deployment. These are not one-time setup items — they require periodic attention or automation.

Borg Backup Automation

Currently: Borg backup is manual (borg create from shell). Automate with a systemd timer.

Create systemd service

# ~/.config/systemd/user/borg-backup.service
cat << 'EOF'
[Unit]
Description=Borg backup to NAS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=%h/bin/borg-backup.sh
EOF

Create systemd timer

# ~/.config/systemd/user/borg-backup.timer
cat << 'EOF'
[Unit]
Description=Run Borg backup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

Enable

systemctl --user enable --now borg-backup.timer

# Verify timer registered
systemctl --user list-timers | grep borg

Check Status

Check	Status
`borg-backup.sh` script in `~/bin/` and working	[ ]
systemd user service created	[ ]
systemd user timer created and enabled	[ ]
Timer fires correctly (`systemctl --user list-timers`)	[ ]

borg-backup.sh script in ~/bin/ and working

[ ]

systemd user service created

[ ]

systemd user timer created and enabled

[ ]

Timer fires correctly (systemctl --user list-timers)

[ ]

TLP Battery Management

ThinkPad-specific power management. Set charge thresholds to extend battery lifespan.

# Verify TLP is installed and running
sudo tlp-stat -b

Configure charge thresholds

# /etc/tlp.conf — set charge thresholds
# Start charging below 40%, stop at 80%
sudo sed -i 's/^#\?START_CHARGE_THRESH_BAT0=.*/START_CHARGE_THRESH_BAT0=40/' /etc/tlp.conf
sudo sed -i 's/^#\?STOP_CHARGE_THRESH_BAT0=.*/STOP_CHARGE_THRESH_BAT0=80/' /etc/tlp.conf

# Apply changes
sudo tlp start

# Verify thresholds active
sudo tlp-stat -b | grep -E 'charge_control|threshold'

Check Status

Check	Status
TLP installed and running	[ ]
Charge thresholds set (40/80)	[ ]
Thresholds verified with `tlp-stat -b`	[ ]

TLP installed and running

[ ]

Charge thresholds set (40/80)

[ ]

Thresholds verified with tlp-stat -b

[ ]

Vault Certificate Renewal

EAP-TLS certificates from Vault PKI have a finite TTL. Renewal workflow:

# Check current cert expiration
openssl x509 -in /etc/ssl/certs/modestus-p16g-eaptls.pem -noout -enddate

# When expired: re-issue from Vault (see Phase 8b for full workflow)
# 1. Load Vault credentials
# 2. vault write pki_int/issue/domus-client common_name="modestus-p16g.inside.domusdigitalis.dev"
# 3. Extract cert + key with jq
# 4. Install to /etc/ssl/certs and /etc/ssl/private
# 5. Restart NetworkManager

Check	Status
Current cert expiration date known	[ ]
Renewal workflow documented (xref Phase 8b)	[ ]
Calendar reminder set for renewal	[ ]

Check

Status

Current cert expiration date known

[ ]

Renewal workflow documented (xref Phase 8b)

[ ]

Calendar reminder set for renewal

[ ]

Vault SSH Certificate Renewal

SSH certificates issued by Vault have an 8-hour TTL and must be renewed per session.

# Check cert validity
ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep -E 'Valid|Principal'

Vault SSH cert for machine-to-machine auth is currently blocked by VLAN anti-pivot segmentation. Using authorized_keys + YubiKey FIDO2 as workaround.

Pacman Hook Validation

The ESP kernel sync hook was written in Phase 4 but has not been tested with a real kernel update.

# Trigger on next kernel update
sudo pacman -Syu

# After kernel update: verify kernels synced to ESP
ls -la /boot/efi/vmlinuz-linux /boot/efi/initramfs-linux.img

Check	Status
Pacman hook fires on kernel update	[ ]
Kernels synced to ESP after update	[ ]
Both default and LTS kernels present	[ ]

Check

Status

Pacman hook fires on kernel update

[ ]

Kernels synced to ESP after update

[ ]

Both default and LTS kernels present

[ ]

ssh-agent Persistence

Currently: SSH keys require manual ssh-add + gopass passphrase every session. Three options under evaluation:

Option Pros Cons

Option	Pros	Cons
`keychain`	Simple, well-tested, re-uses agents across sessions	Extra package, shell init overhead
systemd user unit (`ssh-agent.service`)	Native, no extra packages, clean lifecycle	Requires `SSH_AUTH_SOCK` export in shell
gpg-agent with SSH support	Already running for GPG/YubiKey, single agent	Configuration complexity, pinentry interactions

keychain

Simple, well-tested, re-uses agents across sessions

Extra package, shell init overhead

systemd user unit (ssh-agent.service)

Native, no extra packages, clean lifecycle

Requires SSH_AUTH_SOCK export in shell

gpg-agent with SSH support

Already running for GPG/YubiKey, single agent

Configuration complexity, pinentry interactions

Check	Status
Strategy chosen	[ ]
Implemented and tested	[ ]
Keys persist across terminal sessions	[ ]

Check

Status

Strategy chosen

[ ]

Implemented and tested

[ ]

Keys persist across terminal sessions

[ ]

Wired EAP-TLS

Deferred until a wired connection (cable + switch port) is available.

# When ready: create wired 802.1X connection (see Phase 8b for cert workflow)
# nmcli connection add type ethernet con-name "Domus-Wired-EAP-TLS" \
#   ifname enp134s0 \
#   802-1x.eap tls \
#   802-1x.identity "modestus-p16g.inside.domusdigitalis.dev" \
#   802-1x.ca-cert /etc/ssl/certs/domus-ca-chain.pem \
#   802-1x.client-cert /etc/ssl/certs/modestus-p16g-eaptls.pem \
#   802-1x.private-key /etc/ssl/private/modestus-p16g-eaptls.key \
#   802-1x.private-key-password-flags 4

Check	Status
Physical cable connected to managed switch port	[ ]
Wired EAP-TLS connection created	[ ]
ISE authentication verified with netapi	[ ]

Check

Status

Physical cable connected to managed switch port

[ ]

Wired EAP-TLS connection created

[ ]

ISE authentication verified with netapi

[ ]

Btrfs Scrub Schedule

Periodic scrub verifies filesystem integrity by reading all data and checking checksums.

# Create systemd timer for monthly scrub
sudo tee /etc/systemd/system/btrfs-scrub@.timer << 'EOF'
[Unit]
Description=Monthly btrfs scrub on %f

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo tee /etc/systemd/system/btrfs-scrub@.service << 'EOF'
[Unit]
Description=btrfs scrub on %f

[Service]
Type=oneshot
ExecStart=/usr/bin/btrfs scrub start -B %f
IOSchedulingClass=idle
CPUSchedulingPolicy=idle
EOF

# Enable for root and home
sudo systemctl enable btrfs-scrub@-.timer
sudo systemctl enable btrfs-scrub@home.timer

# Verify timers
systemctl list-timers | grep btrfs

Manual btrfs subvolume snapshot works but requires discipline before every change. Snapper automates this — particularly valuable given the P16g’s history of kernel update regressions (INC-2026-05-23-002, CR-2026-06-19-p16g-i915-psr-persistence).

Install

# Before
command -v snapper

# Change
sudo pacman -S snapper snap-pac

# After
snapper --version

snap-pac is the pacman hook that triggers snapper automatically on every pacman -Syu. Every kernel update, mesa upgrade, or hyprland bump gets a pre/post snapshot pair without manual intervention.

Configure for Root Subvolume

Snapper expects to manage /.snapshots itself. If the @snapshots subvolume is already mounted there, snapper’s create-config will conflict. Handle this:

Before — check current state

findmnt /.snapshots -o SOURCE,TARGET,FSTYPE
sudo btrfs subvolume list / | awk '$NF ~ /snapshots/ {print}'

If @snapshots exists and is mounted at /.snapshots:

# Unmount the existing subvolume
sudo umount /.snapshots

# Create snapper config (this creates its own .snapshots subvolume)
sudo snapper -c root create-config /

# Delete snapper's auto-created subvolume (we want to keep @snapshots)
sudo btrfs subvolume delete /.snapshots

# Recreate the mountpoint and remount our @snapshots
sudo mkdir /.snapshots
sudo mount -a

If @snapshots does not exist:

# Simpler path — let snapper create everything
sudo snapper -c root create-config /

After — verify config created

sudo snapper -c root list

Configure Snapshot Policy

Edit snapper config

sudo nvim /etc/snapper/configs/root

Recommended settings for a workstation

# Timeline snapshots — hourly, pruned aggressively
TIMELINE_CREATE="yes"
TIMELINE_LIMIT_HOURLY="5"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="2"
TIMELINE_LIMIT_MONTHLY="1"
TIMELINE_LIMIT_YEARLY="0"

# Cleanup — prevent disk fill
NUMBER_CLEANUP="yes"
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="50"
NUMBER_LIMIT_IMPORTANT="10"

Enable Timers

sudo systemctl enable --now snapper-timeline.timer
sudo systemctl enable --now snapper-cleanup.timer

# Verify timers active
systemctl list-timers | grep snapper

Verify snap-pac Integration

After the next pacman -Syu, verify pre/post snapshot pair was created:

sudo snapper -c root list --type pre-post

Expected output (after a pacman transaction)

Type   | # | Pre # | Date                     | Cleanup  | Description     | Userdata
-------+---+-------+--------------------------+----------+-----------------+---------
pre    | 1 |       | Thu Jun 19 10:00:00 2026 | number   | pacman -Syu     |
post   | 2 |     1 | Thu Jun 19 10:05:00 2026 | number   | pacman -Syu     |

Snapper Rollback (Post-Update)

If a pacman -Syu breaks the system and snapper has a pre-update snapshot:

# List snapshots — find the pre-update number
sudo snapper -c root list

# Compare what changed between pre and post
sudo snapper -c root status <pre-number>..<post-number>

# Undo the changes (restores files to pre-update state)
sudo snapper -c root undochange <pre-number>..<post-number>

# Reboot
sudo reboot

For a full subvolume rollback (if undochange is insufficient), use the manual btrfs procedure in the Rollback Procedures section below.

Check Status

Check	Status
`snapper` installed	[ ]
`snap-pac` installed (pacman hook)	[ ]
`snapper -c root create-config /` completed	[ ]
Timeline policy configured	[ ]
`snapper-timeline.timer` enabled	[ ]
`snapper-cleanup.timer` enabled	[ ]
First `pacman -Syu` creates pre/post pair	[ ]

snapper installed

[ ]

snap-pac installed (pacman hook)

[ ]

snapper -c root create-config / completed

[ ]

Timeline policy configured

[ ]

snapper-timeline.timer enabled

[ ]

snapper-cleanup.timer enabled

[ ]

First pacman -Syu creates pre/post pair

[ ]

Kernel Update Procedure

# 1. Update packages (kernel included)
# snap-pac will automatically create pre/post snapshots
sudo pacman -Syu

# 2. Verify ESP sync hook ran
ls -la /boot/efi/vmlinuz-linux /boot/efi/initramfs-linux.img

# 3. Verify /etc/kernel/cmdline preserved custom parameters
# This file is the persistent source for kernel-install.
# Without it, kernel updates strip custom boot parameters (learned via CR-2026-06-19).
grep -o 'i915.enable_psr=[0-9]' /etc/kernel/cmdline
# Expected: i915.enable_psr=0

# 4. Verify ESP boot entry has custom parameters
grep -o 'i915.enable_psr=[0-9]' /boot/efi/loader/entries/arch.conf
# Expected: i915.enable_psr=0
# If missing: kernel-install did not read /etc/kernel/cmdline — investigate

# 5. Reboot into new kernel
sudo reboot

# 6. Verify new kernel loaded
uname -r

# 7. Verify PSR still disabled (critical for Arrow Lake-S display stability)
grep -o 'i915.enable_psr=[0-9]' /proc/cmdline

# 8. If new kernel breaks: reboot, select LTS from systemd-boot menu
# Press any key during 3-second boot timeout, then select "Arch Linux LTS"
#
# If snapper is configured, you can also undo the update:
# sudo snapper -c root undochange <pre-number>..<post-number>

AppArmor Profile Maintenance

After pacman -Syu updates the apparmor package, modified browser profiles may be overwritten with .pacnew files.

# Check for .pacnew files after apparmor update
find /etc/apparmor.d/ -name "*.pacnew" | head

# If .pacnew found: diff against our version
diff /etc/apparmor.d/firefox /etc/apparmor.d/firefox.pacnew

# Re-apply our customizations if overwritten:
# 1. Remove flags=(unconfined)
# 2. Add allow-all baseline
# 3. Add flags=(attach_disconnected)
# 4. Reload: sudo apparmor_parser -r /etc/apparmor.d/firefox

Rollback Procedures

Three tiers, escalating in scope. Try the least destructive option first.

Tier 1: Snapper Undo (Surgical)

Reverses specific file changes from a pacman transaction without replacing the entire subvolume. Preferred when snapper is configured and the pre/post snapshot pair exists.

# List pre/post snapshot pairs
sudo snapper -c root list --type pre-post

# See what changed
sudo snapper -c root status <pre-number>..<post-number>

# Undo the changes
sudo snapper -c root undochange <pre-number>..<post-number>

# Reboot
sudo reboot

Tier 2: Btrfs Subvolume Rollback (Full Root Restore)

Replaces the entire @ subvolume with a snapshot. Use when snapper undo is insufficient or the system is unbootable. Requires LTS kernel or live USB.

# List available snapshots
sudo btrfs subvolume list /.snapshots

# Rollback to a snapshot (from live USB or LTS kernel):
# 1. Mount the btrfs volume
# 2. Rename current @ to @broken
# 3. Snapshot the target to new @
sudo mount /dev/mapper/cryptroot /mnt
sudo mv /mnt/@ /mnt/@broken
sudo btrfs subvolume snapshot /mnt/@snapshots/<snapshot-name> /mnt/@
sudo umount /mnt
sudo reboot

/home is on a separate LUKS volume (crypthome) with its own @home subvolume. Root rollback does not touch user data.

Post-rollback cleanup

# Remove the broken subvolume after confirming restoration
sudo mount /dev/mapper/cryptroot /mnt
sudo btrfs subvolume delete /mnt/@broken
sudo umount /mnt

Tier 3: Kernel Rollback (Boot-Level)

If mainline kernel breaks, boot into LTS:

Reboot
Press any key during systemd-boot 3-second timeout
Select "Arch Linux LTS"
Report the issue, wait for kernel fix in next pacman -Syu

Verify LTS boot entry also has custom kernel parameters (i915.enable_psr=0, etc.). Both ESP entries (arch.conf and arch-lts.conf) must be maintained — see CR-2026-06-19-p16g-i915-psr-persistence.

Check Status

Check	Status
Borg backup automation deployed	[ ]
Borg timer fires correctly	[ ]
TLP charge thresholds set (40/80)	[ ]
Vault EAP-TLS cert expiration tracked	[ ]
Vault SSH cert renewal documented	[ ]
ESP sync hook validated on real update	[ ]
`/etc/kernel/cmdline` exists with all custom parameters	[ ]
ssh-agent persistence implemented	[ ]
Btrfs scrub timer enabled (monthly)	[ ]
Snapper configured with snap-pac	[ ]
Snapper timeline + cleanup timers enabled	[ ]
Kernel update procedure tested (with cmdline verification)	[ ]
AppArmor .pacnew handling documented	[ ]
Rollback procedure tested (snapper, snapshot, or kernel)	[ ]

Borg backup automation deployed

[ ]

Borg timer fires correctly

[ ]

TLP charge thresholds set (40/80)

[ ]

Vault EAP-TLS cert expiration tracked

[ ]

Vault SSH cert renewal documented

[ ]

ESP sync hook validated on real update

[ ]

/etc/kernel/cmdline exists with all custom parameters

[ ]

ssh-agent persistence implemented

[ ]

Btrfs scrub timer enabled (monthly)

[ ]

Snapper configured with snap-pac

[ ]

Snapper timeline + cleanup timers enabled

[ ]

Kernel update procedure tested (with cmdline verification)

[ ]

AppArmor .pacnew handling documented

[ ]

Rollback procedure tested (snapper, snapshot, or kernel)

[ ]