STD-002: Deployment Validation Standards

Systematic post-deployment verification. Every check has a command, expected output, and pass/fail criterion. Organized by domain so the same standard applies whether you’re deploying Arch on a ThinkPad or RHEL on a KVM guest.

Principles

  1. Command + Expected + Criterion. Every check is a triple: the command to run, what the output should look like, and how to judge pass/fail. "It works" is not a criterion.

  2. Organize by domain, not phase. Deployment phases are project-specific; validation domains are universal. A security check is a security check regardless of which phase installed the package.

  3. Platform-specific values are variables. Hardcoded IPs, hostnames, sink IDs, and device paths make standards non-portable. Use descriptions ("the default audio sink") or attributes where possible.

  4. Run as a complete suite. Spot-checking creates blind spots. Run every domain, every time. A check that "probably still works" is a check that hasn’t been verified.

  5. Snapshot after all checks pass. A known-good state is only known-good if you captured it.

Domain 1: System Health

Baseline system state. Run first β€” a failed service or kernel error can invalidate downstream checks.

Check Command Pass Criterion

No failed services

systemctl --failed

0 units listed

No kernel errors

journalctl -b -p err --no-pager | tail -20

No unexpected errors (firmware warnings are typically noise)

Boot time analysis

systemd-analyze

Userspace < 30s to graphical target (kernel time dominated by LUKS unlock is expected)

Slowest services

systemd-analyze blame | head -10

No single service > 15s unless justified (e.g., LUKS, firmware init)

Domain 2: Boot & Encryption

Verify the boot chain end-to-end: kernel loaded, encryption active, filesystems mounted, bootloader entries valid.

Check Command Pass Criterion

Kernel version

uname -r

Expected kernel (mainline or LTS)

LUKS volumes open

lsblk -f | grep -E 'crypto_LUKS|crypt'

All expected encrypted volumes open and mounted

Filesystem subvolumes

sudo btrfs subvolume list / | awk '{print $NF}'

Expected subvolumes present (e.g., @, @snapshots, @var_log)

Swap active

zramctl or swapon --show

Swap device present with expected size and algorithm

Boot entries valid

bootctl list

All entries resolve β€” no "No such file or directory" warnings

ESP kernel sync

diff <(md5sum /boot/vmlinuz-linux | awk '{print $1}') <(md5sum /boot/efi/vmlinuz-linux | awk '{print $1}')

MATCH (only applies when kernels are synced to ESP)

Kernel parameters

cat /proc/cmdline

Expected parameters present (LUKS UUID, rootflags, security modules, GPU modesetting)

The ESP sync check assumes a split-boot layout where kernels on /boot are copied to the ESP. Skip if using a unified ESP mount.

Domain 3: Security

Verify the security posture: mandatory access control, firewall, SSH hardening, attack surface.

Check Command Pass Criterion

LSM stack

cat /sys/kernel/security/lsm

Expected modules present (e.g., capability,landlock,lockdown,yama,apparmor,bpf)

MAC framework active

sudo aa-status | head -5 (AppArmor) or getenforce (SELinux)

Profiles loaded and enforcing / Enforcing mode

MAC in kernel cmdline

grep -E 'apparmor|selinux' /proc/cmdline

Security module enabled at boot (not just userspace)

Critical profiles enforced

sudo aa-status | awk '/in enforce/{p="enforce"} /in unconfined/{p="unconfined"} /firefox|chrome|chromium/{print p": "$0}'

Browsers and sensitive applications show "enforce", not "unconfined"

Firewall status

sudo ufw status verbose or sudo firewall-cmd --state

Active with expected rules

SSH hardening

sudo sshd -T | grep -iE 'permitroot|passwordauth|maxauthtries'

PermitRootLogin no, PasswordAuthentication no (or as policy dictates)

Open ports audit

ss -tlnp | awk 'NR>1{print $4, $6}' | sort

Only expected listeners β€” SSH, localhost services. No unexpected public listeners.

Domain 4: Desktop Environment

Compositor, GPU, audio, bluetooth, session processes. Skip this domain for headless servers.

Check Command Pass Criterion

Display server

echo $XDG_SESSION_TYPE

wayland or x11 as expected

GPU detected

nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader or lspci | grep -i vga

Correct GPU, driver version, VRAM

GPU modesetting

cat /sys/module/nvidia_drm/parameters/modeset or check kernel cmdline

Y or nvidia_drm.modeset=1 present

GPU persistence

systemctl is-active nvidia-persistenced.service

active (NVIDIA only β€” prevents cold-start delays)

Audio β€” not dummy

wpctl status | grep -A5 'Sinks:'

Real audio devices listed β€” NOT "Dummy Output"

Audio playback

pw-play /usr/share/sounds/freedesktop/stereo/bell.oga

Sound plays through expected output device

Audio β€” SOF firmware

journalctl -b --grep='sof|cs35l56|cs42l43' --no-pager | tail -5

Firmware loaded, calibration applied (Intel laptops with SOF audio)

Bluetooth controller

bluetoothctl show | grep -E 'Name|Powered|Address'

Controller present and powered

Session processes

for proc in waybar mako pipewire wireplumber; do pgrep -x $proc > /dev/null && echo "βœ“ $proc" || echo "βœ— $proc"; done

All expected session processes running (adapt list to your compositor)

Monitor config

Compositor-specific: hyprctl monitors, swaymsg -t get_outputs, or xrandr

Correct resolution, refresh rate, scaling

Notifications

notify-send "Validation" "Desktop operational"

Notification appears on screen

Screenshot pipeline

grim -g "$(slurp)" /tmp/test-screenshot.png && ls -lh /tmp/test-screenshot.png

File created (Wayland); adapt for X11

Clipboard

echo "test" | wl-copy && wl-paste

Output matches input

The SOF firmware check is Intel-specific. ThinkPad P-series and T-series with Cirrus Logic CS35L56 speaker amps require sof-firmware β€” without it, you get "Dummy Output" only. The dummy regulator warnings for VDD_B/VDD_AMP are harmless (no ACPI supply regulators exposed).

Audio tool choice

pw-play is the native PipeWire playback tool β€” no dependency on alsa-utils. Use it instead of speaker-test unless you specifically need ALSA-level diagnostics.

# Default sink
pw-play /usr/share/sounds/freedesktop/stereo/bell.oga

# Target a specific sink by ID (find IDs with: wpctl status)
pw-play --target <SINK_ID> /usr/share/sounds/freedesktop/stereo/bell.oga

Domain 5: Development Tools

Languages, package managers, editor config, version control connectivity.

Check Command Pass Criterion

Languages installed

python3 --version && node --version && rustc --version && go version

All expected languages present with expected versions

Package managers

uv --version && npm --version && cargo --version

All expected package managers present

Editor config

nvim --version | head -1 && ls ~/.config/nvim/init.lua

Editor installed, config loaded

AI coding tools

claude --version

Installed and version reported

Git SSH connectivity

ssh -T git@github.com

"Hi <user>! You’ve successfully authenticated"

Git remotes

git -C <repo-path> remote -v | awk '{print $1, $2}' | sort -u

Expected remotes configured (origin, upstream, etc.)

SSH askpass

Non-interactive contexts (tools, scripts, cron) fail when SSH keys require a passphrase prompt. If your workflow includes non-terminal SSH:

# X11/Wayland askpass helper
sudo pacman -S x11-ssh-askpass    # Arch
sudo dnf install openssh-askpass  # RHEL/Fedora

Domain 6: AI Stack

Local inference engine, models, storage. Skip for deployments without local AI.

Check Command Pass Criterion

Inference engine running

systemctl is-active ollama.service

active

Models loaded

ollama list

Expected models present

Model storage

mount | grep ollama-models or du -sh /usr/share/ollama/.ollama/models/

Models on correct partition (not root, if large)

Inference test

ollama run <model> "What is 2+2?" 2>/dev/null | head -1

Coherent response returned

Domain 7: Secrets & Credentials

GPG, SSH keys, certificate validity, encrypted vaults. Every check here touches sensitive material β€” verify presence, not content.

Check Command Pass Criterion

GPG key present

gpg --list-secret-keys --keyid-format long | grep -c sec

β‰₯ 1 secret key

SSH keys present

ls ~/.ssh/id_ed25519_* | wc -l

Expected key count

SSH cert validity

ssh-keygen -L -f ~/.ssh/id_ed25519_vault-cert.pub | grep Valid

Certificate not expired (if Vault SSH certs are used)

age identities

test -f ~/.age/identities && echo "OK" || echo "MISSING"

OK

Password manager

gopass ls | head -5

Store accessible, entries listed

Encrypted vault

ls ~/.credentials/ && echo "mounted" || echo "not mounted"

Mounted (if gocryptfs or similar is used)

Never print secret values during validation. Check for presence and accessibility, not content. Paths to secret stores are acceptable to log; values are not.

Domain 8: Network & Connectivity

WiFi, DNS, certificate expiration. Validates the machine can reach what it needs to reach.

Check Command Pass Criterion

Active connection

nmcli connection show --active

Expected connection active (WiFi EAP-TLS, wired, VPN, etc.)

DNS resolution

dig +short <hostname> or getent hosts <hostname>

IP address returned for internal hostnames

DNS tools available

dig -v or drill -v

Installed (if DNS diagnostics are part of your workflow)

EAP-TLS cert expiration

openssl x509 -in <cert-path> -noout -enddate

notAfter date is > 30 days out

External connectivity

curl -sI github.com | head -1

HTTP/2 200

If dig is unavailable, getent hosts uses the system resolver and works without bind-tools. For full diagnostic capability, install bind (Arch) or bind-utils (RHEL).

Requirements

  1. Every validation check MUST be a triple: command, expected output, pass/fail criterion. "It works" is not a criterion.

  2. Validation MUST be organized by domain (System Health, Boot, Security, Desktop, Dev Tools, AI, Secrets, Network), not by deployment phase.

  3. Platform-specific values (IPs, hostnames, device IDs) MUST NOT be hardcoded in validation commands. Use descriptions or {attribute} references.

  4. Validation MUST be run as a complete suite β€” every domain, every time. Spot-checking is non-compliant.

  5. A clean-state snapshot MUST be captured after all checks pass (btrfs snapshot, LVM snapshot, or equivalent).

  6. Desktop Environment (Domain 4) and AI Stack (Domain 6) domains MAY be skipped for headless server deployments.

  7. Audio validation MUST use pw-play (native PipeWire), not speaker-test (requires alsa-utils).

  8. Secret values MUST NOT be printed during validation. Check for presence and accessibility, not content (STD-006).

Platform Adaptation

Not every check applies to every deployment. Adapt per platform:

Domain Arch Linux Workstation RHEL 9 Server/Workstation Common

Boot

systemd-boot, ESP sync, btrfs

GRUB2, LVM, XFS/ext4

LUKS, kernel params

Security

AppArmor, aa-status

SELinux, getenforce

LSM stack, firewall, SSH

Desktop

Hyprland, PipeWire, pw-play

GNOME, PipeWire or PulseAudio

Skip for headless

GPU

nvidia-open (Blackwell+)

nvidia or nouveau

nvidia-smi, modesetting

Audio firmware

sof-firmware (Intel laptops)

Usually kernel-included

Check wpctl status regardless

Package managers

pacman, yay, uv, npm, cargo

dnf, pip, npm, cargo

Language-specific are universal

Firewall

UFW

firewalld

Verify rules match policy

Clean-State Snapshot

After ALL checks pass, capture a known-good state for rollback.

btrfs

sudo btrfs subvolume snapshot -r / /.snapshots/fresh-deploy-$(date +%Y%m%d)
sudo btrfs subvolume list /.snapshots | tail -1

LVM

sudo lvcreate --size 10G --snapshot --name fresh-deploy-$(date +%Y%m%d) /dev/vg0/root
sudo lvs | grep fresh-deploy

Alternatives

For ext4 without LVM: use Timeshift, rsync-based backup, or Borg before declaring the deployment verified.

Summary Checklist Template

Copy this table into your project’s verification phase and fill in status:

[cols="3,1"]
|===
| Check | Status

| No failed systemd services | [ ]
| Kernel booted (expected version) | [ ]
| Encryption volumes open | [ ]
| Filesystem subvolumes correct | [ ]
| Swap active | [ ]
| MAC framework enforcing | [ ]
| Firewall active with expected rules | [ ]
| SSH hardened | [ ]
| Open ports clean | [ ]
| GPU detected with correct driver | [ ]
| Audio output works (not Dummy Output) | [ ]
| Bluetooth controller present | [ ]
| Desktop compositor running | [ ]
| Session processes running | [ ]
| Languages installed (expected versions) | [ ]
| Editor + config loaded | [ ]
| Git SSH connectivity | [ ]
| AI tools installed | [ ]
| Secrets accessible (GPG, SSH, gopass) | [ ]
| Encrypted vault mountable | [ ]
| Network connected | [ ]
| DNS resolving | [ ]
| Certificates valid (> 30 days) | [ ]
| Clean-state snapshot captured | [ ]
|===