STD-002: Deployment Validation Standards
Systematic post-deployment verification. Every check has a command, expected output, and pass/fail criterion. Organized by domain so the same standard applies whether you’re deploying Arch on a ThinkPad or RHEL on a KVM guest.
Principles
-
Command + Expected + Criterion. Every check is a triple: the command to run, what the output should look like, and how to judge pass/fail. "It works" is not a criterion.
-
Organize by domain, not phase. Deployment phases are project-specific; validation domains are universal. A security check is a security check regardless of which phase installed the package.
-
Platform-specific values are variables. Hardcoded IPs, hostnames, sink IDs, and device paths make standards non-portable. Use descriptions ("the default audio sink") or attributes where possible.
-
Run as a complete suite. Spot-checking creates blind spots. Run every domain, every time. A check that "probably still works" is a check that hasn’t been verified.
-
Snapshot after all checks pass. A known-good state is only known-good if you captured it.
Domain 1: System Health
Baseline system state. Run first β a failed service or kernel error can invalidate downstream checks.
| Check | Command | Pass Criterion |
|---|---|---|
No failed services |
|
0 units listed |
No kernel errors |
|
No unexpected errors (firmware warnings are typically noise) |
Boot time analysis |
|
Userspace < 30s to graphical target (kernel time dominated by LUKS unlock is expected) |
Slowest services |
|
No single service > 15s unless justified (e.g., LUKS, firmware init) |
Domain 2: Boot & Encryption
Verify the boot chain end-to-end: kernel loaded, encryption active, filesystems mounted, bootloader entries valid.
| Check | Command | Pass Criterion |
|---|---|---|
Kernel version |
|
Expected kernel (mainline or LTS) |
LUKS volumes open |
|
All expected encrypted volumes open and mounted |
Filesystem subvolumes |
|
Expected subvolumes present (e.g., @, @snapshots, @var_log) |
Swap active |
|
Swap device present with expected size and algorithm |
Boot entries valid |
|
All entries resolve β no "No such file or directory" warnings |
ESP kernel sync |
|
MATCH (only applies when kernels are synced to ESP) |
Kernel parameters |
|
Expected parameters present (LUKS UUID, rootflags, security modules, GPU modesetting) |
The ESP sync check assumes a split-boot layout where kernels on /boot are copied to the ESP. Skip if using a unified ESP mount.
|
Domain 3: Security
Verify the security posture: mandatory access control, firewall, SSH hardening, attack surface.
| Check | Command | Pass Criterion |
|---|---|---|
LSM stack |
|
Expected modules present (e.g., |
MAC framework active |
|
Profiles loaded and enforcing / Enforcing mode |
MAC in kernel cmdline |
|
Security module enabled at boot (not just userspace) |
Critical profiles enforced |
|
Browsers and sensitive applications show "enforce", not "unconfined" |
Firewall status |
|
Active with expected rules |
SSH hardening |
|
PermitRootLogin no, PasswordAuthentication no (or as policy dictates) |
Open ports audit |
|
Only expected listeners β SSH, localhost services. No unexpected public listeners. |
Domain 4: Desktop Environment
Compositor, GPU, audio, bluetooth, session processes. Skip this domain for headless servers.
| Check | Command | Pass Criterion |
|---|---|---|
Display server |
|
|
GPU detected |
|
Correct GPU, driver version, VRAM |
GPU modesetting |
|
|
GPU persistence |
|
|
Audio β not dummy |
|
Real audio devices listed β NOT "Dummy Output" |
Audio playback |
|
Sound plays through expected output device |
Audio β SOF firmware |
|
Firmware loaded, calibration applied (Intel laptops with SOF audio) |
Bluetooth controller |
|
Controller present and powered |
Session processes |
|
All expected session processes running (adapt list to your compositor) |
Monitor config |
Compositor-specific: |
Correct resolution, refresh rate, scaling |
Notifications |
|
Notification appears on screen |
Screenshot pipeline |
|
File created (Wayland); adapt for X11 |
Clipboard |
|
Output matches input |
The SOF firmware check is Intel-specific. ThinkPad P-series and T-series with Cirrus Logic CS35L56 speaker amps require sof-firmware β without it, you get "Dummy Output" only. The dummy regulator warnings for VDD_B/VDD_AMP are harmless (no ACPI supply regulators exposed).
|
Audio tool choice
pw-play is the native PipeWire playback tool β no dependency on alsa-utils. Use it instead of speaker-test unless you specifically need ALSA-level diagnostics.
# Default sink
pw-play /usr/share/sounds/freedesktop/stereo/bell.oga
# Target a specific sink by ID (find IDs with: wpctl status)
pw-play --target <SINK_ID> /usr/share/sounds/freedesktop/stereo/bell.oga
Domain 5: Development Tools
Languages, package managers, editor config, version control connectivity.
| Check | Command | Pass Criterion |
|---|---|---|
Languages installed |
|
All expected languages present with expected versions |
Package managers |
|
All expected package managers present |
Editor config |
|
Editor installed, config loaded |
AI coding tools |
|
Installed and version reported |
Git SSH connectivity |
|
"Hi <user>! You’ve successfully authenticated" |
Git remotes |
|
Expected remotes configured (origin, upstream, etc.) |
SSH askpass
Non-interactive contexts (tools, scripts, cron) fail when SSH keys require a passphrase prompt. If your workflow includes non-terminal SSH:
# X11/Wayland askpass helper
sudo pacman -S x11-ssh-askpass # Arch
sudo dnf install openssh-askpass # RHEL/Fedora
Domain 6: AI Stack
Local inference engine, models, storage. Skip for deployments without local AI.
| Check | Command | Pass Criterion |
|---|---|---|
Inference engine running |
|
|
Models loaded |
|
Expected models present |
Model storage |
|
Models on correct partition (not root, if large) |
Inference test |
|
Coherent response returned |
Domain 7: Secrets & Credentials
GPG, SSH keys, certificate validity, encrypted vaults. Every check here touches sensitive material β verify presence, not content.
| Check | Command | Pass Criterion |
|---|---|---|
GPG key present |
|
β₯ 1 secret key |
SSH keys present |
|
Expected key count |
SSH cert validity |
|
Certificate not expired (if Vault SSH certs are used) |
age identities |
|
OK |
Password manager |
|
Store accessible, entries listed |
Encrypted vault |
|
Mounted (if gocryptfs or similar is used) |
| Never print secret values during validation. Check for presence and accessibility, not content. Paths to secret stores are acceptable to log; values are not. |
Domain 8: Network & Connectivity
WiFi, DNS, certificate expiration. Validates the machine can reach what it needs to reach.
| Check | Command | Pass Criterion |
|---|---|---|
Active connection |
|
Expected connection active (WiFi EAP-TLS, wired, VPN, etc.) |
DNS resolution |
|
IP address returned for internal hostnames |
DNS tools available |
|
Installed (if DNS diagnostics are part of your workflow) |
EAP-TLS cert expiration |
|
notAfter date is > 30 days out |
External connectivity |
|
HTTP/2 200 |
If dig is unavailable, getent hosts uses the system resolver and works without bind-tools. For full diagnostic capability, install bind (Arch) or bind-utils (RHEL).
|
Requirements
-
Every validation check MUST be a triple: command, expected output, pass/fail criterion. "It works" is not a criterion.
-
Validation MUST be organized by domain (System Health, Boot, Security, Desktop, Dev Tools, AI, Secrets, Network), not by deployment phase.
-
Platform-specific values (IPs, hostnames, device IDs) MUST NOT be hardcoded in validation commands. Use descriptions or
{attribute}references. -
Validation MUST be run as a complete suite β every domain, every time. Spot-checking is non-compliant.
-
A clean-state snapshot MUST be captured after all checks pass (btrfs snapshot, LVM snapshot, or equivalent).
-
Desktop Environment (Domain 4) and AI Stack (Domain 6) domains MAY be skipped for headless server deployments.
-
Audio validation MUST use
pw-play(native PipeWire), notspeaker-test(requiresalsa-utils). -
Secret values MUST NOT be printed during validation. Check for presence and accessibility, not content (STD-006).
Platform Adaptation
Not every check applies to every deployment. Adapt per platform:
| Domain | Arch Linux Workstation | RHEL 9 Server/Workstation | Common |
|---|---|---|---|
Boot |
systemd-boot, ESP sync, btrfs |
GRUB2, LVM, XFS/ext4 |
LUKS, kernel params |
Security |
AppArmor, |
SELinux, |
LSM stack, firewall, SSH |
Desktop |
Hyprland, PipeWire, |
GNOME, PipeWire or PulseAudio |
Skip for headless |
GPU |
|
|
|
Audio firmware |
|
Usually kernel-included |
Check |
Package managers |
pacman, yay, uv, npm, cargo |
dnf, pip, npm, cargo |
Language-specific are universal |
Firewall |
UFW |
firewalld |
Verify rules match policy |
Clean-State Snapshot
After ALL checks pass, capture a known-good state for rollback.
btrfs
sudo btrfs subvolume snapshot -r / /.snapshots/fresh-deploy-$(date +%Y%m%d)
sudo btrfs subvolume list /.snapshots | tail -1
LVM
sudo lvcreate --size 10G --snapshot --name fresh-deploy-$(date +%Y%m%d) /dev/vg0/root
sudo lvs | grep fresh-deploy
Alternatives
For ext4 without LVM: use Timeshift, rsync-based backup, or Borg before declaring the deployment verified.
Summary Checklist Template
Copy this table into your project’s verification phase and fill in status:
[cols="3,1"]
|===
| Check | Status
| No failed systemd services | [ ]
| Kernel booted (expected version) | [ ]
| Encryption volumes open | [ ]
| Filesystem subvolumes correct | [ ]
| Swap active | [ ]
| MAC framework enforcing | [ ]
| Firewall active with expected rules | [ ]
| SSH hardened | [ ]
| Open ports clean | [ ]
| GPU detected with correct driver | [ ]
| Audio output works (not Dummy Output) | [ ]
| Bluetooth controller present | [ ]
| Desktop compositor running | [ ]
| Session processes running | [ ]
| Languages installed (expected versions) | [ ]
| Editor + config loaded | [ ]
| Git SSH connectivity | [ ]
| AI tools installed | [ ]
| Secrets accessible (GPG, SSH, gopass) | [ ]
| Encrypted vault mountable | [ ]
| Network connected | [ ]
| DNS resolving | [ ]
| Certificates valid (> 30 days) | [ ]
| Clean-state snapshot captured | [ ]
|===
Related
-
Change Management β the verify-change-verify principle that motivates systematic validation
-
Root Cause Analysis β what to do when validation fails
-
ThinkPad P16g Phase 11 β the originating implementation (Arch Linux, 30+ checks)
-
P16g Validation Run (2026-04-06) β first execution of this standard