WRKLOG-2026-02-20

Summary

Vault SSH Certificate Authority deployment completed. Successfully authenticated to modestus-aw using Vault-signed certificate.

Vault SSH CA Deployment

Phase 4: CA Deployed to modestus-aw

Two-hop transfer from workstation:

scp vault-01:/tmp/vault-ssh-ca.pub /tmp/
scp /tmp/vault-ssh-ca.pub modestus-aw:/tmp/

Installed on modestus-aw:

sudo mv /tmp/vault-ssh-ca.pub /etc/ssh/vault-ca.pub
sudo chmod 644 /etc/ssh/vault-ca.pub
sudo tee /etc/ssh/sshd_config.d/vault-ca.conf << 'CONF'
TrustedUserCAKeys /etc/ssh/vault-ca.pub
CONF
sudo sshd -t && sudo systemctl restart sshd

Phase 5: Key Generation and Signing

Generated dedicated Vault-signing key:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_vault -C "vault-signed-20260220"

Stored passphrase in gopass v3:

gopass generate -p v3/domains/d000/identity/ssh/id_ed25519_vault 32

Signed from vault-01 (localhost-only Vault):

cat ~/.ssh/id_ed25519_vault.pub | ssh vault-01 "cat > /tmp/id_ed25519_vault.pub"
ssh vault-01
export VAULT_ADDR='http://127.0.0.1:8200'
vault write -field=signed_key ssh/sign/domus-client \
  public_key=@/tmp/id_ed25519_vault.pub > /tmp/id_ed25519_vault-cert.pub
exit
scp vault-01:/tmp/id_ed25519_vault-cert.pub ~/.ssh/

Phase 6: Successful Test

ssh -i ~/.ssh/id_ed25519_vault -o CertificateFile=~/.ssh/id_ed25519_vault-cert.pub modestus-aw

Result: Connected successfully with Vault-signed certificate.

Certificate validity:

Valid: from 2026-02-20T00:14:05 to 2026-02-20T08:14:35

Issues Discovered

dsec Vault Token Stale

dsource d000 dev/vault loads VAULT_ROOT_TOKEN but the stored value is invalid/expired.

Current working token (from vault token lookup on vault-01):

hvs.rkZPXMDnMGde779ln0jloyvs

Workaround: Sign from vault-01 directly (Option A in runbook).

Fix: Update VAULT_ROOT_TOKEN in dsec:

dsec edit d000 dev/vault

Update the VAULT_ROOT_TOKEN line with the current value. Keep the structure - only the token value changed.

hvault Alias Needed

Local vault command is aliased to gocryptfs vault manager. HashiCorp Vault binary at /usr/bin/vault.

alias hvault=/usr/bin/vault

Add to ~/.zshrc for persistence.

Runbook Updates

Updated vault-ssh-ca.adoc with:

  • Three signing options (A: vault-01, B: SSH tunnel, C: External TLS)

  • gopass v3 heredoc pattern for passphrase storage

  • Key existence check before generation

  • dsource workflow with proper variable export

Alias Conflict Resolution

Renamed gocryptfs vault manager alias to avoid conflict with HashiCorp Vault:

# Before (conflict)
alias vault='~/atelier/_vaults/bin/vault-manager.sh'

# After (clear separation)
alias gcvault='~/atelier/_vaults/bin/vault-manager.sh'
alias vault=/usr/bin/vault

Updated in dotfiles-optimus (both atelier.shell and atelier.fish).

Vault External TLS Runbook Created

Assessment: Not a Setback

Localhost-only Vault is a valid security posture (bastion host pattern). External TLS is a 30-60 minute configuration task, not a redesign.

Phase Time Notes

Issue TLS cert

5 min

Same commands as PKI cert issuance

Install cert

2 min

Copy files to /opt/vault/tls/

Update vault.hcl

5 min

Add TLS listener block

Restart + unseal

5 min

Standard procedure

Client config

10 min

CA chain + dsec update

Test

5 min

vault status from workstation

Total: ~30 minutes. SSH CA deployment was the hard part - this just enables the network listener.

Created vault-tls-external.adoc for enterprise-grade Vault access:

Architecture:

Current:  Workstation → SSH tunnel → vault-01:8200 (localhost)
Target:   Workstation → HTTPS → vault-01:8200 (TLS, network)

Key phases: 1. Issue TLS cert from Vault PKI for Vault itself 2. Install cert in /opt/vault/tls/ 3. Update vault.hcl with TLS listener on 0.0.0.0:8200 4. Restart Vault, unseal 5. Configure clients with CA chain 6. Update dsec with VAULT_ADDR, VAULT_CACERT

Updated vault-ssh-ca.adoc: - Option C now links to vault-tls-external.adoc - Automation script validates all required env vars - No SSH tunnels required for enterprise deployment

Morning: Spanish Tutoring (Rescheduled)

Carried over from 2026-02-14 - teacher rescheduled to tomorrow morning.

  • Practice Spanish oral presentation (Ciudad vs Campo)

Phase 7: Access Policy and AppRole

Created restricted policy for SSH signing (non-root usage):

vault policy write ssh-client - << 'EOF'
path "ssh/sign/domus-client" {
  capabilities = ["create", "update"]
}
path "ssh/config/ca" {
  capabilities = ["read"]
}
EOF

Enabled AppRole auth and created role:

vault auth enable approle
vault write auth/approle/role/ssh-user \
  token_policies="ssh-client" \
  token_ttl="1h" \
  token_max_ttl="4h"

Updated vault-ssh-ca.adoc with missing step (7.3 Enable AppRole before 7.4 Create Role).

Infrastructure Rename: certmgr-01 → vault-01

Rationale: Enterprise naming clarity. "vault-01" indicates primary Vault server, enables clean HA naming (vault-01, vault-02, vault-03).

Documentation updated:

Item Count

antora.yml attributes

7 (vault-primary-hostname, vault-addr, etc.)

.adoc file references

189

D2 diagrams

14 (re-rendered to SVG)

Batch replace command:

grep -rl "certmgr-01" docs/ | xargs sed -i 's/certmgr-01/vault-01/g'
for f in docs/asciidoc/modules/ROOT/images/diagrams/*.d2; do
  d2 "$f" "${f%.d2}.svg"
done

Completed (actual infrastructure):

  • Rename VM hostname (hostnamectl set-hostname vault-01…​)

  • Update DNS (BIND forward + reverse zones, pfSense overrides)

  • Update workstation SSH config

  • Update dsec references

Server Rename Execution (Live Session)

Phase 1: VM Hostname

sudo hostnamectl set-hostname vault-01.inside.domusdigitalis.dev
echo "10.50.1.60  vault-01.inside.domusdigitalis.dev vault-01" | sudo tee -a /etc/hosts

Phase 2: BIND DNS

Check current state with awk:

sudo awk '/certmgr-01/ {print NR": "$0}' /var/named/inside.domusdigitalis.dev.zone

Extract SOA record (multi-line range pattern):

sudo awk '/SOA/,/\)/ {print NR": "$0}' /var/named/inside.domusdigitalis.dev.zone

Preview sed changes before applying:

sudo sed -n 's/certmgr-01/vault-01/p' /var/named/inside.domusdigitalis.dev.zone

Phase 3: pfSense (jq as SQL for JSON)

# Query like SELECT * WHERE host LIKE '%certmgr%'
netapi pfsense dns list --format json | jq '.[] | select(.host | test("certmgr"))'

# Update entries
netapi pfsense dns update --id 6 -h vault-01 -d inside.domusdigitalis.dev -i 10.50.1.60 --descr "HashiCorp Vault primary"

# Verify empty (all deleted)
netapi pfsense dns list --format json | jq '.[] | select(.host | test("certmgr"))'

Phase 4: SSH Config

# View with character class [12] (not [1|2] - pipe is literal inside brackets)
awk '/certmgr-0[12]/,/^$/ {print NR": "$0}' ~/.ssh/config

# Preview and apply
sed -n 's/certmgr-01/vault-01/p' ~/.ssh/config
sed -i 's/certmgr-01/vault-01/g' ~/.ssh/config

# Remove stale known_hosts
ssh-keygen -R certmgr-01.inside.domusdigitalis.dev 2>/dev/null

# Verify
ssh vault-01 "hostname"

Phase 5: dsec

dsec edit d000 dev/vault
# In neovim: :%s/certmgr-01/vault-01/gc

NVIM_APPNAME Configuration for dsec

Goal: Make dsec use domus-instrumentum (custom neovim config) as editor.

Check current state:

awk '/NVIM_APPNAME|EDITOR/ {print NR": "$0}' ~/.zshrc

Insert NVIM_APPNAME after EDITOR (line 31):

sed -i '31a export NVIM_APPNAME="nvim-domus"' ~/.zshrc

Verify range:

awk 'NR>=31&&NR<=33 {print NR": "$0}' ~/.zshrc

Result:

31: export EDITOR='nvim'
32: export NVIM_APPNAME="nvim-domus"
33: export VISUAL='nvim'

Key learning: Neovim looks for config in ~/.config/$NVIM_APPNAME/ when set. Symlink nvim-domus → domus-instrumentum already exists.

ASCII Diagram Replacement

Replaced ASCII box-drawing diagrams with D2:

  • vault-tls-architecture.d2 → Shows current (localhost) vs target (external TLS) state

  • Professional styling with color-coded sections

Policy: No ASCII diagrams - use D2/Mermaid with SVG output.

Follow-up Tasks

  • Update dsec with current root token

  • ~~Add alias hvault=/usr/bin/vault to /.zshrc~ → Reclaimed vault directly

  • ~~Create Vault External TLS runbook~~ → vault-tls-external.adoc

  • ~~Phase 7 AppRole~~ → ssh-client policy + ssh-user role created

  • ~~Documentation rename~~ → certmgr-01 → vault-01 (82 files)

  • ~~Rename actual VM~~ → hostname, BIND, pfSense, SSH config, dsec all updated

  • EXECUTE Vault External TLS runbook on vault-01 ← IN PROGRESS

  • Deploy SSH CA to additional hosts (nas-01, kvm-01, etc.)

Vault External TLS Execution (Live Session)

Phase 1: Certificate Issuance

DNS validation before cert issuance:

dig +short vault-01.inside.domusdigitalis.dev
dig +short -x 10.50.1.60

PKI role constraints check:

vault read pki_int/roles/domus-server | awk '/allowed_domains|allow_subdomains|allow_bare_domains|allow_localhost/'

Key finding: allow_bare_domains: false - short hostnames not allowed (correct enterprise practice).

Certificate issued (FQDN only, no short hostname):

vault write -format=json pki_int/issue/domus-server \
  common_name="vault-01.inside.domusdigitalis.dev" \
  alt_names="localhost" \
  ip_sans="10.50.1.60,127.0.0.1" \
  ttl="8760h" > /tmp/vault-tls-cert.json

jq Exploration Skills

Explore JSON structure before extracting:

# Show keys
jq '.data | keys' /tmp/vault-tls-cert.json

# Show keys with types
jq -r '.data | to_entries[] | "\(.key): \(.value | type)"' /tmp/vault-tls-cert.json

# Preview values (first 50 chars)
jq -r '.data | to_entries[] | "\(.key): \(.value | tostring | .[0:50])..."' /tmp/vault-tls-cert.json

Why no colors with -r: Raw output strips JSON formatting. To get colors with custom output, keep it as JSON or pipe to jq again.

Certificate verification via jq + openssl pipeline:

jq -r '.data.certificate' /tmp/vault-tls-cert.json | openssl x509 -noout -subject -dates

Phase 2: Certificate Installation

File verification with for loop:

for f in /tmp/vault-tls.{crt,key} /tmp/vault-ca-chain.crt; do
  [[ -s "$f" ]] && echo "✓ $f ($(wc -c < "$f") bytes)" || echo "✗ $f MISSING"
done

Installation verification (sudo required - 750 permissions):

sudo ls -la /opt/vault/tls/ | awk 'NR>1 {printf "%-12s %-6s %-6s %8s  %s\n", $1, $3, $4, $5, $NF}'
Result
-rw-r-----   vault  vault     4248  ca-chain.crt
-rw-r-----   vault  vault     1879  vault.crt
-rw-------   vault  vault     1675  vault.key

Unicode Symbols for Terminal Scripts

Symbol Unicode Bash

✓

U+2713

$'\u2713'

✗

U+2717

$'\u2717'

Phase 3: Configuration Complete

Vault 1.20+ Breaking Change Discovered:

Error: disable_mlock must be configured 'true' or 'false'

  • Vault 1.20+ requires explicit disable_mlock setting

  • Safe to set true when LUKS disk encryption is in use

  • Added to runbook with IMPORTANT admonition

Security Hardening Applied:

  • Bound to 10.50.1.60:8200 instead of 0.0.0.0:8200

  • Only accepts connections on specific interface

Phase 4: CA Chain Permission Fix

Issue: After restart, vault status failed with permission denied:

Error loading CA File: open /opt/vault/tls/ca-chain.crt: permission denied

Root Cause: CA chain file has 640 permissions (vault:vault). The ansible user cannot read it.

Fix: Copy CA chain to world-readable location:

sudo cp /opt/vault/tls/ca-chain.crt /etc/ssl/certs/DOMUS-CA-CHAIN.pem
sudo chmod 644 /etc/ssl/certs/DOMUS-CA-CHAIN.pem

Key Learning: CA chains are public information - meant to be distributed to all clients. Only private keys need restricted permissions.

Updated environment:

export VAULT_ADDR='https://10.50.1.60:8200'
export VAULT_CACERT='/etc/ssl/certs/DOMUS-CA-CHAIN.pem'

CRITICAL: Storage Backend Mismatch

Issue: After applying new config with storage "raft", Vault showed Initialized: false.

Root Cause: Original Vault was configured with storage "file". Storage backends are NOT interchangeable - the data format is completely different.

Backend Use Case Data Format

storage "file"

Single node

Directory structure under path

storage "raft"

HA cluster

Raft consensus log + BoltDB

Fix: Check backup config and preserve original storage backend:

awk '/storage/ {print NR": "$0}' /etc/vault.d/vault.hcl.bak.*

If it shows storage "file", keep using file storage with TLS:

storage "file" {
  path = "/opt/vault/data"
}

Lesson: Always check existing storage backend before changing Vault config. Changing backends requires proper migration (export/import), not just config change.

Phase 5: External TLS COMPLETE

Workstation test successful:

export VAULT_ADDR='https://vault-01.inside.domusdigitalis.dev:8200'
export VAULT_CACERT='/etc/ssl/certs/DOMUS-CA-CHAIN.pem'
/usr/bin/vault status | awk 'NR <= 5'
Result
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false

Vault is now accessible over TLS from the network without SSH tunnels.

Phase 6: dsec Configuration

Issue: Quotes embedded in values caused path errors:

echo "$VAULT_CACERT" | xxd | head -1
00000000: 222f 6574 632f 7373 6c2f 6365 7274 732f  "/etc/ssl/certs/

The 22 hex = " character. Vault was looking for "/etc/ssl/…​" with literal quotes.

Fix: Remove quotes around values in dsec file:

# Wrong (quotes embedded)
VAULT_ADDR="https://vault-01.inside.domusdigitalis.dev:8200"

# Correct (no quotes)
VAULT_ADDR=https://vault-01.inside.domusdigitalis.dev:8200

Verification:

eval "$(dsource d000 dev/vault)"
/usr/bin/vault status | awk 'NR <= 5'
Result
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false

Vault External TLS deployment COMPLETE.

Summary: Vault External TLS

Phase Status

1. Certificate issuance

✓ Complete

2. Certificate installation

✓ Complete

3. Vault configuration

✓ Complete (file storage, not raft)

4. Restart and unseal

✓ Complete

5. Workstation test

✓ Complete

6. dsec configuration

✓ Complete

Key learnings:

  1. Storage backends (file vs raft) are NOT interchangeable

  2. CA chains are public - copy to /etc/ssl/certs/ for client access

  3. Vault 1.20+ requires explicit disable_mlock

  4. dsec values should NOT have embedded quotes

  5. Use xxd to debug string encoding issues

SSH CA Validation (Session 2)

Validating Vault SSH CA on infrastructure hosts: kvm-01, bind-01, ipsk-mgr-01.

Certificate Status Check

ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub | awk '/Type:|Valid:|Principals:/{print} /Principals:/{getline; print}'
Result
Type: ssh-ed25519-cert-v01@openssh.com user certificate
Valid: from 2026-02-20T17:22:55 to 2026-02-21T17:23:25
Principals:
        adminerosado
        evanusmodestus

Issue: GPG Pinentry Terminal Too Small

Error:

gpg: public key decryption failed: Screen or window too small
gpg: decryption failed: Screen or window too small

Root cause: pinentry-curses requires minimum terminal size (~80x24).

Fix: Switch to pinentry-tty (no dialog box):

echo "pinentry-program /usr/bin/pinentry-tty" >> ~/.gnupg/gpg-agent.conf && gpgconf --kill gpg-agent

Alternative: Use pinentry-gnome3 for GUI dialog:

echo "pinentry-program /usr/bin/pinentry-gnome3" >> ~/.gnupg/gpg-agent.conf && gpgconf --kill gpg-agent

Issue: Wrong Key Offered First

Symptom:

Enter passphrase for key '/home/evanusmodestus/.ssh/id_ed25519_sk_rk_d000':
Connection closed by 192.168.1.225 port 22

Root cause: SSH config for kvm-01 doesn’t include vault cert key:

Host kvm-01
    HostName 192.168.1.225
    User evanusmodestus
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    ...

SSH tries keys in order listed. Vault key not present → YubiKey tried → fails if server only trusts CA.

Test Command: Force Vault Key Only

Bypass SSH config entirely:

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault kvm-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'

Flags explained:

Flag Purpose

-o ControlPath=none

Bypass SSH multiplexing (force fresh connection)

-o IdentitiesOnly=yes

ONLY use specified key, ignore agent and config

-i ~/.ssh/id_ed25519_vault

Specify vault-signed key explicitly

kvm-01 Validation: SUCCESS

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault kvm-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'
Result
Static hostname: supermicro300-9d1
Operating System: Arch Linux
Kernel: Linux 6.17.5-arch1-1
Architecture: x86-64

SSH Config: Add Vault Key as Primary

Goal: Use vault cert first, fall back to YubiKey if CA not trusted.

Updated config:

Host kvm-01
    HostName 192.168.1.225
    User evanusmodestus
    IdentityFile ~/.ssh/id_ed25519_vault
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_d000
    PasswordAuthentication yes
    PreferredAuthentications publickey,password

Key points:

  1. Vault key + cert at TOP = tried first

  2. CertificateFile tells SSH to offer cert with that key

  3. YubiKey keys remain as fallback

  4. Order matters - SSH tries in sequence

Verify cert auth used (not fallback):

ssh -o ControlPath=none -vvv kvm-01 'exit' 2>&1 | awk '/Server accepts.*CERT|Accepted.*CERT/'

Validation Results

Host IP OS Status

kvm-01

192.168.1.225

Arch Linux

✓ VERIFIED

bind-01

10.50.1.90

Rocky Linux 9.7

✓ VERIFIED

ipsk-mgr-01

10.50.1.30

Ubuntu 22.04.5

✓ VERIFIED

ipa-01

10.50.1.100

Rocky Linux 9.7

✓ VERIFIED

keycloak-01

10.50.1.80

Fedora Linux 43

✓ VERIFIED

nas-01

10.50.1.70

Synology DSM 7.x

✓ VERIFIED

pfsense-01

10.50.1.1

pfSense 2.7.2 (FreeBSD)

✓ VERIFIED

vault-01

10.50.1.60

Rocky Linux 9.x

✓ VERIFIED

home-dc01

10.50.1.50

Windows Server 2025

✓ VERIFIED

home-dc01 Deployment (Windows OpenSSH)

Windows OpenSSH paths:

  • CA: C:\ProgramData\ssh\vault-ca.pub

  • Config: C:\ProgramData\ssh\sshd_config

Transfer CA:

scp /tmp/vault-ssh-ca.pub home-dc01:C:/ProgramData/ssh/vault-ca.pub

Configure via PowerShell:

ssh home-dc01 'powershell -Command "Add-Content -Path C:\ProgramData\ssh\sshd_config -Value \"`nTrustedUserCAKeys C:\ProgramData\ssh\vault-ca.pub\""'
ssh home-dc01 'powershell -Command "Restart-Service sshd"'

Issue: Windows user is domus\administrator - domain format required in principals.

Update Vault role for Windows domain users:

vault write ssh/roles/domus-client \
  key_type="ca" \
  allow_user_certificates=true \
  allowed_users='evanusmodestus,adminerosado,admin,administrator,domus\administrator,ansible,root' \
  default_user="evanusmodestus" \
  allowed_extensions="permit-pty,permit-port-forwarding" \
  ttl="8h" \
  max_ttl="24h"
Use single quotes to preserve backslash in domus\administrator.

Re-sign cert with domain principal:

vault write -field=signed_key ssh/sign/domus-client \
  public_key=@$HOME/.ssh/id_ed25519_vault.pub \
  valid_principals='evanusmodestus,adminerosado,admin,administrator,domus\administrator,ansible,root' >| ~/.ssh/id_ed25519_vault-cert.pub

Validation:

ssh -o ControlPath=none -o IdentityAgent=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault home-dc01 'hostname'
Result
home-dc01

ipa-01 Deployment

CA Transfer:

scp /tmp/vault-ssh-ca.pub ipa-01:/tmp/

Install and Configure (single command):

ssh ipa-01 'sudo mv /tmp/vault-ssh-ca.pub /etc/ssh/vault-ca.pub && sudo chmod 644 /etc/ssh/vault-ca.pub && echo "TrustedUserCAKeys /etc/ssh/vault-ca.pub" | sudo tee /etc/ssh/sshd_config.d/vault-ca.conf && sudo sshd -t && sudo systemctl restart sshd'

Validation:

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault ipa-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'
Result
Static hostname: ipa-01.inside.domusdigitalis.dev
Operating System: Rocky Linux 9.7 (Blue Onyx)
Kernel: Linux 5.14.0-611.5.1.el9_7.x86_64
Architecture: x86-64

keycloak-01 Deployment

Issue: Nano key not in authorized_keys. Used regular key to get in.

Check authorized_keys:

ssh -o IdentityAgent=none -i ~/.ssh/id_ed25519_d000 keycloak-01
awk '{print NR": "$NF}' ~/.ssh/authorized_keys
Before
1: evanusmodestus@d000-yubikey
2: evanusmodestus@d000-secondary
3: evanusmodestus@d000
4: certmgr-01-deploy

Add nano from workstation:

cat ~/.ssh/id_ed25519_sk_rk_d000_nano.pub | ssh -o IdentityAgent=none -i ~/.ssh/id_ed25519_d000 keycloak-01 'tee -a ~/.ssh/authorized_keys'

Clean up (remove certmgr, dedupe):

awk 'NF && !/certmgr/ {gsub(/^[[:space:]]+/, ""); if (!seen[$0]++) print}' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
After
1: evanusmodestus@d000-yubikey
2: evanusmodestus@d000-secondary
3: evanusmodestus@d000
4: d000-nano-35641207

Deploy CA:

scp /tmp/vault-ssh-ca.pub keycloak-01:/tmp/
ssh keycloak-01 'sudo mv /tmp/vault-ssh-ca.pub /etc/ssh/vault-ca.pub && sudo chmod 644 /etc/ssh/vault-ca.pub && echo "TrustedUserCAKeys /etc/ssh/vault-ca.pub" | sudo tee /etc/ssh/sshd_config.d/vault-ca.conf && sudo sshd -t && sudo systemctl restart sshd'

Validation:

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault keycloak-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'
Result
Static hostname: keycloak-01.inside.domusdigitalis.dev
Operating System: Fedora Linux 43 (Cloud Edition)
Kernel: Linux 6.17.1-300.fc43.x86_64
Architecture: x86-64

nas-01 Deployment

Issue: Different user (adminerosado), nano key missing, messy authorized_keys, Synology DSM quirks.

SSH in with regular key:

ssh nas-01  # Uses id_ed25519_sk_rk_d000 per config

Check authorized_keys:

awk '{print NR": "$NF}' ~/.ssh/authorized_keys
Before
1: evanusmodestus@d000
2: evanusmodestus@d000-yubikey
3:
4: evanusmodestus@d000-secondary
5: certmgr-01-deploy

Add nano from workstation:

cat ~/.ssh/id_ed25519_sk_rk_d000_nano.pub | ssh nas-01 'tee -a ~/.ssh/authorized_keys'

Clean up (remove empty lines, certmgr, dedupe):

awk 'NF && !/certmgr/ {if (!seen[$0]++) print}' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
After
1: evanusmodestus@d000
2: evanusmodestus@d000-yubikey
3: evanusmodestus@d000-secondary
4: d000-nano-35641207

Synology DSM SSH CA Deployment

Synology-specific challenges:

  1. No sshd_config.d/ - must edit main config directly

  2. scp to /tmp blocked - use home directory or pipe through cat

  3. sudo requires TTY - SSH in interactively for sudo commands

  4. synosystemctl not in PATH - use full path /usr/syno/bin/synosystemctl

Transfer CA (scp blocked, use cat pipe):

cat /tmp/vault-ssh-ca.pub | ssh nas-01 'cat > ~/vault-ca.pub'

Verify transfer:

ssh nas-01 'md5sum ~/vault-ca.pub'
# Compare with: md5sum /tmp/vault-ssh-ca.pub

Install CA (interactive session required):

ssh nas-01
sudo mv ~/vault-ca.pub /etc/ssh/vault-ca.pub
sudo chmod 644 /etc/ssh/vault-ca.pub

CRITICAL: sshd_config Match Block Positioning

TrustedUserCAKeys MUST appear BEFORE any Match blocks!

In sshd_config, once a Match directive appears, ALL subsequent directives apply ONLY to that match context until another Match block.

WRONG - Directive inside Match block:

125: Match User root
126:     AllowTcpForwarding yes
127: Match User anonymous
128:     AllowTcpForwarding no
129:     GatewayPorts no
130: TrustedUserCAKeys /etc/ssh/vault-ca.pub    # <-- WRONG! Only applies to anonymous!

CORRECT - Directive in global section:

1: # Global settings
2: TrustedUserCAKeys /etc/ssh/vault-ca.pub      # <-- CORRECT! Applies to all users
...
125: Match User root
126:     AllowTcpForwarding yes

Diagnosis command:

# This shows RUNTIME config, not file contents
sudo sshd -T | grep trustedusercakeys
# Output: "trustedusercakeys none" = BROKEN (in Match block)
# Output: "trustedusercakeys /etc/ssh/vault-ca.pub" = WORKING

Fix sequence:

# Remove from current (wrong) location
sudo sed -i '/TrustedUserCAKeys/d' /etc/ssh/sshd_config

# Find first Match block
sudo grep -n "^Match" /etc/ssh/sshd_config | head -1

# Insert at line 2 (global section)
sudo sed -i '1a TrustedUserCAKeys /etc/ssh/vault-ca.pub' /etc/ssh/sshd_config

# Verify runtime config
sudo sshd -T | grep trustedusercakeys

# Restart (Synology)
sudo /usr/syno/bin/synosystemctl restart sshd

Validation:

# Get passphrase first
gopass show -c v3/domains/d000/identity/ssh/id_ed25519_vault

# Test vault cert auth
ssh -o ControlPath=none -o IdentityAgent=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault nas-01 'hostname'
Result
nas-01

Key Lesson: sshd_config Match Block Rules

Rule Explanation

Match blocks are "sticky"

All directives after Match apply to that match only

Global directives first

TrustedUserCAKeys, PasswordAuthentication, etc. must be BEFORE any Match

sshd -T shows runtime

Use this to verify what sshd actually sees, not what’s in the file

sshd -t checks syntax

No output = syntax OK, but doesn’t catch Match positioning issues

Match ends at next Match

Or end of file - no explicit "end match" directive

Memory aid: Think of sshd_config like a waterfall:

[Global Settings]     ← TrustedUserCAKeys goes HERE
        ↓
[Match User root]     ← Settings below apply only to root
        ↓
[Match User anon]     ← Settings below apply only to anon
        ↓
[End of file]

pfsense-01 Deployment

pfSense-specific challenges:

  1. No sudo - use pfSense shell menu (option 8)

  2. Config regeneration - sshd_config regenerated on restart from pfSense database

  3. User is admin - not evanusmodestus, requires Vault role update

  4. FreeBSD-based - different service commands

Add nano key:

cat ~/.ssh/id_ed25519_sk_rk_d000_nano.pub | ssh pfsense-01 'cat >> ~/.ssh/authorized_keys'

Transfer CA:

scp /tmp/vault-ssh-ca.pub pfsense-01:/tmp/

Deploy CA (via pfSense shell):

ssh pfsense-01
# Select option 8 (Shell)
mv /tmp/vault-ssh-ca.pub /etc/ssh/vault-ca.pub
chmod 644 /etc/ssh/vault-ca.pub

CRITICAL: pfSense sshd_config Persistence

pfSense regenerates /etc/ssh/sshd_config on service restart!

Direct edits to sshd_config are LOST. Use /etc/sshd_extra instead.

WRONG - Direct edit (lost on restart):

echo "TrustedUserCAKeys /etc/ssh/vault-ca.pub" >> /etc/ssh/sshd_config
pfSsh.php playback svc restart sshd
grep TrustedUserCAKeys /etc/ssh/sshd_config  # EMPTY!

CORRECT - Use /etc/sshd_extra:

echo "TrustedUserCAKeys /etc/ssh/vault-ca.pub" > /etc/sshd_extra
pfSsh.php playback svc restart sshd
grep TrustedUserCAKeys /etc/ssh/sshd_config  # Present!

Verify runtime config:

sshd -T | grep trusteduser
# Output: trustedusercakeys /etc/ssh/vault-ca.pub

Vault Role Update for admin Principal

Issue: pfSense user is admin, not in Vault allowed_users list.

Check current role:

vault read ssh/roles/domus-client | grep allowed_users
# Output: allowed_users  evanusmodestus,adminerosado,ansible,root

Update role to include admin:

vault write ssh/roles/domus-client \
  key_type="ca" \
  allow_user_certificates=true \
  allowed_users="evanusmodestus,adminerosado,admin,ansible,root" \
  default_user="evanusmodestus" \
  allowed_extensions="permit-pty,permit-port-forwarding" \
  ttl="8h" \
  max_ttl="24h"

Re-sign certificate with new principals:

vault write -field=signed_key ssh/sign/domus-client \
  public_key=@$HOME/.ssh/id_ed25519_vault.pub \
  valid_principals="evanusmodestus,adminerosado,admin,ansible,root" >| ~/.ssh/id_ed25519_vault-cert.pub
Use $HOME not ~ in the public_key path - Vault doesn’t expand tilde.
Use >| to force overwrite when zsh noclobber is enabled.

Validation:

ssh -o ControlPath=none -o IdentityAgent=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault pfsense-01 'hostname'
Result
pfSense-FW01.home.local

pfSense SSH Quick Reference

Task Command

Enter shell

SSH in, select option 8

Restart sshd

pfSsh.php playback svc restart sshd

Persistent sshd config

/etc/sshd_extra (not sshd_config directly)

Check runtime config

sshd -T | grep <option>

authorized_keys Cleanup Pattern

The universal cleanup command:

awk 'NF && !/certmgr/ {gsub(/^[[:space:]]+/, ""); if (!seen[$0]++) print}' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
Component Purpose

NF

Skip empty lines (NF=0 when empty)

!/certmgr/

Skip lines containing "certmgr" (old deploy key)

gsub(/^+/, "")

Trim leading whitespace

!seen[$0]++

Dedupe - print only first occurrence

> /tmp/ak && mv

Atomic write (safe editing)

chmod 600

Required SSH permissions

Workflow:

  1. SSH in with working key

  2. Check: awk '{print NR": "$NF}' ~/.ssh/authorized_keys

  3. Add nano: cat ~/.ssh/id_ed25519_sk_rk_d000_nano.pub | ssh HOST 'tee -a ~/.ssh/authorized_keys'

  4. Clean: (command above)

  5. Verify: awk '{print NR": "$NF}' ~/.ssh/authorized_keys

bind-01 Validation

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault bind-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'
Result
Static hostname: bind-01.inside.domusdigitalis.dev
Operating System: Rocky Linux 9.7 (Blue Onyx)
Kernel: Linux 5.14.0-611.5.1.el9_7.x86_64
Architecture: x86-64

ipsk-mgr-01 Validation

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault ipsk-mgr-01 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'
Result
Static hostname: ipsk-mgr-01
Operating System: Ubuntu 22.04.5 LTS
Kernel: Linux 5.15.0-164-generic
Architecture: x86-64

Appendix: Vault SSH CA Quick Reference

Certificate Status

ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub | awk '/Type:|Valid:|Principals:/{print} /Principals:/{getline; print}'

Re-sign Expired Certificate

dsource d000 dev/vault && vault write -field=signed_key ssh/sign/domus-client public_key=@~/.ssh/id_ed25519_vault.pub >| ~/.ssh/id_ed25519_vault-cert.pub

Test SSH CA (Bypass Config)

Force vault key only - ignores SSH config and agent:

ssh -o ControlPath=none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault <HOST> 'hostnamectl' | awk '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, ""); print}'

SSH Config Pattern (Vault + YubiKey Fallback)

Add vault key at TOP of IdentityFile list:

Host <hostname>
    HostName <ip>
    User evanusmodestus
    IdentityFile ~/.ssh/id_ed25519_vault
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_d000
    PasswordAuthentication yes
    PreferredAuthentications publickey,password

Order matters:

  1. Vault cert tried first (short-lived, CA-signed)

  2. YubiKey keys as fallback (hardware-bound)

  3. Password last resort

Verify Cert Auth Used

ssh -o ControlPath=none -vvv <HOST> 'exit' 2>&1 | awk '/Server accepts.*CERT|Accepted.*CERT/'

Troubleshooting

GPG Pinentry Too Small

Error: Screen or window too small

Fix:

echo "pinentry-program /usr/bin/pinentry-tty" >> ~/.gnupg/gpg-agent.conf && gpgconf --kill gpg-agent

Wrong Key Offered

Symptom: SSH tries YubiKey before vault cert

Cause: Vault key not in SSH config or not at TOP

Test: Use -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_vault to force vault key

Connection Closed

Symptom: Connection closed by <ip> port 22

Causes:

  • Server doesn’t trust Vault CA (check /etc/ssh/vault-ca.pub)

  • Certificate expired (check with ssh-keygen -Lf)

  • Principal mismatch (cert principals vs login user)

AWK Range Patterns for SSH Config

View Specific Host Blocks

Pattern anatomy:

awk '/^Host (kvm-01|bind-01|ipsk-mgr-01)$/,/^Host / {print NR": "$0}' ~/.ssh/config
     |___________________________________|  |______|  |______________|
                   START                      END        ACTION
Part Meaning

/^Host …​/

START: line begins with "Host " + one of the names

(a|b|c)

Alternation - matches a OR b OR c

$

End of line (exact match, not "kvm-01-test")

,

Range operator - "from START to END"

/^Host /

END: next line starting with "Host "

{print NR": "$0}

Print line number + content

Problem: Range includes the END pattern line. To see full blocks without next header:

# By line number (after finding start line)
awk 'NR>=267 && NR<=280' ~/.ssh/config

# Exclude terminating pattern
awk '/^Host kvm-01$/,/^Host /{if(!/^Host / || /^Host kvm-01$/)print NR": "$0}' ~/.ssh/config

Insert Lines After Match (sed)

Goal: Insert vault key lines after User evanusmodestus in a specific Host block.

WRONG (nested braces don’t work this way):

# This fails with "unmatched `{'"
sed -i '/^Host kvm-01$/,/^Host /{/User evanusmodestus/a\...' ~/.ssh/config

CORRECT (GNU sed with line continuation):

sed -i '/^Host kvm-01$/,/^Host /{
  /User evanusmodestus/a\
    IdentityFile ~/.ssh/id_ed25519_vault\
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
}' ~/.ssh/config

Or use heredoc for clarity:

sed -i '/^Host kvm-01$/,/^Host /{
/User evanusmodestus/a\
    IdentityFile ~/.ssh/id_ed25519_vault\
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
}' ~/.ssh/config

Safest approach - line number insertion:

# Find the User line number first
awk '/^Host kvm-01$/,/^Host /{if(/User/)print NR}' ~/.ssh/config
# Output: 270

# Insert after that line
sed -i '270a\    IdentityFile ~/.ssh/id_ed25519_vault\n    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub' ~/.ssh/config

Manual Edit (Recommended for 3 Hosts)

For small changes, manual editing is faster and safer:

nvim +267 ~/.ssh/config   # Jump to kvm-01
nvim +347 ~/.ssh/config   # Jump to bind-01

Add after User evanusmodestus:

    IdentityFile ~/.ssh/id_ed25519_vault
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub

Verify Changes

awk '/^Host kvm-01$/,/^Host /{print NR": "$0}' ~/.ssh/config | head -12

authorized_keys Management with AWK

View with Line Numbers

awk '{print NR": "$0}' ~/.ssh/authorized_keys

Clean authorized_keys (Dedupe + Trim + Remove Junk)

The full cleanup command:

awk 'NF && !/EOF/ {gsub(/^[[:space:]]+/, ""); if (!seen[$0]++) print}' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys

Breakdown:

Component Purpose

NF

Skip empty lines (NF = number of fields, 0 = empty)

!/EOF/

Skip lines containing literal "EOF" (heredoc accidents)

gsub(/^+/, "")

Trim leading whitespace

!seen[$0]++

Dedupe - only print first occurrence of each line

> /tmp/ak && mv /tmp/ak …​

Write to temp, then atomic replace (safe editing)

chmod 600

Fix permissions (required for SSH)

Add Key via SSH + Heredoc

From workstation to remote host:

cat ~/.ssh/id_ed25519_sk_rk_d000_nano.pub | ssh <host> 'tee -a ~/.ssh/authorized_keys'

Or with inline key:

ssh <host> 'tee -a ~/.ssh/authorized_keys << EOF
sk-ssh-ed25519@openssh.com AAAA... comment
EOF'

Verify Key Exists

ssh <host> 'awk "/nano/ {print NR\": \"\$0}" ~/.ssh/authorized_keys'

Check Which Keys Are Authorized

awk -F' ' '{print NR": "$NF}' ~/.ssh/authorized_keys

Shows line number and comment (last field) only.

Colored Output with jq

Explore JSON Keys

hostnamectl --json=short outputs JSON. Explore available fields:

ssh -o ControlPath=none kvm-01 'hostnamectl --json=short' | jq 'keys'
Available Keys (systemd 256+)
BootID, Chassis, ChassisAssetTag, DefaultHostname, Deployment,
FirmwareDate, FirmwareVendor, FirmwareVersion, HardwareModel,
HardwareSKU, HardwareSerial, HardwareVendor, HardwareVersion,
Hostname, HostnameSource, IconName, KernelName, KernelRelease,
KernelVersion, Location, MachineID, OperatingSystemCPEName,
OperatingSystemHomeURL, OperatingSystemPrettyName, StaticHostname...
No .Architecture field in JSON output. Use awk conversion for that field.

Native JSON (Missing Architecture)

for host in kvm-01 bind-01 ipsk-mgr-01; do
  echo "=== $host ==="
  ssh -o ControlPath=none $host 'hostnamectl --json=short' | jq '{Hostname: .StaticHostname, OS: .OperatingSystemPrettyName, Kernel: .KernelRelease}'
done

AWK to JSON Conversion (All Fields + Colors)

Convert text output to JSON for jq coloring:

for host in kvm-01 bind-01 ipsk-mgr-01; do
  echo "=== $host ==="
  ssh -o ControlPath=none $host 'hostnamectl' | awk -F': ' '/Static hostname|Operating System|Kernel|Architecture/ {gsub(/^[[:space:]]+/, "", $1); gsub(/^[[:space:]]+/, "", $2); printf "\"%s\": \"%s\",\n", $1, $2}' | sed '1s/^/{/; $s/,$/}/' | jq
done
Output (colored in terminal)
=== kvm-01 ===
{
  "Static hostname": "supermicro300-9d1",
  "Operating System": "Arch Linux",
  "Kernel": "Linux 6.17.5-arch1-1",
  "Architecture": "x86-64"
}
=== bind-01 ===
{
  "Static hostname": "bind-01.inside.domusdigitalis.dev",
  "Operating System": "Rocky Linux 9.7 (Blue Onyx)",
  "Kernel": "Linux 5.14.0-611.5.1.el9_7.x86_64",
  "Architecture": "x86-64"
}
=== ipsk-mgr-01 ===
{
  "Static hostname": "ipsk-mgr-01",
  "Operating System": "Ubuntu 22.04.5 LTS",
  "Kernel": "Linux 5.15.0-164-generic",
  "Architecture": "x86-64"
}

AWK to JSON Pattern Explained

awk -F': ' '...' | sed '...' | jq
    |_____|  |_|   |______|   |__|
      (1)    (2)     (3)      (4)

(1) -F': '     Field separator is colon+space
(2) Pattern    Match specific fields, format as JSON key:value
(3) sed        Wrap in { } braces (add { at start, replace trailing , with } at end)
(4) jq         Parse JSON and colorize

AWK breakdown:

Part Purpose

-F': '

Split on `: ` (colon-space)

/Static hostname|…​/

Match lines containing these fields

gsub(/^+/, "", $1)

Trim leading whitespace from field name

gsub(/^+/, "", $2)

Trim leading whitespace from value

printf "\"%s\": \"%s\",\n"

Output as JSON key-value pair with trailing comma

sed breakdown:

Part Purpose

1s/^/{/

Line 1: insert { at beginning

$s/,$/}/

Last line: replace trailing , with }

Bonus: Advanced AWK Patterns

Process Multiple Files with Context

# Process all authorized_keys across hosts
for host in kvm-01 bind-01 ipsk-mgr-01; do
  echo "=== $host ==="
  ssh $host 'awk -F" " "{print NR\": \"\$NF}" ~/.ssh/authorized_keys'
done

Conditional Field Extraction

# Extract only sk-ssh (YubiKey) entries
awk '/sk-ssh/ {print NR": "$NF}' ~/.ssh/authorized_keys

Field Count Validation

# Validate authorized_keys format (SSH keys have 3+ fields)
awk 'NF < 3 {print "INVALID LINE " NR": " $0}' ~/.ssh/authorized_keys

Combine Multiple Transformations

# Trim + dedupe + validate + count
awk 'NF >= 3 && !/^#/ && !seen[$0]++ {
  gsub(/^[[:space:]]+/, "")
  count++
  print count": "$NF
} END {print "\n"count" valid keys"}' ~/.ssh/authorized_keys

Parse systemd Unit Status

# Extract service state info
systemctl status sshd | awk -F'[;:]' '/Active:|Main PID:|CGroup:/ {gsub(/^[[:space:]]+/, ""); print}'

Bonus: sed Patterns for Config Files

In-place Editing with Backup

# Create .bak backup before editing
sed -i.bak 's/old/new/g' /etc/ssh/sshd_config

Insert After Match

# Insert line after PermitRootLogin
sed -i '/^PermitRootLogin/a TrustedUserCAKeys /etc/ssh/vault-ca.pub' /etc/ssh/sshd_config

Delete Lines Matching Pattern

# Remove all comment lines
sed -i '/^#/d' /etc/ssh/sshd_config

Range Delete

# Delete from pattern to end of file
sed -i '/^# Legacy config/,$d' config.txt

Multi-line Address

# Operate only within range
sed -n '/^Host kvm-01/,/^Host /p' ~/.ssh/config

Bonus: jq Advanced Patterns

Flatten Nested Arrays

jq '[.. | select(type == "string")]' data.json

Object to Array of Key-Value Pairs

jq 'to_entries | .[] | "\(.key)=\(.value)"' data.json

Merge Multiple JSON Objects

jq -s 'add' file1.json file2.json

Dynamic Key Access

KEY="hostname"
jq --arg k "$KEY" '.[$k]' data.json

Recursive Key Search

# Find all "id" values at any depth
jq '.. | .id? // empty' data.json