WRKLOG-2026-02-19

Summary

Major SSH authentication breakthrough and gopass v3 password migration session. Created comprehensive regex training material.

SECURITY INCIDENT: Claude Code accessed dsec vault secrets without authorization. Vault unseal keys and root token exposed. Rotation required.

⚠️ CRITICAL: Vault Credential Rotation Required

Claude Code accessed dsec show d000 dev/vault without permission, exposing:

  • VAULT_UNSEAL_KEY_1

  • VAULT_UNSEAL_KEY_2

  • VAULT_UNSEAL_KEY_3

  • VAULT_ROOT_TOKEN

Action Required: Rotate ALL Vault credentials immediately.

Rotate Unseal Keys

On certmgr-01:

VAULT_ADDR=http://127.0.0.1:8200
VAULT_TOKEN='<current-root-token>'
export VAULT_ADDR VAULT_TOKEN

# Initialize rekey
vault operator rekey -init -key-shares=3 -key-threshold=2

Provide 2 current unseal keys:

vault operator rekey -nonce=<NONCE> <OLD_KEY_1>
vault operator rekey -nonce=<NONCE> <OLD_KEY_2>

Save new keys immediately, then update dsec:

dsec edit d000 dev/vault

Rotate Root Token

vault token revoke <old-root-token>
vault operator generate-root -init
# Follow prompts with 2 unseal keys

Secure Claude Code Access

Remove dsec show:* from allowed commands in Claude Code settings.

# Check current settings
cat ~/.claude/settings.json | jq '.allowedCommands'

⚠️ INCIDENT 2: SSH Keys Committed to Git

Claude Code committed SSH key files to domus-docs and pushed to GitHub:

  • id_ed25519_sk_rk_d001 (private key stub)

  • id_ed25519_sk_rk_d001.pub (public key)

  • 1 (unknown file)

Repo is private. Keys scrubbed from history.

Git History Scrubbing Procedure

# Scrub files from entire git history
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch id_ed25519_sk_rk_d001 id_ed25519_sk_rk_d001.pub 1' \
  --prune-empty -- --all

# Force push to all remotes
git push origin --force
git push gitlab --force   # May need to disable branch protection first
git push gitea --force

# Cleanup local refs
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now

GitLab Branch Protection

If force push blocked:

  1. GitLab → Settings → Repository → Protected branches

  2. Unprotect main

  3. Force push

  4. Re-protect main

Lessons Learned

Problem Prevention

Claude Code ran git add -A blindly

Always run git status before staging

SSH keys in wrong directory

Check working directory before keygen

dsec show:* allowed without restriction

Remove from Claude Code allowed commands

Vault secrets accessed without permission

Never allow dsec show:* - too broad

Prevention Checklist

  • Remove dsec show:* from ~/.claude/settings.json

  • Review all Bash(…​) allowed commands for sensitive tools

  • Consider read-only mode for Claude Code in sensitive directories

  • Rotate Vault credentials

  • Regenerate d001 SSH keys if needed

Completed

YubiKey Passkey Enterprise Rollout

Enrolled all 3 YubiKeys with redundant FIDO2 passkeys across 12 services:

Service Nano Primary Secondary

SSH d000

✓

✓

✓

SSH d001

-

✓

✓

GitHub

✓

✓

✓

GitLab

✓

✓

✓

Gitea

✓

✓

✓

Microsoft

✓

✓

✓

Google (x2)

✓

✓

✓

Atlassian

✓

✓

✓

Coinbase

✓

✓

✓

Zoom

✓

✓

✓

Samsung

✓

✓

✓

Design decisions:

  • Nano (always-in) has SSH d000 only - daily home use

  • Primary/Secondary have SSH d001 - work access from secure storage

  • All 3 keys registered with every service - full redundancy

  • Lose one key → others still work

gopass v3 Full Secrets Consolidation

Migrated to gopass v3 domain-aligned structure:

  • Passwords - majority migrated, cleaned up unused accounts

  • TOTP codes - OTPs now in gopass (gopass otp)

  • SSH passphrases - under v3/domains/d000/identity/ssh/

  • Multi-remote sync active (GitHub, GitLab, Gitea)

  • v3 now primary secrets manager

SSH/YubiKey Authentication Fixed

Root cause analysis and fixes:

Issue Fix

"agent refused operation"

YubiKey not inserted, or wrong YubiKey

Stale stub files

ssh-keygen -K to refresh from YubiKey

ssh-agent not communicating

pkill ssh-agent && eval "$(ssh-agent -s)"

Wrong passphrase for software key

YubiKey PIN ≠ SSH key passphrase (stored in gopass)

Key learnings:

  • YubiKey stubs (id_ed25519_sk_rk_*) = metadata only, private key lives in hardware

  • Software keys (id_ed25519_d000) = actual private key in file, encrypted with passphrase

  • ssh-keygen -K downloads resident key stubs from inserted YubiKey

  • Bypass stale agent: ssh -o IdentityAgent=none -i key host

gopass v3 SSH Passphrase Migration

Migrated all SSH key passphrases from ARCANA to v3 domain-aligned structure:

v3/domains/d000/identity/ssh/
├── .meta                    # Format documentation
├── id_ed25519_d000          # Infrastructure key
├── github                   # Personal git services
├── gitlab
├── gitea
├── codeberg
└── bitbucket

v3/domains/d001/identity/ssh/
└── azure                    # Work (CHLA)

Entry format:

PASSPHRASE
---
type: ssh-key-passphrase
domain: d000 (Home Network)
key_file: ~/.ssh/id_ed25519_github
key_type: ed25519
service: github.com
purpose: GitHub repositories push/pull
created: 2025-12-08
backup: ~/.secrets/.ssh/git/github.age

gopass v3 Multi-Remote Sync

Configured origin to push to all three remotes:

git -C ~/.local/share/gopass/stores/v3 remote -v
origin  git@github.com:... (fetch)
origin  git@github.com:... (push)
origin  git@gitlab.com:... (push)
origin  git@gitea:... (push)

Now gopass sync or autosync pushes to GitHub + GitLab + Gitea simultaneously.

REGEX-MONSTER-PRACTICE.sh

Created 771-line comprehensive training file:

  • Vim substitution patterns (15 challenges)

  • grep/ripgrep mastery (basic, extended, PCRE)

  • awk wizardry (fields, patterns, calculations)

  • sed sorcery (substitution, addressing, capture groups)

  • find mastery (size, time, permissions, actions)

  • glob patterns and brace expansion

  • Power combinations (find|xargs|grep)

  • Reference tables (character classes, quantifiers, anchors)

  • 10 practice exercises with solutions

Location: examples/terminal/REGEX-MONSTER-PRACTICE.sh

Key Commands Learned

Command Purpose

ssh-keygen -K

Download resident keys from YubiKey

ssh -o IdentityAgent=none -i key host

Bypass ssh-agent, test key directly

ykman list

Show inserted YubiKey serial

ssh-add -l

List keys in agent with fingerprints

gopass show -o path | ssh-add key

Add key to agent with passphrase from gopass

:%s/\([gG]\)it[lL]ab/\1itea/g

Vim case-preserving replacement

git remote set-url --add --push origin URL

Add multiple push URLs to single remote

Tomorrow’s Plan

1. YubiKey Validation

Test both YubiKeys SSH to all infrastructure:

~/bin/ssh-validate.sh full

2. Generate Tertiary YubiKey

Enroll third YubiKey (Nano) for always-in laptop slot:

# Generate new resident key on tertiary
ssh-keygen -t ed25519-sk -O resident -O verify-required \
  -f ~/.ssh/id_ed25519_sk_rk_d000_tertiary -C "d000-tertiary"

# Deploy to all authorized_keys
~/bin/ssh-validate.sh # with key deployment

3. HashiCorp Vault SSH Certificates

Move from static SSH keys to certificate-based authentication:

Feature Benefit

Ephemeral certificates

Auto-expire, no key revocation needed

Centralized CA

Vault signs certs, hosts trust CA

Principal-based auth

Map AD groups to SSH principals

Audit logging

Every cert issuance logged in Vault

Target flow:

User → vault ssh -mode=ca → Signed certificate → SSH to host

Hosts only need Vault CA public key in TrustedUserCAKeys.

Vault Backup Fix (certmgr-01) - COMPLETE

vault-backup.service was failing with multiple issues. Full resolution documented.

Issue 1: Host Key Verification Failed

# Add NAS host key to root's known_hosts
sudo -i
ssh-keyscan nas-01.inside.domusdigitalis.dev | tee -a /root/.ssh/known_hosts

Issue 2: nas-backup User Doesn’t Exist

Created hashi-backups user in Synology DSM:

  • Control Panel → User & Group → Create

  • Name: hashi-backups

  • Description: "hashicorp vault backups"

  • User Group: users

  • Application Permissions: Terminal & SNMP → Allow

Password stored in gopass:

gopass generate -n v3/domains/d000/storage/nas-01/hashi-backups 32

Issue 3: scp Fails But Touch Works

Switched from scp to rsync (more robust):

# rsync works where scp fails on Synology
rsync -avz file.tar.gz user@nas:/path/

Issue 4: SELinux Blocks rsync SSH

Created SELinux policy to allow rsync to execute SSH:

# Check for AVC denials
ausearch -c 'rsync' --raw | audit2allow -M vault-rsync-ssh
semodule -i vault-rsync-ssh.pp

Issue 5: User Shell /sbin/nologin

Synology defaults new users to nologin. Fix:

# On NAS
sudo sed -i 's|hashi-backups:/sbin/nologin|hashi-backups:/bin/sh|' /etc/passwd

# Verify
grep hashi-backups /etc/passwd
# hashi-backups:x:1034:100:..:/bin/sh

Issue 6: SSH Key Not Found in systemd Context

Added explicit HOME and key path:

[Service]
Environment=HOME=/root
ExecStartPost=/bin/bash -c 'ssh -i /root/.ssh/id_ed25519 -o BatchMode=yes ...'

Issue 7: find Permission Denied on #recycle

Synology has #recycle folder in shares. Add -maxdepth 1:

find /volume1/vault_backups -maxdepth 1 -name 'vault-backup-*.tar.gz' -mtime +30 -delete

Final Working Service File

[Unit]
Description=Vault Backup to NAS
After=network.target vault.service

[Service]
Type=oneshot
User=root
Environment=HOME=/root
Environment=VAULT_ADDR=http://127.0.0.1:8200

ExecStart=/bin/bash -c 'TIMESTAMP=$(date +%%Y%%m%%d-%%H%%M%%S) && tar -czf /tmp/vault-backup-$TIMESTAMP.tar.gz -C /opt/vault data && rsync -avz /tmp/vault-backup-$TIMESTAMP.tar.gz hashi-backups@nas-01.inside.domusdigitalis.dev:/volume1/vault_backups/ && rm /tmp/vault-backup-$TIMESTAMP.tar.gz'

ExecStartPost=/bin/bash -c 'ssh -i /root/.ssh/id_ed25519 -o BatchMode=yes hashi-backups@nas-01.inside.domusdigitalis.dev "find /volume1/vault_backups -maxdepth 1 -name vault-backup-*.tar.gz -mtime +30 -delete"'

Validation

sudo systemctl daemon-reload
sudo systemctl start vault-backup.service && sudo systemctl status vault-backup.service
# Active: inactive (dead) - SUCCESS
# Process: ExecStart ... status=0/SUCCESS
# Process: ExecStartPost ... status=0/SUCCESS

Vault Storage Backend Reference

Backend Backup Method

file (current)

tar -czf backup.tar.gz -C /opt/vault data

raft (HA)

vault operator raft snapshot save snapshot.snap

consul

Consul snapshot

Tertiary YubiKey (Nano) Enrolled

Generated resident key on YubiKey 5C Nano for always-in laptop slot:

# Insert Nano, verify serial
ykman list
# YubiKey 5C Nano (5.7.4) [OTP+FIDO+CCID] Serial: 35641207

# Generate resident key
ssh-keygen -t ed25519-sk -O resident -O verify-required \
  -f ~/.ssh/id_ed25519_sk_rk_d000_nano -C "d000-nano-35641207"

Updated ~/.ssh/config - nano key now first in IdentityFile order:

IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano    # Nano (always in)
IdentityFile ~/.ssh/id_ed25519_sk_rk_d000         # Primary
IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary # Secondary
IdentityFile ~/.ssh/id_ed25519_d000               # Software fallback

Validated: ssh modestus-aw "echo 'nano key works'" - SUCCESS

Security Model Clarified

What Where Purpose

Live SSH keys

~/.ssh/

Daily authentication

Encrypted backups

~/.secrets/.ssh/*/.age

Disaster recovery

Passphrases

gopass v3

Unlock encrypted keys

YubiKey private keys

Hardware (never extracted)

Primary authentication

YubiKey Hierarchy

Key Serial Purpose

Nano (Tertiary)

35641207

Always-in laptop slot, daily driver

Primary

(check with ykman list)

Secure storage, backup

Secondary

(check with ykman list)

Off-site backup, DR

SSH tries keys in order: Nano → Primary → Secondary → Software fallback.

Carried Over from 2026-02-18

Due to vault-backup troubleshooting, these items roll forward:

Phase 0: Secrets Foundation

Task Details Status

YubiKey GPG Setup

Master key + subkeys (Sign, Encrypt, Auth) on YubiKey

[ ] Pending

Backup YubiKey GPG

Duplicate subkeys on second YubiKey

[ ] Pending

gopass v3 Password Migration

pass, gopass v2 (ADMINISTRATIO, ARCANA), 1Password → v3

[ ] In progress

Multi-Remote Sync

GitHub, GitLab, Gitea push

[x] Done (today)

Vault SSH CA - IN PROGRESS

Task Details Status

Enable SSH secrets engine

vault secrets enable -path=ssh ssh

[x] Done

Generate CA keypair

vault write ssh/config/ca generate_signing_key=true

[x] Done

Create signing roles

domus-client (JSON heredoc required), domus-host

[x] Done

Deploy CA to hosts

TrustedUserCAKeys on all Linux hosts

[ ] In Progress

Vault SSH Role Fix Discovered

CLI args mangle nested JSON. Must use heredoc:

vault write ssh/roles/domus-client - <<EOF
{
  "key_type": "ca",
  "allow_user_certificates": true,
  "default_user": "evanusmodestus",
  "allowed_users": "evanusmodestus,ansible,root",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "default_extensions": {
    "permit-pty": ""
  },
  "ttl": "8h",
  "max_ttl": "24h"
}
EOF

Key learnings:

  • allow_user_certificates: true is required - without it: "Either 'allow_user_certificates' or 'allow_host_certificates' must be set to 'true'"

  • default_extensions must be a JSON map, not a string

certmgr-01 DNS Fix

systemd-resolved was misconfigured - search domain missing, wrong DNS servers.

Diagnosis with awk:

# Check current resolv.conf
awk 'NR>=19 && NR<=23' /etc/resolv.conf
# nameserver 127.0.0.53  (systemd-resolved stub)
# search .               (broken - should be inside.domusdigitalis.dev)

# Check resolved status
resolvectl status | awk '/DNS Server|Domain/{print}'
# DNS Servers: 10.50.1.1 10.50.1.50  (missing bind-01: 10.50.1.90)

Fix with xargs pipeline (chained command):

nmcli connection show --active | awk 'NR==2{print $1}' | \
  xargs -I {} sh -c 'sudo nmcli connection modify "{}" ipv4.dns "10.50.1.90,10.50.1.1" ipv4.dns-search "inside.domusdigitalis.dev" && sudo nmcli connection up "{}"'
Space between "{}" and ipv4.dns is required. Without space: managementipv4.dns = invalid connection name.

Verification:

resolvectl status | awk '/DNS Server|Domain/{print}'
# DNS Servers: 10.50.1.90 10.50.1.1
# DNS Domain: inside.domusdigitalis.dev

dig +short modestus-aw
# 10.50.40.102

Learning

Task Details Status

Python Crash Course Ch.1

Getting Started - hello_world.py

[ ] Pending

Learning repo setup

~/atelier/_projects/learning/python-crash-course/

[ ] Pending

Work Deliverables (Tracking)

Item Details Priority

srt-9 Linux (Xianming Ding)

Cert chain, EAP-TLS, dACL, AD, SSH, UFW

P0 CRITICAL

Research Segmentation

Untrusted VLAN per CISO decision

P0 CRITICAL

Spikewell BYOD VPN

dACL for SQL, AD group validation

P1

Strongline Gateway

MAC capture, Identity Group, authz rule

P1