Daily Worklog: 2026-02-16

Overview

Date: 2026-02-16 (Monday)

Location: Remote

Focus: ZSH History Cookbook, Xianming Linux Delivery, CISO Research VLAN Meeting, iPSK HA with Ben Castillo

Critical Deliverables

1. Xianming Ding Linux Workstation Delivery

Deadline: Today

Location: SRT 9th Floor / Research / Xianming Ding

Item Status Notes

Certificate chain

[ ]

Root CA + Issuing CA + Client cert

Private key

[ ]

Mode 0600, owned by root

NetworkManager 802.1X

[ ]

EAP-TLS, password flags fixed

EAP-TLS authentication

[ ]

MUST be EAP-TLS, NOT MAB

dACL applied

[ ]

DACL_LINUX_RESEARCH_AD_AUTH

AD connectivity

[ ]

kinit, ldapsearch verified

SSH access

[ ]

From admin workstation

Authentication Flow:

EAP-TLS Authentication Flow

Verification Before Handoff:

# On the Linux workstation
nmcli connection show "Wired 802.1X" | grep -E "802-1x|connection.id"

# Check ISE for EAP-TLS (not Lookup/MAB)
netapi ise mnt session <MAC>

# Verify from ISE DataConnect
netapi ise dc query "
SELECT
    USERNAME,
    AUTHENTICATION_PROTOCOL,
    AUTHORIZATION_PROFILES,
    PASSED
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID = '<MAC>'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 1 ROWS ONLY
"
# Expected: AUTHENTICATION_PROTOCOL = 'EAP-TLS'

2. CISO Research VLAN Meeting

Attendees: CISO, Network Manager, Security Team

Objective: Advocate for dedicated Research VLAN with defense-in-depth architecture

Reference: Principia/02_Assets/TAB-CAPTURES/research/research-vlan-vs-dacl.adoc

Defense in Depth Architecture

Elevator Pitch

"dACLs provide per-user L3/L4 filtering, but they don’t protect against Layer 2 attacks like ARP spoofing, MAC flooding, or broadcast-based reconnaissance. A dedicated Research VLAN adds a second independent security control at Layer 2, following defense-in-depth principles required by NIST, CIS, and ISO 27001."

Counter-Arguments Ready

Their Objection Your Response

"L3 routing already configured"

"L3 routing handles traffic AFTER it leaves the VLAN. L2 attacks like ARP spoofing happen BEFORE traffic is routed."

"Only gain is broadcast isolation"

"Broadcast isolation prevents ARP spoofing, rogue DHCP, and discovery protocol leakage. These are documented attack vectors in penetration testing frameworks."

"Management overhead"

"One-time VLAN creation with automated provisioning via ISE. Ongoing cost is zero. Risk of L2 attacks is ongoing."

"Our dACLs are strong enough"

"dACLs are L3/L4 only. They cannot block L2 attacks by design. This is a TCP/IP stack limitation, not a configuration issue."

3. iPSK HA with Ben Castillo

Topic: iPSK Manager High Availability for CHLA

iPSK HA Architecture with Vault PKI

Agenda:

  • Review current iPSK Manager deployment

  • Discuss HA architecture options

  • PostgreSQL replication strategy

  • ISE ODBC failover configuration

  • Vault/ACME for server certificate issuance

Reference Docs:

  • PRJ-ISE-IPSK-CHLA-ANTORA/ha-design/overview.adoc

  • domus-infra-ops/runbooks/ipsk-manager-deployment.adoc

  • domus-infra-ops/runbooks/ipsk-failover.adoc

Session: ZSH History Cookbook

Created comprehensive command reference document from real zsh history patterns. Published to domus-linux-ops spoke site.

Document Location

Content Added

Interface and IP Listing (Network Diagnostics)

ip -4 -o addr show | awk '{print $2, $4}' | grep -E "^(enp|eth|wlan)"
ip -4 -o addr show | awk '{print $2, $4}' | grep -v "^lo"
ip -o link show | awk -F': ' '{print $2}'
ip route | awk '/default/ {print $3}'
ip route | awk '/default/ {print $5}'

Context: The -o flag gives one-line output per interface, making it awk-friendly. The grep -E "^(enp|eth|wlan)" filters to physical interfaces only, excluding docker bridges, virbr0, and other virtual interfaces.

AWK Block/Context Extraction

awk '/modestus-aw/{found=NR} found && NR<=found+5' ~/.ssh/config
awk '/^Host.*modestus-aw/,/^Host [^*]|^$/' ~/.ssh/config
awk '/10\.50\.40/{print NR": "$0}' ~/.ssh/config
awk '/\[section\]/,/^\[/' config.ini
awk '/BEGIN/,/END/' logfile.txt

Context:

  • {found=NR} found && NR⇐found+5 — Store line number when pattern matches, then print that line plus next N lines. Essential for extracting config blocks.

  • /pattern1/,/pattern2/ — Range pattern. Prints all lines from first match of pattern1 to first match of pattern2. The SSH config example extracts a complete Host block.

  • {print NR": "$0} — Prepend line numbers to output for quick reference.

GSSAPI/Kerberos SSH Debugging

ssh -vv evanusmodestus@inside.domusdigitalis.dev@modestus-aw 2>&1 | grep -i -E "gssapi|authentic"
grep -n GSSAPIAuthentication /etc/ssh/sshd_config
sudo sed -i '73s/#GSSAPIAuthentication no/GSSAPIAuthentication yes/' /etc/ssh/sshd_config
awk 'NR==73' /etc/ssh/sshd_config
sudo systemctl restart sshd
sudo sshd -T | grep -i gssapi

Context: Complete workflow for enabling GSSAPI authentication. The sshd -T command shows effective configuration after all includes are processed.

Session: Cloudflare Pages Deployment

Discovered that Cloudflare Pages only triggers on domus-docs hub commits, not spoke repo commits. Spoke repos (domus-linux-ops, domus-captures, etc.) require a trigger commit to domus-docs.

Deployment Pattern

# After pushing to a spoke repo, trigger hub rebuild:
cd ~/atelier/_bibliotheca/domus-docs
git commit --allow-empty -m "trigger rebuild: <spoke-name> <change-summary>"
git push

Commits Today

Repo Commit Description

domus-linux-ops

057f44b

Initial zsh-history-cookbook (976 lines)

domus-linux-ops

82b8fab

Add ip/awk interface listing patterns

domus-linux-ops

056ebab

Prefer grep -E for physical interfaces

domus-linux-ops

190d62e

Add awk block/context extraction patterns

domus-docs

bc30965

Trigger rebuild for zsh-history-cookbook

AWK Learning Resources

Curated references for deepening awk mastery:

Resource URL

Grymoire AWK Tutorial (Bruce Barnett)

www.grymoire.com/Unix/Awk.html

TutorialsPoint AWK

www.tutorialspoint.com/awk/index.htm

GeeksforGeeks AWK Examples

www.geeksforgeeks.org/linux-unix/awk-command-unixlinux-examples/

Simple AWK (GitHub)

github.com/adrianlarion/simple-awk

Hackr.io AWK Tutorials

hackr.io/tutorials/learn-awk

Reflection

The zsh-history-cookbook approach — extracting real commands from actual history rather than inventing examples — produces documentation that’s immediately useful. The patterns reflect actual operational needs, not theoretical exercises.

Key awk insights captured today:

  • Range patterns (/start/,/end/) are underutilized for config file parsing

  • The {found=NR} trick for context extraction is cleaner than grep -A

  • Combining ip -o with awk eliminates the need for complex parsing

Session: Spoke Validation and Date Corrections

Problem

  1. Antora build warnings for unescaped {id} and {realm} in WRKLOG-2026-02-14.adoc

  2. Day-of-week errors across 13 worklog files (Feb 02, 03, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17)

Root Cause

Date drift from copy-paste workflow. When creating new worklogs from templates or previous day’s files, the day-of-week wasn’t recalculated—it was manually typed or carried over incorrectly. Each successive file inherited and propagated the error, causing a systematic one-day offset across the entire week.

Contributing factors:

  • No automated date validation in the worklog creation workflow

  • Human error compounded across multiple files

  • No pre-commit hook to catch inconsistencies before push

The fix process itself was incomplete initially—first pass only caught Feb 03, 07, 13-17. Second pass required to fix Feb 02, 03, 06, 08-12. This demonstrates the importance of validating ALL files, not just the ones you think are broken.

Validation Loop (All Spokes)

for d in 01 02 03 06 07 08 09 10 11 12 13 14 15 16 17; do
  echo "Feb $d: $(date -d '2026-02-'$d +%A)"
done

Date Check Pattern

actual_day=$(date -d "2026-02-15" +%A)
in_file=$(grep -m1 "Date:" WRKLOG-2026-02-15.adoc)
echo "$actual_day | $in_file"

Bulk Fix with sed

Pass 1: Initial fixes (incomplete)

sed -i 's/(Thursday)/(Tuesday)/' WRKLOG-2026-02-03.adoc
sed -i 's/2026-02-07 (Friday)/2026-02-07 (Saturday)/' WRKLOG-2026-02-07.adoc
sed -i 's/2026-02-13 (Thursday)/2026-02-13 (Friday)/' WRKLOG-2026-02-13.adoc
sed -i 's/2026-02-14 (Friday)/2026-02-14 (Saturday)/' WRKLOG-2026-02-14.adoc
sed -i 's/2026-02-15 (Saturday)/2026-02-15 (Sunday)/' WRKLOG-2026-02-15.adoc
sed -i 's/2026-02-16 (Sunday)/2026-02-16 (Monday)/' WRKLOG-2026-02-16.adoc
sed -i 's/2026-02-17 (Monday)/2026-02-17 (Tuesday)/' WRKLOG-2026-02-17.adoc

Pass 2: Remaining files caught on second validation

sed -i 's/{capture-date} (Sunday)/{capture-date} (Monday)/' WRKLOG-2026-02-02.adoc
sed -i 's/{capture-date} (Monday)/{capture-date} (Tuesday)/' WRKLOG-2026-02-03.adoc
sed -i 's/{capture-date} (Thursday)/{capture-date} (Friday)/' WRKLOG-2026-02-06.adoc
sed -i 's/2026-02-08$/2026-02-08 (Sunday)/' WRKLOG-2026-02-08.adoc
sed -i 's/2026-02-09$/2026-02-09 (Monday)/' WRKLOG-2026-02-09.adoc
sed -i 's/2026-02-10$/2026-02-10 (Tuesday)/' WRKLOG-2026-02-10.adoc
sed -i 's/2026-02-11$/2026-02-11 (Wednesday)/' WRKLOG-2026-02-11.adoc
sed -i 's/2026-02-12$/2026-02-12 (Thursday)/' WRKLOG-2026-02-12.adoc

WRKLOG-2026-02-17 (Tomorrow’s Prep)

Note: WRKLOG-2026-02-17 exists as a pre-planning document for tomorrow’s deliverables (Xianming Linux delivery, CISO Research VLAN meeting). Keeping it as-is—tasks apply to both today’s prep and tomorrow’s execution.

Spoke Build Validation Script

Created domus-docs/scripts/validate-all-spokes.sh:

SPOKES=(domus-captures domus-linux-ops domus-infra-ops ...)

for repo in "${SPOKES[@]}"; do
    path="$BASE/$repo"
    if [[ -d "$path" && -f "$path/Makefile" ]]; then
        echo "=== $repo ==="
        WARNINGS=$(make 2>&1 | grep -E "WARN|ERROR")
        if [[ -n "$WARNINGS" ]]; then
            echo "$WARNINGS"
            FAILED+=("$repo")
        else
            echo "CLEAN"
        fi
    fi
done

Pre-commit Hook (domus-captures)

Created .git/hooks/pre-commit to block commits with build warnings:

OUTPUT=$(make 2>&1)
WARNINGS=$(echo "$OUTPUT" | grep -E "WARN|ERROR")
if [ -n "$WARNINGS" ]; then
    echo "BUILD WARNINGS/ERRORS DETECTED:"
    echo "$WARNINGS"
    exit 1
fi

Lesson

Always validate builds before pushing. Configure AND test.

Session: BIND DNS Server Deployment (bind-01)

Successfully deployed internal BIND DNS server at 10.50.1.90. This eliminates dependency on pfSense DNS and provides proper AD SRV record support for Linux domain join.

Zone File: inside.domusdigitalis.dev

$TTL 86400
@   IN  SOA     bind-01.inside.domusdigitalis.dev. admin.inside.domusdigitalis.dev. (
                2026021401  ; Serial (YYYYMMDDNN)
                3600        ; Refresh
                1800        ; Retry
                604800      ; Expire
                86400 )     ; Minimum TTL

; Name servers
@               IN  NS      bind-01.inside.domusdigitalis.dev.

; Gateway (.1)
pfsense-01      IN  A       10.50.1.1

; Network Devices (.10-19)
3560cx-01       IN  A       10.50.1.10
9300-01         IN  A       10.50.1.11

; Identity Services (.20-29)
ise-01          IN  A       10.50.1.20
ise-02          IN  A       10.50.1.21

; iPSK Manager (.30-39)
ipsk-mgr-01     IN  A       10.50.1.30
ipsk-mgr-02     IN  A       10.50.1.31

; Wireless (.40-49)
9800-wlc-01     IN  A       10.50.1.40
wlc-01          IN  A       10.50.1.40

; Windows Servers (.50-59)
home-dc01       IN  A       10.50.1.50
home-dc02       IN  A       10.50.1.51

; Active Directory SRV Records (CRITICAL for Kerberos/LDAP)
_kerberos._tcp          IN  SRV 0 100 88   home-dc01
_kerberos._udp          IN  SRV 0 100 88   home-dc01
_ldap._tcp              IN  SRV 0 100 389  home-dc01
_kpasswd._tcp           IN  SRV 0 100 464  home-dc01
_kpasswd._udp           IN  SRV 0 100 464  home-dc01
_gc._tcp                IN  SRV 0 100 3268 home-dc01
_ldap._tcp.dc._msdcs    IN  SRV 0 100 389  home-dc01
_kerberos._tcp.dc._msdcs IN SRV 0 100 88   home-dc01

; PKI Services (.60-69)
certmgr-01      IN  A       10.50.1.60
certmgr-02      IN  A       10.50.1.61

; Storage/Git (.70-79)
nas-01          IN  A       10.50.1.70
nas-02          IN  A       10.50.1.71
gitea-01        IN  A       10.50.1.70

; IdP/SSO (.80-89)
keycloak-01     IN  A       10.50.1.80
keycloak-02     IN  A       10.50.1.81

; DNS Services (.90-99)
bind-01         IN  A       10.50.1.90
bind-02         IN  A       10.50.1.91
kvm-01          IN  A       10.50.1.99

; LDAP/Directory (.100-109)
ipa-01          IN  A       10.50.1.100
ipa-02          IN  A       10.50.1.101

; Load Balancers (.110-119)
netscaler-01    IN  A       10.50.1.110
netscaler-02    IN  A       10.50.1.111

; IPMI/BMC (.200-209)
ipmi-01         IN  A       10.50.1.200

; Aliases
ise             IN  CNAME   ise-01
keycloak        IN  CNAME   keycloak-01
ipsk            IN  CNAME   ipsk-mgr-01
dc              IN  CNAME   home-dc01
ipa             IN  CNAME   ipa-01
dns             IN  CNAME   bind-01
vault           IN  CNAME   certmgr-01
nas             IN  CNAME   nas-01
gitea           IN  CNAME   gitea-01
wlc             IN  CNAME   9800-wlc-01
lb              IN  CNAME   netscaler-01

Design Decisions

IP Addressing Scheme

Functional grouping by /24 subnet range:

Range Function Examples

.1

Gateway

pfsense-01

.10-19

Network devices

3560cx-01, 9300-01

.20-29

Identity services (ISE)

ise-01, ise-02

.30-39

iPSK Manager

ipsk-mgr-01, ipsk-mgr-02

.40-49

Wireless controllers

9800-wlc-01

.50-59

Windows/AD

home-dc01, home-dc02

.60-69

PKI/Vault

certmgr-01, certmgr-02

.70-79

Storage/Git

nas-01, gitea-01

.80-89

IdP/SSO

keycloak-01, keycloak-02

.90-99

DNS/Hypervisor

bind-01, kvm-01

.100-109

FreeIPA

ipa-01, ipa-02

.110-119

Load balancers

netscaler-01

.200-209

IPMI/BMC

ipmi-01

AD SRV Records

Critical for Linux domain join. Without these, realm discover and kinit fail because SSSD/Kerberos cannot locate the domain controller.

Record Port Purpose

_kerberos._tcp

88

Kerberos KDC discovery

_kerberos._udp

88

Kerberos KDC (UDP fallback)

_ldap._tcp

389

LDAP directory service

_kpasswd._tcp

464

Password change service

_gc._tcp

3268

Global Catalog (multi-domain)

_ldap._tcp.dc._msdcs

389

DC locator (Microsoft-specific)

_kerberos._tcp.dc._msdcs

88

Kerberos DC locator

CNAME Aliases

Operational convenience - use ssh dc instead of ssh home-dc01:

  • ise → ise-01 (ISE primary)

  • dc → home-dc01 (Domain controller)

  • vault → certmgr-01 (HashiCorp Vault)

  • nas → nas-01 (NFS/SMB storage)

Validation

Server-side (from bind-01)

ssh bind-01 "sudo rndc zonestatus inside.domusdigitalis.dev"
name: inside.domusdigitalis.dev
type: primary
files: inside.domusdigitalis.dev.zone
serial: 2026021601
nodes: 51
last loaded: Mon, 16 Feb 2026 18:45:42 GMT
secure: no
dynamic: no
reconfigurable via modzone: no
ssh bind-01 "sudo named-checkzone inside.domusdigitalis.dev /var/named/inside.domusdigitalis.dev.zone"
zone inside.domusdigitalis.dev/IN: loaded serial 2026021601
OK

Client-side (from workstation)

dig _kerberos._tcp.inside.domusdigitalis.dev SRV +short
0 100 88 home-dc01.inside.domusdigitalis.dev.
dig ipa-01.inside.domusdigitalis.dev +short
10.50.1.100

Kerberos Discovery

realm discover inside.domusdigitalis.dev

Impact

  • Linux domain join now works - SSSD can discover AD via SRV records

  • Eliminates pfSense DNS dependency - Internal DNS fully self-hosted

  • Operational shortcuts - ssh vault, ssh ise, ssh dc

  • HA-ready - bind-02 (.91) reserved for secondary

Client Configuration Required

Workstations must be configured to use bind-01 as DNS. Without this, queries go to pfSense/upstream and fail:

# WRONG - queries pfSense (10.50.10.1), fails with NXDOMAIN
dig ipa-01

# RIGHT - explicit server and FQDN
dig @10.50.1.90 ipa-01.inside.domusdigitalis.dev +short

Option 1: NetworkManager (per-connection)

Find your 802.1X connection name:

nmcli connection show | grep -E "802\.1X"
Wired-802.1X-Vault   3f9fefb8-4da8-4a45-9c86-971b20afee42  ethernet  enp130s0
Domus-Secure-802.1X  23672874-9cc0-48cf-9c72-d0e0dee53a7a  wifi      wlan0

Configure DNS for the connection:

nmcli connection modify "Wired-802.1X-Vault" ipv4.dns "10.50.1.90"
nmcli connection modify "Wired-802.1X-Vault" ipv4.dns-search "inside.domusdigitalis.dev"
nmcli connection modify "Wired-802.1X-Vault" ipv4.ignore-auto-dns yes
nmcli connection up "Wired-802.1X-Vault"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/63)

Option 2: systemd-resolved (system-wide)

# /etc/systemd/resolved.conf
[Resolve]
DNS=10.50.1.90
Domains=inside.domusdigitalis.dev
sudo systemctl restart systemd-resolved
resolvectl status

Option 3: DHCP (recommended for domain clients)

Configure pfSense DHCP to distribute: - DNS Server: 10.50.1.90 - Domain: inside.domusdigitalis.dev

Pending: FreeIPA Integration

FreeIPA servers reserved at .100-109 but not yet deployed:

Host IP Status

ipa-01

10.50.1.100

DNS record ready, VM pending

ipa-02

10.50.1.101

DNS record ready, VM pending

When deployed, add FreeIPA SRV records:

; FreeIPA SRV Records (add when ipa-01 is live)
_kerberos._tcp.ipa      IN  SRV 0 100 88   ipa-01
_kerberos._udp.ipa      IN  SRV 0 100 88   ipa-01
_ldap._tcp.ipa          IN  SRV 0 100 389  ipa-01
_kpasswd._tcp.ipa       IN  SRV 0 100 464  ipa-01

Session: GSSAPI SSH Troubleshooting (modestus-aw)

Successfully enabled Kerberos/GSSAPI SSH authentication from modestus-razer to modestus-aw. This was a multi-issue troubleshooting session that mirrors the CHLA deployment pattern.

Problem Statement

SSH with GSSAPI authentication was failing with Permission denied (publickey,gssapi-with-mic) despite:

  • Valid Kerberos ticket on client

  • SSSD running on target

  • GSSAPIAuthentication enabled in sshd_config

Root Cause Analysis

Issue Root Cause Fix

Realm join failed

dACL missing port 464 (kpasswd)

Added TCP/UDP 464 + response rules

Realm join failed

dACL missing Kerberos response rules

Added permit tcp/udp any eq 88 any gt 1023

GSSAPI SSH failed

Keytab had wrong SPN (.localdomain)

Rejoin realm with correct FQDN hostname

GSSAPI SSH failed

Hostname not FQDN

hostnamectl set-hostname modestus-aw.inside.domusdigitalis.dev

DNS not resolving

Bind forwarding + serial not updated

Added to /etc/hosts (workaround)

dACL Fix: Missing AD Protocol Ports

Original dACL (broken):

permit tcp any host 10.50.1.50 eq 88   # Kerberos outbound - OK
# MISSING: permit tcp any eq 88 any gt 1023  # No response rule!
# MISSING: port 464 entirely (kpasswd)

Fixed dACL:

remark AD/Kerberos - auth and password change
permit tcp any host 10.50.1.50 eq 88
permit tcp any eq 88 any gt 1023
permit udp any host 10.50.1.50 eq 88
permit udp any eq 88 any gt 1023
permit tcp any host 10.50.1.50 eq 464
permit tcp any eq 464 any gt 1023
permit udp any host 10.50.1.50 eq 464
permit udp any eq 464 any gt 1023
permit tcp any host 10.50.1.50 eq 389
permit tcp any eq 389 any gt 1023
permit tcp any host 10.50.1.50 eq 636
permit tcp any eq 636 any gt 1023
permit tcp any host 10.50.1.50 eq 445
permit tcp any eq 445 any gt 1023
permit tcp any host 10.50.1.50 eq 3268
permit tcp any eq 3268 any gt 1023

dACL Update Workflow

# 1. Detach dACL from profile
netapi ise update-authz-profile "Linux_EAPTLS_Permit" --no-dacl

# 2. Delete old dACL
netapi ise delete-dacl "Linux_Research_Zero_Trust_v2" --force

# 3. Create new dACL from file
netapi ise create-dacl "Linux_Research_Zero_Trust_v2" \
    --file /tmp/dacl-research-v5.txt \
    --descr "Zero-trust + AD auth + kpasswd + SSH"

# 4. Reattach dACL to profile
netapi ise update-authz-profile "Linux_EAPTLS_Permit" \
    --dacl "Linux_Research_Zero_Trust_v2"

# 5. Bounce port to apply
netapi ios bounce gi1/0/5

Hostname and Realm Rejoin

The keytab had wrong SPNs because hostname wasn’t FQDN:

Wrong (in keytab):

host/modestus-aw.localdomain@INSIDE.DOMUSDIGITALIS.DEV

Correct (after fix):

host/modestus-aw.inside.domusdigitalis.dev@INSIDE.DOMUSDIGITALIS.DEV

Fix procedure:

# 1. Set proper FQDN
sudo hostnamectl set-hostname modestus-aw.inside.domusdigitalis.dev

# 2. Leave realm
sudo realm leave

# 3. CRITICAL: kinit BEFORE realm join (avoids "Preauthentication failed")
kinit Administrator@INSIDE.DOMUSDIGITALIS.DEV

# 4. Rejoin realm
sudo realm join -v inside.domusdigitalis.dev

# 5. Verify keytab has correct SPNs
sudo klist -k /etc/krb5.keytab | grep -i host

SSH Verification Commands

Client-side (modestus-razer):

# Force GSSAPI only (no ControlMaster interference)
ssh -vvv -o ControlPath=none -o GSSAPIAuthentication=yes \
    -o PreferredAuthentications=gssapi-with-mic \
    evanusmodestus@10.50.40.102 2>&1 | grep -E "gssapi|GSSAPI|userauth"

Server-side (modestus-aw) - watch auth in real-time:

journalctl -u sshd -f

Server-side - post-hoc analysis:

journalctl -u sshd --since "2 minutes ago" | grep -iE "gssapi|krb|pam|auth"

Success Indicators

sshd log showing successful GSSAPI auth:

Feb 16 10:59:29 modestus-aw sshd-session[730555]: Authorized to evanusmodestus, krb5 principal evanusmodestus@INSIDE.DOMUSDIGITALIS.DEV (krb5_kuserok)
Feb 16 10:59:29 modestus-aw sshd-session[730555]: Accepted gssapi-with-mic for evanusmodestus from 10.50.10.130 port 60216 ssh2: evanusmodestus@INSIDE.DOMUSDIGITALIS.DEV
Feb 16 10:59:29 modestus-aw sshd-session[730555]: pam_unix(sshd:session): session opened for user evanusmodestus(uid=1000) by evanusmodestus(uid=0)

SSH Failure Diagnostics Table

Symptom Root Cause Action

Connection refused

sshd not running/listening

sudo systemctl enable --now sshd

Timeout (no response)

dACL/firewall blocking

Check dACL rules, verify port 22 permitted

Permission denied

Auth method mismatch

Check sshd_config (GSSAPI, PAM), verify Kerberos ticket

Quick Port Test

timeout 3 bash -c "</dev/tcp/10.50.40.102/22" && echo "OPEN" || echo "BLOCKED"

  • OPEN → sshd listening, proceed with auth troubleshooting

  • Connection refused → sshd not running, need local access

  • Timeout → Network/dACL blocking

Hexdump for Zone File Debugging

# Check for hidden characters in bind zone file
ssh bind-01 "sudo hexdump -C /var/named/inside.domusdigitalis.dev.zone | grep -A1 -B1 'modestus-aw'"

# Show invisible characters ($ = newline)
ssh bind-01 "sudo cat -A /var/named/inside.domusdigitalis.dev.zone | grep modestus"

NSSwitch Resolution Order Fix

Issue: GSSAPI SSH failed by hostname but worked by IP.

Root Cause: /etc/nsswitch.conf had resolve before files. systemd-resolved returned IPv6 link-local from mDNS before checking /etc/hosts.

# Diagnosis - shows IPv6 link-local instead of /etc/hosts entry
getent hosts modestus-aw
# Returns: fe80::xxxx%enp0s31f6 modestus-aw

# Wrong order
awk '/^hosts/' /etc/nsswitch.conf
# hosts: mymachines resolve [!UNAVAIL=return] files myhostname dns

# Fix - put files before resolve
sudo sed -i 's/^hosts:.*/hosts: files mymachines resolve [!UNAVAIL=return] myhostname dns/' /etc/nsswitch.conf

# Verify
getent hosts modestus-aw
# Returns: 10.50.40.102 modestus-aw.inside.domusdigitalis.dev modestus-aw

Key Learning: For GSSAPI/Kerberos SSH, hostname must resolve to correct IPv4 from /etc/hosts before systemd-resolved attempts mDNS/LLMNR.

SSH Client PreferredAuthentications Override

Issue: GSSAPI SSH never attempted despite server supporting it.

Root Cause: Custom ~/.ssh/config had PreferredAuthentications publickey,password - GSSAPI wasn’t in the list.

# Diagnosis
awk '/^Host modestus-aw/{found=1} found && /PreferredAuth/{print; exit}' ~/.ssh/config

# Fix - add gssapi-with-mic to front
sed -i '/^Host modestus-aw$/,/^Host /{s/PreferredAuthentications publickey,password/PreferredAuthentications gssapi-with-mic,publickey,password/}' ~/.ssh/config

Key Finding: Researchers with NO custom ~/.ssh/config get GSSAPI automatically:

# Test without custom config (simulates researcher)
ssh -F /dev/null -o GSSAPIAuthentication=yes user@host

Impact

  • GSSAPI SSH now works - Passwordless Kerberos authentication from admin workstation (hostname AND IP)

  • NSSwitch order critical - files before resolve for hostname-based GSSAPI

  • SSH config aware - Custom PreferredAuthentications can block GSSAPI

  • Researchers need NO config - System defaults work out of box

  • dACL template corrected - All AD ports with response rules documented

  • Runbook updated - Appendix C v2.7 with complete troubleshooting guide

  • Mirrors CHLA pattern - Same issues, same fixes apply to production

Session: P50 Cert Deployment (Gabriel)

Successfully deployed Vault-issued certificate to Gabriel’s P50 workstation for 802.1X authentication on both wired and wireless interfaces.

Certificate Details

Field Value

Common Name

modestus-p50.inside.domusdigitalis.dev

Organization (O)

Domus-Infrastructure

Organizational Unit (OU)

Domus-Users

Vault Role

domus-client-users

TTL

8760h (1 year)

Two-Hop SCP Workflow

P50 cannot resolve certmgr-01 directly. Used razer as intermediate hop.

# Step 1: From razer - pull from certmgr-01
scp certmgr-01:/tmp/modestus-p50-eaptls.{pem,key} certmgr-01:/tmp/domus-ca-chain.pem /tmp/

# Step 2: Verify cert fields with awk
openssl x509 -in /tmp/modestus-p50-eaptls.pem -noout -subject | \
  awk -F', ' '{gsub(/^subject=/, "", $1); for(i=1;i<=NF;i++) print $i}'
Output
O=Domus-Infrastructure
OU=Domus-Users
CN=modestus-p50.inside.domusdigitalis.dev
# Step 3: Push to P50
scp /tmp/modestus-p50-eaptls.{pem,key} /tmp/domus-ca-chain.pem modestus-p50:/tmp/

P50 Cert Installation

sudo cp /tmp/modestus-p50-eaptls.pem /etc/ssl/certs/
sudo cp /tmp/modestus-p50-eaptls.key /etc/ssl/private/
sudo chmod 600 /etc/ssl/private/modestus-p50-eaptls.key

Validation Results

Wired (enp0s31f6)

netapi ios exec "show access-s int g1/0/5 detail"
Field Value

MAC

c8:5b:76:c6:59:62

User-Name

modestus-p50.inside.domusdigitalis.dev

Status

Authorized

VLAN

10

IPv4

10.50.10.136

Session timeout

28800s (8hr)

Network Device

3560CX-8PC (switch)

Wireless (wlan0)

netapi ise mnt auth-status "14:f6:d8:7b:31:80"
Field Value

MAC

14:f6:d8:7b:31:80

User-Name

modestus-p50.inside.domusdigitalis.dev

AuthZ Profile

Domus_Secure_Profile

Endpoint Profile

Linux-Workstation

IPv4

10.50.10.121

Network Device

Home-9800-WLC

Key awk Commands Used

Cert field extraction:

openssl x509 -in /tmp/modestus-p50-eaptls.pem -noout -subject | \
  awk -F', ' '{gsub(/^subject=/, "", $1); for(i=1;i<=NF;i++) print $i}'

Find wifi SSIDs matching pattern:

nmcli device wifi list | awk '/Domus|Secure/ {print $2}'

Find wired interface name:

ip link | awk '/state UP|enp|eth/ && !/lo/ {gsub(/:/, "", $2); print $2}'

Check wifi connection status:

nmcli conn show | awk '/wifi/ {print $1, $2, $3}'
nmcli device status | awk '/wifi|wlan/ \{print}'

Session: xargs/awk Command Mastery

Reference patterns for batch operations and text processing.

xargs Fundamentals

When to use xargs:

  • Taking stdin and passing as arguments to a command

  • Batch operations on multiple items

  • Parallel execution with -P

When NOT to use xargs:

  • Sequential dependent operations (use && instead)

  • Single items (just run the command directly)

xargs Patterns

Batch NetworkManager Operations

# Bounce ALL wifi connections
nmcli conn show | awk '/wifi/ {print $1}' | xargs -I{} sh -c 'nmcli conn down "{}" && nmcli conn up "{}"'

# Delete old connections matching pattern
nmcli conn show | awk '/old-wifi/ {print $1}' | xargs -I{} nmcli conn delete "{}"

ISE Profile Operations

# Update reauth timer on all Domus profiles
echo "Domus_Admin_Profile Domus_Research_Profile Domus_Secure_Profile Domus_IoT_Profile" | \
  tr ' ' '\n' | xargs -I{} netapi ise update-authz-profile "{}" --reauth-timer 28800

# Get dACL attached to multiple profiles
echo "Profile1 Profile2 Profile3" | tr ' ' '\n' | \
  xargs -I{} sh -c 'echo "=== {} ===" && netapi ise get-authz-profile "{}" | jq .daclName'

File Operations

# Find and delete files matching pattern
find /tmp -name "*.bak" -print0 | xargs -0 rm -v

# Copy multiple certs to remote host
ls /tmp/*.pem | xargs -I{} scp {} remote-host:/etc/ssl/certs/

# Parallel execution (4 jobs)
cat hosts.txt | xargs -P4 -I{} ssh {} "uptime"

Git Operations

# Add all modified .adoc files
git status --porcelain | awk '/\.adoc$/ {print $2}' | xargs git add

# Show commits touching specific files
find . -name "*.py" | xargs git log --oneline --

# Bulk rename with pattern
git status --porcelain | awk '/old-name/ {print $2}' | \
  xargs -I{} sh -c 'git mv "{}" "$(echo {} | sed s/old-name/new-name/)"'

awk Quick Reference

Pattern Description Example

{print $N}

Print Nth field

awk '{print $2}'

-F','

Set field separator

awk -F',' '{print $1}'

/regex/

Match lines

awk '/ERROR/ {print}'

!/regex/

Exclude lines

awk '!/DEBUG/ {print}'

NR

Line number

awk 'NR==5 {print}'

NF

Number of fields

awk '{print $NF}' (last field)

gsub()

Global substitution

awk '{gsub(/old/, "new"); print}'

/p1/,/p2/

Range pattern

awk '/START/,/END/ {print}'

{found=NR}

Store line number

Context extraction (see below)

awk Context Extraction Patterns

# Print line + next N lines after match
awk '/pattern/{found=NR} found && NR<=found+5' file

# Print complete config block
awk '/^Host modestus-aw/,/^Host [^*]|^$/' ~/.ssh/config

# Print with line numbers
awk '/pattern/{print NR": "$0}' file

# Extract between markers (inclusive)
awk '/BEGIN/,/END/' file

# Extract between markers (exclusive)
awk '/BEGIN/{flag=1; next} /END/{flag=0} flag' file

Combining awk + xargs

# Find files by content and process them
grep -l "pattern" *.txt | xargs -I{} awk '/specific/{print FILENAME": "$0}' {}

# Extract IPs and ping them
awk '/server/ {print $2}' config.txt | xargs -I{} ping -c1 {}

# Process ISE policy sets
netapi ise get-policy-sets | awk 'NR>2 {print $1}' | \
  xargs -I{} sh -c 'echo "=== {} ===" && netapi ise authz "{}"'

Next Actions

  • Add more patterns as they emerge from daily operations

  • Cross-reference cookbook from other domus-* components

  • Consider domus-captures REF-2026-02-awk-siem-mastery.adoc consolidation

  • Pre-commit hook for domus-captures

  • validate-all-spokes.sh script for domus-docs

  • P50 cert deployment documented

  • xargs/awk mastery patterns documented

  • GSSAPI SSH troubleshooting documented

  • use_fully_qualified_names = False requirement documented

Session: GSSAPI SSH Troubleshooting

Successfully debugged and documented GSSAPI SSH authentication for researcher simulation.

Root Cause: use_fully_qualified_names

use_fully_qualified_names = True in /etc/sssd/sssd.conf BREAKS GSSAPI SSH.

  • True: Local username = user@domain.com, Kerberos principal = user@REALM → NO MATCH

  • False: Local username = user, Kerberos principal = user@REALM → MAPS via krb5_kuserok

Fix Applied on modestus-aw

sudo sed -i 's/use_fully_qualified_names = True/use_fully_qualified_names = False/' /etc/sssd/sssd.conf
sudo systemctl restart sssd

Researcher Simulation Test

Simulates researcher with NO custom SSH config:

ssh -F /dev/null -o GSSAPIAuthentication=yes -o PreferredAuthentications=gssapi-with-mic -o PubkeyAuthentication=no evanusmodestus@modestus-aw

Result: SUCCESS - connected via GSSAPI without password/keys.

Troubleshooting Commands Used

Check Command

Kerberos ticket valid?

klist | head -5

Get service ticket

kvno host/modestus-aw.inside.domusdigitalis.dev@INSIDE.DOMUSDIGITALIS.DEV

Effective sshd GSSAPI config

sudo sshd -T | grep -i gssapi

Server keytab SPNs

sudo klist -k /etc/krb5.keytab | grep host

use_fully_qualified_names

awk '/use_fully_qualified/' /etc/sssd/sssd.conf

krb5 renewal settings

sudo awk '/krb5/' /etc/sssd/sssd.conf

Kill frozen SSH

pkill -f "ssh.*modestus-aw"

Watch sshd auth live

sudo journalctl -u sshd -f

Other Fixes Applied

  1. krb5 ticket auto-renewal added to sssd.conf template:

    • krb5_renewable_lifetime = 7d

    • krb5_renew_interval = 60

  2. Static IP set on modestus-aw (was DHCP, IP kept changing)

  3. GSSAPIStrictAcceptorCheck no added to sshd_config (helps with SPN mismatches)

Impact

  • GSSAPI SSH now works for researchers

  • No custom SSH config needed

  • No SSH keys needed

  • Just kinit at login, then ssh user@host

  • Runbook updated with complete troubleshooting guide