WRKLOG-2026-03-12

Summary

EAP-TLS and work prep day. Fix modestus-razer WiFi authentication (missing intermediate CA in client cert chain). Vocera phone 802.1X triage at work.

Today’s Priorities (2026-03-12)

Priority Task Status Notes

P0

Fix modestus-razer EAP-TLS

[x] RESOLVED 03-13

Actual root cause: IP conflict + MAC randomization (not cert chain). See RCA-2026-03-13-001

P0

Vocera EAP-TLS Supplicant Fix (Work)

[x] DOCUMENTED

~10 phones failing 802.1X, TAC case for supplicant config

P1

Monad Pipeline Evaluation

→DEFERRED

Carried forward - see WRKLOG-2026-03-14

P1

k3s NAT verification

→DEFERRED

Carried forward since 03-09

P2

Wazuh indexer recovery

→DEFERRED

Depends on NAT - carried forward

Carried Over from 2026-03-11

Task Details Status

modestus-razer EAP-TLS

Root cause: IP conflict + MAC randomization (not cert chain). ISE auth succeeded but DHCP config failed

[x] RESOLVED 03-13

k3s NAT verification

NAT rule 170 for 10.42.0.0/16 pod network

→DEFERRED

Wazuh indexer recovery

Restart after NAT confirmed

→DEFERRED

Monad Pipeline

Test pipeline creation, input sources

→DEFERRED

PacketFence VM

Deploy packetfence-01 on kvm-02 for FreeRADIUS learning

→DEFERRED

Session Log

Session 1: modestus-razer EAP-TLS Fix

Objective: Fix intermittent WiFi authentication by adding intermediate CA to client cert.

Root cause (from 03-11):

Client cert file contains only leaf cert. ISE expects full chain during EAP-TLS handshake.

Fix commands:

# Create chain file (leaf + intermediate)
cat /etc/ssl/certs/modestus-razer-eaptls.pem \
    /etc/ssl/certs/DOMUS-ISSUING-CA.pem \
    > /tmp/modestus-razer-chain.pem
# Verify chain has 2 certs
openssl crl2pkcs7 -nocrl -certfile /tmp/modestus-razer-chain.pem | \
    openssl pkcs7 -print_certs -noout | grep -c "subject="
# Expected: 2
# Install chain file
sudo cp /tmp/modestus-razer-chain.pem /etc/ssl/certs/modestus-razer-eaptls-chain.pem
# Update NM connection
sudo nmcli con modify "Domus-WiFi-EAP-TLS" \
    802-1x.client-cert /etc/ssl/certs/modestus-razer-eaptls-chain.pem
# Reconnect and verify
nmcli con down "Domus-WiFi-EAP-TLS" && nmcli con up "Domus-WiFi-EAP-TLS"
# Check ISE for successful auth
netapi ise dc query "SELECT acs_timestamp, passed, failure_reason
    FROM mnt.radius_auth_48_live
    WHERE calling_station_id LIKE '%razer%'
    ORDER BY acs_timestamp DESC LIMIT 5"

Result: Cert chain fix was applied but issue persisted. Actual root cause identified on 03-13: IP conflict + MAC randomization. See RCA-2026-03-13-001

Session 2: Security Hardening & M-Disc Archival

Objective: Complete all backup tiers and harden workstation in response to elevated threat environment.

Context: Iran/state-sponsored APT activity increasing. Full defense-in-depth audit initiated.

Backup Tiers Completed

Tier Description Status

Tier 1 HOT

~/.secrets sync

✓ DONE

Tier 2 WARM

Borg → NAS (nas-01)

✓ DONE

Tier 3 COLD

Seagate SSDs (primary + secondary sync)

✓ DONE

Tier 4 ARCHIVAL

M-Disc BD-R (31GB FULL + 14MB P0-CRITICAL)

✓ VERIFIED

M-Disc Verification

cd /mnt/cdrom && sha256sum -c SHA256SUMS.txt
# FULL-BACKUP-2026-03-11.tar.age: OK
# P0-CRITICAL-2026-03-11.tar.age: OK
# RECOVERY-README.txt: OK

Disc labeled and stored in fireproof safe.

Workstation Security Audit Findings

Issue Finding Action

Docker exposed

Redis (6379), PostgreSQL (5432, 5433) on 0.0.0.0

✓ FIXED - docker stop

rpcbind exposed

NFS portmapper on 0.0.0.0:111

✓ FIXED - systemctl mask

No firewall

nftables/iptables empty

⚠️ PENDING

Key Commands Learned

# Show what's listening on all interfaces (exposed to network)
sudo ss -tlnp | grep '0.0.0.0' | awk '{printf "%-20s %s\n", $4, $6}'

# Mask a service permanently (symlink to /dev/null)
sudo systemctl mask rpcbind.service rpcbind.socket

# Verify service is gone
ss -tlnp | grep :111 && echo "STILL RUNNING" || echo "CLEAN"

Optical Disc Operations Learned

# Check disc type and status
dvd+rw-mediainfo /dev/sr0 2>&1 | awk '/Mounted Media|Disc status/'

# BD-R burning (NOT wodim!)
growisofs -dvd-compat -Z /dev/sr0=file.iso

# Large files (>4GB) require -iso-level 3
genisoimage -iso-level 3 -J -r -o backup.iso files/

Key learning: wodim = CD/DVD only. BD-R requires growisofs.

Session 3: CHLA ISE TAC Case - 802.1X Auth Failures

Objective: Open and resolve S1 TAC case for ~500 endpoint authentication failures at work.

Problem: - ~500 endpoints failing 802.1X authentication - Live Logs showing "No data available" when clicking session details - Patient care devices affected (WOWs, Chromebooks)

Root Cause Identified: - Primary MNT (pmnt.ise.chla.org) RabbitMQ service at 109% CPU - Message queue saturated, preventing session data replication - Replication alarms visible on PAN dashboard

Resolution:

# SSH to Primary MNT
ssh pmnt.ise.chla.org

# Stop ISE services
application stop ise

# Reboot node
reload
# Save ade-os: yes

# Post-reboot verification (after ~10 min)
show application status ise
# All services running

# Verify logging restored
show logging application rabbitmq.log tail count 50

Result: Live Logs restored, session details now displaying correctly.

Follow-up Items from TAC:

Item Description Status

ISE Messaging Service

Enable "Use ISE Messaging Service for UDP syslogs delivery to MnT"

PENDING

Upgrade to Patch 9

ISE 3.2 Patch 9 addresses known replication issues

PENDING

MNT Replication

Monitor for recurrence, may need node deregister/re-register

MONITORING

Documentation Created: - TAC-2026-03-chla-8021x-auth-failures.adoc - full case notes - D2 diagram of ISE deployment with problem areas - ECCRB submission (retroactive emergency change) - Added MNT diagnostics to ise-tac-case-prep.adoc

Key Learning: RabbitMQ handles inter-node messaging in ISE. When overloaded, session data doesn’t replicate to MNT, causing "no data available" errors in Live Logs.

Session 4: Advanced Bash Training & Student Prep

Objective: Create training materials for advanced bash patterns; prepare Ubuntu installation guide for student migrations.

Documentation Created

File Content Lines

examples/codex/linux/ubuntu-install-encrypted.adoc

Ubuntu 24.04 LUKS install guide with file search appendix

537

pages/education/systems/advanced-bash-patterns.adoc

Process substitution, FIFOs, signals, subshells, coprocesses

635

Skills Practiced

Process substitution:

diff <(grep -E "^    [a-z]" ~/atelier/_bibliotheca/domus-infra-ops/docs/asciidoc/antora.yml | sort) \
     <(grep -E "^    [a-z]" ~/atelier/_bibliotheca/domus-captures/docs/antora.yml | sort)

comm for set operations:

comm -3 \
  <(find ~/atelier/_bibliotheca/domus-infra-ops -name "*.adoc" -printf "%f\n" | sort -u) \
  <(find ~/atelier/_bibliotheca/domus-captures -name "*.adoc" -printf "%f\n" | sort -u) \
  | awk '/^\t/{r++} /^[^\t]/{l++} END{print "infra-ops only:", l, "\ncaptures only:", r}'
# Result: infra-ops only: 275, captures only: 437

Skill Assessment

Level Status

Beginner

PASSED

Intermediate

CURRENT - terminal-first, pipelines, find+grep combos

Advanced

IN TRAINING - process substitution, FIFOs, signals

Eyes-closed typing practice initiated - building touch-typing muscle memory.

Session 5: CLI Training - Process Substitution, comm, Git Ranges

Objective: Hands-on practice with advanced CLI patterns.

Process Substitution <()

Creates a temporary file descriptor containing command output. Receiving command sees a path like /dev/fd/63.

# Compare two command outputs without temp files
diff <(command1) <(command2)

# Real example: compare antora.yml attributes across repos
diff <(grep -E "^    [a-z]" ~/atelier/_bibliotheca/domus-infra-ops/docs/asciidoc/antora.yml | sort) \
     <(grep -E "^    [a-z]" ~/atelier/_bibliotheca/domus-captures/docs/antora.yml | sort)
# No output = no differences (attributes synchronized)

comm - Set Operations

Compares two sorted files, outputs 3 columns:

Column Meaning

1 (no indent)

Only in file 1

2 (one tab)

Only in file 2

3 (two tabs)

In both files

 
# -3 suppresses column 3 (common lines)
comm -3 <(sort file1) <(sort file2)

# Real example: find .adoc files unique to each repo
comm -3 \
  <(find ~/atelier/_bibliotheca/domus-infra-ops -name "*.adoc" -printf "%f\n" | sort -u) \
  <(find ~/atelier/_bibliotheca/domus-captures -name "*.adoc" -printf "%f\n" | sort -u) \
  | head -20

comm + awk - Count Unique Files

comm -3 \
  <(find ~/atelier/_bibliotheca/domus-infra-ops -name "*.adoc" -printf "%f\n" | sort -u) \
  <(find ~/atelier/_bibliotheca/domus-captures -name "*.adoc" -printf "%f\n" | sort -u) \
  | awk '/^\t/{r++} /^[^\t]/{l++} END{print "infra-ops only:", l, "\ncaptures only:", r}'
# Result: infra-ops only: 275, captures only: 437

awk pattern breakdown:

Pattern Meaning

/^\t/

Line starts with tab (captures-only)

/[\t]/

Line starts with non-tab (infra-ops-only)

{r++}

Increment right counter

{l++}

Increment left counter

END{print …​}

Print totals after all lines processed

Git Log Range Syntax

# Show last N commits
git log HEAD~1..HEAD    # Last 1 commit
git log HEAD~2..HEAD    # Last 2 commits
git log HEAD~4..HEAD    # Last 4 commits

# The .. means "commits reachable from right but NOT from left"

Git Reflog - The Safety Net

Every HEAD movement is recorded. Commits "lost" to reset/rebase live here ~90 days.

git reflog | head -10

Output format:

45163b7 HEAD@{0}: commit: docs(growth): Add CLI Mastery Assessment
a4a04bf HEAD@{1}: commit: docs(worklog): Add Session 4
05153d4 HEAD@{2}: commit: feat(education): Add advanced bash patterns

Recovery pattern:

# Oops, reset too far!
git reset --hard HEAD~5

# Find lost commit in reflog
git reflog

# Recover to before the reset
git reset --hard HEAD@{1}

Training Reference

Full training doc: Advanced Bash Patterns

Covers 5 modules:

  1. Process substitution - diff <() <() patterns

  2. Named pipes (FIFOs) - mkfifo, reader/writer IPC

  3. Signal handling - trap, cleanup, graceful shutdown

  4. Subshells vs grouping - () isolation vs {} current shell

  5. Coprocesses - bidirectional pipe communication

HA Deployment Queue Status

Priority System Status Next Action

P1

BIND

COMPLETE

bind-01 + bind-02 (AXFR)

P2

Vault

COMPLETE

vault-01/02/03 (Raft HA)

P3

Keycloak

NEXT

Rebuild from scratch (corrupted)

P4

FreeIPA

PLANNED

ipa-01 + ipa-02 (IPA Replication)

P5

AD DC

PLANNED

home-dc01 + home-dc02 (AD Replication)

P6

iPSK

PLANNED

ipsk-mgr-01 + ipsk-mgr-02 (MySQL Replication)

P7

ISE

DEFERRED

ise-01 reconfigure after ise-02 stable

Key Learnings from 2026-03-11

EAP-TLS Chain Lesson

Issue Solution

ISE error 5411 "Supplicant stopped responding"

Client cert file missing intermediate CA - ISE waits 120s for full chain

"Secrets were required, but not provided"

Red herring - NM error when wpa_supplicant can’t complete handshake

Leaf-only cert + 2-tier PKI

Always include intermediate in client cert file for EAP-TLS

netapi JSON Output Lesson

Command Type JSON Support

ERS commands (get-, create-, update-*)

-f json works

api-call (raw API)

-f json works

MnT commands (mnt sessions, mnt coa)

NO JSON - human-readable only

DataConnect (dc query)

NO JSON - human-readable only

Tomorrow (2026-03-13)

Carried Over (If Not Complete Today)

  • k3s NAT verification

  • Wazuh indexer recovery

  • PacketFence VM deployment

HA Queue

  • Keycloak rebuild (P3)

  • FreeIPA ipa-02 replica (P4)

Runbook References

Runbook URL

Vault HA Deployment

Vault HA Deployment (infra-ops runbook)

KVM Operations

KVM Operations (infra-ops runbook)

ISE EAP-TLS Troubleshooting

NetworkManager Troubleshooting (TODO: create in ise-linux)