WRKLOG-2026-03-01

Summary

Saturday. Started RHCSA studies (command priority, I/O redirection). Home infrastructure: kvm-02 IPMI configuration, rack reorganization, Supermicro hardware documentation updates.

Today’s Priority Tasks

Priority Task Status

P0

CHLA Linux SSH issue (Xianming Ding)

[ ] CARRY-OVER

P1

kvm-02 Full Deployment

[x] COMPLETE (22/22)

P1

iPSK Manager - DB replication

[ ] CARRY-OVER

P1

MSCHAPv2 Migration Planning

[ ] CARRY-OVER

P2

RHCSA 9 study - Chapter 2 (Essential Tools)

[x] In Progress

RHCSA Study - Chapter 2: Essential Tools

Command Priority (Shell Resolution Order)

When you type a command, the shell resolves it in this order:

Priority Type Example

1 (highest)

Alias

alias ls='ls --color=auto'

2

Shell builtin

cd, echo, type, source

3 (lowest)

External command (from $PATH)

/usr/bin/ls, /usr/bin/grep

Key learnings:

  • Alias takes precedence even if external command exists in $PATH

  • To force external command: use full path (/usr/bin/ls) or command ls

  • type -a <cmd> shows all versions (alias, builtin, external)

  • which <cmd> only shows external commands (ignores aliases/builtins)

# See resolution order
type -a ls
# ls is aliased to `ls --color=auto'
# ls is /usr/bin/ls

# Force external
/usr/bin/ls
command ls
\ls          # backslash escapes alias

I/O Redirection, Streams, and Pipes

Three standard streams:

Stream FD Default Purpose

stdin

0

keyboard

Input to command

stdout

1

terminal

Normal output

stderr

2

terminal

Error messages

Redirection operators:

# stdout to file (overwrite)
command > file

# stdout to file (append)
command >> file

# stderr to file
command 2> file

# Both stdout and stderr to file
command > file 2>&1
command &> file          # bash shorthand

# Discard output
command > /dev/null 2>&1

# stdin from file
command < file

# Here document (stdin from literal)
command <<EOF
content
EOF

# Here string
command <<< "string"

Pipes:

# stdout of cmd1 becomes stdin of cmd2
cmd1 | cmd2

# Include stderr in pipe (bash 4+)
cmd1 |& cmd2

Questions for later:

  • How does process substitution <(cmd) differ from pipes?

  • When to use tee vs redirection?

  • Difference between 2>&1 and &> in different shells?

Personal Infrastructure

kvm-02 Deployment Progress

Hardware: Supermicro SYS-E300-9D-8CN8TP (P/N: E300-15)

Today’s work:

  • IPMI configuration (unknown IP - need ipmitool from booted OS)

  • Rack reorganization and cabling

  • Hardware documentation corrected (was wrong model number)

  • Switch port mapping documented (D2 diagram)

IPMI Recovery Plan:

BMC IP unknown from previous configuration. No jumper reset available on X11SDV.

# Boot Rocky Linux, then:
sudo modprobe ipmi_devintf && sudo modprobe ipmi_si
sudo ipmitool lan print 1

# Configure static IP
sudo ipmitool lan set 1 ipsrc static
sudo ipmitool lan set 1 ipaddr 10.50.1.201
sudo ipmitool lan set 1 netmask 255.255.255.0
sudo ipmitool lan set 1 defgw ipaddr 10.50.1.1

Documentation updated:

  • domus-infra-ops/pages/hardware/supermicro-e300-9d-8cn8tp.adoc - Corrected model, added BMC reset, added upgrade path

  • domus-captures/examples/commands/network/discovery-patterns.sh - IPMI discovery patterns

Documentation Created Today

  • domus-captures/examples/commands/network/discovery-patterns.sh - nmcli, masscan, nmap, dnsmasq, ipmitool patterns

  • domus-captures/examples/commands/awk/infrastructure.sh - Added line-range, field-extraction, nmcli-formatting, ip-interface tags

  • domus-infra-ops/pages/hardware/supermicro-e300-9d-8cn8tp.adoc - Major update with correct model and upgrade path

Carried Over

Professional (CHLA)

Critical (P0)

Project Description Status Blocker

Mandiant Remediation

dACL enforcement, posture/ACL remediation, ISE patch

ACTIVE — Q2 assessment

Project

Linux Research (Xiangming)

EAP-TLS for Linux workstations

BEHIND (due 02-24)

Certificate "password required" — nmcli flags fix documented

iPSK Manager HA

Pre-shared key automation — HA deployment

BEHIND

DB replication issues

MSCHAPv2 Migration

Legacy auth deprecation → EAP-TLS

BEHIND

No progress on planning

High Priority (P1)

Project Description Status

CHLA Antora Setup

8-phase Antora documentation deployment at work

ACTIVE (Project)

SIEM Migration

QRadar → Microsoft Sentinel — SDK integration

ACTIVE (Project)

ISE 3.4 Migration

Upgrade from 3.2p9

Planned — blocked by P0 items

ISE Hardware Refresh

PSN/MnT lifecycle replacement

Planned

Switch Upgrades

IOS-XE fleet update

Pending

Standard (P2)

Project Description Status

HHS Regulatory Compliance

New HHS security policies

NOT STARTED

InfoSec Reporting Dashboard

PowerBI metrics for executives

NOT STARTED

EDR Migration (AMP → Defender)

Endpoint protection consolidation

NOT STARTED

Azure Legacy Migration

Modern landing zone

In progress

Personal Infrastructure

Recently Completed

Project Description Date

domus-api v0.1.0

44-endpoint REST API — multi-spoke, DI, cache invalidation, 55 tests

2026-04-07

IOT_WAN VPN Passthrough

4 firewall rules applied — IPsec ESP, NAT-T, IKE, TCP 2443

2026-04-07

VyOS HA Migration

vyos-01 deployed, replacing pfSense

2026-03-07

C9130AX WiFi6 AP

Catalyst 9130AX access point deployment

2026-03-10

enterprise-linux-8021x

Standalone 802.1X EAP-TLS documentation spoke

2026-02-26

CLI Mastery Documentation

openssl/curl/awk/sed/xargs/pipelines

2026-02-26

Vault SSH CA

8h certs, 9 hosts configured

2026-02-21

k3s + Prometheus/Grafana

Monitoring stack on k3s

2026-02-23

In Progress

Project Description Status

ThinkPad P16g Deploy

Phase 11 verification, Phase 12 security hardening

Active

EVE-NG Lab

Network simulation lab — 8-phase rollout

Phase 0

RHEL 9 Workstation

Dr. Shahab’s workstation — 12-phase deployment

Draft

kvm-02 Hardware Upgrade

Supermicro B deployment

Hardware ready

Planned

Project Description Blocked By

Vault HA (3-node)

vault-02, vault-03 on kvm-02

kvm-02 deployment

DNS HA (bind-02)

Secondary with zone transfers from bind-01

kvm-02 deployment

k3s HA (3-node)

Control plane HA

kvm-02 deployment

Wazuh Agents

Deploy to all infrastructure hosts

k3s NAT fix (29 days blocked)

Cold Storage (M-DISC)

Offline archival of keys/headers

Time

SanDisk USB Offsite

Third backup drive rotation

Time

Learning Tracks

Track Description Status Resources

API Development (FastAPI)

REST API design, Pydantic, DI, async, testing

ACTIVE

domus-api (44 endpoints), API CLI Mastery

Claude Code + AI Engineering

Claude Code mastery, hooks, skills, agents

ACTIVE

Anthropic docs, Project

RHCSA 9 (EX200)

Red Hat system administration — 21-phase curriculum

ACTIVE

Sander van Vugt book, Project

CISSP

10-domain security certification

ACTIVE

Project

Spanish (DELE C1/C2)

Advanced Spanish certification

ACTIVE

Connectors reference, essay structure, Don Quijote

API CLI Mastery

jq/curl/awk/httpx pipeline composition

COMPLETE

Codex — 6-level curriculum

Terminal Mastery

awk/sed/jq/xargs/grep/find patterns

COMPLETE

Education

College Algebra

Functions, polynomials, exponentials, logarithms

ACTIVE

Education

Shell Scripting Guide

Comprehensive shell scripting (23 chapters)

In progress

Education

Linux Bible 11e

Linux administration reference

In progress

Education

D2 Diagrams

Infrastructure visualization

Ongoing

Practice with every runbook

Biblical Studies

Study notes and teachings

In progress

Private notes

Ruby Metaprogramming

DSL for infrastructure generation

PARKED (P3)

domus-captures/education/ruby/

AP Authentication Issue - Resolved (2026-02-28)

Cisco AP on Gi1/0/3 was failing MAB auth. Root cause: wrong endpoint group + rejected endpoints list.

Fix documented: domus-netapi-docs/examples/ise-api/operations/troubleshoot-mab-auth.sh

Switch SSH - Legacy Key Exchange

3560CX-01 requires legacy key exchange:

ssh -oKexAlgorithms=+diffie-hellman-group14-sha1 admin@10.50.1.5

Session Log

Session 1: RHCSA Study

Time: Morning

Topics covered:

  • Command resolution order (alias > builtin > external)

  • I/O redirection (stdin/stdout/stderr)

  • Pipes and stream manipulation

Next: File system hierarchy, essential file operations

Session 2: kvm-02 Infrastructure

Time: Afternoon

Tasks:

  • Rack reorganization

  • Cable management

  • Switch port mapping (D2 diagram created)

  • Hardware docs corrected

  • IPMI configuration (pending OS boot)

Session 3: kvm-02 Deployment Completion

Time: Evening

kvm-02 Deployment COMPLETE - 22/22 health checks passing

  • Rocky Linux 9.7 on 118GB SuperDOM SSD

  • NVMe with LVM thin provisioning (1.5TB pool)

  • IPMI configured at 10.50.1.201

  • Network bridge (br-mgmt) on eno8 (10GbE)

  • KVM/QEMU with nested virtualization

  • 16GB huge pages, CPU governor performance, swappiness=1

  • Wazuh agent connected to 10.50.1.134

  • Vault SSH CA trusted, SELinux enforcing

Health check script created: /usr/local/bin/kvm-health-check

CLI Mastery practiced:

# First/last N lines with awk
awk 'NR<=5 {print NR": "$0; next} {buf[NR]=$0} END {print "..."; for(i=NR-4;i<=NR;i++) print i": "buf[i]}' file

# Extract section between pattern and empty line
/usr/local/bin/kvm-health-check 2>&1 | awk '/Virtualization/,/^$/'

# Focused nmcli output
nmcli -t -f NAME,DEVICE,STATE connection show --active | column -t -s:

# Bridge validation
bridge link show | grep -E "eno8.*master"
ip -br addr show br-mgmt; ip -br link show eno8

kvm-01 Migration Plan created: domus-infra-ops/pages/runbooks/kvm-01-migration-plan.adoc

  • Documented dual-path network issue (192.168.1.x vs 10.50.1.x)

  • VM migration priority order defined

  • Network cleanup post-migration planned

Deferred:

  • NAS storage pools (need Synology NFS permissions for 10.50.1.111)

  • kvm-01 VM migration (pending validation)

Session 4: HA Infrastructure Planning

Time: Late Evening

HA-First Strategy Approved: Deploy HA infrastructure on kvm-02 BEFORE migrating VMs from kvm-01. This ensures true high availability across hypervisors, not just moving single points of failure.

Runbooks created:

  • domus-infra-ops/pages/runbooks/vault-ha-deployment.adoc - Deploy vault-02/vault-03 on kvm-02 for 3-node Raft cluster

  • domus-infra-ops/pages/runbooks/bind-02-deployment.adoc - Deploy secondary DNS with zone transfers from bind-01

  • Updated nas-share-management.adoc with Phase 0: kvm-02 NFS access

  • Updated kvm-01-migration-plan.adoc with HA prerequisites

HA Deployment Phases:

Phase Task Status

0

NAS NFS permissions for kvm-02

[ ] Pending

1

Vault HA (file→raft, vault-02/03)

[ ] Pending

2

DNS HA (bind-02)

[ ] Pending

3

Non-critical VM migration

[ ] Pending

4

Critical infrastructure HA

[ ] Future

Key insight: Instead of migrating primaries (which would move the SPOF), deploy secondaries on kvm-02:

  • vault-01 stays on kvm-01, vault-02/03 on kvm-02 = 3-node Raft

  • bind-01 stays on kvm-01, bind-02 on kvm-02 = zone transfer HA

  • pfSense-FW01 stays on kvm-01, pfSense-FW02 on kvm-02 = CARP HA (future)

Session 5: VyOS HA & FreeIPA Replica Documentation

Time: Night

VyOS Firewall HA (VRRP):

  • Created vyos-ha-topology.d2 diagram showing VRRP master/backup with VIP

  • Updated vyos-deployment.adoc runbook (already existed, added diagram reference)

  • Added VyOS attributes to antora.yml: vyos-vip (10.50.1.1), vyos-01-ip (10.50.1.3), vyos-02-ip (10.50.1.2)

VRRP allows using 10.50.1.1 (current pfSense gateway) as VIP. Brief cutover outage during pfSense→VyOS transition, then clients keep same gateway forever.

FreeIPA HA (ipa-02 replica):

  • Added Phase 10 to freeipa-deployment.adoc - complete ipa-02 replica deployment

  • Covers: DNS records, VM creation on kvm-02, ipa-replica-install with CA replication

  • FreeIPA multi-master replication is simpler than Vault Raft - single command

Infrastructure Inventory Updates:

  • Added ipa-02 (10.50.1.101) and vyos-02 (10.50.1.2) to:

    • kvm-02-vms.adoc

    • system-inventory-planned.adoc

    • infrastructure-radial-v3.d2 (diagram)

  • Added VyOS HA section to diagrams.adoc

Build Fixes (domus-captures):

  • Fixed Antora xref errors - must use full paths from pages root

  • Fixed shell-mathematics.adoc source block wrapping

  • Created ha-deployment-phases.adoc partial

Keycloak HA - DEFERRED:

Current keycloak runbook is single-instance only. HA requires PostgreSQL clustering (Patroni/repmgr) before deploying keycloak-02. Lower priority than core infrastructure HA.

Deployment Queue (runbooks ready):

Priority Service Runbook

1

vyos-02 (VRRP backup)

vyos-deployment.adoc

2

bind-02 (DNS secondary)

bind-02-deployment.adoc

3

ipa-02 (FreeIPA replica)

freeipa-deployment.adoc Phase 10

Deferred

keycloak-02 (IdP HA)

Needs PostgreSQL HA first

Notes