WRKLOG-2026-02-26
Summary
Wednesday. CATCH-UP DAY. Three work items behind deadline: Linux Research (Xiangming), iPSK Manager, MSCHAPv2 Migration.
Work Status: BEHIND
|
These items are overdue. No more personal infrastructure until work items are delivered. |
Professional (CHLA) - CRITICAL
| Priority | Project | Status | Blocker |
|---|---|---|---|
P0 |
Linux Research (Xiangming) |
BEHIND (due 02-24) |
Certificate "password required" error |
P0 |
iPSK Manager |
BEHIND |
DB replication issues |
P1 |
MSCHAPv2 Migration |
BEHIND |
No progress on planning |
P1 |
ISE 3.4 Migration |
Pending |
Blocked by above items |
P2 |
HHS Regulatory Compliance |
NOT STARTED |
Need to review new HHS security policies |
P2 |
InfoSec Reporting Dashboard |
NOT STARTED |
PowerBI metrics for executives |
P2 |
EDR Migration (AMP → Defender) |
NOT STARTED |
Endpoint protection consolidation |
Professional (CHLA) - Backlog
| Project | Description | Status |
|---|---|---|
QRadar → Sentinel Migration |
Full SIEM platform transition |
Partial |
Azure Legacy Migration |
Modern landing zone |
In progress |
Research Workstation Segmentation |
Trusted/Untrusted VLAN model |
Documented |
ChromeOS EAP-TEAP Migration |
ISE CSV endpoint analysis |
Partial |
Personal Infrastructure
| Project | Description | Status |
|---|---|---|
enterprise-linux-8021x |
Standalone 802.1X documentation |
DONE TODAY |
Cold Storage (M-DISC) |
Offline archival of keys/headers |
NOT STARTED |
SanDisk USB Offsite Rotation |
Third backup drive for offsite |
NOT STARTED |
Wazuh Archives Indexing |
Enable archives in Filebeat |
IN PROGRESS - PVC fix pending |
Shell Session Logging |
infra-session, tlog, SSH wrapper |
DONE TODAY |
Audio Functions Fix |
Rename source() → audio-input() |
DONE TODAY |
Home Lab Projects |
ISE, pfSense, PKI, SOC |
DEFERRED (12+ items) |
System Upgrade (pacman -Syu) |
Arch Linux update |
DONE (02-25) |
Shell functions (fe, fec, fef) |
Add to .zshrc |
NOT DONE |
Certifications
| Certification | Target | Status |
|---|---|---|
CISSP |
2026 |
Studying (Plan created) |
Cisco DevNet (DEVASC) |
2026 |
Planning (Plan created) |
GIAC GSEC |
2026 |
Planning (Plan created) |
RHEL (RHCSA/RHCE) |
2026 |
Planning (Plan created) |
SIELE C1 (Spanish) |
2026 |
Planning (Plan created) |
Learning Tracks
| Track | Description | Status |
|---|---|---|
Terminal Mastery |
awk/sed/jq patterns |
Assessment created |
CLI Data Processing |
Senior engineer patterns |
Draft started |
Biblical Teachings |
Study notes |
In progress |
Time Tracking (PeopleSoft)
Funded Projects (Billable)
| Project # | Combo Code | Description | My Role |
|---|---|---|---|
000017633 |
Replacing Cisco Secure Endpoint (AMP → Defender) |
EDR migration support |
|
000017706 |
Windows 11 Device Hardening |
Security policy review |
|
000016444 |
Spectrum TV App & Getwell SSID Design (iPad Refresh) |
iPSK Manager, ISE configuration |
|
000017481 |
Immunity Lab Move |
Network segmentation |
|
000017956 |
000018452 |
Mind DLP Proof of Value |
DLP evaluation |
TBD |
Cisco Catalyst Center |
DNA Center migration |
Internal Projects (Non-Funded)
| Project | Description | Status |
|---|---|---|
Hoxhunt |
Phishing simulation platform |
In progress |
Linux EAP-TLS (Xiangming) |
Research workstation 802.1X |
BEHIND |
iPSK Manager HA |
DB replication fix |
BEHIND |
MSCHAPv2 → EAP-TLS Migration |
Corporate SSID security |
Planning |
ISE 3.4 Migration |
Version upgrade |
Pending |
QRadar → Sentinel |
SIEM migration |
Partial |
Time Entry Reference
| Code Type | Purpose |
|---|---|
Project Number |
PeopleSoft project tracking (e.g., 000017633) |
Combo Code |
Budget/cost center allocation (e.g., 000018452) |
Internal |
Non-billable work (use department default code) |
Priority 1: Xiangming Linux EAP-TLS Delivery
Current State
-
Research dACL tested at HOME (modestus-aw)
-
Research dACL tested PARTIALLY at CHLA
-
Certificate deployment on Xiangming’s workstation
-
Certificate "password required" error needs resolution
-
Final EAP-TLS authentication validation
Known Issue: Certificate Password Required Error
Symptom: NetworkManager prompts "Secrets were required, but not provided" even though EAP-TLS is certificate-based (no password needed).
Root Cause: NetworkManager connection profile has incorrect password flags. The connection is configured to PROMPT for secrets instead of using stored/none.
Fix (from 2026-02-01 troubleshooting):
# Check current password flags
sudo nmcli -s conn show "Wired-802.1X-Vault" | grep -E '802-1x\.(identity|private-key-password)-flags'
# Fix: Set identity-flags=0 (none required), private-key-password-flags=4 (not required)
sudo nmcli conn modify "Wired-802.1X-Vault" \
802-1x.identity-flags 0 \
802-1x.private-key-password-flags 4
Flag Values:
| Flag | Value | Meaning |
|---|---|---|
|
|
No flags, secret stored |
|
|
Secret owned by agent (keyring) |
|
|
Secret not saved |
|
|
Secret not required |
Today’s Tasks (Xiangming)
1. Verify Certificate Subject (CHLA Machine)
# SSH to Xiangming's machine and verify cert
ssh xding@la.ad.chla.org@<hostname>
openssl x509 -in /etc/ssl/certs/<hostname>-eaptls.pem -noout -subject
# Should show: O=Research, OU=Research-Users, CN=<hostname>
2. Check NetworkManager Connection Profile
# List 802.1X connections
sudo nmcli conn show | grep -i 802
# Show full connection details (password flags)
sudo nmcli -s conn show "Wired-802.1X" | grep -E '802-1x\.'
3. Fix Password Flags
# Set flags to not prompt for password
sudo nmcli conn modify "Wired-802.1X" \
802-1x.identity-flags 0 \
802-1x.private-key-password-flags 4
# Verify change
sudo nmcli -s conn show "Wired-802.1X" | grep -E 'flags'
4. Restart Connection and Validate
# Restart connection
sudo nmcli conn down "Wired-802.1X"; sleep 3; sudo nmcli conn up "Wired-802.1X"
# Check ISE for auth result
netapi ise dc query "
SELECT TIMESTAMP_TIMEZONE, USERNAME, AUTHENTICATION_PROTOCOL, AUTHORIZATION_RULE, AUTHORIZATION_PROFILES
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%<hostname>%' OR CALLING_STATION_ID = '<MAC>'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY
"
5. Verify dACL Application on Switch
# Get switch and port from ISE
netapi ise mnt session <MAC>
# Verify dACL on switch
netapi ios exec "show access-session interface <port> de"
# Should show: ACS ACL: xACSACLx-IP-DACL_LINUX_RESEARCH_AD_AUTH-*
Research dACL Rules Reference
remark ### ACTIVE DIRECTORY AUTHENTICATION ###
permit tcp any host {ad-dc-ip} eq 88 # Kerberos
permit udp any host {ad-dc-ip} eq 88
permit tcp any host {ad-dc-ip} eq 389 # LDAP
permit tcp any host {ad-dc-ip} eq 636 # LDAPS
permit tcp any host {ad-dc-ip} eq 3268 # Global Catalog
permit tcp any host {ad-dc-ip} eq 3269 # GC SSL
permit udp any host {ad-dc-ip} eq 53 # DNS
permit tcp any host {ad-dc-ip} eq 53
remark ### SSH MANAGEMENT ###
permit tcp any host {admin-workstation} eq 22
remark ### INFRASTRUCTURE ###
permit udp any host {dns-server} eq 53
permit udp any host {ntp-server} eq 123
permit icmp any any
remark ### BLOCK RFC1918 LATERAL MOVEMENT ###
deny ip any 10.0.0.0 0.255.255.255 log
deny ip any 172.16.0.0 0.15.255.255 log
deny ip any 192.168.0.0 0.0.255.255 log
remark ### PERMIT INTERNET ###
permit ip any any
Priority 2: iPSK Manager
Status
-
DB replication between iPSK nodes needs troubleshooting
-
Manager functionality issues
Today’s Tasks
1. Check DB Replication Status
# Check MySQL replication status on primary
ssh ipsk-mgr-01 "sudo mysql -e 'SHOW MASTER STATUS\\G'"
# Check replication status on secondary
ssh ipsk-mgr-02 "sudo mysql -e 'SHOW SLAVE STATUS\\G'" | grep -E '(Slave_IO_Running|Slave_SQL_Running|Seconds_Behind_Master|Last_Error)'
2. Identify Replication Errors
# Check for errors
ssh ipsk-mgr-02 "sudo mysql -e 'SHOW SLAVE STATUS\\G'" | grep -E 'Error'
# Check MySQL error log
ssh ipsk-mgr-01 "sudo tail -50 /var/log/mysql/error.log"
ssh ipsk-mgr-02 "sudo tail -50 /var/log/mysql/error.log"
3. Test iPSK Manager Functionality
# Test API endpoint
curl -ks https://ipsk.la.ad.chla.org/api/health
# Check recent authentications
netapi ise dc query "
SELECT TIMESTAMP_TIMEZONE, CALLING_STATION_ID, AUTHORIZATION_RULE
FROM RADIUS_AUTHENTICATIONS
WHERE AUTHORIZATION_RULE LIKE '%iPSK%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY
"
Priority 3: MSCHAPv2 Migration Planning
Current State
-
MSCHAPv2 on CHLA-Corporate SSID (security risk from pentest)
-
No migration planning started
-
Need client inventory
Today’s Tasks
1. Generate MSCHAPv2 Client Inventory
# Find all PEAP-MSCHAPv2 authentications in last 7 days
netapi ise dc query "
SELECT DISTINCT USERNAME, ENDPOINT_PROFILE, NAS_PORT_ID
FROM RADIUS_AUTHENTICATIONS
WHERE AUTHENTICATION_PROTOCOL = 'PEAP-MSCHAPV2'
AND TIMESTAMP_TIMEZONE > SYSDATE - 7
ORDER BY USERNAME
"
2. Identify Device Types
# Group by endpoint profile (device type)
netapi ise dc query "
SELECT ENDPOINT_PROFILE, COUNT(*) as COUNT
FROM RADIUS_AUTHENTICATIONS
WHERE AUTHENTICATION_PROTOCOL = 'PEAP-MSCHAPV2'
AND TIMESTAMP_TIMEZONE > SYSDATE - 7
GROUP BY ENDPOINT_PROFILE
ORDER BY COUNT DESC
"
3. Document Migration Plan Outline
-
Phase 1: Client inventory (TODAY)
-
Phase 2: EAP-TLS compatibility testing
-
Phase 3: Certificate enrollment automation
-
Phase 4: Pilot rollout (IT department)
-
Phase 5: Organization-wide migration
-
Phase 6: Disable MSCHAPv2
Carried Over from 2026-02-25
| Task | Status | Notes |
|---|---|---|
System upgrade (pacman -Syu) |
Done |
Completed 02-25 |
Wazuh montage scripts |
Done |
8 scripts created |
File hunting runbook |
Done |
378 lines |
Session Log
Session 1: Xiangming Linux Delivery
Morning
Objective: Resolve certificate password error, get workstation authenticated
Steps:
-
[ ] SSH to Xiangming’s workstation
-
[ ] Verify certificate installed correctly
-
[ ] Fix NetworkManager password flags
-
[ ] Restart 802.1X connection
-
[ ] Validate EAP-TLS auth in ISE
-
[ ] Verify dACL applied on switch
-
[ ] Test AD authentication (Kerberos/LDAP)
-
[ ] Test SSH from admin workstation
Session 2: iPSK Troubleshooting
Afternoon
Objective: Identify and fix DB replication issues
Steps:
-
[ ] Check replication status on both nodes
-
[ ] Identify any replication errors
-
[ ] Fix replication if broken
-
[ ] Test iPSK authentication flow
-
[ ] Document findings
Session 3: MSCHAPv2 Planning
End of day
Objective: Generate client inventory for migration planning
Steps:
-
[ ] Run ISE queries for MSCHAPv2 clients
-
[ ] Export client list
-
[ ] Categorize by device type
-
[ ] Draft migration timeline
Key Commands Reference
NetworkManager Password Flags
# View all secrets and flags
sudo nmcli -s conn show "<connection>" | grep -E '(secret|password|flags)'
# Set flags to not require password
sudo nmcli conn modify "<connection>" \
802-1x.identity-flags 0 \
802-1x.private-key-password-flags 4
ISE Session Debugging
# Quick session lookup by MAC
netapi ise mnt session <MAC>
# Data Connect query with DC IP
netapi ise dc query "<SQL>"
Switch dACL Verification
# Show access session details
netapi ios exec "show access-session interface <port> de"
# Show applied ACL
netapi ios exec "show ip access-lists | include ACS"
Session 4: enterprise-linux-8021x Documentation Repository
Evening
Objective: Create standalone enterprise-grade 802.1X documentation repository with professional theming
Completed
-
Created
enterprise-linux-8021xrepository structure -
Migrated content from PRJ-ISE-CHLA-LINUX-ANTORA
-
Created complete supplemental-ui with enterprise theming
-
Added PDF/DOCX/HTML export targets to Makefile
-
Fixed admonition rendering (after extensive troubleshooting)
Root Cause Analysis: Admonition Icon Rendering Failure
Symptom
Admonition icons appeared "sunken into the page background" - the icon cell was visible but not properly integrated with the admonition block. Custom CSS overrides failed repeatedly.
Investigation Timeline
-
Initial attempt: Added background colors to
td.iconcell - created "squared color that looks incomplete" -
Second attempt: Set
td.icon { display: none }and used CSS::beforepseudo-element with Font Awesome unicode - icon disappeared entirely -
Third attempt: Reset to
position: staticto remove absolute positioning - icon still not rendering -
Fourth attempt: Styled the
i::aftercontent (which renders the text label) - looked worse -
Multiple iterations: Various combinations of positioning, backgrounds, and icon styling - all failed
Root Cause
The Antora default UI bundle does NOT include Font Awesome.
The HTML output shows:
<i class="fa icon-tip" title="Tip"></i>
The default Antora UI renders this as a text label via CSS:
.doc .admonitionblock td.icon i::after {
content: attr(title); /* Shows "Tip", "Note", etc. as TEXT */
}
Additionally, the default UI uses absolute positioning for the icon cell:
.doc .admonitionblock td.icon {
position: absolute;
transform: translate(-.5rem, -50%); /* Floats as badge above content */
}
This explains why:
-
Custom backgrounds on
td.icondidn’t work - the cell is positioned absolutely outside normal flow -
Hiding
::afterremoved the visible content - there was no actual Font Awesome icon, just CSS-generated text -
The icon appeared "sunken" - absolute positioning placed it outside the table layout
Solution
Load Font Awesome from CDN and remove all custom admonition CSS.
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">
With Font Awesome loaded, the Antora default UI’s admonition styling works correctly - colored pill badges with text labels.
Lessons Learned
| Lesson | Detail |
|---|---|
Read the source |
Should have inspected the default UI CSS first to understand how admonitions render |
Don’t fight the framework |
Hours spent overriding CSS that was never going to work with the existing icon setup |
Check dependencies |
The |
Antora default UI quirks |
Uses text labels via |
Prevention
For future Antora projects:
-
Always load Font Awesome explicitly if using
icons: font -
Test admonitions early in the build process
-
Inspect browser dev tools before writing custom CSS
-
Don’t assume
:icons: fontattribute loads the font library
Files Modified
| File | Changes |
|---|---|
|
Added Font Awesome CDN link, removed broken admonition CSS |
|
Added |
|
Expanded with netplan fixes, ISE verification, full diagnostic script |
Repository Status
-
Location:
~/atelier/_bibliotheca/enterprise-linux-8021x -
Build:
npx antora antora-playbook.yml -
Serve:
python3 -m http.server -d build/site 8080 -
Ready for GitHub push
Session 5: Wazuh Archives Diagnostic (domus-infra-ops)
Evening
Objective: Enable Wazuh archives indexing for raw log retention
Root Cause
-
ConfigMap
filebeat-confighas correct setting:archives: enabled: true -
PVC
wazuh-manager-master-wazuh-manager-master-0has stale filebeat.yml witharchives: enabled: false -
Init container copies from ConfigMap to emptyDir, but PVC mount overrides it
Fix (Pending)
kubectl delete pod -n wazuh wazuh-manager-master-0
kubectl delete pvc -n wazuh wazuh-manager-master-wazuh-manager-master-0
kubectl get pod -n wazuh -w
kubectl exec -n wazuh wazuh-manager-master-0 -- grep -A2 archives /etc/filebeat/filebeat.yml
Session Log
-
Full diagnostic captured:
domus-infra-ops/attachments/sessions/2026-02-26-wazuh-archives-diag.log
Also Completed
-
Created session logging workflow (infra-session, tlog, iend, istat)
-
Added SSH wrapper for session reminders
-
Expanded shell ready banner with all features
-
Fixed audio-functions.sh source() → audio-input() conflict
-
Committed and pushed all dotfiles changes
Session 6: Vault SSH CA Principal Update
Morning
Objective: Add Administrator principal for Windows DC SSH CA authentication
Runbook: Vault SSH CA
The Flow (TL;DR)
┌─────────────────────────────────────────────────────────────────┐
│ VAULT SSH CA WORKFLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. SIGN CERTIFICATE (daily, 8h TTL) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ vault-ssh- │ ──► │ Vault CA │ │
│ │ sign │ │ signs key │ │
│ └──────────────┘ └──────────────┘ │
│ │ │
│ ▼ │
│ ~/.ssh/id_ed25519_vault-cert.pub (8h lifetime) │
│ │
│ 2. SSH CONNECTION │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ ssh host │ ──► │ Server │ │
│ │ (offers │ │ checks CA │ │
│ │ cert) │ │ trust │ │
│ └──────────────┘ └──────────────┘ │
│ │ │ │
│ │ cert valid? │ │
│ │◄────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────┐ │
│ │ YES: Vault cert auth ✓ │ │
│ │ NO: Falls back to YubiKey (by design) │ │
│ └─────────────────────────────────────────┘ │
│ │
│ 3. PRINCIPALS (who you can SSH as) │
│ • Administrator - Windows DC │
│ • adminerosado - Synology NAS │
│ • admin - pfSense, network devices │
│ • evanusmodestus - Linux servers │
│ • gabriel - modestus-p50 │
│ • ansible, root - automation/emergency │
│ │
└─────────────────────────────────────────────────────────────────┘
Why This Matters
| Traditional SSH | Vault SSH CA |
|---|---|
Static keys (never expire) |
8-hour certificates (auto-expire) |
Revoke = change keys everywhere |
Revoke = don’t re-sign |
Keys on every server |
CA public key on servers (one file) |
No audit trail |
Full Vault audit log |
Problem
Windows DC (home-dc01) SSH fell back to YubiKey instead of using Vault certificate. Debug showed cert offered but rejected - missing Administrator principal.
Root Cause Analysis with jq
# Extract current allowed_users from Vault role
vault read ssh/roles/domus-client -format=json | jq -r '.data.allowed_users | split(",") | .[]'
adminerosado admin ansible evanusmodestus gabriel root
Fix: Update Vault Role with jq Pipeline
Vault requires full role config on update. Used jq to read, modify, and write in one pipeline:
# Read current config → add Administrator → write back
vault read ssh/roles/domus-client -format=json | jq '
.data |
.allowed_users = (.allowed_users | split(",") + ["Administrator"] | unique | join(","))
' | vault write ssh/roles/domus-client -
jq breakdown:
| Pattern | Purpose |
|---|---|
|
Vault wraps response in |
|
Convert comma-string to array |
|
Append new principal to array |
|
Deduplicate (idempotent) |
|
Convert back to comma-string |
|
Pipe JSON directly to Vault CLI |
Update Signing Script
# Add Administrator to signing script
sed -i 's/PRINCIPALS="adminerosado/PRINCIPALS="Administrator,adminerosado/' ~/.local/bin/vault-ssh-sign
# Verify
grep PRINCIPALS ~/.local/bin/vault-ssh-sign
Re-sign and Verify
vault-ssh-sign
Certificate signed successfully:
Valid: from 2026-02-26T11:38:10 to 2026-02-26T19:38:40
Principals:
Administrator
admin
adminerosado
ansible
evanusmodestus
gabriel
root
Key jq Patterns Demonstrated
| Pattern | Use Case |
|---|---|
|
Extract Vault response payload |
|
Convert comma-string to line-per-item |
|
Append to array idempotently |
|
Pipe modified JSON back to Vault |
|
String interpolation for formatted output |
Remaining Issue
Windows DC still falls back to YubiKey - Vault CA not deployed to Windows. Separate task: deploy /etc/ssh/vault-ca.pub equivalent to C:\ProgramData\ssh\.
Reference
-
Runbook: Vault SSH CA (Section: "Add New Principal to Vault Role")
-
Script:
~/.local/bin/vault-ssh-sign -
Test:
vault-ssh-test
Session 7: Sentinel Log Optimization - Vendor Evaluation
Evening
Context: Vendor pitched log pre-processing solution to reduce Sentinel ingestion costs. Claims to use jq for "data massage/hydration" before forwarding to Sentinel.
Vendor Claim
-
Pre-process logs with jq transformations
-
Filter noise, extract necessary fields, hydrate with context
-
Reduce data volume → reduce Sentinel costs
-
"Just ask AI for the jq filters"
Technical Validity
The concept is sound - log pre-processing is a legitimate cost optimization:
| Approach | Cost Impact |
|---|---|
Filter debug/heartbeat logs |
20-40% reduction |
Extract only necessary fields |
10-30% reduction |
Deduplicate events |
5-15% reduction |
Normalize formats |
Better query performance |
Red Flags
| Concern | Issue |
|---|---|
"Ask AI for jq" |
No in-house expertise - you maintain it |
jq at scale |
Single-threaded CLI, not enterprise log pipeline |
Unknown architecture |
Is this Cribl? Logstash? Scripts on a VM? |
Compliance |
Where are filtered logs retained for audit? |
Industry Standard Tools
| Tool | Purpose | Enterprise Grade |
|---|---|---|
Cribl |
Observability pipeline (market leader) |
Yes |
Logstash |
Log processing (Elastic ecosystem) |
Yes |
Fluent Bit |
Lightweight log processor |
Yes |
Azure DCR |
Native Data Collection Rules |
Yes (free) |
jq scripts |
Ad-hoc transformation |
No |
Recommendation: In-House with Azure DCR
Azure Data Collection Rules transform logs before ingestion billing - no additional cost:
// Example: Filter Windows Security noise
source
| where EventID !in (4688, 4689, 5156, 5157) // Process/firewall noise
| where EventID != 4634 or TargetUserName != "SYSTEM"
| project-away RawEventData, EventData
Action Items
-
Identify top 5 high-volume log sources in Sentinel
-
Write DCR transformations for each
-
Test in dev workspace before production
-
Document patterns in domus-infra-ops runbooks
-
Estimate cost savings (compare 30-day before/after)
Cost Analysis Query
// Find what's costing money
Usage
| where TimeGenerated > ago(30d)
| summarize GB = sum(Quantity) / 1024 by DataType
| order by GB desc
| take 20
Decision
Build in-house with Azure DCR. The vendor’s "ask AI" approach provides no value beyond what we can do ourselves with native Azure tooling. Plus: better compliance posture, full control, no vendor lock-in.
Learning Track Addition
This aligns with Terminal Mastery track - add jq/curl/awk patterns for log analysis to domus-linux-ops.