WRKLOG-2026-03-05
Summary
Wednesday. Full ISE 3.5 migration from ise-01 to ise-02. IPMI-01 troubleshooting (dedicated LAN mode). Major runbook updates: Phase 3a restore procedure, gopass v3 paths.
Today’s Priority Tasks
| Priority | Task | Status |
|---|---|---|
P0 |
ISE 3.5 migration (ise-01 → ise-02) |
[ ] In Progress |
P0 |
IPMI-01 connectivity (kvm-01 out-of-band) |
[x] DONE |
P1 |
Update ise-35-deployment.adoc with Phase 3a |
[x] DONE |
P1 |
Update gopass paths to v3 structure |
[x] DONE |
P1 |
CHLA Linux SSH issue (Xianming Ding) |
[ ] CARRY-OVER |
P2 |
Restructure ad.adoc into granular directory |
[ ] CARRY-OVER |
P2 |
Restructure remoting.adoc into granular directory |
[ ] CARRY-OVER |
P2 |
iPSK Manager - DB replication |
[ ] CARRY-OVER |
ISE 3.5 Migration Sequence
| Step | Task | Status |
|---|---|---|
1 |
Backup from ise-01 |
[x] DONE |
2 |
Install ISE 3.5 on ise-02 (kvm-02) |
[x] DONE |
3 |
Configure NFS repository |
[x] DONE |
4 |
Restore backup to ise-02 |
[x] DONE |
5 |
Wait for services (~15-20 min) |
[x] DONE |
6 |
Verify restored config |
[x] DONE |
7 |
Issue ise-02 certs from Vault PKI |
[x] DONE |
8 |
Import certs to ISE GUI |
[x] DONE |
9 |
ISE restart after cert import |
[x] DONE |
10 |
NAS OUTAGE - VM CORRUPTED |
⚠️ INCIDENT |
11 |
Redeploy ise-02 VM from scratch |
IN PROGRESS |
12 |
Restore backup (again) |
[ ] Pending |
13 |
Apply Patch 2 |
[ ] Pending |
14 |
Re-import Vault PKI certs |
[ ] Pending |
15 |
Test single device auth |
[ ] Pending |
16 |
Shutdown ise-01 (NAD failover) |
[ ] Pending |
17 |
Verify all authentications |
[ ] Pending |
Session Log
Session 1: IPMI-01 Troubleshooting
Time: Morning
Problem: IPMI-01 (kvm-01 BMC) unreachable at 10.50.1.200
Diagnosis:
-
Switch MAC table showed eno2 MAC, not IPMI MAC
-
BMC LAN mode was
02(Failover), needed00(Dedicated) -
Physical cable was in wrong port (eno2 instead of dedicated IPMI)
Commands used:
# Check BMC LAN mode (from kvm-01)
sudo ipmitool raw 0x30 0x70 0x0c 0
# Returns: 02 = Failover, 00 = Dedicated
# Set to Dedicated mode
sudo ipmitool raw 0x30 0x70 0x0c 1 0
# Reset BMC
sudo ipmitool mc reset cold
# Verify physical port (shows state UP = cable connected)
ip link show eno2
Resolution: Moved cable to dedicated IPMI port. Now reachable at 10.50.1.200.
Documentation updated:
-
kvm-01-migration-plan.adoc - Added Phase 0 IPMI Configuration
-
kvm-02-deployment.adoc - Added section 0.2.1 LAN Mode verification
Session 2: ISE 3.5 Restore
Time: Midday
Fresh backup from ise-01:
dsource d000 dev/storage
netapi ise backup --repo nas-01 --name "pre-ise35-restore" --wait
Configure repository on ise-02 (ISE CLI):
configure terminal
repository nas-01
url nfs://10.50.1.70:/volume1/ise_backups
exit
Verify repository:
show repository nas-01 | include pre-ise35-restore
Restore command:
restore pre-ise35-restore-CFG10-260305-1159.tar.gpg repository nas-01 encryption-key plain <KEY>
Runbook updates:
-
Added Phase 3a: Restore from Backup (Alternative)
-
Updated credentials table from gopass v2 to v3 paths
-
Fixed ISE_BACKUP_KEY location:
dev/storagenotdev/network
Session 3: CLI Patterns Used Today
awk - Extract specific lines from runbook:
# View specific line numbers
awk 'NR==689 || NR==700' docs/asciidoc/modules/ROOT/pages/runbooks/ise-35-deployment.adoc
# Output:
# restore <backup-filename> repository nas-01 encryption-key plain <YOUR_KEY>
# restore scheduled-CFG10-260222-1948.tar.gpg repository nas-01 encryption-key plain MySecretKey123
grep - Search for patterns in runbook:
# Find all restore references
grep -n "restore" ise-35-deployment.adoc | head -20
# Check for ISE_BACKUP_KEY location
grep -i "ISE_BACKUP_KEY" ise-35-deployment.adoc
ISE CLI - Pipe filtering:
# Filter repository listing
show repository nas-01 | include pre-ise35-restore-CFG10-260305-1159.tar.gpg
Session 3: Vault PKI Cert for ise-02
Time: Afternoon
Key Learning: Vault accessible from workstation - no SSH to vault-01 required.
Issue cert from workstation:
dsource d000 dev/vault
vault write pki_int/issue/domus-server \
common_name="ise-02.inside.domusdigitalis.dev" \
ip_sans="10.50.1.21" \
ttl="8760h" \
-format=json > /dev/shm/ise-02-cert.json
Extract certificate, key, CA chain:
jq -r '.data.certificate' /dev/shm/ise-02-cert.json > /dev/shm/ise-02.crt
jq -r '.data.private_key' /dev/shm/ise-02-cert.json > /dev/shm/ise-02.key
jq -r '.data.ca_chain[]' /dev/shm/ise-02-cert.json > /dev/shm/ise-02-chain.pem
Split CA chain into individual certs:
awk '/-----BEGIN CERTIFICATE-----/{n++} n==1' /dev/shm/ise-02-chain.pem > /dev/shm/DOMUS-ISSUING-CA.pem
awk '/-----BEGIN CERTIFICATE-----/{n++} n==2' /dev/shm/ise-02-chain.pem > /dev/shm/DOMUS-ROOT-CA.pem
Verify certs:
openssl x509 -in /dev/shm/ise-02.crt -subject -issuer -noout
openssl x509 -in /dev/shm/DOMUS-ISSUING-CA.pem -subject -noout
openssl x509 -in /dev/shm/DOMUS-ROOT-CA.pem -subject -noout
ISE Import Order:
-
Administration → System → Certificates → Trusted Certificates → Import
-
DOMUS-ROOT-CA.pem (Trust for ISE Auth + Client Auth)
-
DOMUS-ISSUING-CA.pem (Trust for ISE Auth + Client Auth)
-
-
Administration → System → Certificates → System Certificates → Import
-
Certificate: ise-02.crt
-
Private Key: ise-02.key
-
Usage: Admin, EAP, pxGrid
-
-
ISE restarts automatically after system cert import
Runbook updated: ise-35-deployment.adoc - Added 4.0 Quick Method (workstation-based PKI)
Session 4: WLC-01 Single NIC Fix
Time: Afternoon
Problem: Two NICs on kvm-01/virbr0 causes IOS-XE to detect only Gi2, not Gi1.
Solution: Recreate VM with single NIC using existing qcow2:
sudo virt-install \
--name 9800-WLC-01 \
--memory 16384 \
--vcpus 4 \
--import \
--disk path=/mnt/onboard-ssd/vms/C9800-CL-universalk9.17.15.03.qcow2,format=qcow2 \
--network bridge=virbr0,model=virtio \
--os-variant generic \
--graphics vnc \
--noautoconsole
Verification: WLC-01 and WLC-02 CAN ping each other (80-100% success).
Runbook updated: wlc-ha-sso.adoc - Added single NIC requirement + Appendix D Troubleshooting
Session 5: NAS Outage - ISE-02 VM Corrupted
Time: Evening
INCIDENT: NAS (10.50.1.70) became unreachable during ISE migration.
Root Cause: Switch port Gi1/0/6 had 802.1X template AND wrong VLAN:
# BEFORE (broken)
interface GigabitEthernet1/0/6
source template DefaultWiredDot1xClosedAuth <-- 802.1X ON INFRASTRUCTURE PORT
switchport access vlan 10 <-- WRONG VLAN (should be 100)
Impact:
-
NFS disconnect corrupted ise-02 VM filesystem
-
VM booted to emergency mode with I/O errors:
blk_update_request: I/O error, dev vda, sector 93521729 EXT4-fs (vda3): error loading journal
Resolution:
-
Remove 802.1X template from switch port
-
Fix VLAN to 100
-
Decision: Redeploy ise-02 from scratch (filesystem unrecoverable)
Key Learning: NEVER put 802.1X on infrastructure ports (NAS, KVM hosts, switches).
Session 6: ISE-02 Redeployment
Time: Evening
Problem 1: virt-install failed - "Size must be specified for non existent volume"
Fix: Add size=600 to disk parameter.
Problem 2: VM showed "No bootable media" - ISO not attached
Diagnosis:
sudo virsh domblklist ise-02
# sda showed "-" (empty)
Fix:
sudo virsh change-media ise-02 sda /mnt/nas/isos/Cisco-ISE-3.5.0.527.SPA.x86_64.iso --insert --config
sudo virsh destroy ise-02 && sudo virsh start ise-02
Problem 3: ZTP Auto-Install halted
Symptom:
***** The ZTP configuration image is missing or improper. Automatic installation flow exited. ***** Power off and attach the proper ZTP configuration image or choose manual boot to proceed. [ 44.710453] reboot: System halted
Fix: Must select Option 1 (Keyboard/Monitor) from graphical boot menu in Cockpit. ISE defaults to ZTP mode without manual selection.
Current Status: ISE-02 installing from ISO (30-45 min)
Runbook updated: ise-35-deployment.adoc - Added troubleshooting sections for all three issues
Session 7: VyOS VLAN 40 DHCP Validation
Time: Evening
Context: Testing vyos-02 DHCP for VLAN 40 (IOT) per vyos-vlan-fasttrack-migration.adoc.
Network Path (from runbooks):
-
Te1/0/1 → kvm-02 → vyos-02 eth0 (trunk with VLANs 20,30,40,100,999)
-
Te1/0/2 → kvm-01 → pfSense (VLANs 10,100,999 - VLAN 40 intentionally removed)
VyOS verification (from vyos-02 console):
# Confirm eth0.40 interface up
show interfaces | grep -E 'eth0\.(20|30|40)'
# Expected: eth0.40 10.50.40.1/24 u/u IOT
# Confirm DHCP pool available
show dhcp server statistics
# Expected: IOT pool with 100+ available addresses
Test setup - backup ethernet on modestus-razer:
# Create backup connection for USB ethernet adapter
nmcli conn add type ethernet con-name Domus-Wired-Backup ifname "enp0s13f0u2u1"
# View active connections with awk
nmcli conn show --active | awk 'NR>1 {print $1, $3}'
Switch port for VLAN 40 test (G1/0/2):
interface GigabitEthernet1/0/2
description [DOT1X] User Access Port
switchport access vlan 40
switchport mode access
switchport voice vlan 20
ip arp inspection trust
spanning-tree portfast edge
end
Validation commands:
# Force DHCP on backup interface
sudo nmcli conn down Domus-Wired-Backup && sudo nmcli conn up Domus-Wired-Backup
# Check IP from vyos-02 DHCP (should be 10.50.40.x)
ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'
# Verify gateway is vyos-02
ip route | awk '/enp0s13f0u2u1/ && /default|10.50.40/ {print}'
# Check DHCP source in journal
journalctl -u NetworkManager --since "2 min ago" | awk '/enp0s13f0u2u1.*DHCP/ {print}'
Key Learning: Use separate test interface to validate VyOS DHCP without disrupting primary 802.1X session.
Session 8: VyOS Firewall Fix - LOCAL_IOT Rule
Time: Evening
Root Cause: LOCAL_IOT firewall was EMPTY - no rules to allow DHCP replies.
DHCP Flow Analysis:
| Step | Direction | Firewall | Result |
|---|---|---|---|
1. Discover |
Client → VyOS (port 67) |
IOT_LOCAL rule 30 |
✓ Accept (6 packets) |
2. Offer |
VyOS → Client (port 68) |
LOCAL_IOT (empty) |
✗ DROPPED |
Diagnosis commands (VyOS):
show firewall ipv4 name IOT_LOCAL
# rule 30: accept UDP port 67 - packets matched
show firewall ipv4 name LOCAL_IOT
# Ruleset Information (EMPTY - no rules!)
Fix applied (VyOS):
configure
set firewall ipv4 name LOCAL_IOT rule 10 action accept
set firewall ipv4 name LOCAL_IOT rule 10 protocol udp
set firewall ipv4 name LOCAL_IOT rule 10 destination port 68
set firewall ipv4 name LOCAL_IOT rule 10 description "DHCP replies"
commit
save
Verification (VyOS):
show firewall ipv4 name LOCAL_IOT
# rule 10 { action accept; destination { port 68 }; protocol udp }
Test from workstation:
# Force DHCP renewal
sudo nmcli conn down Domus-Wired-Backup && sudo nmcli conn up Domus-Wired-Backup
# Verify IP from vyos-02 DHCP pool (10.50.40.x)
ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'
# Verify gateway is vyos-02
ip route | awk '/enp0s13f0u2u1/{print}'
Key Learning: VyOS zone-based firewall requires BIDIRECTIONAL rules:
-
IOT_LOCAL= Traffic from IOT zone TO the router (DHCP requests, DNS queries) -
LOCAL_IOT= Traffic from router TO IOT zone (DHCP replies, DNS responses)
Without LOCAL_IOT rule for port 68, DHCP Offer packets are silently dropped.
Session 9: VyOS DHCP Fixed - kvm-02 Bridge VLAN
Time: Evening
Problem persisted after VyOS firewall fix. DHCP still failing.
Root Cause: kvm-02 bridge interface vnet1 (vyos-02’s eth0) wasn’t passing VLAN 40 tags.
Diagnosis (kvm-02):
# Check bridge VLAN assignments
sudo bridge vlan show
# Output showed:
# vnet1 1 PVID Egress Untagged <-- NO VLAN 40!
# br-mgmt had VLANs 10,20,30,40,100,110,120 ✓
# eno8 (trunk) had same VLANs ✓
# Verify vyos-02 uses br-mgmt, not virbr0
sudo virsh domiflist vyos-02
# vnet1 → br-mgmt ✓
Fix (kvm-02):
sudo bridge vlan add vid 40 dev vnet1
Verification:
sudo bridge vlan show dev vnet1
# vnet1 1 PVID Egress Untagged
# 40 <-- NOW PRESENT
Result (workstation):
ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'
# 10.50.40.100/24 ✓ FROM VYOS!
ip route | awk '/enp0s13f0u2u1/ && /default/{print}'
# default via 10.50.40.1 ... ✓ VYOS GATEWAY
Key Learning: KVM bridge VLAN filtering requires VLANs on BOTH:
-
Physical trunk interface (eno8) ✓
-
Bridge itself (br-mgmt) ✓
-
VM’s vnet interface ← This was missing!
WARNING: bridge vlan add is NOT persistent. Needs NetworkManager or libvirt hook.
Carried Over
Professional (CHLA)
Critical (P0)
| Project | Description | Status | Blocker |
|---|---|---|---|
Mandiant Remediation |
dACL enforcement, posture/ACL remediation, ISE patch |
ACTIVE — Q2 assessment |
|
Linux Research (Xiangming) |
EAP-TLS for Linux workstations |
BEHIND (due 02-24) |
Certificate "password required" — nmcli flags fix documented |
iPSK Manager HA |
Pre-shared key automation — HA deployment |
BEHIND |
DB replication issues |
MSCHAPv2 Migration |
Legacy auth deprecation → EAP-TLS |
BEHIND |
No progress on planning |
High Priority (P1)
| Project | Description | Status |
|---|---|---|
CHLA Antora Setup |
8-phase Antora documentation deployment at work |
ACTIVE (Project) |
SIEM Migration |
QRadar → Microsoft Sentinel — SDK integration |
ACTIVE (Project) |
ISE 3.4 Migration |
Upgrade from 3.2p9 |
Planned — blocked by P0 items |
ISE Hardware Refresh |
PSN/MnT lifecycle replacement |
Planned |
Switch Upgrades |
IOS-XE fleet update |
Pending |
Standard (P2)
| Project | Description | Status |
|---|---|---|
HHS Regulatory Compliance |
New HHS security policies |
NOT STARTED |
InfoSec Reporting Dashboard |
PowerBI metrics for executives |
NOT STARTED |
EDR Migration (AMP → Defender) |
Endpoint protection consolidation |
NOT STARTED |
Azure Legacy Migration |
Modern landing zone |
In progress |
Personal Infrastructure
Recently Completed
| Project | Description | Date |
|---|---|---|
domus-api v0.1.0 |
44-endpoint REST API — multi-spoke, DI, cache invalidation, 55 tests |
2026-04-07 |
IOT_WAN VPN Passthrough |
4 firewall rules applied — IPsec ESP, NAT-T, IKE, TCP 2443 |
2026-04-07 |
VyOS HA Migration |
vyos-01 deployed, replacing pfSense |
2026-03-07 |
C9130AX WiFi6 AP |
Catalyst 9130AX access point deployment |
2026-03-10 |
enterprise-linux-8021x |
Standalone 802.1X EAP-TLS documentation spoke |
2026-02-26 |
CLI Mastery Documentation |
openssl/curl/awk/sed/xargs/pipelines |
2026-02-26 |
Vault SSH CA |
8h certs, 9 hosts configured |
2026-02-21 |
k3s + Prometheus/Grafana |
Monitoring stack on k3s |
2026-02-23 |
In Progress
| Project | Description | Status |
|---|---|---|
ThinkPad P16g Deploy |
Phase 11 verification, Phase 12 security hardening |
|
EVE-NG Lab |
Network simulation lab — 8-phase rollout |
|
RHEL 9 Workstation |
Dr. Shahab’s workstation — 12-phase deployment |
|
kvm-02 Hardware Upgrade |
Supermicro B deployment |
Hardware ready |
Planned
| Project | Description | Blocked By |
|---|---|---|
Vault HA (3-node) |
vault-02, vault-03 on kvm-02 |
kvm-02 deployment |
DNS HA (bind-02) |
Secondary with zone transfers from bind-01 |
kvm-02 deployment |
k3s HA (3-node) |
Control plane HA |
kvm-02 deployment |
Wazuh Agents |
Deploy to all infrastructure hosts |
k3s NAT fix (29 days blocked) |
Cold Storage (M-DISC) |
Offline archival of keys/headers |
Time |
SanDisk USB Offsite |
Third backup drive rotation |
Time |
Learning Tracks
| Track | Description | Status | Resources |
|---|---|---|---|
API Development (FastAPI) |
REST API design, Pydantic, DI, async, testing |
ACTIVE |
domus-api (44 endpoints), API CLI Mastery |
Claude Code + AI Engineering |
Claude Code mastery, hooks, skills, agents |
ACTIVE |
Anthropic docs, Project |
RHCSA 9 (EX200) |
Red Hat system administration — 21-phase curriculum |
ACTIVE |
Sander van Vugt book, Project |
CISSP |
10-domain security certification |
ACTIVE |
|
Spanish (DELE C1/C2) |
Advanced Spanish certification |
ACTIVE |
Connectors reference, essay structure, Don Quijote |
API CLI Mastery |
jq/curl/awk/httpx pipeline composition |
COMPLETE |
Codex — 6-level curriculum |
Terminal Mastery |
awk/sed/jq/xargs/grep/find patterns |
COMPLETE |
|
College Algebra |
Functions, polynomials, exponentials, logarithms |
ACTIVE |
|
Shell Scripting Guide |
Comprehensive shell scripting (23 chapters) |
In progress |
|
Linux Bible 11e |
Linux administration reference |
In progress |
|
D2 Diagrams |
Infrastructure visualization |
Ongoing |
Practice with every runbook |
Biblical Studies |
Study notes and teachings |
In progress |
Private notes |
Ruby Metaprogramming |
DSL for infrastructure generation |
PARKED (P3) |
domus-captures/education/ruby/ |
Key Learnings
IPMI LAN Modes (Supermicro)
| Value | Mode | Description |
|---|---|---|
|
Dedicated |
Uses dedicated IPMI port only (required for separate IPMI NIC) |
|
Shared |
Shares with onboard NIC1 |
|
Failover |
Tries dedicated, falls back to shared |
Lesson: Always verify LAN mode when IPMI is unreachable. Physical cabling matters.
ISE Restore Methods
| Method | When to Use | Notes |
|---|---|---|
|
Existing ISE with API configured |
Requires dsec pointing to target ISE |
ISE CLI |
Fresh install, API not configured |
Works universally, no dsec changes needed |
gopass v3 Structure
ISE credentials now in identity hierarchy:
v3/domains/d000/identity/ise/ise-01/{admin,cli,dataconnect}
v3/domains/d000/identity/ise/ise-02/{admin,cli,dataconnect}
Vault PKI from Workstation
Key Pattern: If vault-ssh-sign works from workstation, so does cert issuance.
dsource d000 dev/vault && vault-ssh-sign # If this works, Vault is accessible
No need to SSH to vault-01 for cert operations.
WLC Single NIC on kvm-01/virbr0
Problem: IOS-XE with 2 NICs on virbr0 shows only Gi2 (Gi1 missing).
Solution: Use single NIC with virt-install --import.
Verification: Both WLCs can ping each other (80-100% success).
awk CA Chain Split Pattern
# Full BEGIN CERTIFICATE string required (not just "BEGIN CERT")
awk '/-----BEGIN CERTIFICATE-----/{n++} n==1' chain.pem > first-cert.pem
awk '/-----BEGIN CERTIFICATE-----/{n++} n==2' chain.pem > second-cert.pem
ISE ISO ZTP Boot Trap
Problem: ISE ISO defaults to ZTP (Zero Touch Provisioning) mode. Without a ZTP config image, it halts with:
***** The ZTP configuration image is missing or improper.
Solution: Select Option 1 (Keyboard/Monitor) from graphical boot menu within ~10 seconds. Access via Cockpit console.
Infrastructure Port 802.1X = Catastrophic
NEVER apply 802.1X templates to infrastructure ports:
-
NAS (storage for VMs)
-
KVM hosts
-
Switches
-
DNS servers
-
Domain controllers
One misconfiguration caused NAS outage → VM filesystem corruption → complete rebuild.
Remaining Tonight
-
Complete ISE-02 install wizard (in progress)
-
Restore backup to ISE-02
-
Apply Patch 2
-
Re-import Vault PKI certs
-
Test single device auth
-
Shutdown ise-01 (NAD failover)
-
Verify all authentications
Tomorrow
-
WLC HA SSO investigation (ping works, HA shows Down)
-
kvm-01 networking parity with kvm-02 (if ISE migration stable)
-
CHLA Linux SSH issue