WRKLOG-2026-03-05

Summary

Wednesday. Full ISE 3.5 migration from ise-01 to ise-02. IPMI-01 troubleshooting (dedicated LAN mode). Major runbook updates: Phase 3a restore procedure, gopass v3 paths.

Today’s Priority Tasks

Priority Task Status

P0

ISE 3.5 migration (ise-01 → ise-02)

[ ] In Progress

P0

IPMI-01 connectivity (kvm-01 out-of-band)

[x] DONE

P1

Update ise-35-deployment.adoc with Phase 3a

[x] DONE

P1

Update gopass paths to v3 structure

[x] DONE

P1

CHLA Linux SSH issue (Xianming Ding)

[ ] CARRY-OVER

P2

Restructure ad.adoc into granular directory

[ ] CARRY-OVER

P2

Restructure remoting.adoc into granular directory

[ ] CARRY-OVER

P2

iPSK Manager - DB replication

[ ] CARRY-OVER

ISE 3.5 Migration Sequence

Step Task Status

1

Backup from ise-01

[x] DONE

2

Install ISE 3.5 on ise-02 (kvm-02)

[x] DONE

3

Configure NFS repository

[x] DONE

4

Restore backup to ise-02

[x] DONE

5

Wait for services (~15-20 min)

[x] DONE

6

Verify restored config

[x] DONE

7

Issue ise-02 certs from Vault PKI

[x] DONE

8

Import certs to ISE GUI

[x] DONE

9

ISE restart after cert import

[x] DONE

10

NAS OUTAGE - VM CORRUPTED

⚠️ INCIDENT

11

Redeploy ise-02 VM from scratch

IN PROGRESS

12

Restore backup (again)

[ ] Pending

13

Apply Patch 2

[ ] Pending

14

Re-import Vault PKI certs

[ ] Pending

15

Test single device auth

[ ] Pending

16

Shutdown ise-01 (NAD failover)

[ ] Pending

17

Verify all authentications

[ ] Pending

Session Log

Session 1: IPMI-01 Troubleshooting

Time: Morning

Problem: IPMI-01 (kvm-01 BMC) unreachable at 10.50.1.200

Diagnosis:

  • Switch MAC table showed eno2 MAC, not IPMI MAC

  • BMC LAN mode was 02 (Failover), needed 00 (Dedicated)

  • Physical cable was in wrong port (eno2 instead of dedicated IPMI)

Commands used:

# Check BMC LAN mode (from kvm-01)
sudo ipmitool raw 0x30 0x70 0x0c 0
# Returns: 02 = Failover, 00 = Dedicated

# Set to Dedicated mode
sudo ipmitool raw 0x30 0x70 0x0c 1 0

# Reset BMC
sudo ipmitool mc reset cold

# Verify physical port (shows state UP = cable connected)
ip link show eno2

Resolution: Moved cable to dedicated IPMI port. Now reachable at 10.50.1.200.

Documentation updated:

  • kvm-01-migration-plan.adoc - Added Phase 0 IPMI Configuration

  • kvm-02-deployment.adoc - Added section 0.2.1 LAN Mode verification

Session 2: ISE 3.5 Restore

Time: Midday

Fresh backup from ise-01:

dsource d000 dev/storage
netapi ise backup --repo nas-01 --name "pre-ise35-restore" --wait

Configure repository on ise-02 (ISE CLI):

configure terminal
repository nas-01
  url nfs://10.50.1.70:/volume1/ise_backups
exit

Verify repository:

show repository nas-01 | include pre-ise35-restore

Restore command:

restore pre-ise35-restore-CFG10-260305-1159.tar.gpg repository nas-01 encryption-key plain <KEY>

Runbook updates:

  • Added Phase 3a: Restore from Backup (Alternative)

  • Updated credentials table from gopass v2 to v3 paths

  • Fixed ISE_BACKUP_KEY location: dev/storage not dev/network

Session 3: CLI Patterns Used Today

awk - Extract specific lines from runbook:

# View specific line numbers
awk 'NR==689 || NR==700' docs/asciidoc/modules/ROOT/pages/runbooks/ise-35-deployment.adoc

# Output:
# restore <backup-filename> repository nas-01 encryption-key plain <YOUR_KEY>
# restore scheduled-CFG10-260222-1948.tar.gpg repository nas-01 encryption-key plain MySecretKey123

grep - Search for patterns in runbook:

# Find all restore references
grep -n "restore" ise-35-deployment.adoc | head -20

# Check for ISE_BACKUP_KEY location
grep -i "ISE_BACKUP_KEY" ise-35-deployment.adoc

ISE CLI - Pipe filtering:

# Filter repository listing
show repository nas-01 | include pre-ise35-restore-CFG10-260305-1159.tar.gpg

Session 3: Vault PKI Cert for ise-02

Time: Afternoon

Key Learning: Vault accessible from workstation - no SSH to vault-01 required.

Issue cert from workstation:

dsource d000 dev/vault
vault write pki_int/issue/domus-server \
  common_name="ise-02.inside.domusdigitalis.dev" \
  ip_sans="10.50.1.21" \
  ttl="8760h" \
  -format=json > /dev/shm/ise-02-cert.json

Extract certificate, key, CA chain:

jq -r '.data.certificate' /dev/shm/ise-02-cert.json > /dev/shm/ise-02.crt
jq -r '.data.private_key' /dev/shm/ise-02-cert.json > /dev/shm/ise-02.key
jq -r '.data.ca_chain[]' /dev/shm/ise-02-cert.json > /dev/shm/ise-02-chain.pem

Split CA chain into individual certs:

awk '/-----BEGIN CERTIFICATE-----/{n++} n==1' /dev/shm/ise-02-chain.pem > /dev/shm/DOMUS-ISSUING-CA.pem
awk '/-----BEGIN CERTIFICATE-----/{n++} n==2' /dev/shm/ise-02-chain.pem > /dev/shm/DOMUS-ROOT-CA.pem

Verify certs:

openssl x509 -in /dev/shm/ise-02.crt -subject -issuer -noout
openssl x509 -in /dev/shm/DOMUS-ISSUING-CA.pem -subject -noout
openssl x509 -in /dev/shm/DOMUS-ROOT-CA.pem -subject -noout

ISE Import Order:

  1. Administration → System → Certificates → Trusted Certificates → Import

    • DOMUS-ROOT-CA.pem (Trust for ISE Auth + Client Auth)

    • DOMUS-ISSUING-CA.pem (Trust for ISE Auth + Client Auth)

  2. Administration → System → Certificates → System Certificates → Import

    • Certificate: ise-02.crt

    • Private Key: ise-02.key

    • Usage: Admin, EAP, pxGrid

  3. ISE restarts automatically after system cert import

Runbook updated: ise-35-deployment.adoc - Added 4.0 Quick Method (workstation-based PKI)

Session 4: WLC-01 Single NIC Fix

Time: Afternoon

Problem: Two NICs on kvm-01/virbr0 causes IOS-XE to detect only Gi2, not Gi1.

Solution: Recreate VM with single NIC using existing qcow2:

sudo virt-install \
  --name 9800-WLC-01 \
  --memory 16384 \
  --vcpus 4 \
  --import \
  --disk path=/mnt/onboard-ssd/vms/C9800-CL-universalk9.17.15.03.qcow2,format=qcow2 \
  --network bridge=virbr0,model=virtio \
  --os-variant generic \
  --graphics vnc \
  --noautoconsole

Verification: WLC-01 and WLC-02 CAN ping each other (80-100% success).

Runbook updated: wlc-ha-sso.adoc - Added single NIC requirement + Appendix D Troubleshooting

Session 5: NAS Outage - ISE-02 VM Corrupted

Time: Evening

INCIDENT: NAS (10.50.1.70) became unreachable during ISE migration.

Root Cause: Switch port Gi1/0/6 had 802.1X template AND wrong VLAN:

# BEFORE (broken)
interface GigabitEthernet1/0/6
 source template DefaultWiredDot1xClosedAuth  <-- 802.1X ON INFRASTRUCTURE PORT
 switchport access vlan 10                      <-- WRONG VLAN (should be 100)

Impact:

  • NFS disconnect corrupted ise-02 VM filesystem

  • VM booted to emergency mode with I/O errors:

    blk_update_request: I/O error, dev vda, sector 93521729
    EXT4-fs (vda3): error loading journal

Resolution:

  1. Remove 802.1X template from switch port

  2. Fix VLAN to 100

  3. Decision: Redeploy ise-02 from scratch (filesystem unrecoverable)

Key Learning: NEVER put 802.1X on infrastructure ports (NAS, KVM hosts, switches).

Session 6: ISE-02 Redeployment

Time: Evening

Problem 1: virt-install failed - "Size must be specified for non existent volume"

Fix: Add size=600 to disk parameter.

Problem 2: VM showed "No bootable media" - ISO not attached

Diagnosis:

sudo virsh domblklist ise-02
# sda showed "-" (empty)

Fix:

sudo virsh change-media ise-02 sda /mnt/nas/isos/Cisco-ISE-3.5.0.527.SPA.x86_64.iso --insert --config
sudo virsh destroy ise-02 && sudo virsh start ise-02

Problem 3: ZTP Auto-Install halted

Symptom:

***** The ZTP configuration image is missing or improper. Automatic installation flow exited.
***** Power off and attach the proper ZTP configuration image or choose manual boot to proceed.
[   44.710453] reboot: System halted

Fix: Must select Option 1 (Keyboard/Monitor) from graphical boot menu in Cockpit. ISE defaults to ZTP mode without manual selection.

Current Status: ISE-02 installing from ISO (30-45 min)

Runbook updated: ise-35-deployment.adoc - Added troubleshooting sections for all three issues

Session 7: VyOS VLAN 40 DHCP Validation

Time: Evening

Context: Testing vyos-02 DHCP for VLAN 40 (IOT) per vyos-vlan-fasttrack-migration.adoc.

Network Path (from runbooks):

  • Te1/0/1 → kvm-02 → vyos-02 eth0 (trunk with VLANs 20,30,40,100,999)

  • Te1/0/2 → kvm-01 → pfSense (VLANs 10,100,999 - VLAN 40 intentionally removed)

VyOS verification (from vyos-02 console):

# Confirm eth0.40 interface up
show interfaces | grep -E 'eth0\.(20|30|40)'
# Expected: eth0.40  10.50.40.1/24  u/u  IOT

# Confirm DHCP pool available
show dhcp server statistics
# Expected: IOT pool with 100+ available addresses

Test setup - backup ethernet on modestus-razer:

# Create backup connection for USB ethernet adapter
nmcli conn add type ethernet con-name Domus-Wired-Backup ifname "enp0s13f0u2u1"

# View active connections with awk
nmcli conn show --active | awk 'NR>1 {print $1, $3}'

Switch port for VLAN 40 test (G1/0/2):

interface GigabitEthernet1/0/2
 description [DOT1X] User Access Port
 switchport access vlan 40
 switchport mode access
 switchport voice vlan 20
 ip arp inspection trust
 spanning-tree portfast edge
end

Validation commands:

# Force DHCP on backup interface
sudo nmcli conn down Domus-Wired-Backup && sudo nmcli conn up Domus-Wired-Backup

# Check IP from vyos-02 DHCP (should be 10.50.40.x)
ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'

# Verify gateway is vyos-02
ip route | awk '/enp0s13f0u2u1/ && /default|10.50.40/ {print}'

# Check DHCP source in journal
journalctl -u NetworkManager --since "2 min ago" | awk '/enp0s13f0u2u1.*DHCP/ {print}'

Key Learning: Use separate test interface to validate VyOS DHCP without disrupting primary 802.1X session.

Session 8: VyOS Firewall Fix - LOCAL_IOT Rule

Time: Evening

Root Cause: LOCAL_IOT firewall was EMPTY - no rules to allow DHCP replies.

DHCP Flow Analysis:

Step Direction Firewall Result

1. Discover

Client → VyOS (port 67)

IOT_LOCAL rule 30

✓ Accept (6 packets)

2. Offer

VyOS → Client (port 68)

LOCAL_IOT (empty)

✗ DROPPED

Diagnosis commands (VyOS):

show firewall ipv4 name IOT_LOCAL
# rule 30: accept UDP port 67 - packets matched

show firewall ipv4 name LOCAL_IOT
# Ruleset Information (EMPTY - no rules!)

Fix applied (VyOS):

configure
set firewall ipv4 name LOCAL_IOT rule 10 action accept
set firewall ipv4 name LOCAL_IOT rule 10 protocol udp
set firewall ipv4 name LOCAL_IOT rule 10 destination port 68
set firewall ipv4 name LOCAL_IOT rule 10 description "DHCP replies"
commit
save

Verification (VyOS):

show firewall ipv4 name LOCAL_IOT
# rule 10 { action accept; destination { port 68 }; protocol udp }

Test from workstation:

# Force DHCP renewal
sudo nmcli conn down Domus-Wired-Backup && sudo nmcli conn up Domus-Wired-Backup

# Verify IP from vyos-02 DHCP pool (10.50.40.x)
ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'

# Verify gateway is vyos-02
ip route | awk '/enp0s13f0u2u1/{print}'

Key Learning: VyOS zone-based firewall requires BIDIRECTIONAL rules:

  • IOT_LOCAL = Traffic from IOT zone TO the router (DHCP requests, DNS queries)

  • LOCAL_IOT = Traffic from router TO IOT zone (DHCP replies, DNS responses)

Without LOCAL_IOT rule for port 68, DHCP Offer packets are silently dropped.

Session 9: VyOS DHCP Fixed - kvm-02 Bridge VLAN

Time: Evening

Problem persisted after VyOS firewall fix. DHCP still failing.

Root Cause: kvm-02 bridge interface vnet1 (vyos-02’s eth0) wasn’t passing VLAN 40 tags.

Diagnosis (kvm-02):

# Check bridge VLAN assignments
sudo bridge vlan show

# Output showed:
# vnet1             1 PVID Egress Untagged  <-- NO VLAN 40!
# br-mgmt had VLANs 10,20,30,40,100,110,120 ✓
# eno8 (trunk) had same VLANs ✓
# Verify vyos-02 uses br-mgmt, not virbr0
sudo virsh domiflist vyos-02
# vnet1 → br-mgmt ✓

Fix (kvm-02):

sudo bridge vlan add vid 40 dev vnet1

Verification:

sudo bridge vlan show dev vnet1
# vnet1             1 PVID Egress Untagged
#                   40  <-- NOW PRESENT

Result (workstation):

ip -4 addr show enp0s13f0u2u1 | awk '/inet/{print $2}'
# 10.50.40.100/24  ✓ FROM VYOS!

ip route | awk '/enp0s13f0u2u1/ && /default/{print}'
# default via 10.50.40.1 ... ✓ VYOS GATEWAY

Key Learning: KVM bridge VLAN filtering requires VLANs on BOTH:

  1. Physical trunk interface (eno8) ✓

  2. Bridge itself (br-mgmt) ✓

  3. VM’s vnet interface ← This was missing!

WARNING: bridge vlan add is NOT persistent. Needs NetworkManager or libvirt hook.

Carried Over

Professional (CHLA)

Critical (P0)

Project Description Status Blocker

Mandiant Remediation

dACL enforcement, posture/ACL remediation, ISE patch

ACTIVE — Q2 assessment

Project

Linux Research (Xiangming)

EAP-TLS for Linux workstations

BEHIND (due 02-24)

Certificate "password required" — nmcli flags fix documented

iPSK Manager HA

Pre-shared key automation — HA deployment

BEHIND

DB replication issues

MSCHAPv2 Migration

Legacy auth deprecation → EAP-TLS

BEHIND

No progress on planning

High Priority (P1)

Project Description Status

CHLA Antora Setup

8-phase Antora documentation deployment at work

ACTIVE (Project)

SIEM Migration

QRadar → Microsoft Sentinel — SDK integration

ACTIVE (Project)

ISE 3.4 Migration

Upgrade from 3.2p9

Planned — blocked by P0 items

ISE Hardware Refresh

PSN/MnT lifecycle replacement

Planned

Switch Upgrades

IOS-XE fleet update

Pending

Standard (P2)

Project Description Status

HHS Regulatory Compliance

New HHS security policies

NOT STARTED

InfoSec Reporting Dashboard

PowerBI metrics for executives

NOT STARTED

EDR Migration (AMP → Defender)

Endpoint protection consolidation

NOT STARTED

Azure Legacy Migration

Modern landing zone

In progress

Personal Infrastructure

Recently Completed

Project Description Date

domus-api v0.1.0

44-endpoint REST API — multi-spoke, DI, cache invalidation, 55 tests

2026-04-07

IOT_WAN VPN Passthrough

4 firewall rules applied — IPsec ESP, NAT-T, IKE, TCP 2443

2026-04-07

VyOS HA Migration

vyos-01 deployed, replacing pfSense

2026-03-07

C9130AX WiFi6 AP

Catalyst 9130AX access point deployment

2026-03-10

enterprise-linux-8021x

Standalone 802.1X EAP-TLS documentation spoke

2026-02-26

CLI Mastery Documentation

openssl/curl/awk/sed/xargs/pipelines

2026-02-26

Vault SSH CA

8h certs, 9 hosts configured

2026-02-21

k3s + Prometheus/Grafana

Monitoring stack on k3s

2026-02-23

In Progress

Project Description Status

ThinkPad P16g Deploy

Phase 11 verification, Phase 12 security hardening

Active

EVE-NG Lab

Network simulation lab — 8-phase rollout

Phase 0

RHEL 9 Workstation

Dr. Shahab’s workstation — 12-phase deployment

Draft

kvm-02 Hardware Upgrade

Supermicro B deployment

Hardware ready

Planned

Project Description Blocked By

Vault HA (3-node)

vault-02, vault-03 on kvm-02

kvm-02 deployment

DNS HA (bind-02)

Secondary with zone transfers from bind-01

kvm-02 deployment

k3s HA (3-node)

Control plane HA

kvm-02 deployment

Wazuh Agents

Deploy to all infrastructure hosts

k3s NAT fix (29 days blocked)

Cold Storage (M-DISC)

Offline archival of keys/headers

Time

SanDisk USB Offsite

Third backup drive rotation

Time

Learning Tracks

Track Description Status Resources

API Development (FastAPI)

REST API design, Pydantic, DI, async, testing

ACTIVE

domus-api (44 endpoints), API CLI Mastery

Claude Code + AI Engineering

Claude Code mastery, hooks, skills, agents

ACTIVE

Anthropic docs, Project

RHCSA 9 (EX200)

Red Hat system administration — 21-phase curriculum

ACTIVE

Sander van Vugt book, Project

CISSP

10-domain security certification

ACTIVE

Project

Spanish (DELE C1/C2)

Advanced Spanish certification

ACTIVE

Connectors reference, essay structure, Don Quijote

API CLI Mastery

jq/curl/awk/httpx pipeline composition

COMPLETE

Codex — 6-level curriculum

Terminal Mastery

awk/sed/jq/xargs/grep/find patterns

COMPLETE

Education

College Algebra

Functions, polynomials, exponentials, logarithms

ACTIVE

Education

Shell Scripting Guide

Comprehensive shell scripting (23 chapters)

In progress

Education

Linux Bible 11e

Linux administration reference

In progress

Education

D2 Diagrams

Infrastructure visualization

Ongoing

Practice with every runbook

Biblical Studies

Study notes and teachings

In progress

Private notes

Ruby Metaprogramming

DSL for infrastructure generation

PARKED (P3)

domus-captures/education/ruby/

Key Learnings

IPMI LAN Modes (Supermicro)

Value Mode Description

00

Dedicated

Uses dedicated IPMI port only (required for separate IPMI NIC)

01

Shared

Shares with onboard NIC1

02

Failover

Tries dedicated, falls back to shared

Lesson: Always verify LAN mode when IPMI is unreachable. Physical cabling matters.

ISE Restore Methods

Method When to Use Notes

netapi ise restore

Existing ISE with API configured

Requires dsec pointing to target ISE

ISE CLI restore

Fresh install, API not configured

Works universally, no dsec changes needed

gopass v3 Structure

ISE credentials now in identity hierarchy:

v3/domains/d000/identity/ise/ise-01/{admin,cli,dataconnect}
v3/domains/d000/identity/ise/ise-02/{admin,cli,dataconnect}

Vault PKI from Workstation

Key Pattern: If vault-ssh-sign works from workstation, so does cert issuance.

dsource d000 dev/vault && vault-ssh-sign  # If this works, Vault is accessible

No need to SSH to vault-01 for cert operations.

WLC Single NIC on kvm-01/virbr0

Problem: IOS-XE with 2 NICs on virbr0 shows only Gi2 (Gi1 missing).

Solution: Use single NIC with virt-install --import.

Verification: Both WLCs can ping each other (80-100% success).

awk CA Chain Split Pattern

# Full BEGIN CERTIFICATE string required (not just "BEGIN CERT")
awk '/-----BEGIN CERTIFICATE-----/{n++} n==1' chain.pem > first-cert.pem
awk '/-----BEGIN CERTIFICATE-----/{n++} n==2' chain.pem > second-cert.pem

ISE ISO ZTP Boot Trap

Problem: ISE ISO defaults to ZTP (Zero Touch Provisioning) mode. Without a ZTP config image, it halts with:

***** The ZTP configuration image is missing or improper.

Solution: Select Option 1 (Keyboard/Monitor) from graphical boot menu within ~10 seconds. Access via Cockpit console.

Infrastructure Port 802.1X = Catastrophic

NEVER apply 802.1X templates to infrastructure ports:

  • NAS (storage for VMs)

  • KVM hosts

  • Switches

  • DNS servers

  • Domain controllers

One misconfiguration caused NAS outage → VM filesystem corruption → complete rebuild.

Remaining Tonight

  • Complete ISE-02 install wizard (in progress)

  • Restore backup to ISE-02

  • Apply Patch 2

  • Re-import Vault PKI certs

  • Test single device auth

  • Shutdown ise-01 (NAD failover)

  • Verify all authentications

Tomorrow

  • WLC HA SSO investigation (ping works, HA shows Down)

  • kvm-01 networking parity with kvm-02 (if ISE migration stable)

  • CHLA Linux SSH issue