WRKLOG-2026-03-06

Summary

Friday. WLC HA SSO configuration ongoing - both WLCs reloaded, ha-interface configured, peer still DISABLED. WLC-02 required single NIC recreation (same as WLC-01). ISE-02 restore initiated earlier. Monad evaluation prep for tomorrow.

Today’s Priority Tasks

Priority	Task	Status
P0	ISE-02 restore from backup	[x] DONE
P0	WLC HA SSO configuration	[x] RELOADED (peer DISABLED)
P1	Import Vault PKI certs to ISE-02	[x] DONE (Admin + EAP-TLS)
P1	Test 802.1X authentication	[x] DONE (bound to ise-02)
P1	Reissue pxGrid cert for ise-02	[ ] Pending
P1	CHLA Linux SSH issue (Xianming Ding)	[ ] CARRY-OVER
P2	iPSK Manager - DB replication	[ ] CARRY-OVER

Priority

Task

Status

ISE-02 restore from backup

[x] DONE

WLC HA SSO configuration

[x] RELOADED (peer DISABLED)

Import Vault PKI certs to ISE-02

[x] DONE (Admin + EAP-TLS)

Test 802.1X authentication

[x] DONE (bound to ise-02)

Reissue pxGrid cert for ise-02

[ ] Pending

CHLA Linux SSH issue (Xianming Ding)

[ ] CARRY-OVER

iPSK Manager - DB replication

[ ] CARRY-OVER

NEW: Monad Security Data Pipeline (CISO Priority)

CISO has prioritized Monad evaluation. Starting tomorrow (2026-03-07).

What is Monad?

Security ETL platform - sits "to the left of your SIEM" for data pipeline management.

Aspect	Details
Purpose	Normalize, filter, route security data before SIEM ingestion
Position	Security tools → Monad → SIEM/Data Lake
Value Prop	Reduce SIEM costs (filter noise), normalize formats, intelligent routing
Integrations	250+ pre-built: Splunk, AWS CloudTrail, Azure, Okta, CrowdStrike, Wiz, etc.
Deployment	SaaS, On-prem, or Hybrid
URL	www.monad.com/

Aspect

Details

Purpose

Normalize, filter, route security data before SIEM ingestion

Position

Security tools → Monad → SIEM/Data Lake

Value Prop

Reduce SIEM costs (filter noise), normalize formats, intelligent routing

Integrations

250+ pre-built: Splunk, AWS CloudTrail, Azure, Okta, CrowdStrike, Wiz, etc.

Deployment

SaaS, On-prem, or Hybrid

URL

www.monad.com/

Key Capabilities

Feature	Description
Normalization	Convert diverse log formats to consistent schema
Filtering	Drop low-value logs before expensive SIEM storage (claims 70% reduction)
Routing	Rules-based: high-value → SIEM, low-value → S3 archive
Enrichment	Add context in-flight (GeoIP, threat intel, asset data)
Custom Transforms	Python, Go, REST APIs for custom logic

Feature

Description

Normalization

Convert diverse log formats to consistent schema

Filtering

Drop low-value logs before expensive SIEM storage (claims 70% reduction)

Routing

Rules-based: high-value → SIEM, low-value → S3 archive

Enrichment

Add context in-flight (GeoIP, threat intel, asset data)

Custom Transforms

Python, Go, REST APIs for custom logic

Monad Evaluation Prep (Tomorrow)

Step	Task	Status
1	Request demo/trial access from Monad	[ ]
2	Inventory current log sources (ISE, pfSense, WLC, AD, Wazuh)	[ ]
3	Document SIEM costs and pain points (QRadar/Sentinel?)	[ ]
4	Identify use cases: cost reduction, normalization, compliance	[ ]
5	Review Monad architecture docs	[ ]
6	Prepare questions for vendor call	[ ]

Step

Task

Status

Request demo/trial access from Monad

[ ]

Inventory current log sources (ISE, pfSense, WLC, AD, Wazuh)

[ ]

Document SIEM costs and pain points (QRadar/Sentinel?)

[ ]

Identify use cases: cost reduction, normalization, compliance

[ ]

Review Monad architecture docs

[ ]

Prepare questions for vendor call

[ ]

Questions for Monad Evaluation

Pricing model - Per GB ingested? Per source? Flat rate?
On-prem requirements - What infra needed for self-hosted?
ISE integration - Does 250+ include Cisco ISE syslog/pxGrid?
Latency impact - ETL in-line delay for real-time detection?
Compliance - SOC 2, HIPAA, PCI-DSS certifications?
Data residency - Where does data flow for SaaS option?
Custom parsers - How easy to add new log sources?
HA/DR - Failover and data durability guarantees?

SIEM Context (Document Before Call)

Current SIEM: __
Monthly ingest volume: __ GB
Top log sources by volume: __
Current pain points: __
Compliance requirements: __

WLC HA SSO Configuration

Status

Both WLCs configured, require reload for SSO activation.

Aspect	WLC-01	WLC-02
IP	10.50.1.40	10.50.1.41
Location	kvm-01	kvm-02
redundancy mode	sso	sso
ha-interface	GigabitEthernet 1	GigabitEthernet 1
local-ip	10.50.1.40/24	10.50.1.41/24
remote-ip	10.50.1.41	10.50.1.40
Priority	1 (default)	1 (default)
Communications	Down (pre-reload)	Down (pre-reload)

Aspect

WLC-01

WLC-02

10.50.1.40

10.50.1.41

Location

kvm-01

kvm-02

redundancy mode

sso

ha-interface

GigabitEthernet 1

local-ip

10.50.1.40/24

10.50.1.41/24

remote-ip

10.50.1.41

10.50.1.40

Priority

1 (default)

Communications

Down (pre-reload)

Key Finding

Log message explains why SSO isn’t active:

%VOICE_HA-7-STATUS: NONE->SSO; SSO mode will not take effect until after a platform reload

Commands Applied

! On both WLCs
configure terminal
redundancy
 mode sso
 exit

! WLC-01
chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.40 /24 remote-ip 10.50.1.41

! WLC-02
chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.41 /24 remote-ip 10.50.1.40

write memory

Next Steps (After ISE)

Reload WLC-01 first (becomes Active)
Reload WLC-02 (joins as Standby)
Verify show redundancy shows Communications = Up

Issue: chassis 1 priority Failed

chassis 1 priority 100 failed with syntax error on 17.15.x. Priority defaults to 1 on both. After reload, whichever boots first becomes Active.

Need to research correct 17.x priority syntax.

ISE-02 Restore

Backup File

pre-ise35-restore-CFG10-260305-1159.tar.gpg (from 2026-03-05)

Commands

! Configure NFS repository (already done from previous session)
configure terminal
repository nas-01
  url nfs://10.50.1.70:/volume1/ise_backups
exit

! List backups
show repository nas-01

! Restore
restore pre-ise35-restore-CFG10-260305-1159.tar.gpg repository nas-01 encryption-key plain <KEY>

Progress

Initiating restore.  Please wait...
% restore in progress: Starting Restore...10% completed
% restore in progress: Retrieving backup file from Repository...20% completed
% restore in progress: Decrypting backup data...25% completed
% restore in progress: Extracting backup data...30% completed

ISE will reboot automatically after restore. ~15-20 min total.

After Restore

Wait for all ISE services to start (show application status ise)
Import Vault PKI certificates (ise-02.crt, DOMUS-ISSUING-CA.pem, DOMUS-ROOT-CA.pem)
Test 802.1X authentication

Session Log

Session 1: WLC HA SSO Investigation

Time: Evening

Problem: WLC-01 and WLC-02 showed Communications = Down despite pings working.

Diagnosis:

show redundancy
# Communications = Down      Reason: Failure
# Peer (slot: 0) information is not available because it is in 'DISABLED' state

telnet 10.50.1.40 9800
% Connection refused by remote host

Root Cause: RP service (port 9800) not running - requires reload.

Key Log Message:

%VOICE_HA-7-STATUS: NONE->SSO; SSO mode will not take effect until after a platform reload

Runbook Finding: wlc-ha-sso.adoc Phase 3 commands had issues: - chassis 1 priority 100 - Invalid syntax on 17.15.x - Space before /24 caused confusion but worked

Session 2: ISE-02 Restore

Time: Evening

Backup selected: pre-ise35-restore-CFG10-260305-1159.tar.gpg

Repository already configured from previous session:

show repository nas-01
# Lists all backups including pre-ise35-restore-CFG10-260305-1159.tar.gpg

Restore initiated with encryption key from $ISE_BACKUP_KEY.

Session 3: WLC-02 Gi1 Fix and HA Reconfiguration

Time: Early Morning (02:00-06:00 PST)

Problem: WLC-02 SSH not working, Gi1 not detected (same issue as WLC-01).

Root Cause: 9800-CL requires single NIC on both kvm-01/virbr0 AND kvm-02/br-mgmt.

Fix Applied:

# On kvm-02
sudo virsh destroy 9800-WLC-02
sudo virsh undefine 9800-WLC-02

sudo virt-install \
  --name 9800-WLC-02 \
  --memory 16384 \
  --vcpus 4 \
  --import \
  --disk path=/mnt/nas/vms/9800-WLC-02.qcow2,format=qcow2 \
  --network bridge=br-mgmt,model=virtio \
  --os-variant generic \
  --graphics vnc \
  --noautoconsole

Post-Recreation Config:

! Timezone and NTP (both WLCs)
configure terminal
clock timezone PST -8
clock summer-time PDT recurring
ntp server 10.50.1.1
end

! HA Interface (priv exec, NOT config mode)
chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.41 /24 remote-ip 10.50.1.40
write memory

Current Status:

show chassis ha-status local

My state = ACTIVE
Peer state = DISABLED
Local-IP: 10.50.1.40 / 10.50.1.41
Remote-IP: 10.50.1.41 / 10.50.1.40
HA-Interface: GigabitEthernet1

Key Findings:

chassis redundancy ha-interface is priv exec command, not config mode
Config stored separately - not visible in show run | sec chassis
Use show chassis ha-status local to verify HA config
Both WLCs show peer DISABLED despite config + reload

Runbook Updated: wlc-ha-sso.adoc - Added WLC-02 Gi1 troubleshooting section.

Session 6: WLC HA SSO - Version Mismatch and kvm-01 DNS Fix

Time: Evening

Problem: WLC HA SSO showing peer DISABLED on both controllers.

Diagnosis:

9800-WLC-01# telnet 10.50.1.41 9800
% Connection refused by remote host

9800-WLC-02# telnet 10.50.1.40 9800
% Connection refused by remote host

Root Cause: Version mismatch - WLC-01 is 17.15.3, WLC-02 is 17.15.4d. SSO requires same version.

Fix: Upgrade WLC-01 to 17.15.4d. Image on NAS at /volume1/isos/C9800-CL-universalk9.17.15.04d.SPA.bin

Secondary Issue: kvm-01 couldn’t resolve nas-01 hostname.

sudo mount -t nfs nas-01:/volume1/isos /mnt/nas/isos
# mount.nfs: Failed to resolve server nas-01: Name or service not known

cat /etc/resolv.conf
# nameserver 10.50.1.1   <-- Only pfSense, no bind-01!

Fix: Add bind-01 as primary nameserver:

sudo tee /etc/resolv.conf <<'EOF'
nameserver 10.50.1.90
nameserver 10.50.1.1
EOF

Key Learnings:

Learning Details

Learning	Details
`tee` vs `tee -a`	`tee` overwrites, `tee -a` appends (no clobber)
WLC SSO version match	Both controllers must run identical IOS-XE version
kvm-01 DNS config	Needs bind-01 (10.50.1.90) to resolve internal names
Port 9800 refused	RP service not running = version mismatch or SSO not activated

tee vs tee -a

tee overwrites, tee -a appends (no clobber)

WLC SSO version match

Both controllers must run identical IOS-XE version

kvm-01 DNS config

Needs bind-01 (10.50.1.90) to resolve internal names

Port 9800 refused

RP service not running = version mismatch or SSO not activated

Session 5: Keycloak SAML Client Update for ISE-02

Time: Evening

Problem: SAML login failing with "invalid redirect url" - Keycloak client had ise-01 URIs only.

Solution: Update Keycloak SAML client to support BOTH ise-01 and ise-02 via REST API.

# Load identity secrets
dsource d000 dev/identity

# Get admin token
KC_TOKEN=$(curl -s -X POST \
  "https://keycloak-01.inside.domusdigitalis.dev:8443/realms/master/protocol/openid-connect/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin" \
  -d "password=$KC_ADMIN_PASS" \
  -d "grant_type=password" \
  -d "client_id=admin-cli" \
  --insecure | jq -r '.access_token')

# Find ISE SAML client UUID
curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure | jq '.[] | select(.protocol=="saml") | {id, clientId, name, redirectUris}'

# Export full client config
CLIENT_UUID="<uuid-from-above>"
curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure > /dev/shm/ise-saml-client.json

# Edit JSON to add BOTH ise-01 and ise-02 to redirectUris array
# Then apply:
curl -s -X PUT \
  "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  -H "Content-Type: application/json" \
  -d @/dev/shm/ise-saml-client-updated.json \
  --insecure -w "\nHTTP_STATUS: %{http_code}\n"

# Verify and cleanup
curl -s "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/clients/$CLIENT_UUID" \
  -H "Authorization: Bearer $KC_TOKEN" \
  --insecure | jq '{name, redirectUris}'

shred -u /dev/shm/ise-saml-client*.json

Key Learning: netapi doesn’t have update-client yet - use Keycloak Admin REST API directly.

Session 4: ISE-02 Cutover - ise-01 Shutdown

Time: Morning

Status: ISE-02 now primary. ise-01 shutdown.

sudo virsh shutdown ise-01
# Domain 'ise-01' is being shutdown

Vault PKI Certs Imported to ISE-02:

Service	Certificate	Status
Admin Portal	Vault PKI (DOMUS-ISSUING-CA)	✓ Active
EAP-TLS	Vault PKI (DOMUS-ISSUING-CA)	✓ Active
pxGrid	Need reissue for ise-02	Pending
Guest Portal	Let’s Encrypt (future)	Planned

Service

Certificate

Status

Admin Portal

Vault PKI (DOMUS-ISSUING-CA)

✓ Active

EAP-TLS

Vault PKI (DOMUS-ISSUING-CA)

✓ Active

pxGrid

Need reissue for ise-02

Pending

Guest Portal

Let’s Encrypt (future)

Planned

Current pxGrid Cert (exported for reference):

openssl x509 -in ~/Downloads/DOMUSPXGRID.pem -noout -subject -issuer -dates -ext subjectAltName
# subject=CN=ise-01.inside.domusdigitalis.dev  <-- Wrong, needs ise-02
# issuer=CN=DOMUS-ISSUING-CA
# SAN: DNS:ise-01.inside.domusdigitalis.dev, IP:10.50.1.20

pxGrid Cert Reissuance Workflow (ise-02):

# Issue new cert from Vault PKI
vault write pki_int/issue/domus-server \
  common_name="ise-02.inside.domusdigitalis.dev" \
  alt_names="ise-02.inside.domusdigitalis.dev" \
  ip_sans="10.50.1.21" \
  ttl="8760h" \
  -format=json > /dev/shm/ise-02-pxgrid.json

# Extract certificate
jq -r '.data.certificate' /dev/shm/ise-02-pxgrid.json > /dev/shm/ise-02-pxgrid.crt

# Extract private key
jq -r '.data.private_key' /dev/shm/ise-02-pxgrid.json > /dev/shm/ise-02-pxgrid.key

# Extract CA chain
jq -r '.data.ca_chain[]' /dev/shm/ise-02-pxgrid.json > /dev/shm/ise-02-pxgrid-chain.pem

# Verify
openssl x509 -in /dev/shm/ise-02-pxgrid.crt -noout -subject -issuer -dates -ext subjectAltName

# Cleanup after import to ISE
rm -f /dev/shm/ise-02-pxgrid.*

Why /dev/shm? RAM-backed tmpfs - private key never touches disk.

Carried Over

Professional (CHLA)

Linux SSH issue (Xianming Ding) - AD-joined Linux SSH troubleshooting
iPSK Manager DB replication
ISE 3.4 Patch 9 or 3.5 migration planning

Personal Infrastructure

WLC reload for SSO (after ISE stable)
Import Vault PKI certs to ISE-02
Test 802.1X authentication
kvm-01 bridge VLAN persistence (libvirt hook)

Learning Tracks

RHCSA study session (Thursdays)
DELE C1 Spanish tutor prep

restore <filename> repository <repo-name> encryption-key plain <key>

ISE automatically reboots after restore completes.

Monad Architecture Pattern

Log Sources → Monad (ETL) → SIEM
                  ↓
              Archive (S3)

Filter noise before expensive SIEM storage. Route by value.

WLC 9800-CL Single NIC Requirement

BOTH kvm-01/virbr0 AND kvm-02/br-mgmt require single NIC for 9800-CL.

The runbook incorrectly stated br-mgmt could use dual NIC. Updated.

WLC HA Interface Command Syntax

chassis redundancy ha-interface is a privileged exec command, NOT config mode:

9800-WLC-01# chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.40 /24 remote-ip 10.50.1.41

Config is stored separately from running-config. Verify with:

show chassis ha-status local

Use /dev/shm for Sensitive Files

Pattern: Store private keys and sensitive data in /dev/shm/ (RAM-backed tmpfs):

Never written to disk
Cleared on reboot
No swap risk (if swappiness=0)

# Good: RAM-backed
vault write ... -format=json > /dev/shm/cert.json
jq -r '.data.private_key' /dev/shm/cert.json > /dev/shm/cert.key

# Bad: Disk-backed
vault write ... -format=json > /tmp/cert.json  # May persist, recoverable

Tomorrow (2026-03-07)

CISO Priority: Monad

Request Monad demo/trial
Document current SIEM infrastructure
Prepare evaluation questions
Schedule vendor call

Infrastructure

Verify ISE-02 restore complete
Import Vault PKI certificates
Test 802.1X authentication
Reload WLCs for SSO

CHLA

Linux SSH troubleshooting (Xianming Ding)