WRKLOG-2026-03-16

Summary

Sunday. Preparing for potential ISE questions from leadership regarding ongoing TAC case (P2). Strongline gateway identity group issue needs resolution. Regex CLI drill system completed and pushed. Carryover tasks from multiple days require focused attack.

ISE Incident Prep

Stakeholders Expecting Updates

Name	Title	Concern
Sarah Clizer	CISO	Overall ISE stability and security posture
Jonathan Carr	Assoc. Dir. Field Support Services	End-user connectivity impact
Albert Rodriguez	Manager, Collaboration Services	Network-wide auth issues

Name

Title

Concern

Sarah Clizer

CISO

Overall ISE stability and security posture

Jonathan Carr

Assoc. Dir. Field Support Services

End-user connectivity impact

Albert Rodriguez

Manager, Collaboration Services

Network-wide auth issues

TAC Case Status

SR Priority: S2 (downgraded from S1 after reboot)
Status: Open - Monitoring post-reboot
Primary MNT: Rebooted 2026-03-12 16:19, RabbitMQ CPU normalized
Pending: ISE 3.2 Patch 9 upgrade, ISE Messaging Service enable

Defense Points

If Asked	Response
"Is ISE causing the auth failures?"	ISE is processing authentications correctly. The MNT logging backlog (RabbitMQ) affected visibility, not authentication. PSNs never stopped processing RADIUS. See post-reboot validation showing all services running.
"Why did RabbitMQ spike to 100%+ CPU?"	MNT message queue saturation from high session volume + possible replication lag. TAC confirmed this is a known issue addressed in Patch 9. Reboot cleared the backlog.
"Is the network safe?"	All 4 PSNs continue processing 802.1X authentications. Secondary MNT provides redundancy. Authentication flow is completely separate from monitoring/logging.
"What’s the root cause?"	Still under TAC investigation. Working theory: replication delay between MNT nodes caused queue buildup. Patch 9 addresses known replication issues.
"What’s the timeline to full resolution?"	Pending: ISE Messaging Service enable (maintenance window), Patch 9 upgrade (TAC coordinated). No authentication impact in current state.

If Asked

Response

"Is ISE causing the auth failures?"

ISE is processing authentications correctly. The MNT logging backlog (RabbitMQ) affected visibility, not authentication. PSNs never stopped processing RADIUS. See post-reboot validation showing all services running.

"Why did RabbitMQ spike to 100%+ CPU?"

MNT message queue saturation from high session volume + possible replication lag. TAC confirmed this is a known issue addressed in Patch 9. Reboot cleared the backlog.

"Is the network safe?"

All 4 PSNs continue processing 802.1X authentications. Secondary MNT provides redundancy. Authentication flow is completely separate from monitoring/logging.

"What’s the root cause?"

Still under TAC investigation. Working theory: replication delay between MNT nodes caused queue buildup. Patch 9 addresses known replication issues.

"What’s the timeline to full resolution?"

Pending: ISE Messaging Service enable (maintenance window), Patch 9 upgrade (TAC coordinated). No authentication impact in current state.

Quick Diagnostic Commands

# Check PSN auth processing (should show active sessions)
netapi ise -f json mnt sessions | jq 'length'

# Verify no auth failures in last hour
netapi ise -f json mnt failures --hours 1 | jq 'length'

# Check deployment health
netapi ise -f json api-call openapi GET "/api/v1/deployment/node" | jq -r '.response[] | [.hostname, .nodeStatus] | @tsv'

Active Incidents

INC-2026-03-16: Strongline Gateway VLAN Assignment

Reported by: Arin Khachikyan (Network Engineer)
Assigned to: David Rukiza, Ntashamaje (Security Analyst)
Issue: 8 Strongline gateways in wrong ISE identity group → wrong VLAN
Background: CSV import moved some devices; 8 remain misassigned
Expected: Devices should be in designated identity group for correct VLAN assignment

See: Full Incident Doc

Today’s Priorities (2026-03-16)

Priority	Task	Status	Notes
P0	ISE Incident Prep	[x] DONE	This worklog + defense document
P0	Strongline Gateway Remediation	[ ] IN PROGRESS	Move 8 devices to correct identity group
P1	Regex Mastery Training	[ ] PENDING	CLI drills created, practice time
P1	Monad Pipeline Evaluation	[ ] PENDING	Carried over since 03-11 - 5 days
P1	k3s NAT verification	[ ] PENDING	Carried over since 03-09 - 7 days
P2	Wazuh indexer recovery	[ ] PENDING	Depends on NAT working

Priority

Task

Status

Notes

ISE Incident Prep

[x] DONE

This worklog + defense document

Strongline Gateway Remediation

[ ] IN PROGRESS

Move 8 devices to correct identity group

Regex Mastery Training

[ ] PENDING

CLI drills created, practice time

Monad Pipeline Evaluation

[ ] PENDING

Carried over since 03-11 - 5 days

k3s NAT verification

[ ] PENDING

Carried over since 03-09 - 7 days

Wazuh indexer recovery

[ ] PENDING

Depends on NAT working

Comprehensive Backlog Audit

Professional (CHLA) - CRITICAL

Task	Details	Original Date	Days Carried
Monad Pipeline Evaluation	Test pipeline creation, input sources, transforms	2026-03-11	5 days
k3s NAT verification	NAT rule 170 for 10.42.0.0/16 pod network - test internet	2026-03-09	7 days
Wazuh indexer recovery	Restart pod after NAT confirmed working	2026-03-09	7 days
Strongline Gateway VLAN fix	8 devices in wrong identity group	2026-03-16	NEW
Vocera EAP-TLS Supplicant Fix	~10 phones failing 802.1X, missing supplicant config	2026-03-12	4 days
modestus-razer EAP-TLS fix	Add intermediate CA to client cert chain (ISE 5411 error)	2026-03-11	5 days
ISE MnT Messaging Service	Enable "Use ISE Messaging Service for UDP syslogs delivery"	2026-03-12	4 days
ISE Patch 9 upgrade	ISE 3.2 Patch 9 addresses known replication issues	2026-03-12	4 days

Task

Details

Original Date

Days Carried

Monad Pipeline Evaluation

Test pipeline creation, input sources, transforms

2026-03-11

5 days

k3s NAT verification

NAT rule 170 for 10.42.0.0/16 pod network - test internet

2026-03-09

7 days

Wazuh indexer recovery

Restart pod after NAT confirmed working

2026-03-09

7 days

Strongline Gateway VLAN fix

8 devices in wrong identity group

2026-03-16

NEW

Vocera EAP-TLS Supplicant Fix

~10 phones failing 802.1X, missing supplicant config

2026-03-12

4 days

modestus-razer EAP-TLS fix

Add intermediate CA to client cert chain (ISE 5411 error)

2026-03-11

5 days

ISE MnT Messaging Service

Enable "Use ISE Messaging Service for UDP syslogs delivery"

2026-03-12

4 days

ISE Patch 9 upgrade

ISE 3.2 Patch 9 addresses known replication issues

2026-03-12

4 days

Personal - ONGOING

Task	Details	Status
AirBnB	Saturday through Sunday next week	CHECKED IN
Housing search	San Diego vs affordable Pasadena alternative	PENDING
X1 Carbon Ubuntu installs	2 laptops, LUKS encryption	IN PROGRESS
P50 Arch to Ubuntu migration	CR-2026-03-12 - Hyprland, Firefox, Steam (blocked)	IN PROGRESS

Task

Details

Status

AirBnB

Saturday through Sunday next week

CHECKED IN

Housing search

San Diego vs affordable Pasadena alternative

PENDING

X1 Carbon Ubuntu installs

2 laptops, LUKS encryption

IN PROGRESS

P50 Arch to Ubuntu migration

CR-2026-03-12 - Hyprland, Firefox, Steam (blocked)

IN PROGRESS

Accomplishments (2026-03-16)

Task	Result	Notes
Regex CLI drills	COMPLETED	9 drill scripts + browser-viewable pages
Cross-reference improvements	COMPLETED	Added curriculum links to REF files
ISE incident prep	COMPLETED	Defense document created
P50 Ubuntu migration	IN PROGRESS	Phase 4b (Hyprland), Phase 6 (oh-my-posh), Firefox, CA Secure Browser
domus-gabriel-docs	COMPLETED	Fixed standalone build (no domus-docs dependency)
domus-nvim public	COMPLETED	Added neovim 0.11+ version check, updated README

Task

Result

Notes

Regex CLI drills

COMPLETED

9 drill scripts + browser-viewable pages

Cross-reference improvements

COMPLETED

Added curriculum links to REF files

ISE incident prep

COMPLETED

Defense document created

P50 Ubuntu migration

IN PROGRESS

Phase 4b (Hyprland), Phase 6 (oh-my-posh), Firefox, CA Secure Browser

domus-gabriel-docs

COMPLETED

Fixed standalone build (no domus-docs dependency)

domus-nvim public

COMPLETED

Added neovim 0.11+ version check, updated README

P50 Migration Session (Evening)

Continued CR-2026-03-12 P50 Arch to Ubuntu migration for Gabriel:

Completed:

Firefox snap → deb replacement (Wayland fix)
Hyprland 0.41.2 with minimal config (Ubuntu repo version)
Nerd Fonts manual installation (waybar icons)
Volume controls (pamixer + keybinds)
oh-my-posh with custom theme
CA Secure Browser (dpkg --force-depends workaround)
domus-gabriel-docs standalone build

Issue Encountered:

Steam installation entered dependency hell:

Ubuntu 25.10 32-bit library conflicts
libglx-mesa0 version mismatch (i386 vs amd64)
Cascading xcb/mesa/glibc dependencies
Attempted: dpkg --force-depends for each lib
Result: System in broken state

Resolution in progress:

Remove all broken i386 packages
Clean up casecurebrowser (nonexistent libgdk-pixbuf2.0-0 dependency)
Use Flatpak for Steam (sandboxed, no system lib conflicts)

See: Change Request

March 2026 Worklog Inventory

Date	File	Status
2026-03-01	WRKLOG-2026-03-01.adoc	✓ Created
2026-03-02	WRKLOG-2026-03-02.adoc	✓ Created
2026-03-03	WRKLOG-2026-03-03.adoc	✓ Created
2026-03-04	none	Weekend
2026-03-05	WRKLOG-2026-03-05.adoc	✓ Created
2026-03-06	WRKLOG-2026-03-06.adoc	✓ Created
2026-03-07	WRKLOG-2026-03-07.adoc	✓ Created
2026-03-08	WRKLOG-2026-03-08.adoc	✓ Created
2026-03-09	WRKLOG-2026-03-09.adoc	✓ Created
2026-03-10	none	Weekend
2026-03-11	WRKLOG-2026-03-11.adoc	✓ Created
2026-03-12	WRKLOG-2026-03-12.adoc	✓ Created
2026-03-13	WRKLOG-2026-03-13.adoc	✓ Created
2026-03-14	WRKLOG-2026-03-14.adoc	✓ Created
2026-03-15	WRKLOG-2026-03-15.adoc	✓ Created
2026-03-16	WRKLOG-2026-03-16.adoc	✓ Current

Date

File

Status

2026-03-01

WRKLOG-2026-03-01.adoc

✓ Created

2026-03-02

WRKLOG-2026-03-02.adoc

✓ Created

2026-03-03

WRKLOG-2026-03-03.adoc

✓ Created

2026-03-04

none

Weekend

2026-03-05

WRKLOG-2026-03-05.adoc

✓ Created

2026-03-06

WRKLOG-2026-03-06.adoc

✓ Created

2026-03-07

WRKLOG-2026-03-07.adoc

✓ Created

2026-03-08

WRKLOG-2026-03-08.adoc

✓ Created

2026-03-09

WRKLOG-2026-03-09.adoc

✓ Created

2026-03-10

none

Weekend

2026-03-11

WRKLOG-2026-03-11.adoc

✓ Created

2026-03-12

WRKLOG-2026-03-12.adoc

✓ Created

2026-03-13

WRKLOG-2026-03-13.adoc

✓ Created

2026-03-14

WRKLOG-2026-03-14.adoc

✓ Created

2026-03-15

WRKLOG-2026-03-15.adoc

✓ Created

2026-03-16

WRKLOG-2026-03-16.adoc

✓ Current

Total: 14 worklogs created for March 2026 (as of 03-16)

Notes

TAC case remains P2 - monitoring phase
Strongline gateway issue is high visibility - Arin + David involved
Professional backlog growing critically - need weekday focus
regex CLI drills ready for daily practice