Daily Worklog
1. Overview
Date: 2026-02-02 (Monday)
Location: Remote
Focus: Sentinel migration planning, Linux workstation, iPSK, MSCHAPv2, certifications, Monday prep
Strategic Priorities:
-
Sentinel migration from QRadar (PRIORITY #1)
-
Dr. Shahab Linux workstation completion (PRIORITY #2)
-
iPSK deployment planning (PRIORITY #3)
-
MSCHAPv2 โ EAP-TLS migration
-
Certification roadmap (CISSP, DevNet, LPIC-1)
2. Session: Security Tools Learning Roadmap Finalization
|
Planning Session: Monday, February 3, 2026 This roadmap defines Q1/Q2 2026 learning objectives for critical security platforms. Budget approval required: $5,000 |
2.1. Context
Preparing comprehensive learning roadmap for Monday, February 3, 2026 planning session. This roadmap documents security tools and platforms requiring hands-on learning and operational integration for Q1/Q2 2026.
2.2. Work Completed
2.2.1. 1. Security Tools & Platforms Roadmap Created
File: PLAN-2026-02-security-tools-learning-roadmap.adoc
Tools Documented:
|
Threat Intelligence & Analysis (4 platforms):
|
|
Extended Detection & Response (XDR):
|
|
SIEM MIGRATION (CRITICAL PRIORITY #1): Microsoft Sentinel (Target SIEM) - Microsoft Sentinel Migration from IBM QRadar SIEM (Legacy - migrating from)
This is the #1 priority for Monday - Sentinel migration is the strategic direction. IBM QRadar SIEM (Legacy - migrating from) Legacy Knowledge: * Maintain operational knowledge during transition (80-120 hours (legacy knowledge)) * Document existing rules and workflows for Sentinel migration * Support legacy system until cutover complete |
2.2.2. 2. Implementation Phases Defined
Phase 1 (30 days - February 2026):
-
QRadar foundation and access
-
AbuseIPDB/VirusTotal API integration
-
First 10 offenses investigated
Phase 2 (90 days - March-April 2026):
-
QRadar custom rules (5+ deployed)
-
URLScan.io phishing workflows
-
Defender XDR evaluation
Phase 3 (180 days - May-July 2026):
-
XDR POC deployment
-
SOAR playbooks (3+ automated responses)
-
Advanced threat hunting
2.2.3. 3. Success Metrics Established
| Metric | Current State (Jan 2026) | Target State (Q2 2026) |
|---|---|---|
Threat Intelligence Lookups/Week |
~10 (manual, ad-hoc) |
>50 (automated, integrated) |
QRadar Independent Investigations |
0 (100% vendor-dependent) |
10+ per week (fully independent) |
SIEM Custom Rules Created |
0 |
โฅ10 rules (CHLA-specific) |
Mean Time to Investigate (MTTI) |
2-4 hours |
<30 minutes |
Incident Response Time |
Baseline |
30% reduction |
2.2.4. 4. Training Budget Estimated
|
Q1/Q2 2026 Training Costs:
Total Budget: $5,000 ROI Justification:
|
2.2.5. 5. Document Properties
-
Lines: ~700 (comprehensive planning doc)
-
Status: All tools marked NOT STARTED
-
Ready for: Monday, February 3, 2026 planning session
-
Output formats: HTML, PDF, DOCX generated via build.sh
2.3. Key Priorities for Monday
|
Monday Feb 3 Strategic Priorities (in order):
These priorities align with CHLA strategic direction - cloud-native security, zero-trust architecture, professional development. |
2.3.1. Immediate Actions (Week of Feb 3-7)
|
Week 1 Deliverables - Must Complete: Sentinel Migration (PRIORITY #1): * [ ] Monday: Research Microsoft Sentinel workspace setup and pricing * [ ] Monday: Document IBM QRadar SIEM (Legacy - migrating from) โ Microsoft Sentinel (Target SIEM) migration requirements * [ ] Tuesday: KQL query language foundation (Microsoft Learn modules) * [ ] Wednesday-Thursday: Sentinel POC lab deployment (Azure trial) * [ ] Friday: Executive briefing document draft (cost-benefit, timeline, risks) Linux Workstation (PRIORITY #2): * [ ] Monday: Resolve Microsoft Defender for Endpoint connectivity issue (Dr. Shahab) * [ ] Tuesday: EAP-TLS migration testing (MAB โ certificate auth) * [ ] Wednesday: User acceptance testing with Dr. Shahab iPSK Deployment (PRIORITY #3): * [ ] Monday: iPSK architecture research (Cisco ISE 3.2 capabilities) * [ ] Tuesday: Use case documentation (IoT, BYOD, guests) These are gate items for Phase 1 success. Blockers must be escalated immediately. |
2.3.2. Sentinel Migration Requirements
|
Microsoft Sentinel Migration Planning: Current State: * SIEM: IBM QRadar SIEM (Legacy - migrating from) (legacy, on-premises) * Limitations: Vendor dependency, limited cloud integration, aging platform * Annual cost: TBD (licensing + maintenance + vendor support) Target State: * SIEM: Microsoft Sentinel (Target SIEM) (cloud-native, Azure-integrated) * Benefits: - Native integration with Microsoft Defender for Endpoint for Endpoint, Cloud, Identity - KQL query language (industry-standard, same as Defender XDR) - AI/ML threat detection (UEBA, anomaly detection) - Elastic scaling (pay-per-GB ingestion) - Reduced vendor dependency (in-house management) Phase 1 Requirements (Feb 2026): * [ ] Azure subscription with Sentinel workspace * [ ] KQL training (Microsoft Learn paths: 20-30 hours) * [ ] Migration assessment tool (QRadar โ Sentinel rule mapping) * [ ] POC environment (trial workspace, sample log ingestion) * [ ] Cost modeling (ingest GB/day, retention, analytics rules) Migration Timeline (Proposed): * Feb 2026: Research, POC, executive briefing * Mar 2026: Sentinel workspace deployment, pilot log sources * Apr-May 2026: Rule migration, dashboard recreation * Jun 2026: Parallel operation (QRadar + Sentinel) * Jul 2026: Cutover to Sentinel primary, QRadar decommission Business Justification: * Cloud-first strategy - Aligns with Microsoft 365, Azure, Defender stack * Cost reduction - Elastic pricing vs fixed licensing * Faster incident response - Native XDR integration (no API latency) * Compliance - HIPAA, HITRUST log retention and audit trails * Future-proof - Microsoft investing heavily in Sentinel (not QRadar) |
2.3.3. iPSK Deployment Planning (PRIORITY #3)
|
Identity PSK (iPSK) for CHLA: Use Cases: * IoT devices - Lab equipment, medical devices, building automation (no 802.1X support) * BYOD onboarding - Secure guest/contractor access without certificates * Legacy devices - Equipment that cannot be upgraded to 802.1X Current Status: 0% (not started) ISE 3.2 Capabilities: * Per-user or per-device PSK assignment * Dynamic VLAN assignment based on identity * Integration with Active Directory for user-based iPSK * Self-service portal for PSK retrieval * PSK rotation policies (30/60/90 day expiration) Week 1 Research Tasks: * [ ] Review Cisco ISE 3.2 iPSK configuration guide * [ ] Document CHLA use cases (IoT inventory, BYOD policy) * [ ] Design iPSK architecture (VLANs, SSIDs, authorization policies) * [ ] Identify pilot devices for testing * [ ] Estimate deployment timeline and resources |
2.3.4. MSCHAPv2 Migration Planning
|
MSCHAPv2 Vulnerability Remediation: Current Risk: MSCHAPv2 on CHLA-Corporate SSID susceptible to credential harvesting (identified in pentest) Migration Strategy: * Primary path: EAP-TLS (certificate-based, most secure) * Fallback path: EAP-TEAP (Protected EAP, for clients that don’t support EAP-TLS) Status: 0% (planning phase) Client Assessment Needed: * [ ] Inventory all wireless clients (laptops, tablets, phones) * [ ] Test EAP-TLS support (Windows 10/11, macOS, iOS, Android versions) * [ ] Identify non-EAP-TLS clients (legacy devices, IoT) * [ ] Document EAP-TEAP configuration for fallback clients Migration Phases: * Phase 1 (Feb-Mar): Client inventory and EAP-TLS compatibility testing * Phase 2 (Apr): Certificate enrollment automation (ADCS + SCEP/Intune) * Phase 3 (May): Pilot EAP-TLS with IT department (20-30 devices) * Phase 4 (Jun-Jul): Organization-wide EAP-TLS rollout * Phase 5 (Aug): Disable MSCHAPv2 on CHLA-Corporate SSID (security hardening) Critical Success Factors: * Certificate template design (machine + user certs) * Automated enrollment (Intune for managed devices, SCEP for BYOD) * User communication plan (change management) * Help desk training (EAP-TLS troubleshooting) * Rollback plan (if critical business disruption) |
2.3.5. Certification Roadmap
|
Professional Development - 2026 Goals: CISSP (Certified Information Systems Security Professional) * Value: Industry-standard security certification, CISO career path * Cost: $749 exam + $699/year membership * Prep time: 120-160 hours study * Timeline: Exam Q4 2026 (Oct-Dec) * Study plan: Official Study Guide + practice exams + Bootcamp (optional) * Experience requirement: 5 years security work (already met) Cisco Certified DevNet Associate * Value: Network automation, Python, APIs, DevOps for NetOps * Cost: $300 exam * Prep time: 80-100 hours study * Timeline: Exam Q2 2026 (Apr-Jun) * Study plan: Cisco DevNet Learning Labs + Python scripting practice * Relevance: ISE automation (netapi skills), network programmability LPIC-1 (Linux Professional Institute Certification) * Value: Linux systems administration, validates PRJ-ISE-CHLA-LINUX (PRIORITY #2 for Monday) skills * Cost: $200 per exam (2 exams required) * Prep time: 40-60 hours study * Timeline: Exams Q1 2026 (Feb-Mar) - EARLIEST TARGET * Study plan: Linux Academy + hands-on lab (home enterprise servers) * Relevance: Linux workstation deployment, RHEL/Ubuntu server management Recommended Order: 1. LPIC-1 (Linux Professional Institute Certification) (Q1 2026) - Quickest win, validates current Linux work 2. Cisco Certified DevNet Associate (Q2 2026) - Aligns with netapi development, ISE automation 3. CISSP (Certified Information Systems Security Professional) (Q4 2026) - Long-term goal, requires 120+ hours prep Total Investment: $749 + $300 + $400 + $699/year = ~$2,150 (exams) + $699/year (CISSP membership) Budget Request: Include in $5,000 for Monday planning session |
2.3.6. Tool Account Setup Checklist
-
AbuseIPDB: Create account, obtain API key (FREE tier)
-
VirusTotal: Create account, obtain API key (FREE tier: 500/day)
-
URLScan.io: Create account, obtain API key (FREE tier)
-
Cisco Talos: Subscribe to Intelligence Blog (daily reading)
-
Microsoft Sentinel: Azure trial account for POC workspace
2.4. Learning Resources Prepared
2.4.1. Sentinel Training Path (PRIORITY #1)
-
Week 1: Microsoft Learn - Sentinel fundamentals (SC-200 modules)
-
Week 2: KQL query language (Kusto Query Language) - 15-20 hours
-
Week 3: Sentinel POC deployment (Azure trial workspace)
-
Week 4: Log ingestion architecture (connectors, data collection rules)
-
Month 2-3: Analytics rules migration (QRadar โ Sentinel)
-
Month 4: Microsoft SC-200 certification (optional - Security Operations Analyst)
Key Microsoft Learn Paths: * SC-200: Microsoft Security Operations Analyst (40+ hours) * KQL for Security Analysts (10 hours) * Sentinel Workshop (hands-on labs, 8 hours)
2.4.2. QRadar Legacy Knowledge (Maintenance)
Limited scope - Understanding existing deployment during migration: 1. Week 1: Shadow CHLA QRadar admin (1-2 sessions) 2. Week 2-3: Document existing rules and dashboards (migration inventory) 3. Week 4: AQL basics (for troubleshooting during parallel operation)
Goal: Maintain operational knowledge, not become QRadar expert
2.4.3. Python Scripts to Develop
Priority automation scripts:
-
abuseipdb-bulk-lookup.py- Bulk IP reputation checks -
virustotal-hash-lookup.py- Automated malware hash analysis -
qradar-api-query.py- QRadar API automation (offense investigation) -
threat-intel-dashboard.py- Unified threat intel view (all sources)
2.5. Risks & Mitigations
| Risk | Impact | Mitigation |
|---|---|---|
Limited QRadar Access |
Cannot complete learning phases |
Escalate to CISO with business justification |
Operational Incidents Delay Training |
Learning roadmap deprioritized |
Block dedicated learning time (Fridays 1-3 PM) |
Budget Constraints |
Cannot evaluate paid tools |
Maximize free tiers first, build ROI case |
QRadar Admin Unavailable |
Limited mentorship |
IBM training resources, user forums, Udemy |
2.6. Week of Feb 3-7: Consolidated Task List
2.6.1. CRITICAL Priority (Must Complete This Week)
1. Sentinel Migration (PRIORITY #1): * [ ] Monday: Research Microsoft Sentinel workspace setup and pricing model * [ ] Monday: Create Azure trial account for Sentinel POC * [ ] Monday: Document QRadar โ Sentinel migration requirements and timeline * [ ] Tuesday: Microsoft Learn - Sentinel fundamentals (SC-200 Module 1-2) * [ ] Wednesday: KQL query language foundation (4-6 hours) * [ ] Thursday-Friday: Sentinel POC workspace deployment and first log connectors
2. Dr. Shahab Linux Workstation (PRIORITY #2): * [ ] Monday: Resolve Microsoft Defender for Endpoint connectivity issue - Contact IT/Desktop Support - Validate agent installation and health - Test posture policy enforcement - Document troubleshooting in runbook * [ ] Tuesday: EAP-TLS migration testing (MAB โ certificate auth) * [ ] Wednesday: User acceptance testing with Dr. Shahab * [ ] Thursday: Knowledge transfer documentation for Desktop Support
3. iPSK Deployment Planning (PRIORITY #3): * [ ] Monday: Review Cisco ISE 3.2 iPSK configuration guide * [ ] Tuesday: Document CHLA use cases (IoT, BYOD, guests) * [ ] Wednesday: Design iPSK architecture (VLANs, SSIDs, authz policies)
2.6.2. HIGH Priority (Target This Week)
Certification Planning: * [ ] Register for LPIC-1 exam (target: late February) * [ ] Order LPIC-1 study materials (Linux Academy subscription) * [ ] Create certification study schedule (LPIC-1 โ DevNet โ CISSP)
MSCHAPv2 Migration: * [ ] Inventory wireless client devices (IT asset management database) * [ ] Research EAP-TLS support matrix (Windows/macOS/iOS/Android versions) * [ ] Document EAP-TEAP fallback configuration for legacy clients
Security Tools Setup: * [ ] Create AbuseIPDB account and obtain API key * [ ] Create VirusTotal account and obtain API key (FREE tier: 500/day) * [ ] Create URLScan.io account and obtain API key * [ ] Subscribe to Cisco Talos Intelligence Blog (daily reading)
2.6.3. MEDIUM Priority (If Time Permits)
-
Shadow CHLA QRadar admin (1-2 hour session)
-
Document existing QRadar rules for migration inventory
-
Update HOME runbook with netapi validation methods
-
Python script: abuseipdb-bulk-lookup.py (skeleton)
-
Python script: virustotal-hash-lookup.py (skeleton)
-
Python script: sentinel-kql-query.py (skeleton)
2.6.4. DEFERRED (Post-Feb 7)
-
Review pentest formal report (due Feb 7)
-
Prioritize pentest remediation roadmap
-
Create change requests for critical findings
-
Posture redirect ACL remediation planning
2.7. Next Steps
2.7.1. Before Monday Planning Session
-
โ Security tools roadmap document completed
-
โ Implementation phases defined
-
โ Success metrics established
-
โ Training budget updated ($$5,000)
-
โ Sentinel migration priorities documented
-
โ iPSK, MSCHAPv2, certification roadmaps added
-
PENDING Prepare Sentinel migration executive briefing
-
PENDING Research Azure Sentinel pricing calculator
2.7.2. Monday Session Agenda (Feb 3)
Duration: 90 minutes
-
Strategic Priorities Review (30 min)
-
PRIORITY #1: Sentinel migration from QRadar (cloud-first strategy)
-
PRIORITY #2: Dr. Shahab Linux workstation completion
-
PRIORITY #3: iPSK deployment planning (IoT/BYOD)
-
MSCHAPv2 โ EAP-TLS migration (pentest remediation)
-
Certification roadmap (LPIC-1, DevNet, CISSP)
-
-
Budget Approval (20 min)
-
Training & certifications: $5,000
-
Sentinel POC: Azure trial (FREE for 30 days)
-
Tool accounts: FREE tiers (AbuseIPDB, VT, URLScan, Talos)
-
LPIC-1 exam: $200 per exam (2 exams required)
-
Cisco DevNet: $300 exam
-
CISSP: $749 exam + $699/year membership
-
-
Resource Allocation (20 min)
-
Dedicated learning time: Fridays 1-4 PM (12 hours/week)
-
Sentinel POC: Azure subscription approval
-
QRadar access: Shadow admin for migration inventory
-
Dr. Shahab: Defender connectivity troubleshooting with IT
-
-
Week 1 Deliverables Agreement (20 min)
-
Sentinel POC workspace deployed
-
KQL fundamentals complete (Microsoft Learn)
-
Dr. Shahab workstation completion (Defender issue resolved)
-
iPSK architecture document
-
Tool accounts created (4 platforms)
-
2.7.3. Week 1 Deliverables (Feb 3-7)
Sentinel Migration (PRIORITY #1): * [ ] Azure trial account created with Sentinel workspace * [ ] Microsoft Learn SC-200 Module 1-2 complete (Sentinel fundamentals) * [ ] KQL query language foundation (10+ queries practiced) * [ ] QRadar migration inventory documented (rules, dashboards, log sources) * [ ] Executive briefing draft (cost-benefit, timeline, risks)
Dr. Shahab Deployment (PRIORITY #2): * [ ] Defender connectivity issue resolved * [ ] EAP-TLS migration testing complete * [ ] User acceptance testing signed off
iPSK Planning (PRIORITY #3): * [ ] ISE 3.2 iPSK configuration guide reviewed * [ ] CHLA use cases documented (IoT, BYOD, guests) * [ ] iPSK architecture designed (VLANs, SSIDs, policies)
Supporting Tasks: * [ ] 4 tool accounts created (AbuseIPDB, VT, URLScan, Talos, Sentinel) * [ ] LPIC-1 exam registered (target: late February) * [ ] MSCHAPv2 client inventory initiated
3. Notes
3.1. Document Status
-
PLAN-2026-02-security-tools-learning-roadmap.adoc: COMPLETE
-
Location:
03_Captures/2026/02/ -
Follows PREFIX standard (
PLAN-for planning/roadmap) -
Output formats generated (HTML, PDF, DOCX)
-
Ready for Monday presentation
-
3.2. January Carryover Items
3.2.1. Active Tasks from WRKX-2026-01-30
From "Next Steps" section (Jan 30):
-
~~Complete Pat Levitt authentication investigation~~ - RESOLVED (IP misconfiguration)
-
โ ~~Document findings and resolution~~ - COMPLETE (documented in Jan 30 capture)
-
Update HOME runbook with netapi validation methods - PENDING
-
Deferred until operational work complete
-
HOME runbook uses netapi (personal lab)
-
CHLA runbook uses ISE GUI (shareable)
-
-
Verify Dr. Shahab Asgharzadeh’s workstation Microsoft Defender for Endpoint connection status - BLOCKED
|
CRITICAL BLOCKER: Issue: Microsoft Defender for Endpoint reported "not connected" on Dr. Shahab Asgharzadeh’s workstation (MAC: b4:e9:b8:f6:c8:17) Impact: * Posture compliance cannot be validated * Deployment stuck at 90% complete * Cannot migrate from MAB to EAP-TLS until resolved Next Action: Contact IT/Desktop Support to troubleshoot agent connectivity Required for: User acceptance testing, production cutover |
3.2.2. Ongoing Projects (January โ February)
Dr. Shahab Linux Workstation Deployment: * Status: IN PROGRESS (90% complete) * Completed: - MAB onboarding to Research_Onboard profile - LUKS full disk encryption - AD domain join (chla.usc.edu) - Machine certificate enrollment - 802.1X configuration (wpa_supplicant) * Pending: - [ ] Defender connectivity issue (CRITICAL - blocking posture validation) - [ ] EAP-TLS migration (MAB โ certificate-based auth) - [ ] User acceptance testing (Dr. Shahab sign-off) - [ ] Knowledge transfer to Desktop Support
Pentest Findings Remediation: * Status: AWAITING REPORT
|
Timeline: Formal report expected February 7, 2026 Known Critical Findings: * โ ๏ธ Posture redirect ACL too permissive (Kerberos 88, SMB 445) * โ ๏ธ MSCHAPv2 on CHLA-Corporate SSID (credential harvesting risk) Next Actions (Post-February 7, 2026): * [ ] Review formal pentest report * [ ] Prioritize remediation roadmap with CISO * [ ] Create change requests for critical findings * [ ] Estimate remediation timeline and resources |
CHLA Runbook Maintenance: * Status: COMPLETE (Rev 3.0) * Achievements: - 53 AsciiDoc attributes for CI/CD sustainability - All netapi references removed (100% shareable with team) - Professional ISE GUI workflows documented * Next evolution: - Consider Antora site for team-wide access - Add screenshots for complex ISE workflows
3.3. Tools Already Available
No account needed:
-
QRadar (CHLA enterprise) - need access credentials only
-
Microsoft Defender for Endpoint - already deployed
-
ISE Live Logs - already have access
-
NetFlow (QRadar) - via SIEM access
Free tier accounts to create:
-
AbuseIPDB - unlimited lookups (rate-limited)
-
VirusTotal - 500 API requests/day
-
URLScan.io - 5,000 scans/month
-
Cisco Talos - blog subscription only
4. Session: Docs-as-Code Strategy Discussion with Derek Pizzagoni
4.1. Context
Phone call with Derek Pizzagoni regarding docs-as-code methodology and potential collaboration on security documentation projects. Demonstrated the documentation engineering system and discussed how it can be applied to InfoSec workflows.
4.2. Discussion Topics
4.2.1. 1. Documentation Engineering Overview
Covered the core docs-as-code paradigm:
-
Version-controlled documentation using Git
-
AsciiDoc/Antora toolchain for multi-format output
-
Diagrams-as-code (D2, Mermaid)
-
Single source of truth with attributes
-
Modular composition via includes
-
Encryption for sensitive content (age)
4.2.2. 2. Industry Adoption Evidence
Shared examples of organizations using docs-as-code:
-
Google, GitLab, Stripe, Microsoft, Spotify, AWS
-
Security-focused implementations: HashiCorp, Cisco DevNet, NIST OSCAL, CIS Benchmarks, MITRE ATT&CK
4.2.3. 3. Collaboration Framework
Discussed potential collaboration models:
-
Shared private repository - Joint write access, PR-based workflow
-
Fork and contribute - Canonical repo with PR contributions
-
Template repository - Starter templates for customization
4.2.4. 4. Terminal-Based Email (aerc)
Derek expressed interest in the terminal email workflow using aerc. Key points discussed:
-
aerc - Terminal email client (aerc-mail.org)
-
OAuth2 authentication for Gmail/Microsoft 365
-
Custom scripts for address book integration
-
Vim-like keybindings for efficient email handling
-
Integrates with the docs-as-code workflow (compose in terminal, version control drafts)
Project reference: PRJ-AERC (terminal email configuration)
4.2.5. 5. Next Steps
-
Created comprehensive collaboration framework document
-
Will send Derek the PDF/HTML output for review
-
Potential to set up shared Git repository for joint documentation projects
-
Share aerc configuration resources if interested
4.3. Deliverables Created
File: DOC-2026-02-02-docs-as-code-collaboration.adoc
Outputs generated:
-
output/DOC-2026-02-02-docs-as-code-collaboration.html(56K) -
output/DOC-2026-02-02-docs-as-code-collaboration.pdf(175K) -
output/DOC-2026-02-02-docs-as-code-collaboration.docx(18K)
Purpose: Reference document for Derek explaining the docs-as-code methodology and collaboration framework for potential joint security documentation projects.
4.4. Key Value Propositions for Collaboration
| Benefit | Application |
|---|---|
Auditability |
Complete Git history of all documentation changes |
Peer Review |
PR-based workflow for technical accuracy |
Consistency |
Templates enforce standards across projects |
Automation |
CI/CD builds and publishes on commit |
Encryption |
Sensitive procedures protected at rest |
Portability |
Single source, multiple output formats (HTML, PDF, DOCX) |
5. Incident: ISE-01 Power-On Caused Network-Wide Outage
5.1. Timeline
| Time | Event |
|---|---|
~20:15 |
SSH to KVM host (supermicro300-9d1) at 192.168.1.185 via certmgr-01 jump host (10.50.1.60). Direct SSH from workstation blocked by legacy SSH algorithms on KVM host (ssh-rsa/ssh-dss only, no ed25519 host keys). |
~20:20 |
Enabled VM autostart for all critical VMs: ise-02, pfSense-FW01, home-dc01, 9800-CL-WLC, certmgr-01, ipsk-manager, keycloak-01. |
~20:22 |
Started ise-01 ( |
~20:23 |
NETWORK-WIDE OUTAGE - All 802.1X authentication failed. Wired and wireless connections dropped across all VLANs. |
~20:24 |
Force shutdown ise-01 ( |
~20:28 |
All VMs back online via autostart. Wired 802.1X (enp130s0) reconnected automatically. |
~20:34 |
WiFi (wlan0) required manual reconnection: |
5.2. Root Cause Analysis
ISE-01 has a stale configuration from before the PKI migration (HOME-ROOT-CA → DOMUS-ROOT-CA). When ISE-01 booted:
-
Switches and WLC have both ISE-01 (10.50.1.20) and ISE-02 (10.50.1.21) configured as RADIUS servers
-
ISE-01 started responding to RADIUS requests with old certificates and policies
-
Client authentication failed against ISE-01’s stale config
-
All devices on the network experienced authentication failures simultaneously
5.3. ISE Deployment State
| Node | IP | Role | Status |
|---|---|---|---|
ISE-02 |
10.50.1.21 |
Primary Admin Node (PAN) - ADMINISTRATOR |
Running - all services healthy |
ISE-01 |
10.50.1.20 |
Unknown role (stale config) |
Shut off - must not be started without isolation |
5.4. KVM Host State (supermicro300-9d1)
| Resource | Value |
|---|---|
Total RAM |
125 GB |
Used |
50 GB |
Available |
74 GB |
ISE-01 allocation |
16 GB RAM, 4 vCPUs |
ISE-02 allocation |
16 GB RAM, 4 vCPUs |
Running VMs |
7 (pfSense, 9800-WLC, ise-02, certmgr-01, home-dc01, ipsk-manager, keycloak-01) |
5.5. Required Actions Before Starting ISE-01 Again
|
DO NOT start ISE-01 until ALL of the following are completed: |
-
Remove ISE-01 from switch RADIUS config - Remove 10.50.1.20 from all
radius serverstatements on 3560CX-01 -
Remove ISE-01 from WLC RADIUS config - Remove 10.50.1.20 from 9800 WLC RADIUS server list
-
Check pfSense - Verify no RADIUS references to 10.50.1.20
-
Deregister ISE-01 from ISE-02 deployment - Administration → System → Deployment, remove ise-01 node
-
Optionally: Start ISE-01 with network disconnected -
virsh domiflist ise-01then detach NIC before starting -
Restore ISE-02 backup to ISE-01 once isolated
-
Update ISE-01 certs to DOMUS-ROOT-CA chain before reconnecting to network
5.6. WiFi Reconnection
WiFi profile Domus-Secure-802.1X did not auto-reconnect after the outage on any device. Required manual intervention on each.
5.6.1. Workstation (modestus-razer)
nmcli connection up Domus-Secure-802.1X
wlan0: state change: config -> ip-config
dhcp4 (wlan0): new lease, address=10.50.10.103
device (wlan0): Activation: successful, device activated.
5.6.2. Son’s Workstation (P50 - Arch Linux)
Same issue - WiFi 802.1X did not auto-reconnect after the outage. Bringing the interface up manually resolved it.
nmcli connection up Domus-Secure-802.1X
5.6.3. Mobile Devices
Android - Samsung Z Fold 7 (SM_F966U1)
Problem: After ISE-01 outage, phone would not reconnect to Domus-Secure-802.1X. Certificates had been sideloaded previously (DOMUS-ROOT-CA, DOMUS-ISSUING-CA, zfold7-evanusmodestus.p12) but Android installs CA certs to the user trust store only — WiFi enterprise does not trust user-store CAs for server certificate validation.
ISE Auth Log (before fix):
12520 EAP-TLS failed SSL/TLS handshake because the client rejected the ISE local-certificate
Workaround Applied: Connected using "Do not validate" for CA certificate in the WiFi profile settings. This bypasses server certificate validation, allowing the EAP-TLS handshake to complete with the client certificate only.
netapi ise mnt session 9c:83:06:ce:89:46
Session: 9c:83:06:ce:89:46 | Status: FAILED
Username: zfold7-evanusmodestus.byod.inside.domusdigitalis.dev
ISE Node: ise-02
Client IPv4: 10.50.10.100
NAD IP: 10.50.1.40
Port: capwap_90000005
Profiled As: Samsung-Device
"Do not validate" is a temporary workaround. This disables ISE server certificate verification, making the device vulnerable to rogue RADIUS attacks. The proper fix is to provision the device via the ISE BYOD portal flow so the CA chain is installed correctly for WiFi enterprise use. See: byod-certificate-management runbook in PRJ-INFRA-OPS-ANT.
|
Root Cause: Android (non-rooted) cannot install CA certificates to the system trust store via sideloading. Only MDM or the ISE BYOD portal enrollment flow can place CAs in the system store where WiFi enterprise trusts them.
Proper Fix (TODO):
-
Set up ISE BYOD portal provisioning flow
-
Device connects to provisioning SSID
-
ISE enrolls device cert via Vault PKI (SCEP) with full CA chain
-
WiFi profile pushed with proper CA trust anchor
-
Remove "Do not validate" workaround
Validation Test Plan (Home Enterprise):
Verify that the "Do not validate" workaround does not weaken client-side authentication, and that only the server certificate verification is bypassed:
-
Confirm client cert is still presented during EAP-TLS handshake (check ISE Live Logs for cert CN match)
-
Verify dACL and VLAN assignment matches expected policy for
zfold7-evanusmodestus -
Test connectivity scope — confirm device is restricted to the BYOD segment, not flat network access
-
Review ISE auth detail report: confirm
EAP-TLSmethod (not PEAP or open), cert issuer = DOMUS-ISSUING-CA -
Simulate rogue AP test: set up a second SSID with same name on a spare AP, present a self-signed RADIUS cert, verify phone connects blindly (confirms the risk is real)
-
After BYOD portal migration: repeat rogue AP test, verify phone rejects the fake RADIUS cert
-
Document results in PRJ-INFRA-OPS-ANT security validation runbook
-
Android - Connected via "Do not validate" workaround
-
Android - Validate "Do not validate" security posture (test plan above)
-
Android - Migrate to BYOD portal flow (proper CA trust)
-
iPad - Released from rejected list, reconnected to Domus BYOD SSID (EAP-TLS)
iPad (44:1B:88:75:CF:74)
Problem: iPad rejected by ISE anti-RADIUS-spray protection after repeated EAP-TLS handshake failures during ISE-01 outage. Same 12520 EAP-TLS failed SSL/TLS handshake error as Android. iPad has mobileconfig profile (com.domusdigitalis.byod) with cert identity ipad-evanusmodestus.p12 targeting Domus BYOD SSID.
Discovery: iPad was last successfully authenticated on 2026-02-01 02:25 (Corp WIFI policy set, Domus_Secure_Prof). After ISE-01 outage, repeated failures triggered RADIUS spray protection.
netapi ise get-rejected-endpoints
โ Found 2 rejected endpoint(s)
Rejected Endpoints
โโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโ
โ MAC Address โ Reason โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ 44:1B:88:75:CF:74 โ EndPoint โ
โ 3C:EC:EF:43:4D:49 โ EndPoint โ
โโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโ
Fix: Release from rejection, then reconnect:
netapi ise release-rejected 44:1B:88:75:CF:74
MAC address must be uppercase for the ISE ERS release API. The SDK passes the MAC directly as the UNSET path parameter to PUT /ers/config/endpoint/UNSET/releaserejectedendpoint. Lowercase MAC returns "not on rejected endpoints list" even though the endpoint is rejected.
|
-
Release iPad from rejected endpoint list
-
Reconnect iPad to Domus BYOD SSID
-
Verify session
netapi Enhancements (2 bugs fixed):
-
get-rejected-endpoints: Fixed to display MAC addresses and rejection reasons in a table. Previously only showed count without endpoint details. ISE ERS API returns MAC addresses inOperationResult.resultValuefields alongside the count. -
release-rejected: Fixed MAC case sensitivity bug. The ISE ERS API requires uppercase MAC addresses in the URL path forreleaserejectedendpoint. Addedmac.upper()normalization so lowercase input works correctly. Previous code attempted to look up endpoint UUID which was wrong — the rejection API uses MAC directly, not the ERS endpoint UUID.
5.7. KVM Host SSH Access Issue
KVM host (supermicro300-9d1) only offers legacy SSH algorithms:
-
Host keys:
ssh-rsa,ssh-dssonly (no ed25519) -
Ciphers:
aes128-ctr,aes256-ctronly -
MACs:
hmac-sha1-96,hmac-sha1,hmac-sha2-256,hmac-sha2-512
ssh ansible@10.50.1.60 # certmgr-01
ssh evanusmodestus@10.50.1.99 # KVM host from management VLAN
# On KVM host - generate modern host keys
sudo ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""
sudo systemctl restart sshd
5.8. Lessons Learned
-
Never start a decommissioned ISE node without isolating it from RADIUS first
-
VM autostart saved the recovery - all services came back without manual intervention
-
WiFi does not auto-reconnect after network-wide outage; wired does
-
KVM host needs SSH modernization - legacy algorithms block direct access from hardened workstations
6. Tags
infosec sentinel-migration linux-workstation ipsk mschapv2-eaptls certifications cissp lpic1 cisco-devnet learning-roadmap qradar threat-intelligence xdr siem planning monday-prep strategic-priorities kql docs-as-code collaboration antora asciidoc incident ise-outage kvm autostart android byod eap-tls ca-trust wifi-recovery
7. Document Revision History
| Version | Date | Changes |
|---|---|---|
1.0 |
2026-02-02 |
Initial daily worklog - Security tools roadmap planning for Monday Feb 3 session |
2.0 |
2026-02-02 |
MAJOR UPDATE - Added strategic priorities: Sentinel migration (#1), Dr. Shahab Linux workstation (#2), iPSK deployment (#3), MSCHAPv2 โ EAP-TLS migration, Certification roadmap (CISSP, DevNet, LPIC-1). Expanded attributes (60+), updated Monday agenda, revised training budget to $5,000 |
3.0 |
2026-02-02 |
Added Session: Docs-as-Code Strategy Discussion with Derek Pizzagoni. Created collaboration framework document (DOC-2026-02-02-docs-as-code-collaboration.adoc) for potential joint security documentation projects. Updated tags. |
4.0 |
2026-02-02 |
INCIDENT: ISE-01 power-on caused network-wide 802.1X outage. Documented timeline, root cause (stale RADIUS config), KVM host state (125GB RAM, 7 VMs), WiFi reconnection, KVM SSH legacy algorithm issue, and required isolation checklist before next ISE-01 attempt. Enabled VM autostart for all critical VMs. |
5.0 |
2026-02-02 |
Android Z Fold 7 recovery: documented CA trust store limitation (user vs system store), "Do not validate" workaround applied, ISE session verified. Identified proper fix path via ISE BYOD portal enrollment flow. iPhone recovery pending. |