Daily Worklog - CHLA InfoSec Operations

1. Overview

Date: 2026-01-30

Location: CHLA InfoSec

Focus: Runbook maintenance, operational incident response

2. Morning Session: CHLA Linux Runbook Refactoring

2.1. Context

Continued work on Dr. Shahab’s Linux workstation deployment runbook for CHLA. Key requirement: remove all proprietary netapi tool references (personal tool) and replace with shareable ISE GUI instructions.

2.2. Work Completed

2.2.1. 1. Attribute Maximization (53 Total Attributes)

Centralized all configuration values as AsciiDoc attributes for CI/CD sustainability:

Infrastructure (17):

  • DNS servers: dns-primary, dns-secondary, dns-backup

  • AD Domain Controllers: ad-dc-1, ad-dc-2, ad-pdc, ad-dc-3

  • ISE Cluster: ise-ppan, ise-ppan-host, ise-span, ise-span-host, ise-psn-1 through ise-psn-4

  • Research: nas-ip

Device-Specific (10):

  • Device: device-hostname, device-short, device-mac, device-interface, device-ip, device-location

  • Network: switch-name, switch-port

  • Certs: machine-cert, machine-key

ISE Policy (18):

  • Policy Sets: policy-set-wired, policy-set-mab

  • Authorization Profiles: authz-profile-onboard, authz-profile-pending, authz-profile-discovery, authz-profile-full, authz-profile-quarantine

  • Downloadable ACLs: dacl-onboard, dacl-discovery, dacl-compliant, dacl-quarantine

  • Endpoint Groups: endpoint-group-parent, endpoint-group-research, endpoint-group-onboarding

  • VLANs: vlan-research (40), vlan-research-name (CHLA-IoT), vlan-quarantine (999), vlan-quarantine-name (Critical Auth Fallback)

Users (4):

  • Dr. Shahab: user-shahab, user-shahab-short

  • Xiangming Ding: user-ding, user-ding-short

Filesystem (4):

  • cert-dir, key-dir, sssd-conf, crypttab

2.2.2. 2. Netapi Removal - Final Cleanup

Replaced last 34 netapi references in CRITICAL FINDINGS section with ISE GUI instructions:

Before (netapi):

netapi ise get-endpoint "b4:e9:b8:f6:c8:17"
netapi ise mnt coa "b4:e9:b8:f6:c8:17"
netapi ise mnt session "b4:e9:b8:f6:c8:17"

After (ISE GUI):

  • Navigate to: Operations → RADIUS → Live Sessions

  • Filter by MAC, click Disconnect icon (⊗)

  • Verify in Live Logs: Status, AuthZ Profile, VLAN

Result:

  • ✓ All 70 netapi references removed

  • ✓ 100% shareable with CHLA InfoSec team

  • ✓ Professional ISE GUI workflow documentation

2.2.3. 3. Key Replacements

  • All dACL IP addresses now use attributes (DNS, AD, ISE PSNs, NAS)

  • All ISE policy names use attributes

  • All VLAN references use attributes

  • Added ISE SPAN attributes: span.ise.chla.org (10.101.2.122), span.ise.chla.org (span.ise.chla.org)

  • Corrected user info: Xiangming Ding, Senior Bioinformatics Scientist

2.2.4. 4. Build Status

Format Status

HTML

159K ✓

PDF

779K ✓

Gitleaks

PASS ✓

Commit

8e390a7a

2.3. Outcome

Runbook is now:

  • ✓ Attribute-driven (53 centralized config values)

  • ✓ Netapi-free (shareable with team)

  • ✓ CI/CD ready (infrastructure changes = attribute updates only)

  • ✓ Professional (ISE GUI workflows, not proprietary CLI tools)

3. Afternoon Session: Operational Incident - Pat Levitt Authentication Failure

3.1. Alert Received

Time

~3:15 PM

Source

Microsoft Teams

Issue

User authentication failure

3.2. User Details

Name

Levitt, Pat

Title

USC Faculty/Non Physician CWR Neurology CWR

Email

plevit@chla.usc.edu

Status

Authentication issues reported

3.3. Initial Response Plan

3.3.1. Phase 1: Gather Information

  1. Confirm exact failure symptoms (can’t login at all? specific error?)

  2. Determine authentication method (802.1X? VPN? Windows login?)

  3. Check if issue is new or recurring

3.3.2. Phase 2: ISE Diagnostics (netapi)

# Load credentials
dsource d001 dev/network
export ISE_PAN_IP="{ise-ppan-ip}"

# Check for active sessions
netapi ise mnt session <MAC_ADDRESS>

# Review authentication history
netapi ise dc auth-history <MAC_ADDRESS> --limit 10

# Check for failed attempts
netapi ise dc failed --limit 20 | grep -i levitt
netapi ise dc failed --limit 20 | grep -i plevit

# Check user authentication status
netapi ise mnt auth-status <MAC_ADDRESS>

3.3.3. Phase 3: Common Failure Checks

  • AD account status (locked? disabled? password expired?)

  • Certificate validity (if using EAP-TLS)

  • Endpoint group assignment (correct policy applied?)

  • Authorization rule matching (hitting correct rule?)

  • ISE Live Logs review (detailed failure reason)

3.3.4. Phase 4: Resolution Steps

TBD based on findings

3.4. Investigation Log

Root Cause: IP address misconfiguration on user’s workstation.

Resolution: Corrected IP configuration (DHCP/static IP settings).

Time to Resolution: ~30 minutes

Validation: User confirmed network access restored.

3.5. Status

✓ RESOLVED - IP address misconfiguration corrected

4. Notes

4.1. Runbook Work (Backfill from Previous Days)

  • 2026-01-28: Added Cisco Secure Posture Client validation section

  • 2026-01-28: Added Cisco Secure Umbrella Client validation section

  • 2026-01-29: Replaced Phase 1.5 netapi validation with ISE GUI validation

  • 2026-01-29: Replaced Appendix D netapi commands with ISE GUI configuration guide

  • 2026-01-30: Final netapi cleanup in CRITICAL FINDINGS section

  • 2026-01-30: Maximized attributes (53 total)

4.2. Key Learnings

  1. Attribute-driven documentation is critical for CI/CD - centralized config management

  2. Separate work vs personal tools - netapi stays in HOME runbook, CHLA uses GUI

  3. Operational incidents always take priority - runbook work paused for Pat Levitt issue

4.3. Next Steps

  • Complete Pat Levitt authentication investigation

  • Document findings and resolution in this capture

  • Update HOME runbook with netapi validation methods (when operational work complete)

  • Verify Dr. Shahab’s workstation Defender connection status (reported not connected)

5. Tags

chla infosec ise 802.1x incident-response runbook netapi operational

6. Document Revision History

Version Date Changes

1.0

2026-01-30

Initial capture - runbook refactoring + Pat Levitt incident response