Daily Worklog

1. Overview

Date: 2026-02-06 (Friday)

Location: Remote

Focus: chlxsbg SSSD/AD login fix, directory services validation, iPSK GWN deployment validation, runbook attribute cleanup

Strategic Priorities:

  1. chlxsbg AD user login — SSSD access_provider fix for research handover

  2. Directory services validation script and documentation

  3. Runbook attribute cleanup (chla.usc.edula.ad.chla.org)

  4. iPSK GWN deployment traffic validation

2. Session: chlxsbg SSSD / AD Login Fix

2.1. Context

Dr. Shahab’s Linux workstation (chlxsbg) needs to allow AD user login for xding@la.ad.chla.org (the researcher who will use the machine). Sarah (CISO) requires AD accounts for accountability — no local admin usage.

2.2. Problem

SSSD configured with access_provider = ad which checks AD GPO for login permissions. CHLA’s large AD causes this check to timeout:

pam_acct_mgmt: Authentication service cannot retrieve authentication info

User lookup succeeds via sssctl, but login fails.

2.3. Solution

Change access_provider = ad to access_provider = simple with explicit user allowlist:

# /etc/sssd/sssd.conf under [domain/LA.AD.CHLA.ORG]
access_provider = simple
simple_allow_users = xding@la.ad.chla.org, sasgharzadeh@la.ad.chla.org

Then restart SSSD:

sudo systemctl restart sssd && sudo sss_cache -E

2.4. Validation

Created comprehensive directory services validation script. See: PRJ-ISE-CHLA-LINUX-ANTORA/runbooks/notes/validating-directory-services-20260206.adoc

Key findings:

Component Status Notes

Realm membership

PASS

kerberos-member of LA.AD.CHLA.ORG

Kerberos config

PASS

AES256/AES128, DNS KDC lookup

Machine keytab

PASS

CHLXSBG$@LA.AD.CHLA.ORG (KVNO 3)

SSSD config

PASS

AD provider, NSS/PAM services

NSS switch

PASS

passwd/group/shadow β†’ sss

PAM integration

PASS

pam_sss in common-auth/session

AD user lookup

PASS

sssctl resolves xding@la.ad.chla.org

AD user login

FAIL

access_provider = ad timeout

Group enumeration

FAIL

Hangs (CHLA AD scale issue)

2.5. Reference

UNIX/Linux Handbook notes on directory services requirements:

  1. realmd installed and domain joined

  2. SSSD configured with AD identity/auth stores

  3. nsswitch.conf using sss for passwd/group/shadow

  4. PAM configured for sss authentication

  5. Machine keytab with TGT for ongoing auth

All requirements met on chlxsbg. Issue is purely access_provider = ad vs simple.

3. Session: Runbook Attribute Cleanup

3.1. Context

Found chla.usc.edu domain in user attributes instead of correct la.ad.chla.org.

3.2. Changes

File Change

_partials/attributes.adoc

Fixed user-shahab and user-ding domains

asciidoc-reference.adoc

Fixed example domain

_partials/validation-summary.adoc

Fixed 4 hardcoded values (domain, cert template, CA files, machine cert)

team-views/cloud-ad-tasks.adoc

Updated version v3.7 β†’ v3.9

team-views/infosec-tasks.adoc

Updated version v3.8 β†’ v3.9

Rebuilt team-views HTML with corrected attributes.

4. Session: iPSK GWN Deployment Validation

4.1. Context

Validating iPSK authentication for GetWell iPads. Checking ISE live logs for client auth success.

4.2. MAC Address 1: 34:F6:8D:0E:76:E7

Result: Authentication succeeded

Field Value

Event

5200 Authentication succeeded

Username

34:F6:8D:0E:76:E7

Authentication Policy

CHLA IoT >> MAB

Authorization Policy

CHLA IoT >> GetWell iPAD iPSK only

Authorization Result

iPSK-MGR-AuthZ

Identity Store

Internal Endpoints

External Groups

GetWell_iPads

VLAN

1630

PSK

Applied (masked)

Network Device

USC-OEAP-WLC3

Timestamp

2026-01-30 11:47:26.559

Key steps from ISE log:

  1. 11001 - Received RADIUS Access-Request

  2. 24211 - Found Endpoint in Internal Endpoints IDStore

  3. 22037 - Authentication Passed

  4. 24870 - Fetching user groups from external ODBC succeeded

  5. 15016 - Selected Authorization Profile - iPSK-MGR-AuthZ

  6. 11002 - Returned RADIUS Access-Accept

4.3. MAC Address 2: 34:f6:8d:0f:e7:f5

Result: Wrong MAC provided initially

Original MAC given was 34:F6:8D:0E:F7:F5 (incorrect). Correct MAC is 34:f6:8d:0f:e7:f5.

iPad MAC Address IP

iPad #1

34:f6:8d:0e:76:e7

10.218.21.24

iPad #2

34:f6:8d:0f:e7:f5

10.218.21.27

Verify iPad #2 is registered in ISE under GetWell_iPads identity group.

4.4. netapi Commands for ISE Lookup

To query ISE authentication logs via CLI instead of GUI:

4.4.1. MnT API (Live Sessions & Auth Logs)

# Get authentication logs for a MAC address
netapi ise mnt auth-logs 34:F6:8D:0E:76:E7

# Get active session details by MAC
netapi ise mnt session 34:F6:8D:0E:76:E7

# Get authentication status
netapi ise mnt auth-status 34:F6:8D:0E:76:E7

# List all active RADIUS sessions
netapi ise mnt sessions

# Get failed authentication attempts
netapi ise mnt failed

# List all failure reason codes
netapi ise mnt failure-reasons

4.4.2. DataConnect (Database Analytics)

# Get recent authentications (live RADIUS log)
netapi ise dc recent

# Get comprehensive session view for a MAC
netapi ise dc session 34:F6:8D:0E:76:E7

# Get authentication history timeline for a MAC
netapi ise dc auth-history 34:F6:8D:0E:76:E7

# Get failed authentications
netapi ise dc failed

# Get endpoint details
netapi ise dc endpoint 34:F6:8D:0E:76:E7

# List endpoints
netapi ise dc endpoints

# Get top failure reasons with counts
netapi ise dc top-failures

# Get authentication statistics
netapi ise dc stats

# Get authentication method distribution
netapi ise dc auth-methods

# Get hourly authentication trends
netapi ise dc trends

# Get NAS (switch/WLC) health by auth success rate
netapi ise dc nas

# Find stale endpoints not seen in N days
netapi ise dc stale --days 30
MnT logs are retained for ~7 days by default. DataConnect queries the ISE database directly for historical data.

5. Session: Strongline Gateway VLAN Issue

5.1. Problem

9 Strongline gateways are hitting a "misc" IoT subnet instead of the correct VLAN. Root cause: endpoints are assigned to an identity group with "misc" in the name, causing incorrect authorization profile match.

5.2. Identity Groups

Group Name ID

IoT_iPSK_VLAN1610_Strongline

6b8bc440-7d21-11f0-bf95-82b6868ac19f

IoT_iPSK_VLAN1620_Misc

c4dfd4d0-ea3e-11ef-9e89-46637f695c46

ise_medigate_Beacons_Receiver_Strongline_Staff_Safety_System

75cf6930-9bf6-11ec-bf4f-da147ea7a803

5.3. netapi Commands

# List endpoints in wrong group (use -f json to avoid column truncation)
netapi ise get-endpoints --group "IoT_iPSK_VLAN1620_Misc" --all -f json | jq -r '.[].mac'

# Get endpoint details with full output
netapi ise get-endpoint 3c:e0:64:e8:90:6e -f json | jq '.groupId, .identityGroup'

# Check auth history (use -f json for full AuthZ Profile names)
netapi ise dc auth-history 3c:e0:64:e8:90:6e -f json | jq '.[] | {time, status, authz_profile}'

# Update single endpoint to correct group
netapi ise update-endpoint 3c:e0:64:e8:90:6e --group "IoT_iPSK_VLAN1610_Strongline" --static-group

# Bulk update with loop
for mac in 3c:e0:64:e8:88:61 3c:e0:64:e8:90:6e; do
  netapi ise update-endpoint "$mac" --group "IoT_iPSK_VLAN1610_Strongline" --static-group
done

# Alternative: delete and recreate
netapi ise delete-endpoint 3c:e0:64:e8:90:6e
netapi ise create-endpoint 3c:e0:64:e8:90:6e --group "IoT_iPSK_VLAN1610_Strongline" --static-group

5.4. DataConnect Raw SQL Queries

Run via netapi ise dc query "SQL" or directly against Oracle (port 2484, service cpm10).

5.4.1. Endpoint Lookup

-- Get endpoint by MAC
SELECT edf_macaddress AS mac_address,
       edf_endpointpolicy AS endpoint_policy,
       edf_endpointip AS ip_address,
       edf_identitygroupid AS identity_group,
       TO_CHAR(edf_create_time, 'YYYY-MM-DD HH24:MI:SS') AS created,
       TO_CHAR(edf_update_time, 'YYYY-MM-DD HH24:MI:SS') AS updated
FROM cepm.edf_ep_master
WHERE UPPER(edf_macaddress) = '3C:E0:64:E8:90:6E'

5.4.2. Auth History for MAC

-- Get auth history for a MAC (last 24 hours)
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') AS timestamp,
       calling_station_id AS mac_address,
       user_name AS username,
       nas_ip_address,
       authentication_method AS auth_method,
       selected_azn_profiles AS authz_profile,
       policy_set_name AS policy_set,
       CASE WHEN passed = 1 THEN 'PASSED' ELSE 'FAILED' END AS status,
       failure_reason
FROM mnt.radius_auth_48_live
WHERE UPPER(calling_station_id) = '3C:E0:64:E8:90:6E'
  AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 50 ROWS ONLY

5.4.3. Failed Auths (All Endpoints)

-- Get failed auths in last 24 hours
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') as timestamp,
       calling_station_id as mac_address,
       user_name as username,
       nas_ip_address,
       failure_reason
FROM mnt.radius_auth_48_live
WHERE failed = 1
  AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 100 ROWS ONLY

5.4.4. Top Failure Reasons

-- Get top failure reasons with counts
SELECT failure_reason,
       COUNT(*) AS count,
       ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER(), 1) AS pct
FROM mnt.radius_auth_48_live
WHERE failed = 1
  AND acs_timestamp >= SYSDATE - 1
  AND failure_reason IS NOT NULL
GROUP BY failure_reason
ORDER BY count DESC
FETCH FIRST 20 ROWS ONLY

5.4.5. Auth Stats Summary

-- Get auth statistics for last 24 hours
SELECT COUNT(*) as total,
       SUM(CASE WHEN passed = 1 THEN 1 ELSE 0 END) as passed,
       SUM(CASE WHEN failed = 1 THEN 1 ELSE 0 END) as failed
FROM mnt.radius_auth_48_live
WHERE acs_timestamp >= SYSDATE - 1

5.4.6. Key Tables

Table Description

cepm.edf_ep_master

Endpoint master table (MAC, policy, identity group)

mnt.radius_auth_48_live

Live RADIUS auth logs (last 48 hours)

mnt.radius_auth_aggr

Aggregated auth logs (older data)

5.5. Sample Endpoint: 3C:E0:64:E8:90:6E

MnT shows no auth records in last 24 hours, but DataConnect shows history:

Time Status AuthZ Profile

2026-02-06 08:01:40

PASSED

IoT_WiFI_iPSK_VLAN16…​

2026-02-06 00:01:31

PASSED

Wlan-Internet-Only

2026-02-05 16:01:22

PASSED

IoT_WiFI_iPSK_VLAN16…​

  • IP: 10.218.3.108

  • First Seen: 2024-11-14

  • Profiler: Unknown

Alternating between iPSK and Internet-Only profiles suggests identity group mismatch or authorization rule ordering issue.

5.6. Action

Delete existing endpoint entries and reimport via CSV to correct identity group. DHCP lease expiration takes a few days before devices get correct VLAN assignment.

5.7. Status

Reimported endpoints to ISE. Monitoring for VLAN correction after DHCP timer expires.

6. Commits

  • 51976219 - fix(chla): Correct domain attributes and add directory services validation

7. Next Actions

  • Apply access_provider = simple fix on chlxsbg

  • Test xding@la.ad.chla.org SSH login

  • Create home directory for xding if needed

  • Coordinate handover to research (xding)

  • Continue 802.1X EAP-TLS troubleshooting (SSL handshake issue)

  • Verify iPad #2 (34:f6:8d:0f:e7:f5) is registered in GetWell_iPads

  • Monitor Strongline gateways for correct VLAN after DHCP expiration

8. Reference

8.1. QRadar Contact

Alex Mejia — working with SoC on QRadar.

8.1.1. Linux workstation ISE DC Queries =====

 netapi ise dc query "
-- Get auth history for a MAC (last 24 hours)
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') AS timestamp,
       calling_station_id AS mac_address,
       user_name AS username,
       nas_ip_address,
       authentication_method AS auth_method,
       selected_azn_profiles AS authz_profile,
       policy_set_name AS policy_set,
       CASE WHEN passed = 1 THEN 'PASSED' ELSE 'FAILED' END AS status,
       failure_reason
FROM mnt.radius_auth_48_live
WHERE UPPER(calling_station_id) = 'b4:e9:b8:f6:c8:17'
  AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 50 ROWS ONLY"