Daily Worklog
1. Overview
Date: 2026-02-06 (Friday)
Location: Remote
Focus: chlxsbg SSSD/AD login fix, directory services validation, iPSK GWN deployment validation, runbook attribute cleanup
Strategic Priorities:
-
chlxsbg AD user login — SSSD
access_providerfix for research handover -
Directory services validation script and documentation
-
Runbook attribute cleanup (
chla.usc.edu→la.ad.chla.org) -
iPSK GWN deployment traffic validation
2. Session: chlxsbg SSSD / AD Login Fix
2.1. Context
Dr. Shahab’s Linux workstation (chlxsbg) needs to allow AD user login for xding@la.ad.chla.org (the researcher who will use the machine). Sarah (CISO) requires AD accounts for accountability — no local admin usage.
2.2. Problem
SSSD configured with access_provider = ad which checks AD GPO for login permissions. CHLA’s large AD causes this check to timeout:
pam_acct_mgmt: Authentication service cannot retrieve authentication info
User lookup succeeds via sssctl, but login fails.
2.3. Solution
Change access_provider = ad to access_provider = simple with explicit user allowlist:
# /etc/sssd/sssd.conf under [domain/LA.AD.CHLA.ORG]
access_provider = simple
simple_allow_users = xding@la.ad.chla.org, sasgharzadeh@la.ad.chla.org
Then restart SSSD:
sudo systemctl restart sssd && sudo sss_cache -E
2.4. Validation
Created comprehensive directory services validation script. See: PRJ-ISE-CHLA-LINUX-ANTORA/runbooks/notes/validating-directory-services-20260206.adoc
Key findings:
| Component | Status | Notes |
|---|---|---|
Realm membership |
PASS |
kerberos-member of LA.AD.CHLA.ORG |
Kerberos config |
PASS |
AES256/AES128, DNS KDC lookup |
Machine keytab |
PASS |
CHLXSBG$@LA.AD.CHLA.ORG (KVNO 3) |
SSSD config |
PASS |
AD provider, NSS/PAM services |
NSS switch |
PASS |
passwd/group/shadow β sss |
PAM integration |
PASS |
pam_sss in common-auth/session |
AD user lookup |
PASS |
sssctl resolves xding@la.ad.chla.org |
AD user login |
FAIL |
|
Group enumeration |
FAIL |
Hangs (CHLA AD scale issue) |
2.5. Reference
UNIX/Linux Handbook notes on directory services requirements:
-
realmd installed and domain joined
-
SSSD configured with AD identity/auth stores
-
nsswitch.conf using sss for passwd/group/shadow
-
PAM configured for sss authentication
-
Machine keytab with TGT for ongoing auth
All requirements met on chlxsbg. Issue is purely access_provider = ad vs simple.
3. Session: Runbook Attribute Cleanup
3.1. Context
Found chla.usc.edu domain in user attributes instead of correct la.ad.chla.org.
3.2. Changes
| File | Change |
|---|---|
|
Fixed |
|
Fixed example domain |
|
Fixed 4 hardcoded values (domain, cert template, CA files, machine cert) |
|
Updated version v3.7 β v3.9 |
|
Updated version v3.8 β v3.9 |
Rebuilt team-views HTML with corrected attributes.
4. Session: iPSK GWN Deployment Validation
4.1. Context
Validating iPSK authentication for GetWell iPads. Checking ISE live logs for client auth success.
4.2. MAC Address 1: 34:F6:8D:0E:76:E7
Result: Authentication succeeded
| Field | Value |
|---|---|
Event |
5200 Authentication succeeded |
Username |
34:F6:8D:0E:76:E7 |
Authentication Policy |
CHLA IoT >> MAB |
Authorization Policy |
CHLA IoT >> GetWell iPAD iPSK only |
Authorization Result |
iPSK-MGR-AuthZ |
Identity Store |
Internal Endpoints |
External Groups |
GetWell_iPads |
VLAN |
1630 |
PSK |
Applied (masked) |
Network Device |
USC-OEAP-WLC3 |
Timestamp |
2026-01-30 11:47:26.559 |
Key steps from ISE log:
-
11001- Received RADIUS Access-Request -
24211- Found Endpoint in Internal Endpoints IDStore -
22037- Authentication Passed -
24870- Fetching user groups from external ODBC succeeded -
15016- Selected Authorization Profile - iPSK-MGR-AuthZ -
11002- Returned RADIUS Access-Accept
4.3. MAC Address 2: 34:f6:8d:0f:e7:f5
Result: Wrong MAC provided initially
Original MAC given was 34:F6:8D:0E:F7:F5 (incorrect). Correct MAC is 34:f6:8d:0f:e7:f5.
| iPad | MAC Address | IP |
|---|---|---|
iPad #1 |
34:f6:8d:0e:76:e7 |
10.218.21.24 |
iPad #2 |
34:f6:8d:0f:e7:f5 |
10.218.21.27 |
| Verify iPad #2 is registered in ISE under GetWell_iPads identity group. |
4.4. netapi Commands for ISE Lookup
To query ISE authentication logs via CLI instead of GUI:
4.4.1. MnT API (Live Sessions & Auth Logs)
# Get authentication logs for a MAC address
netapi ise mnt auth-logs 34:F6:8D:0E:76:E7
# Get active session details by MAC
netapi ise mnt session 34:F6:8D:0E:76:E7
# Get authentication status
netapi ise mnt auth-status 34:F6:8D:0E:76:E7
# List all active RADIUS sessions
netapi ise mnt sessions
# Get failed authentication attempts
netapi ise mnt failed
# List all failure reason codes
netapi ise mnt failure-reasons
4.4.2. DataConnect (Database Analytics)
# Get recent authentications (live RADIUS log)
netapi ise dc recent
# Get comprehensive session view for a MAC
netapi ise dc session 34:F6:8D:0E:76:E7
# Get authentication history timeline for a MAC
netapi ise dc auth-history 34:F6:8D:0E:76:E7
# Get failed authentications
netapi ise dc failed
# Get endpoint details
netapi ise dc endpoint 34:F6:8D:0E:76:E7
# List endpoints
netapi ise dc endpoints
# Get top failure reasons with counts
netapi ise dc top-failures
# Get authentication statistics
netapi ise dc stats
# Get authentication method distribution
netapi ise dc auth-methods
# Get hourly authentication trends
netapi ise dc trends
# Get NAS (switch/WLC) health by auth success rate
netapi ise dc nas
# Find stale endpoints not seen in N days
netapi ise dc stale --days 30
| MnT logs are retained for ~7 days by default. DataConnect queries the ISE database directly for historical data. |
5. Session: Strongline Gateway VLAN Issue
5.1. Problem
9 Strongline gateways are hitting a "misc" IoT subnet instead of the correct VLAN. Root cause: endpoints are assigned to an identity group with "misc" in the name, causing incorrect authorization profile match.
5.2. Identity Groups
| Group Name | ID |
|---|---|
IoT_iPSK_VLAN1610_Strongline |
6b8bc440-7d21-11f0-bf95-82b6868ac19f |
IoT_iPSK_VLAN1620_Misc |
c4dfd4d0-ea3e-11ef-9e89-46637f695c46 |
ise_medigate_Beacons_Receiver_Strongline_Staff_Safety_System |
75cf6930-9bf6-11ec-bf4f-da147ea7a803 |
5.3. netapi Commands
# List endpoints in wrong group (use -f json to avoid column truncation)
netapi ise get-endpoints --group "IoT_iPSK_VLAN1620_Misc" --all -f json | jq -r '.[].mac'
# Get endpoint details with full output
netapi ise get-endpoint 3c:e0:64:e8:90:6e -f json | jq '.groupId, .identityGroup'
# Check auth history (use -f json for full AuthZ Profile names)
netapi ise dc auth-history 3c:e0:64:e8:90:6e -f json | jq '.[] | {time, status, authz_profile}'
# Update single endpoint to correct group
netapi ise update-endpoint 3c:e0:64:e8:90:6e --group "IoT_iPSK_VLAN1610_Strongline" --static-group
# Bulk update with loop
for mac in 3c:e0:64:e8:88:61 3c:e0:64:e8:90:6e; do
netapi ise update-endpoint "$mac" --group "IoT_iPSK_VLAN1610_Strongline" --static-group
done
# Alternative: delete and recreate
netapi ise delete-endpoint 3c:e0:64:e8:90:6e
netapi ise create-endpoint 3c:e0:64:e8:90:6e --group "IoT_iPSK_VLAN1610_Strongline" --static-group
5.4. DataConnect Raw SQL Queries
Run via netapi ise dc query "SQL" or directly against Oracle (port 2484, service cpm10).
5.4.1. Endpoint Lookup
-- Get endpoint by MAC
SELECT edf_macaddress AS mac_address,
edf_endpointpolicy AS endpoint_policy,
edf_endpointip AS ip_address,
edf_identitygroupid AS identity_group,
TO_CHAR(edf_create_time, 'YYYY-MM-DD HH24:MI:SS') AS created,
TO_CHAR(edf_update_time, 'YYYY-MM-DD HH24:MI:SS') AS updated
FROM cepm.edf_ep_master
WHERE UPPER(edf_macaddress) = '3C:E0:64:E8:90:6E'
5.4.2. Auth History for MAC
-- Get auth history for a MAC (last 24 hours)
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') AS timestamp,
calling_station_id AS mac_address,
user_name AS username,
nas_ip_address,
authentication_method AS auth_method,
selected_azn_profiles AS authz_profile,
policy_set_name AS policy_set,
CASE WHEN passed = 1 THEN 'PASSED' ELSE 'FAILED' END AS status,
failure_reason
FROM mnt.radius_auth_48_live
WHERE UPPER(calling_station_id) = '3C:E0:64:E8:90:6E'
AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 50 ROWS ONLY
5.4.3. Failed Auths (All Endpoints)
-- Get failed auths in last 24 hours
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') as timestamp,
calling_station_id as mac_address,
user_name as username,
nas_ip_address,
failure_reason
FROM mnt.radius_auth_48_live
WHERE failed = 1
AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 100 ROWS ONLY
5.4.4. Top Failure Reasons
-- Get top failure reasons with counts
SELECT failure_reason,
COUNT(*) AS count,
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER(), 1) AS pct
FROM mnt.radius_auth_48_live
WHERE failed = 1
AND acs_timestamp >= SYSDATE - 1
AND failure_reason IS NOT NULL
GROUP BY failure_reason
ORDER BY count DESC
FETCH FIRST 20 ROWS ONLY
5.4.5. Auth Stats Summary
-- Get auth statistics for last 24 hours
SELECT COUNT(*) as total,
SUM(CASE WHEN passed = 1 THEN 1 ELSE 0 END) as passed,
SUM(CASE WHEN failed = 1 THEN 1 ELSE 0 END) as failed
FROM mnt.radius_auth_48_live
WHERE acs_timestamp >= SYSDATE - 1
5.4.6. Key Tables
| Table | Description |
|---|---|
|
Endpoint master table (MAC, policy, identity group) |
|
Live RADIUS auth logs (last 48 hours) |
|
Aggregated auth logs (older data) |
5.5. Sample Endpoint: 3C:E0:64:E8:90:6E
MnT shows no auth records in last 24 hours, but DataConnect shows history:
| Time | Status | AuthZ Profile |
|---|---|---|
2026-02-06 08:01:40 |
PASSED |
IoT_WiFI_iPSK_VLAN16… |
2026-02-06 00:01:31 |
PASSED |
Wlan-Internet-Only |
2026-02-05 16:01:22 |
PASSED |
IoT_WiFI_iPSK_VLAN16… |
-
IP:
10.218.3.108 -
First Seen: 2024-11-14
-
Profiler: Unknown
| Alternating between iPSK and Internet-Only profiles suggests identity group mismatch or authorization rule ordering issue. |
5.6. Action
Delete existing endpoint entries and reimport via CSV to correct identity group. DHCP lease expiration takes a few days before devices get correct VLAN assignment.
5.7. Status
Reimported endpoints to ISE. Monitoring for VLAN correction after DHCP timer expires.
6. Commits
-
51976219- fix(chla): Correct domain attributes and add directory services validation
7. Next Actions
-
Apply
access_provider = simplefix on chlxsbg -
Test xding@la.ad.chla.org SSH login
-
Create home directory for xding if needed
-
Coordinate handover to research (xding)
-
Continue 802.1X EAP-TLS troubleshooting (SSL handshake issue)
-
Verify iPad #2 (34:f6:8d:0f:e7:f5) is registered in GetWell_iPads
-
Monitor Strongline gateways for correct VLAN after DHCP expiration
8. Reference
8.1. QRadar Contact
Alex Mejia — working with SoC on QRadar.
8.1.1. Linux workstation ISE DC Queries =====
netapi ise dc query "
-- Get auth history for a MAC (last 24 hours)
SELECT TO_CHAR(acs_timestamp, 'YYYY-MM-DD HH24:MI:SS') AS timestamp,
calling_station_id AS mac_address,
user_name AS username,
nas_ip_address,
authentication_method AS auth_method,
selected_azn_profiles AS authz_profile,
policy_set_name AS policy_set,
CASE WHEN passed = 1 THEN 'PASSED' ELSE 'FAILED' END AS status,
failure_reason
FROM mnt.radius_auth_48_live
WHERE UPPER(calling_station_id) = 'b4:e9:b8:f6:c8:17'
AND acs_timestamp >= SYSDATE - 1
ORDER BY acs_timestamp DESC
FETCH FIRST 50 ROWS ONLY"