TAC Case: 802.1X Authentication Failures (~500 Endpoints)

# Common failure reasons to look for:
# 12514 - EAP-TLS failed SSL/TLS handshake
# 12308 - Client certificate chain not trusted
# 22056 - Subject not found in identity store
# 24408 - User/machine not found in AD
# 24415 - Could not locate AD domain

# Generate from Primary PAN GUI:
# Administration > System > Logging > Debug Log Configuration
# Set debug levels BEFORE reproducing, then generate bundle

Support bundle generated: [ ] Yes  [ ] No
Bundle filename: ise-support-bundle-YYYY-MM-DD.tar.gz

# Run on each PSN
show application status ise
show logging application ise-psc.log tail count 100

# AD connectivity
test aaa group <AD-join-point> <test-user> <password>

# Database status
show application status ise

# Check replication
application configure ise
# Select option 24: View DB Replication Status

# Disk space (if DB is full, session data won't write)
show disk

# From MNT CLI - check if session database is responding
application configure ise
# Select option 14: Purge Runtime Sessions
# (DO NOT purge - just see if it responds)

# Node overview
netapi ise api info

# All nodes with roles/services
netapi ise -f json api-call openapi GET "/api/v1/deployment/node" | jq -r '["HOSTNAME","IP","ROLES","SERVICES","STATUS"], (.response[] | [.hostname, .ipAddress, (.roles|join("/")), (.services|join(",")), .nodeStatus]) | @tsv' | column -t

# Check MNT node status specifically
netapi ise -f json api-call openapi GET "/api/v1/deployment/node" | jq '.response[] | select(.roles[] | contains("MNT")) | {hostname, ipAddress, status: .nodeStatus, services}'

# Last 24 hours failed authentications
netapi ise -f json mnt failures --hours 24 | jq -r '.[] | [.timestamp, .username, .nas_ip, .failure_reason] | @tsv' | head -100

# Group failures by reason code
netapi ise -f json mnt failures --hours 24 | jq -r '.[].failure_reason' | sort | uniq -c | sort -rn

# Group failures by PSN (which PSN is seeing failures?)
netapi ise -f json mnt failures --hours 24 | jq -r '.[].psn' | sort | uniq -c | sort -rn

# Current session count per PSN
netapi ise -f json mnt sessions | jq -r 'group_by(.psn) | .[] | {psn: .[0].psn, count: length}'

# Total active sessions
netapi ise -f json mnt sessions | jq 'length'

# List authentication policy sets
netapi ise policy-sets

# Check the General-Device-Onboard identity group (workaround)
netapi ise -f json identity-groups | jq '.[] | select(.name | contains("General-Device-Onboard"))'

# AD join point status
netapi ise -f json api-call openapi GET "/api/v1/active-directory" | jq '.response[] | {name, domain, status: .adJoinPointStatus}'

# Dump deployment info to JSON
netapi ise export > /tmp/ise-config-$(date +%Y%m%d).json

- [ ] app ise stop
- [ ] reboot
- [ ] saved ade-os
- [ ] acknowledged reboot
- [ ] ssh'd into server at 2026-03-12 16:21
- [ ] pmnt/admin#show uptime
 16:21:26 up 3 min,  1 user,  load average: 2.77, 1.35, 0.53
- [ ] 2026-03-12 16:29 running show application status ise

ISE PROCESS NAME                       STATE            PROCESS ID
--------------------------------------------------------------------
Database Listener                      running          8851
Database Server                        running          300 PROCESSES
Application Server                     running          28749
Profiler Database                      running          17555
ISE Indexing Engine                    disabled
AD Connector                           running          29781
M&T Session Database                   running          24955
M&T Log Processor                      running          29017
Certificate Authority Service          disabled
EST Service                            running          157854
SXP Engine Service                     disabled
TC-NAC Service                         disabled
PassiveID WMI Service                  disabled
PassiveID Syslog Service               disabled
PassiveID API Service                  disabled
PassiveID Agent Service                disabled
PassiveID Endpoint Service             disabled
PassiveID SPAN Service                 disabled
DHCP Server (dhcpd)                    disabled
DNS Server (named)                     disabled
ISE Messaging Service                  running          12480
ISE API Gateway Database Service       running          16233
ISE API Gateway Service                running          23247
ISE pxGrid Direct Service              disabled
Segmentation Policy Service            disabled
REST Auth Service                      running          145209
SSE Connector                          disabled
Hermes (pxGrid Cloud Agent)            disabled
McTrust (Meraki Sync Service)          disabled
ISE Node Exporter                      running          48340
ISE Prometheus Service                 disabled
ISE Grafana Service                    disabled
ISE MNT LogAnalytics Elasticsearch     running          57535
ISE Logstash Service                   running          75054
ISE Kibana Service                     running          92228

- [ ]





[scratch area for case log / CLI findings]

1. SSH to Primary MNT
   - ssh pmnt.ise.chla.org

2. Initiate reload
   - reload
   - Prompt: "Save ade-os [y/n]" → y

3. Wait for reboot (~5 minutes)

4. Reconnect and verify
   - ssh pmnt.ise.chla.org
   - Wait ~10 minutes for services to initialize
   - show application status ise
   - Confirm critical services running:
     * Database Listener: running
     * Database Server: running
     * Application Server: running
     * M&T Session Database: running
     * M&T Log Processor: running
     * ISE Messaging Service: running

5. Validate logging restored
   - Log into Primary PAN (ppan.ise.chla.org)
   - Navigate to Operations > RADIUS Live Logs
   - Click endpoint details to confirm "No data available" error is resolved