802.1X EAP-TLS Troubleshooting Runbook

Overview

This runbook contains battle-tested commands from real production troubleshooting sessions. Every command here has been validated in the field.

Key insight: When manual wpa_supplicant works but NetworkManager fails, the issue is NM configuration, not certificates or ISE.

Quick Triage

1. Check if endpoint is rejected by ISE

After too many failed attempts, ISE blocks the endpoint entirely:

netapi ise get-rejected-endpoints

Release it:

netapi ise release-rejected "<MAC-ADDRESS>"

Example

netapi ise release-rejected "08:92:04:38:11:9C"

2. Check authentication history by MAC

Most important query - shows complete timeline:

netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%<MAC-with-wildcards>%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Example

netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%08%92%04%38%11%9C%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

3. Check identity store issues

Shows which identity store and group were used:

netapi ise dc query "
SELECT USERNAME, IDENTITY_STORE, IDENTITY_GROUP, PASSED, FAILURE_REASON,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%<MAC-with-wildcards>%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Interpreting results:

IDENTITY_STORE empty = certificate auth (expected for EAP-TLS)
USERNAME = MAC address = MAB fallback (EAP-TLS failed first)
USERNAME = "USERNAME" (literal) = EAP-TLS rejected before identity extraction

4. Check switch 802.1X status

netapi ios exec "show access-session int g1/0/X d"

Success indicators:

Status:  Authorized
Domain:  DATA
dot1x    Authc Success
ACS ACL: <your-dacl-name>

Failure indicators:

Status: Unauthorized
dot1x: Stopped - Client not attempting or auth failed
Domain: UNKNOWN - Not authenticated

Detailed Diagnostics

Check all recent authentications (last hour)

netapi ise dc query "
SELECT USERNAME, CALLING_STATION_ID, NAS_IP_ADDRESS,
       FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 20 ROWS ONLY"

Check all recent failures

netapi ise dc query "
SELECT USERNAME, FAILURE_REASON, AUTHENTICATION_PROTOCOL, POLICY_SET_NAME
FROM RADIUS_AUTHENTICATIONS
WHERE PASSED = 'Fail'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Verify successful auth details

netapi ise dc query "
SELECT USERNAME, IDENTITY_GROUP, AUTHORIZATION_RULE, AUTHORIZATION_PROFILES, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%<MAC-with-wildcards>%'
  AND PASSED = 'Pass'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 3 ROWS ONLY"

Forensics

Check ISE config changes

See what changed in ISE around the time of failure:

netapi ise dc query "
SELECT
    TO_CHAR(ACS_TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS') as TIME,
    ADMIN_NAME,
    OBJECT_TYPE,
    OBJECT_NAME,
    OPERATION_MESSAGE_TEXT
FROM MNT.CONFIG_CHANGE
WHERE ACS_TIMESTAMP > SYSDATE - 1
  AND OBJECT_TYPE IS NOT NULL
ORDER BY ACS_TIMESTAMP DESC
FETCH FIRST 20 ROWS ONLY"

Check specific time window

Investigate exactly what happened during a failure window:

netapi ise dc query "
SELECT USERNAME, IDENTITY_STORE, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE TIMESTAMP_TIMEZONE BETWEEN
      TO_TIMESTAMP('2026-02-08 19:28:00', 'YYYY-MM-DD HH24:MI:SS')
  AND TO_TIMESTAMP('2026-02-08 19:30:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY TIMESTAMP_TIMEZONE"

Client-Side Diagnostics

Manual wpa_supplicant test

The definitive test - bypasses NetworkManager:

HOSTNAME="$HOST"
sudo wpa_supplicant -i enp44s0 -D wired -c /dev/stdin -d << EOF
network={
    key_mgmt=IEEE8021X
    eap=TLS
    identity="${HOSTNAME}.inside.domusdigitalis.dev"
    ca_cert="/etc/ssl/certs/DOMUS-ROOT-CA.pem"
    client_cert="/etc/ssl/certs/${HOSTNAME}-eaptls.pem"
    private_key="/etc/ssl/private/${HOSTNAME}-eaptls.key"
}
EOF

Success indicators:

EAP: Received EAP-Success
CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
State: ASSOCIATED -> COMPLETED
Supplicant port status: Authorized

Failure indicators:

EAP: Received EAP-Failure - ISE rejecting
TLS Alert: unknown CA - Certificate trust issue
No TLS messages at all - Identity rejection before TLS starts

Kill stuck wpa_supplicant

Before testing, ensure clean state:

sudo pkill wpa_supplicant
sleep 3

Check certificate and key match

HOSTNAME="$HOST"
openssl x509 -noout -modulus -in /etc/ssl/certs/${HOSTNAME}-eaptls.pem | md5sum
sudo openssl rsa -noout -modulus -in /etc/ssl/private/${HOSTNAME}-eaptls.key | md5sum

Hashes MUST match.

Verify certificate issuer

HOSTNAME="$HOST"
openssl x509 -in /etc/ssl/certs/${HOSTNAME}-eaptls.pem -issuer -subject -noout

Switch-Side Commands

Bounce the port

Reset 802.1X state machine on switch:

netapi ios config "interface g1/0/X" "shutdown" && sleep 2 && netapi ios config "interface g1/0/X" "no shutdown"

Check RADIUS server status

netapi ios exec "show aaa servers"

Look for:

State: current UP - RADIUS server reachable
State: current DEAD - RADIUS server unreachable

Common Failure Reasons

Error Code	Meaning	Action
12521	EAP-TLS failed SSL/TLS handshake after client alert	Client rejecting ISE cert OR cert chain issue
22056	Subject not found in applicable identity store(s)	AD lookup failing OR Certificate Profile needs matchMode: NEVER
12514	EAP-TLS handshake failed	Certificate chain or trust issue
22045	Identity policy configured for password but received cert	Wrong identity source in auth rule

Error Code

Meaning

Action

12521

EAP-TLS failed SSL/TLS handshake after client alert

Client rejecting ISE cert OR cert chain issue

22056

Subject not found in applicable identity store(s)

AD lookup failing OR Certificate Profile needs matchMode: NEVER

12514

EAP-TLS handshake failed

Certificate chain or trust issue

22045

Identity policy configured for password but received cert

Wrong identity source in auth rule

Troubleshooting Workflow

Check if rejected: netapi ise get-rejected-endpoints
Release if needed: netapi ise release-rejected "<MAC>"
Kill any running supplicant: sudo pkill wpa_supplicant
Test with manual wpa_supplicant (bypasses NetworkManager)
If manual works: Problem is NetworkManager config
If manual fails: Check ISE logs for exact failure reason
Check auth history: Use DataConnect queries above
Verify certs: Check issuer, dates, key match

Key Lessons Learned

Rejected endpoints block ALL auth attempts - must release first
Manual wpa_supplicant is the gold standard test - if it works, certs/ISE are fine
NetworkManager != wpa_supplicant - different config, different behavior
22056 after EAP-TLS = MAB fallback - EAP-TLS failed, fell back to MAC auth
USERNAME = "USERNAME" literal = ISE rejected before extracting cert identity
Check forensics time window to see exactly when/why failure started