ISE Certificate Troubleshooting Runbook

Complete operational runbook for troubleshooting ISE system certificates, OpenAPI import issues, and EAP-TLS authentication failures.

1. Overview

This runbook covers:

ISE system certificate management (Admin, EAP, Portal, pxGrid)
OpenAPI certificate import troubleshooting
Manual certificate import procedures
BYOD/EAP-TLS authentication debugging
Client trust store requirements

ISE PKI Architecture (2026-01)

Internal certificates (Admin, EAP, pxGrid): Issued by Vault DOMUS-ISSUING-CA
Public certificates (Guest/Sponsor Portal): Issued by Let’s Encrypt via certbot
Deprecated: AD CS HOME-ROOT-CA (remove by 2026-07)

2. Client Trust Requirements

2.1. What Clients Need to Trust ISE EAP Certificates

For EAP-TLS authentication to succeed, the client device must trust the CA chain that signed ISE’s EAP certificate.

Table 1. Trust Store Requirements
Client Type	Required CA in Trust Store	Verification Command
Linux (RHEL/Fedora)	DOMUS-ROOT-CA in `/etc/pki/ca-trust/source/anchors/`	`trust list \| grep -i domus`
Linux (Arch)	DOMUS-ROOT-CA in `/etc/ca-certificates/trust-source/anchors/`	`trust list \| grep -i domus`
Linux (Debian/Ubuntu)	DOMUS-ROOT-CA in `/usr/local/share/ca-certificates/`	`trust list \| grep -i domus`
Android phone	DOMUS-ROOT-CA imported via Settings > Security > Encryption & credentials	Settings > Trusted credentials > User tab
iOS device	DOMUS-ROOT-CA profile installed	Settings > General > About > Certificate Trust Settings

2.1.1. Linux Trust Store Paths by Distribution

Distribution Commands

Table 2. Trust Store Installation Commands
Distribution	Commands
Arch Linux	`sudo cp DOMUS-ROOT-CA.crt /etc/ca-certificates/trust-source/anchors/ sudo update-ca-trust`
RHEL/Fedora/CentOS	`sudo cp DOMUS-ROOT-CA.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust`
Debian/Ubuntu	`sudo cp DOMUS-ROOT-CA.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates`

Arch Linux

sudo cp DOMUS-ROOT-CA.crt /etc/ca-certificates/trust-source/anchors/
sudo update-ca-trust

RHEL/Fedora/CentOS

sudo cp DOMUS-ROOT-CA.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Debian/Ubuntu

sudo cp DOMUS-ROOT-CA.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

2.1.2. Downloading DOMUS-ROOT-CA from Vault

# Method 1: Via SSH and Vault CLI (recommended)
ssh certmgr-01.inside.domusdigitalis.dev "export VAULT_ADDR='http://127.0.0.1:8200' && vault read -field=certificate pki/cert/ca" > /tmp/DOMUS-ROOT-CA.crt

# Method 2: If running locally on certmgr-01
export VAULT_ADDR='http://127.0.0.1:8200'
vault read -field=certificate pki/cert/ca > /tmp/DOMUS-ROOT-CA.crt

# Verify the certificate
openssl x509 -in /tmp/DOMUS-ROOT-CA.crt -noout -subject -dates
# Expected: subject=C=US, O=Domus Digitalis, OU=Enterprise PKI, CN=DOMUS-ROOT-CA

The HTTP endpoint certmgr-01:8200/v1/pki/ca returns DER format (binary). Use the Vault CLI with -field=certificate to get PEM format directly.

2.2. Adding "Admin" Usage to ISE Certificate

Question: Will adding "Admin" usage to an EAP certificate break client trust?

Answer: No. Certificate usages (Admin, EAP, Portal, pxGrid) only control what ISE uses the certificate for. They do not affect client trust validation.

Table 3. ISE Certificate Usage Summary
Usage	Purpose
Admin	ISE Admin GUI access (port 8443)
EAP Authentication	Server certificate presented during 802.1X EAP-TLS handshake
Portal	Guest/Sponsor portal HTTPS
pxGrid	pxGrid client/server TLS
RADIUS DTLS	RADIUS over DTLS
SAML	SAML authentication

A single Vault-issued certificate can have multiple usages (e.g., Admin + EAP). The key requirement is that all clients have the issuing CA (DOMUS-ROOT-CA) in their trust stores.

3. ISE Certificate Inventory

3.1. List System Certificates

# Using netapi
dsource d000 dev/network
netapi ise cert list-system

# Using curl directly
curl -sk -X GET "https://$<ise-pan-ip>/api/v1/certs/system-certificate" \
  -H "Authorization: Basic $<ise-api-token>" \
  -H "Accept: application/json" | jq '.response[]'

3.2. List Trusted Certificates (CA Store)

# List all trusted CAs in ISE
netapi ise cert list-trusted

# Filter for DOMUS CAs
netapi ise cert list-trusted | grep -i domus

3.3. Expected Certificates (Vault PKI)

Table 4. ISE System Certificates (Target State)
Certificate Name	Usages	Issued By	Expiration
ISE-VAULT-EAP-ADMIN	Admin, EAP	DOMUS-ISSUING-CA	1 year
LetsEncrypt-Portal	Portal	Let’s Encrypt E8	90 days
ISE-pxGrid	pxGrid	DOMUS-ISSUING-CA	1 year

Table 5. ISE Trusted CA Store (Required)
CA Name	Purpose	Source
DOMUS-ROOT-CA	Root trust anchor	Vault `pki/cert/ca`
DOMUS-ISSUING-CA	Intermediate CA	Vault `pki_int/cert/ca`
ISRG Root X1	Let’s Encrypt root	Let’s Encrypt bundle

4. Manual Certificate Import (GUI Fallback)

Use this procedure when the OpenAPI import fails silently.

4.1. Step 1: Issue Certificate from Vault

# SSH to certmgr-01
ssh certmgr-01.inside.domusdigitalis.dev

# Set Vault environment
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'  # From dsec d000 dev/vault

# Issue EAP certificate for ISE
vault write -format=json pki_int/issue/domus-server \
  common_name="ise-02.inside.domusdigitalis.dev" \
  alt_names="ise-02.inside.domusdigitalis.dev,ise-02.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/ise-eap.json

# Extract certificate and key
jq -r '.data.certificate' /tmp/ise-eap.json > /tmp/ise-eap.crt
jq -r '.data.private_key' /tmp/ise-eap.json > /tmp/ise-eap.key
jq -r '.data.ca_chain[]' /tmp/ise-eap.json > /tmp/ise-eap-chain.crt

# Verify certificate details
openssl x509 -in /tmp/ise-eap.crt -noout -subject -issuer -dates

4.2. Step 2: Transfer to Local Machine

# From your workstation
scp certmgr-01.inside.domusdigitalis.dev:/tmp/ise-eap.crt ~/Downloads/
scp certmgr-01.inside.domusdigitalis.dev:/tmp/ise-eap.key ~/Downloads/
scp certmgr-01.inside.domusdigitalis.dev:/tmp/ise-eap-chain.crt ~/Downloads/

4.3. Step 3: Import via ISE GUI

Login to ISE Admin: ise-02.inside.domusdigitalis.dev:8443
Navigate to: Administration > System > Certificates > System Certificates
Click Import
Fill in:
- Friendly Name: ISE-VAULT-EAP-ADMIN
- Certificate File: Upload ise-eap.crt
- Private Key File: Upload ise-eap.key
- Password: (leave empty for unencrypted key)
Select usages:
- Admin
- EAP Authentication
Click Submit

4.4. Step 4: Verify Import

# List system certificates
netapi ise cert list-system

# Verify EAP usage assigned
netapi ise cert list-system | grep -A5 "ISE-VAULT-EAP-ADMIN"

5. OpenAPI Certificate Import

5.1. Working Payload Format (Reference)

This payload format works for the Let’s Encrypt deploy hook on certmgr-01.inside.domusdigitalis.dev.

{
  "admin": false,
  "allowOutOfDateCert": false,
  "allowExtendedValidity": true,
  "allowPortalTagTransferForSameSubject": true,
  "allowReplacementOfPortalGroupTag": true,
  "allowReplacementOfCertificates": true,
  "allowRoleTransferForSameSubject": true,
  "allowSHA1Certificates": false,
  "allowWildCardCerts": true,
  "data": "<base64-encoded-cert>",
  "name": "LetsEncrypt-Portal",
  "password": "",
  "portal": true,
  "portalGroupTag": "Default Portal Certificate Group",
  "privateKeyData": "<base64-encoded-key>",
  "validateCertificateExtensions": false
}

Critical Field Name: allowWildCardCerts (NOT allowWildCardCertificates)

This field must be set to true even for non-wildcard certificates. Using the wrong field name causes a silent failure.

5.2. Known API Issues

5.2.1. Issue 1: Silent Failure (200 OK with Null Response)

Symptom: API returns HTTP 200 but certificate doesn’t appear in ISE.

{
  "response": {
    "status": null,
    "message": null,
    "id": null
  },
  "version": "1.0.1"
}

Possible Causes:

Certificate already exists with same subject/serial
ISE internal processing error (check ISE logs)
Certificate chain validation failed internally
Field name mismatch (e.g., allowWildCardCertificates vs allowWildCardCerts)

Debugging Steps:

# 1. Check if certificate already exists
netapi ise cert list-system | grep -i "<friendly-name>"

# 2. Check ISE application logs
ssh ise-02.inside.domusdigitalis.dev
show logging application ise-psc.log tail 100

# 3. Compare with working deploy hook
ssh certmgr-01.inside.domusdigitalis.dev
cat /etc/letsencrypt/renewal-hooks/deploy/deploy-certs.sh

# 4. Verify base64 encoding
echo "$CERT_B64" | base64 -d | openssl x509 -noout -text | head -20

5.2.2. Issue 2: 400 Bad Request

Symptom: "request has bad input format in the body"

Common Causes:

Wrong field name (allowWildCardCertificates instead of allowWildCardCerts)
Invalid base64 encoding
Missing required fields
Invalid JSON structure

Fix: Verify payload matches the working format above exactly.

5.2.3. Issue 3: 401 Unauthorized

Symptom: "Authentication failed"

Causes:

Wrong API credentials
OpenAPI not enabled on ISE
User lacks certificate admin permissions

Fix:

# Verify credentials
dsource d000 dev/network
echo "Token: ${ISE_API_TOKEN:0:10}..."

# Test OpenAPI access
curl -sk -X GET "https://$<ise-pan-ip>/api/v1/certs/system-certificate" \
  -H "Authorization: Basic $<ise-api-token>" \
  -H "Accept: application/json"

5.3. netapi cert import Command

# Import EAP/Admin certificate
netapi ise cert import /tmp/ise-eap.crt /tmp/ise-eap.key \
  --name "ISE-VAULT-EAP-ADMIN" \
  --hostname ise-02 \
  --admin \
  --eap

# Import portal certificate
netapi ise cert import /tmp/portal.crt /tmp/portal.key \
  --name "LetsEncrypt-Portal" \
  --portal

6. BYOD/EAP-TLS Troubleshooting

6.1. Check Active Sessions

# Check specific device session
netapi ise mnt session <MAC_ADDRESS>

# Example
netapi ise mnt session b4:e9:b8:f6:c8:17

6.2. Check Authentication Status

# Get auth status for device
netapi ise mnt auth-status <MAC_ADDRESS>

6.3. Common EAP-TLS Failures

6.3.1. Failure: "client rejected the ISE local-certificate"

Meaning: The client device does not trust ISE’s EAP certificate.

Root Cause: ISE’s EAP certificate is signed by a CA not in the client’s trust store.

Diagnosis:

# Check which CA signed ISE's EAP cert
netapi ise cert list-system | grep -A10 "EAP"
# Look for "Issued To" and "Issued By"

# Verify client has the CA
# On Linux:
ls /etc/pki/ca-trust/source/anchors/ | grep -i domus

# On Android:
# Settings > Security > Encryption & credentials > Trusted credentials > User

Resolution:

Import DOMUS-ROOT-CA on the client device
For Android: Download DOMUS-ROOT-CA.crt, tap to install, select "CA certificate"
For Linux: Copy to /etc/pki/ca-trust/source/anchors/ and run update-ca-trust

6.3.2. Failure: "EAP-TLS handshake failed"

Possible Causes:

Client certificate expired
Client certificate revoked
CA chain incomplete on ISE
TLS version mismatch

Diagnosis:

# Check client cert expiration (on client)
openssl x509 -in /etc/pki/tls/certs/client.crt -noout -dates

# Check ISE trusted CA store has full chain
netapi ise cert list-trusted | grep -i domus
# Should see both DOMUS-ROOT-CA and DOMUS-ISSUING-CA

6.3.3. Failure: "No suitable EAP certificate found on ISE"

Meaning: ISE doesn’t have a certificate with "EAP Authentication" usage assigned.

Resolution:

Check current certificate usages: netapi ise cert list-system
Via GUI: Edit the certificate and enable "EAP Authentication" usage
Or import a new certificate with EAP usage enabled

6.4. Android BYOD Troubleshooting

6.4.1. Import Root CA on Android

Transfer DOMUS-ROOT-CA.crt to device (email, ADB, etc.)
Open file, or go to: Settings > Security > Encryption & credentials > Install a certificate
Select CA certificate
Confirm installation

# Transfer via ADB
adb push /tmp/DOMUS-ROOT-CA.crt /sdcard/Download/

6.4.2. Configure EAP-TLS WiFi

Settings > WiFi > Add network
SSID: Domus-Secure
Security: WPA2-Enterprise or WPA3-Enterprise
EAP method: TLS
CA certificate: Select DOMUS-ROOT-CA (user-installed)
User certificate: Select your BYOD certificate
Identity: <device>-<owner>.byod.inside.domusdigitalis.dev
Connect

6.5. Linux Workstation Troubleshooting

6.5.1. Verify CA Trust Store

# Check if DOMUS-ROOT-CA is trusted
trust list | grep -i domus

# Or check directly
ls /etc/pki/ca-trust/source/anchors/

# Update trust store after adding new CAs
sudo update-ca-trust

6.5.2. Check wpa_supplicant Logs

# Check NetworkManager/wpa_supplicant logs
journalctl -u wpa_supplicant -f

# Or NetworkManager logs
journalctl -u NetworkManager -f --grep="802.1X\|EAP"

6.5.3. Manual wpa_supplicant Test

# Create test config
cat > /tmp/test-eap.conf << EOF
network={
  ssid="{ssid-secure}"
  key_mgmt=WPA-EAP
  eap=TLS
  identity="workstation.{domain}"
  ca_cert="/etc/pki/ca-trust/source/anchors/DOMUS-ROOT-CA.crt"
  client_cert="/etc/pki/tls/certs/client.crt"
  private_key="/etc/pki/tls/private/client.key"
}
EOF

# Test connection
sudo wpa_supplicant -i wlan0 -c /tmp/test-eap.conf -d

7. ISE Log Analysis

7.1. Key Log Files

Log Purpose

Log	Purpose
`ise-psc.log`	Certificate and PKI operations
`prrt-server.log`	RADIUS authentication
`guest.log`	Guest portal operations
`pxgrid.log`	pxGrid operations

ise-psc.log

Certificate and PKI operations

prrt-server.log

RADIUS authentication

guest.log

Guest portal operations

pxgrid.log

pxGrid operations

7.2. Viewing Logs

# SSH to ISE
ssh admin@ise-02.inside.domusdigitalis.dev

# View recent cert operations
show logging application ise-psc.log tail 200

# View recent authentications
show logging application prrt-server.log tail 200 | grep -i "EAP\|TLS"

# Export logs for analysis
copy logging application ise-psc.log tftp://<server>/ise-psc.log

8. Recovery Procedures

8.1. Emergency: EAP Certificate Expired/Invalid

Immediate Impact: All 802.1X authentications fail.

Recovery Steps:

Issue new certificate from Vault:

ssh certmgr-01.inside.domusdigitalis.dev
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'

vault write -format=json pki_int/issue/domus-server \
  common_name="ise-02.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/ise-eap-emergency.json

Import via GUI (API may be unreliable):
- Follow manual import procedure above
- Assign EAP Authentication usage
Verify authentication resumes:
```
netapi ise mnt sessions | head -20
```

8.2. Rollback to Previous Certificate

If a new certificate causes issues:

Via ISE GUI: Administration > System > Certificates > System Certificates
Find the previous working certificate
Edit and re-enable the required usages (Admin, EAP, etc.)
Disable usages on the problematic new certificate

9. Related Documentation

10. Appendix: Quick Reference

Certificate Import Commands

# List system certs
netapi ise cert list-system

# List trusted CAs
netapi ise cert list-trusted

# Import cert (API)
netapi ise cert import <cert> <key> --name <name> --eap --admin

# Check session
netapi ise mnt session <MAC>

# Check auth status
netapi ise mnt auth-status <MAC>

Vault Certificate Issuance

# Issue server cert
vault write pki_int/issue/domus-server \
  common_name="<hostname>.inside.domusdigitalis.dev" \
  ttl="8760h"

# Issue BYOD cert
vault write pki_int/issue/domus-byod \
  common_name="<device>.byod.inside.domusdigitalis.dev" \
  ttl="2160h"

Client Trust Store

# Linux: Add CA
sudo cp DOMUS-ROOT-CA.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Android: Install CA
adb push DOMUS-ROOT-CA.crt /sdcard/Download/
# Then install via Settings > Security