TAC-2026-04-gcc-ise-cert-import: GCC ISE Certificate Import Failure

Case Summary

Field Value

Case ID

TAC-2026-04-gcc-ise-cert-import

Date Opened

2026-04-01

Requestor

Matt Comeione (NE-Systems)

Escalated By

Ed Padilla (NE-Systems)

Client

GCC — Glendale Community College (Stanley)

Status

Partially Resolved — PSN-04 cert reissuance pending

Priority

P2

Category

ISE Certificate Management

Problem Statement

GCC has a 4-node ISE deployment. One node will not import a certificate. Their Cisco support has lapsed and they are looking at replacements. NE-Systems asked for ISE assistance.

Additionally, the secondary MNT and a PSN are down. CLI password has been lost. Password recovery required before any diagnostics can begin. GCC has VMware vSphere console access to the nodes. Nodes have not been hardened (password recovery is enabled).

An OVA deployment to ESXi is also in progress (GCC handling download).

Information Requested

Sent to Matt on 2026-04-01. Awaiting response:

  • Certificate type, format, chain completeness

  • Failing node hostname, role, version (show version)

  • Service status (show application status ise)

  • NTP status (show ntp)

  • Current certs (show certificate application-server)

  • Exact error message (screenshot)

  • Context (new cert? expired? recent changes?)

  • GUI/CLI access to failing node

  • The cert chain file

Availability

  • ~~2026-04-01: 12:00-12:30 PM (lunch) or after 4:00 PM~~ — superseded, active call in progress

  • Matt confirmed via email

Timeline

Date Event

2026-04-01 09:43

Matt Comeione emails NE-Systems team requesting ISE help for GCC/Stanley

2026-04-01 10:00

Ed Padilla forwards to Evan

2026-04-01 11:31

Evan replies — available at 12:00 or after 4:00 PM. Sent information checklist.

2026-04-01

Active call with GCC. Scope expanded: secondary MNT and PSN down, CLI password lost. Password recovery is immediate priority. OVA deployment to ESXi in progress (GCC downloading). Cert import deferred until nodes recovered.

2026-04-01

Deregistered secondary admin node and one PSN from the deployment — nodes did not register properly. Nodes are coming back up. Monitoring services.

2026-04-01

Resolved. Deregistered all nodes except PSN-04. All re-registered nodes synced and functional. PSN-04 deferred — requires certificate reissuance before it can rejoin the deployment. Evan led the call. GCC (Glendale Community College) confirmed working state.

Troubleshooting Notes

Step 1: Password Recovery (Secondary MNT and PSN)

Perform this on each down node. Requires ESXi 7 host client and the Cisco ISE installer ISO file (not the OVA).

The ISE installer ISO menu includes options that can wipe and reinstall the node. Read every menu option carefully before pressing any key. If unsure, stop and ask.

Reference: Cisco ISE Password Recovery Mechanisms

1a. Prerequisite — ISE installer ISO

  1. You need the ISE .iso file that matches the installed version (e.g., ISE-3.x.x.xxx.x86_64.iso)

  2. The OVA file will not work for password recovery — it must be the ISO

  3. If GCC does not have the ISO:

    • Download from Cisco Software Center (requires active account, not necessarily active support)

    • Or check if NE-Systems has a copy

  4. Upload the ISO to the ESXi datastore:

    • In the ESXi host client (<esxi-host-ip>/ui), left sidebar: click Storage

    • Click the datastore name

    • Click Datastore browser (top bar)

    • Click Upload, select the ISO file from local machine

    • Wait for upload to complete — note the path (e.g., [datastore1] ISE-3.2.0.542.x86_64.iso)

1b. Power off the ISE VM

  1. Left sidebar: click Virtual Machines

  2. Find the ISE VM (secondary MNT first), click its name

  3. If the VM is powered on:

    • Click Actions > Guest OS > Shut down

    • If greyed out: click Actions > Power > Power off

  4. Wait until the VM status shows Powered off

1c. Mount the ISO to the VM CD/DVD drive

  1. With the VM selected, click Edit (top menu bar)

  2. Scroll to CD/DVD Drive 1

    1. If there is no CD/DVD drive, click Add hard disk dropdown > CD/DVD Drive

  3. Set the dropdown to Datastore ISO file

  4. Click Browse, navigate to the ISO you uploaded in step 1a

  5. Select the ISO file, click Select

  6. Check the box: Connect at power on — this is critical, do not skip

  7. Click Save

1d. Force BIOS on next boot

  1. With the VM still selected, click Edit again

  2. Click VM Options tab (top of the edit window)

  3. Expand Boot Options

  4. Find Force BIOS Setup — check the box to enable it

  5. Click Save

1e. Power on and boot from ISO

  1. Click Actions > Power > Power on

  2. Immediately open Console dropdown > Open browser console

  3. The VM will enter the BIOS setup screen (because we forced it in 1d)

  4. In the BIOS:

    • Navigate to the Boot tab

    • Move CD-ROM Drive to the top of the boot order

    • Press F10 to save and exit (or navigate to Exit > Save Changes and Exit)

  5. The VM will reboot and boot from the ISE ISO

1f. Select System Utilities

  1. The ISE installer menu appears. You will see several numbered options:

    Cisco ISE Installation (Keyboard/Monitor)          [1]
Cisco ISE Installation (Serial Console)             [2]
System Utilities (Keyboard/Monitor)                 [3]
System Utilities (Serial Console)                   [4]

  2. Select option 3 — System Utilities (Keyboard/Monitor)

    Do NOT select option 1 or 2. Those will start a fresh ISE install and wipe the node.

1g. Recover the admin password

  1. The System Utilities menu appears:

    [1] Recover Administrator Password
[2] Virtual Machine Snapshot Management
[q] Quit and Reload

  2. Select option 1 — Recover Administrator Password

  3. A list of administrator accounts appears (usually just admin)

  4. Select the admin user (option 1)

  5. Type the new password — requirements:

    • Minimum 6 characters

    • At least 1 uppercase letter

    • At least 1 lowercase letter

    • At least 1 number

  6. Confirm the password when prompted

  7. Type y to save the new password

1h. Exit and boot from hard disk

  1. You are returned to the System Utilities menu

  2. Type q to Quit and Reload

  3. The VM will reboot — it will boot from the ISO again if we don’t fix the boot order

  4. Immediately go back to the ESXi host client:

    • Select the VM, click Edit

    • CD/DVD Drive 1: uncheck Connect at power on

    • VM Options > Boot Options > Force BIOS Setup: uncheck this

    • Click Save

  5. If the VM already booted back into the ISO menu, select Quit or power cycle from ESXi:

    • Actions > Power > Reset

  6. The VM will now boot from the hard disk into normal ISE

1i. Wait for ISE to come up

  1. ISE takes 8-15 minutes to fully boot — this is normal, do not power cycle

  2. Watch the console. You will see services starting

  3. When boot is complete, you will see:

    ise-hostname login:

  4. Log in:

    • Username: admin

    • Password: the password you just set in step 1g

1j. Verify you are in

  1. After login you should see the ISE CLI prompt:

    ise-hostname/admin#

  2. If you see this, password recovery is done on this node. Proceed to Step 2.

1k. Repeat for the second node

  1. Go back to the ESXi host UI

  2. Select the second ISE VM (the PSN)

  3. Repeat steps 1b through 1j (the ISO is already on the datastore from step 1a)

Step 2: Check service status

Run this on each recovered node immediately after login:

show application status ise
What to look for:

  • All services should show running

  • If any service shows disabled or not running, note which one — that tells us what’s actually broken

  • Pay special attention to:

    • ISE Indexing Engine — needed for MNT

    • ISE Profiler Database — needed for profiling

    • Application Server — the core process, if this is down nothing works

show version
Captures:

  • ISE version and patch level — we need this for the cert issue too

show ntp
What to look for:

  • NTP must be synchronized — if the clock is off, certs will fail validation and replication breaks

  • If NTP is not synced, this could be the root cause of multiple issues

2a. Advanced logging and diagnostics

Use these while nodes are coming up after deregistration/re-registration.

Watch ISE services starting in real time:

show application status ise

Run this every 2-3 minutes. Services come up in order — Database Listener first, Application Server last.

Check ISE application logs for errors:

show logging application ise-psc.log tail count 50
What to look for:

  • ise-psc.log — primary service controller, shows service start/stop and crash reasons

  • Look for ERROR, FATAL, Exception, or OutOfMemory

show logging application ise-psc.log internal tail count 100

More verbose output from the service controller.

Application server log (the core ISE web process):

show logging application server.log tail count 50
What to look for:

  • Started / Ready — good, server is up

  • Connection refused — database or other dependency not up yet

  • CertificateException — cert-related failure on startup

Database and replication status:

show logging application ad_agent.log tail count 30

AD connector status — relevant if ISE is joined to Active Directory.

show logging application caservice.log tail count 30

Certificate Authority service log — directly relevant to the cert import issue.

Registration and replication logs:

show logging application replication.log tail count 50
What to look for after deregistration:

  • Sync completed — data replication finished

  • Unable to connect to primary — network/firewall issue between nodes

  • Certificate error — replication failing due to cert mismatch

show logging application replication-status.log tail count 30

System-level diagnostics:

show logging system messages tail count 50

Kernel and system-level messages — look for disk, memory, or hardware errors.

show logging system boot.log tail count 30

Boot sequence log — useful if a node keeps failing to come up.

Disk and memory (if services won’t start):

show disks
show memory
Red flags:

  • Disk usage above 90% — ISE services will refuse to start

  • Free memory below 2 GB — ISE may OOM during startup

Quick health check one-liner (run all at once):

show application status ise ; show version ; show ntp ; show disks ; show memory

Step 3: Check replication and node health

show run
Captures:

  • Hostname, IP config, DNS, NTP server — note everything

From the PAN (primary admin node) GUI:

  1. Go to Administration > System > Deployment

  2. Check the status of all 4 nodes

  3. The secondary MNT and PSN should show as either:

    • Connected (green) — good, they re-synced after recovery

    • Disconnected (red) — need to re-register or troubleshoot replication

  4. Screenshot this page

Step 4: After recovery — cert import troubleshooting

Once all nodes are up and accessible, circle back to the original cert import failure:

show certificate application-server
show certificate certificate-signing-request
Captures:

  • Which certs are currently installed on each node

  • Whether there’s a pending CSR

Step 5: Remote certificate verification (curl / openssl)

Run these from any workstation that can reach the ISE nodes over the network. Replace <ise-node-ip> with the node’s management IP.

5a. Check what certificate ISE is presenting on the admin portal

curl -vvv -k https://<ise-node-ip>/ 2>&1 | awk '/\* Server certificate:/,/\* SSL connection/'
What to look for:

  • subject: — the CN and O fields, confirm it matches the node hostname

  • start date: / expire date: — is the cert expired?

  • issuer: — who signed it (self-signed, internal CA, public CA)

5b. Pull the full certificate chain

openssl s_client -connect <ise-node-ip>:443 -showcerts </dev/null 2>/dev/null
What to look for:

  • How many certs in the chain (numbered 0 s:, 1 s:, etc.)

  • If only 1 cert and it’s self-signed — no chain, likely the default ISE cert

  • If intermediate or root CA certs are missing — that’s the import problem

5c. Check expiry date on the presented cert

echo | openssl s_client -connect <ise-node-ip>:443 2>/dev/null | openssl x509 -noout -dates
Output:
notBefore=Mar 15 00:00:00 2025 GMT
notAfter=Mar 15 23:59:59 2026 GMT

If notAfter is in the past, the cert is expired.

5d. Inspect the cert details (SAN, issuer, serial)

echo | openssl s_client -connect <ise-node-ip>:443 2>/dev/null | openssl x509 -noout -text | awk '/Subject:|Issuer:|Not Before:|Not After:|DNS:/'
What to look for:

  • Subject Alternative Name DNS entries — must include the node FQDN

  • If SAN is missing the FQDN, the cert won’t work for that node

5e. Compare certs across all nodes

Run this against each node to see if they’re presenting the same or different certs:

for IP in <pan-ip> <smnt-ip> <psn-ip> <mnt-ip>; do
  echo "=== ${IP} ==="
  echo | openssl s_client -connect ${IP}:443 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint
  echo ""
done
Example with GCC IPs (fill in once known):
for IP in 10.x.x.10 10.x.x.11 10.x.x.12 10.x.x.13; do
  echo "=== ${IP} ==="
  echo | openssl s_client -connect ${IP}:443 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint
  echo ""
done
What to look for:

  • All nodes should have the same issuer if using the same CA

  • Fingerprints will differ (each node has its own cert) but issuer and dates should be consistent

  • The node that fails cert import will likely show a different issuer or an expired/self-signed cert

5f. Download a node’s cert to a file for inspection

echo | openssl s_client -connect <ise-node-ip>:443 2>/dev/null | openssl x509 -outform PEM > /tmp/ise-node-cert.pem
openssl x509 -in /tmp/ise-node-cert.pem -noout -text | less

5g. Verify a cert file before importing to ISE

If GCC has the cert file they’re trying to import, verify it locally first:

openssl x509 -in <cert-file.pem> -noout -text | awk '/Subject:|Issuer:|Not Before:|Not After:|DNS:/'

Check the chain is complete (leaf + intermediate + root):

openssl verify -CAfile <root-ca.pem> -untrusted <intermediate-ca.pem> <cert-file.pem>
Output:

  • OK — chain is valid

  • error 2 at 1 depth lookup: unable to get issuer certificate — missing intermediate

  • error 20 at 0 depth lookup: unable to get local issuer certificate — missing root CA

Resolution

Deregistered all nodes except PSN-04 from the ISE deployment and re-registered them. All re-registered nodes synced successfully and the deployment is functional.

Remaining Work

  • PSN-04 — requires certificate reissuance before it can be re-registered to the deployment

  • Confirm all policy sets and endpoint data replicated correctly post-recovery

  • Schedule follow-up with GCC/Matt to address PSN-04 cert

Outcome

  • 3 of 4 nodes restored to full operation

  • Deployment functional for authentication and policy enforcement

  • Evan led the engagement as primary ISE engineer for NE-Systems

Appendix: Cisco References

Topic Source

ISE Password Recovery (all methods)

Cisco: ISE Password Recovery Mechanisms

ISE 3.3 Password Reset/Recovery

Cisco: Reset and Recover ISE 3.3 Password

ISE Password Recovery Product Page

Cisco: ISE Password Recovery List

ISE 3.2 Installation & Post-Install Verification

Cisco: ISE 3.2 Installation Guide — Post-Install Tasks

Related