RCA-2026-03-16-001: 802.1X EAP-TLS WiFi Authentication Failure

Executive Summary

802.1X WiFi EAP-TLS authentication failed on Ubuntu 25.10 (P50) with ISE error 12520 "unknown CA". Root cause was two-fold: (1) client was configured with intermediate CA instead of ROOT CA, preventing verification of ISE’s certificate chain, and (2) NetworkManager’s private-key-password-flags must be set at connection creation time, not modified after. Resolution required using ROOT CA only and recreating the connection with correct flags. This RCA serves as a MODEL for similar EAP-TLS deployments.

Timeline

Time Event

2026-03-16 ~10:00

P50 WiFi connected to Domus-IoT (MAB) instead of Domus-Secure (802.1X)

2026-03-16 ~10:15

Created WiFi EAP-TLS connection with wrong SSID (DomusWifi)

2026-03-16 ~10:30

Fixed SSID to Domus-Secure, connection failing with "Secrets were required"

2026-03-16 ~11:00

Added private-key-password-flags 4 via nmcli con mod - still failing

2026-03-16 ~11:30

Checked ISE logs - error 12520 "client rejected ISE local-certificate"

2026-03-16 ~12:00

Compared working config (modestus-razer) - identified CA difference

2026-03-16 ~12:15

Root cause identified: intermediate CA vs ROOT CA

2026-03-16 ~12:30

Fix implemented: ROOT CA + flags at creation time

2026-03-16 ~12:35

P50 authenticated, VLAN 10, IP 10.50.10.107

Problem Statement

Symptoms

  • WiFi connection showed "Secrets were required, but not provided"

  • wpa_supplicant logs showed TLS handshake failure

  • ISE MNT showed failed authentication attempts

  • Client kept falling back to Domus-IoT (MAB network)

Expected Behavior

Client presents certificate, ISE validates client cert against CA, ISE presents its certificate, client validates ISE cert against CA, mutual TLS authentication succeeds, client placed on VLAN 10 (Data).

Actual Behavior

TLS handshake failed during ISE certificate verification. Client rejected ISE’s certificate because it couldn’t build a trust chain to a known ROOT CA.

Root Cause

5 Whys Analysis

Why # Question and Answer

1

Why did EAP-TLS authentication fail?
Because: Client rejected ISE’s certificate as untrusted

2

Why did client reject ISE’s certificate?
Because: Client couldn’t verify ISE cert chain - "unknown CA"

3

Why couldn’t client verify the chain?
Because: Client only had intermediate CA, not ROOT CA

4

Why did client only have intermediate CA?
Because: Vault pki_int/issue returns issuing_ca (intermediate), not root

5

Why wasn’t ROOT CA explicitly configured?
Because: Documentation didn’t specify ROOT vs intermediate CA requirement

Root Cause Statement

EAP-TLS clients require the ROOT CA certificate to verify the authentication server’s certificate chain. Using only the intermediate CA causes "unknown CA" errors because the client cannot establish trust to the root of the PKI hierarchy.

Secondary Root Cause

NetworkManager’s 802-1x.private-key-password-flags cannot be modified after connection creation. Setting this flag via nmcli con mod has no effect - the flag must be set during nmcli connection add.

Contributing Factors

Factor Description Preventable?

Vault PKI output

vault write pki_int/issue returns intermediate CA in issuing_ca field, easy to assume this is the CA to use

Yes - documentation

nmcli behavior

nmcli con mod silently accepts private-key-password-flags but doesn’t apply it

No - NetworkManager design

SSID confusion

Initial connection used wrong SSID (DomusWifi vs Domus-Secure)

Yes - documentation

No pre-flight check

No verification step to confirm correct CA before deployment

Yes - process

Impact

Severity

Metric Value

Severity

P3 (personal infrastructure)

Duration

~2.5 hours troubleshooting

Users/Systems Affected

1 workstation (P50)

Data Loss

None

Business Impact

  • Lost productivity: 2.5 hours troubleshooting

  • Learning value: HIGH - identified MODEL process for future deployments

  • Similar issue exists at CHLA research Ubuntu workstation

Resolution

Immediate Actions (What was done)

  1. Retrieved ROOT CA from Vault: vault read -field=certificate pki/cert/ca

  2. Installed ROOT CA on client: /etc/ssl/certs/DOMUS-ROOT-CA.pem

  3. Deleted broken connection: nmcli con delete "Domus-WiFi-EAP-TLS"

  4. Created new connection with ALL flags at creation time:

sudo nmcli connection add \
    con-name "Domus-WiFi-EAP-TLS" \
    type wifi \
    ssid "Domus-Secure" \
    wifi-sec.key-mgmt wpa-eap \
    802-1x.eap tls \
    802-1x.identity "p50.inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
    802-1x.client-cert /etc/ssl/certs/p50-client.crt \
    802-1x.private-key /etc/ssl/private/p50-client.key \
    802-1x.private-key-password-flags 4

Verification

# Verify connection active
nmcli con show --active | grep -E 'WiFi.*EAP'

# Verify IP on correct VLAN
ip -4 -o addr show wlp4s0 | awk '{print $4}'
# Expected: 10.50.10.x (VLAN 10 - Data)

# Verify ISE session
netapi ise -f json mnt sessions | jq '.[] | select(.user_name | test("p50"; "i"))'

The MODEL: 802.1X EAP-TLS WiFi Setup

This is the verified working process. Use for all future EAP-TLS deployments.

Prerequisites

# 1. Get ROOT CA from Vault (NOT issuing_ca from pki_int)
dsource d000 dev/vault
vault read -field=certificate pki/cert/ca > /tmp/domus-root-ca.crt

# 2. Issue client certificate
vault write pki_int/issue/domus-client \
    common_name="hostname.inside.domusdigitalis.dev" \
    ttl="8760h" \
    -format=json > /tmp/client-cert.json

# 3. Extract cert and key
jq -r '.data.certificate' /tmp/client-cert.json > /tmp/client.crt
jq -r '.data.private_key' /tmp/client-cert.json > /tmp/client.key

Install on Target

# Transfer files
scp /tmp/domus-root-ca.crt /tmp/client.crt /tmp/client.key user@target:/tmp/

# On target - install certificates
sudo cp /tmp/domus-root-ca.crt /etc/ssl/certs/DOMUS-ROOT-CA.pem
sudo cp /tmp/client.crt /etc/ssl/certs/$(hostname)-client.crt
sudo cp /tmp/client.key /etc/ssl/private/$(hostname)-client.key
sudo chmod 600 /etc/ssl/private/$(hostname)-client.key

Create Connection (Critical: All Flags at Creation)

sudo nmcli connection add \
    con-name "Domus-WiFi-EAP-TLS" \
    type wifi \
    ssid "Domus-Secure" \
    wifi-sec.key-mgmt wpa-eap \
    802-1x.eap tls \
    802-1x.identity "$(hostname).inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
    802-1x.client-cert /etc/ssl/certs/$(hostname)-client.crt \
    802-1x.private-key /etc/ssl/private/$(hostname)-client.key \
    802-1x.private-key-password-flags 4

Common Errors Quick Reference

Error Fix

"Secrets were required, but not provided"

Delete connection, recreate with private-key-password-flags 4 at creation

ISE 12520 "unknown CA"

Use ROOT CA (pki/cert/ca), not intermediate (pki_int/…​/issuing_ca)

Wrong VLAN assigned

Check SSID - must be 802.1X enabled SSID, not MAB SSID

Connection hangs

Check ISE rejected endpoints: netapi ise get-rejected-endpoints

Preventive Measures

Short-term (This week)

Action Owner Status

Update CR-2026-03-12 with MODEL process

Evan

[x] Complete

Create this RCA as reference

Evan

[x] Complete

Test MODEL on CHLA research workstation

Evan

[ ] Pending

Long-term (This quarter)

Action Owner Status

Add pre-flight CA verification to 802.1X runbook

Evan

[ ] Pending

Create netapi vault get-root-ca command

Evan

[ ] Pending

Document ROOT vs intermediate CA in Vault PKI docs

Evan

[ ] Pending

Detection

How was it detected?

  • Manual observation - connection failing

  • ISE MNT logs showing error 12520

  • wpa_supplicant journal logs showing TLS failure

Detection Gap

Could have been detected earlier with:

  • Pre-deployment certificate chain verification:

    # Verify CA can validate ISE cert chain
openssl verify -CAfile /etc/ssl/certs/DOMUS-ROOT-CA.pem /path/to/ise-cert.pem

  • NetworkManager connection validation before nmcli con up

Lessons Learned

What went well

  • ISE MNT logs clearly identified the error (12520)

  • Comparing working config (modestus-razer) quickly revealed the difference

  • Documentation was updated immediately as MODEL

What could be improved

  • Should have compared working config FIRST before hours of troubleshooting

  • Should have verified CA chain before deployment

  • nmcli con mod behavior should have been tested

Key Takeaways

  1. ROOT CA for EAP-TLS - Always use ROOT CA, never intermediate, for client CA-cert

  2. Flags at creation - nmcli con mod cannot change private-key-password-flags

  3. Compare working config - When stuck, diff against a known-working system

  4. Vault PKI paths - pki/cert/ca = ROOT, pki_int/…​/issuing_ca = intermediate

Applicability

This RCA applies to:

  • Any Linux system using NetworkManager for 802.1X EAP-TLS

  • Ubuntu, Fedora, RHEL, Arch with NetworkManager

  • Both WiFi and wired 802.1X (same CA requirements)

  • Any Vault PKI environment with ROOT + intermediate CA hierarchy

Known affected systems:

  • P50 ThinkPad (Ubuntu 25.10) - RESOLVED

  • CHLA research Ubuntu workstation - PENDING (same issue suspected)

Related

Metadata

Field Value

RCA ID

RCA-2026-03-16-001

Author

Evan Rosado

Date Created

2026-03-16

Last Updated

2026-03-16

Status

Final

Review Date

2026-04-16 (30 days)