Vault PKI Quick Reference

Overview

Step-by-step guide to migrate a Linux workstation to Vault PKI certificates.

Workflow: vault-01 (issue) → modestus-razer (relay) → target workstation (install)

Prerequisites

  • SSH access to vault-01 from modestus-razer

  • SSH access to target workstation from modestus-razer

  • Target workstation IP address (hostname may not resolve)

Step 0: Check Current State

Run on target workstation:

openssl x509 -in /etc/ssl/certs/${HOST}-eaptls.pem -issuer -subject -noout 2>/dev/null || echo "No cert"
nmcli connection show | grep -E "ethernet|wifi|802"
Issuer Action

CN=DOMUS-ISSUING-CA

Already migrated. Skip to Step 5.

CN=HOME-ISSUING-CA

Old AD CS. Continue with Step 1.

No cert found

Fresh install. Continue with Step 1.

Step 1: Issue Certificate on vault-01

On modestus-razer:

dsource d000 dev/vault
ssh vault-01

On vault-01:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<paste-token>'
vault status

Issue and extract certificate (replace <hostname> with target hostname):

HOSTNAME="<hostname>"
vault write -format=json pki_int/issue/domus-client \
  common_name="${HOSTNAME}.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/${HOSTNAME}-cert.json

jq -r '.data.certificate' /tmp/${HOSTNAME}-cert.json > /tmp/${HOSTNAME}.crt
jq -r '.data.private_key' /tmp/${HOSTNAME}-cert.json > /tmp/${HOSTNAME}.key
jq -r '.data.ca_chain[]' /tmp/${HOSTNAME}-cert.json > /tmp/domus-ca-chain.crt
chmod 600 /tmp/${HOSTNAME}.key
rm /tmp/${HOSTNAME}-cert.json
Example (modestus-aw)
HOSTNAME="modestus-aw"
vault write -format=json pki_int/issue/domus-client \
  common_name="${HOSTNAME}.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/${HOSTNAME}-cert.json

jq -r '.data.certificate' /tmp/${HOSTNAME}-cert.json > /tmp/${HOSTNAME}.crt
jq -r '.data.private_key' /tmp/${HOSTNAME}-cert.json > /tmp/${HOSTNAME}.key
jq -r '.data.ca_chain[]' /tmp/${HOSTNAME}-cert.json > /tmp/domus-ca-chain.crt
chmod 600 /tmp/${HOSTNAME}.key
rm /tmp/${HOSTNAME}-cert.json

Verify issuer and exit:

openssl x509 -in /tmp/${HOSTNAME}.crt -noout -subject -issuer
exit

Step 2: Transfer Certificates via modestus-razer

On modestus-razer - Pull from vault-01:

HOSTNAME="<hostname>"
scp vault-01:/tmp/${HOSTNAME}.crt /tmp/
scp vault-01:/tmp/${HOSTNAME}.key /tmp/
scp vault-01:/tmp/domus-ca-chain.crt /tmp/
Example (modestus-aw)
HOSTNAME="modestus-aw"
scp vault-01:/tmp/${HOSTNAME}.crt /tmp/
scp vault-01:/tmp/${HOSTNAME}.key /tmp/
scp vault-01:/tmp/domus-ca-chain.crt /tmp/

On modestus-razer - Push to target workstation (use IP if hostname doesn’t resolve):

TARGET_IP="<target-ip>"
scp /tmp/${HOSTNAME}.crt ${TARGET_IP}:/tmp/
scp /tmp/${HOSTNAME}.key ${TARGET_IP}:/tmp/
scp /tmp/domus-ca-chain.crt ${TARGET_IP}:/tmp/
scp /etc/ssl/certs/DOMUS-ROOT-CA.pem ${TARGET_IP}:/tmp/
Example (modestus-aw at 10.50.10.107)
TARGET_IP="10.50.10.107"
scp /tmp/$<hostname>.crt $<target-ip>:/tmp/
scp /tmp/$<hostname>.key $<target-ip>:/tmp/
scp /tmp/domus-ca-chain.crt $<target-ip>:/tmp/
scp /etc/ssl/certs/DOMUS-ROOT-CA.pem $<target-ip>:/tmp/

Step 3: Install Certificates on Target Workstation

On target workstation:

HOSTNAME="$HOST"  # or: HOSTNAME="$(cat /etc/hostname)"
sudo cp /tmp/${HOSTNAME}.crt /etc/ssl/certs/${HOSTNAME}-eaptls.pem
sudo cp /tmp/${HOSTNAME}.key /etc/ssl/private/${HOSTNAME}-eaptls.key
sudo chmod 600 /etc/ssl/private/${HOSTNAME}-eaptls.key
sudo cp /tmp/domus-ca-chain.crt /etc/ssl/certs/DOMUS-CA-CHAIN.pem
sudo cp /tmp/DOMUS-ROOT-CA.pem /etc/ssl/certs/DOMUS-ROOT-CA.pem

Verify chain:

openssl verify -CAfile /etc/ssl/certs/DOMUS-CA-CHAIN.pem /etc/ssl/certs/${HOSTNAME}-eaptls.pem

Verify cert and key match (must have same hash):

openssl x509 -noout -modulus -in /etc/ssl/certs/${HOSTNAME}-eaptls.pem | md5sum
sudo openssl rsa -noout -modulus -in /etc/ssl/private/${HOSTNAME}-eaptls.key | md5sum

Step 4: Configure Wired 802.1X

On target workstation:

HOSTNAME="$HOST"  # or: HOSTNAME="$(cat /etc/hostname)"
sudo nmcli connection add type ethernet con-name "Wired-802.1X-Vault" \
  802-1x.eap tls \
  802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
  802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
  802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
  802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem

sudo nmcli connection modify "Wired-802.1X-Vault" \
  802-1x.private-key-password "" \
  802-1x.private-key-password-flags 4

Step 5: Test Authentication

Terminal 1 - Watch logs:

journalctl -f -u NetworkManager -u wpa_supplicant

Terminal 2 - Activate connection:

sudo nmcli connection up "Wired-802.1X-Vault"
Expected Log Output
CTRL-EVENT-EAP-PEER-CERT depth=2 subject='...DOMUS-ROOT-CA'
CTRL-EVENT-EAP-PEER-CERT depth=1 subject='...DOMUS-ISSUING-CA'
CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully

Step 6: Verify in ISE

Load secrets and check active sessions:

dsource d000 dev/network
netapi ise mnt sessions

Quick Auth Check

netapi ise dc query "
SELECT USERNAME, NAS_IP_ADDRESS, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%<hostname>%'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY"
Example (modestus-aw)
netapi ise dc query "
SELECT USERNAME, NAS_IP_ADDRESS, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%modestus-aw%'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY"

NAS_IP_ADDRESS reveals source:

  • 10.50.1.40 = WLC (WiFi)

  • 10.50.1.10 = Switch (Wired)

Full Authorization Details

Shows policy set, authorization rule, and profile applied:

netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, AUTHORIZATION_RULE,
       AUTHORIZATION_PROFILES,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%<hostname>%'
  AND PASSED = 'Pass'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY"
Example (modestus-aw)
netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, AUTHORIZATION_RULE,
       AUTHORIZATION_PROFILES,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%modestus-aw%'
  AND PASSED = 'Pass'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY"

Wired vs WiFi Side-by-Side

Compare both interfaces for same host:

netapi ise dc query "
SELECT USERNAME,
       CASE WHEN NAS_IP_ADDRESS = '10.50.1.40' THEN 'WiFi'
            WHEN NAS_IP_ADDRESS = '10.50.1.10' THEN 'Wired'
            ELSE NAS_IP_ADDRESS END as INTERFACE,
       PASSED, TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%<hostname>%'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"
Example (modestus-aw)
netapi ise dc query "
SELECT USERNAME,
       CASE WHEN NAS_IP_ADDRESS = '10.50.1.40' THEN 'WiFi'
            WHEN NAS_IP_ADDRESS = '10.50.1.10' THEN 'Wired'
            ELSE NAS_IP_ADDRESS END as INTERFACE,
       PASSED, TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%modestus-aw%'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Step 7: Configure WiFi 802.1X (Optional)

On target workstation:

Check existing WiFi connections:

nmcli connection show | grep wifi

Option A: Create New WiFi 802.1X Connection

Use this if no 802.1X WiFi connection exists (e.g., only guest/HomeRF):

HOSTNAME="$HOST"

sudo nmcli connection add type wifi con-name "Domus-Secure-802.1X" \
  ssid "Domus-Secure" \
  wifi-sec.key-mgmt wpa-eap \
  802-1x.eap tls \
  802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
  802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
  802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
  802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
  802-1x.private-key-password "" \
  802-1x.private-key-password-flags 4 && \
sudo nmcli connection up "Domus-Secure-802.1X"

Option B: Modify Existing WiFi 802.1X Connection

Use this if a Domus-Secure-802.1X connection already exists:

HOSTNAME="$HOST"
WIFI_CONN="Domus-Secure-802.1X"

sudo nmcli connection modify "${WIFI_CONN}" \
  802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
  802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
  802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
  802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
  802-1x.private-key-password "" \
  802-1x.private-key-password-flags 4 && \
sudo nmcli connection down "${WIFI_CONN}" && \
sudo nmcli connection up "${WIFI_CONN}"

Step 8: Cleanup

On target workstation:

rm /tmp/*.crt /tmp/*.key /tmp/*.pem 2>/dev/null

On modestus-razer:

rm /tmp/${HOSTNAME}.crt /tmp/${HOSTNAME}.key /tmp/domus-ca-chain.crt 2>/dev/null
ssh vault-01 "rm /tmp/${HOSTNAME}.crt /tmp/${HOSTNAME}.key /tmp/domus-ca-chain.crt 2>/dev/null"

Step 9: Seal Vault (CRITICAL)

Always seal Vault after certificate operations. An unsealed Vault is a security risk - anyone with network access to vault-01 can issue certificates.

On modestus-razer:

ssh vault-01

On vault-01:

export VAULT_ADDR='http://127.0.0.1:8200'
vault operator seal
Expected Output
Success! Vault is sealed.
exit

Workstation Status

Host IP Wired WiFi Certificate

modestus-razer

10.50.10.x

ACTIVE

ACTIVE

Vault PKI

modestus-p50

10.50.10.x

N/A

ACTIVE

Vault PKI

modestus-aw

10.50.10.107

ACTIVE

ACTIVE

Vault PKI

Troubleshooting

Debug wpa_supplicant directly

Run wpa_supplicant manually with debug output to see TLS handshake details:

HOSTNAME="$HOST"
sudo wpa_supplicant -i enp44s0 -D wired -c /dev/stdin -d << EOF
network={
    key_mgmt=IEEE8021X
    eap=TLS
    identity="${HOSTNAME}.inside.domusdigitalis.dev"
    ca_cert="/etc/ssl/certs/DOMUS-ROOT-CA.pem"
    client_cert="/etc/ssl/certs/${HOSTNAME}-eaptls.pem"
    private_key="/etc/ssl/private/${HOSTNAME}-eaptls.key"
}
EOF

Key things to look for:

Success indicators:

EAP-TLS: Done
EAP: Received EAP-Success
CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
State: ASSOCIATED -> COMPLETED
EAPOL: Supplicant port status: Authorized

Failure indicators: * EAP-Request Identity then Received EAP-Failure - ISE rejecting identity (check ISE policy/AD) * SSL_connect:error - TLS handshake failed (certificate chain issue) * 22056 Subject not found - Identity doesn’t exist in AD/identity store

Manual wpa_supplicant only tests authentication - it doesn’t run DHCP. Use NetworkManager for full connectivity.

Verify cert and key match

openssl x509 -noout -modulus -in /etc/ssl/certs/${HOSTNAME}-eaptls.pem | md5sum
sudo openssl rsa -noout -modulus -in /etc/ssl/private/${HOSTNAME}-eaptls.key | md5sum

Hashes must match.

Check ISE for failure reason

netapi ise dc query "
SELECT USERNAME, FAILURE_REASON, POLICY_SET_NAME
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%${HOSTNAME}%'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 5 ROWS ONLY"

Check authentication history by MAC address

Most important troubleshooting query - shows complete auth history for a specific endpoint:

netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%<MAC-with-percent-wildcards>%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"
Example (modestus-aw)
netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%08%92%04%38%11%9C%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

This query reveals:

  • When authentication started working vs failing

  • Which policy set matched

  • Exact failure reasons with timestamps

  • Username changes (e.g., from certificate CN to MAC address indicates MAB fallback)

Check ALL recent authentications (last hour)

netapi ise dc query "
SELECT USERNAME, CALLING_STATION_ID, NAS_IP_ADDRESS,
       FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 20 ROWS ONLY"

Check ALL recent failures

netapi ise dc query "
SELECT USERNAME, FAILURE_REASON, AUTHENTICATION_PROTOCOL, POLICY_SET_NAME
FROM RADIUS_AUTHENTICATIONS
WHERE PASSED = 'Fail'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1/24
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Common failure reasons:

  • 22056 Subject not found in applicable identity store(s) - Identity doesn’t exist in AD/identity store

  • 22045 Identity policy result configured for password but received certificate - Wrong identity source in auth rule

  • 12514 EAP-TLS handshake failed - Certificate chain or trust issue

Check switch 802.1X status

netapi ios exec "show access-session interface gi1/0/X details"

Success output:

Status:  Authorized
Domain:  DATA
Vlan Group:  Vlan: 10
dot1x   Authc Success

Failure indicators:

  • Status: Unauthorized - Authentication failed

  • dot1x: Running - Client is attempting authentication

  • dot1x: Stopped - Client not responding or auth failed

  • Domain: UNKNOWN - Not yet authenticated

Check if endpoint is rejected by ISE

After too many failed attempts, ISE may reject the endpoint entirely:

netapi ise get-rejected-endpoints

Release the endpoint to allow authentication again:

netapi ise release-rejected "<MAC-ADDRESS>"
Example
netapi ise release-rejected "08:92:04:38:11:9C"

Check identity store issues (22056 error)

Error 22056 "Subject not found in the applicable identity store(s)" means the certificate identity cannot be found in AD or the configured identity store.

Key diagnostic query - shows identity store and group for each attempt:

netapi ise dc query "
SELECT USERNAME, IDENTITY_STORE, IDENTITY_GROUP, PASSED, FAILURE_REASON,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%<MAC-with-percent-wildcards>%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"
Example (modestus-aw)
netapi ise dc query "
SELECT USERNAME, IDENTITY_STORE, IDENTITY_GROUP, PASSED, FAILURE_REASON,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'YYYY-MM-DD HH24:MI:SS') as TIMESTAMP
FROM RADIUS_AUTHENTICATIONS
WHERE CALLING_STATION_ID LIKE '%08%92%04%38%11%9C%'
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

This reveals:

  • IDENTITY_STORE empty = cert auth (expected)

  • IDENTITY_GROUP = Profiled = working

  • USERNAME = MAC address = MAB fallback (EAP-TLS failed)

  • USERNAME = "USERNAME" (literal) = EAP-TLS rejected before identity extraction

Common causes:

  • Computer object doesn’t exist in AD

  • Certificate Authentication Profile is looking for AD group membership

  • Wrong identity store sequence in authentication policy

  • Certificate CN doesn’t match AD computer name

Forensics: Check ISE config changes

See what changed in ISE (endpoint moves, policy edits, etc.):

netapi ise dc query "
SELECT
    TO_CHAR(ACS_TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS') as TIME,
    ADMIN_NAME,
    OBJECT_TYPE,
    OBJECT_NAME,
    OPERATION_MESSAGE_TEXT
FROM MNT.CONFIG_CHANGE
WHERE ACS_TIMESTAMP > SYSDATE - 1
  AND OBJECT_TYPE IS NOT NULL
ORDER BY ACS_TIMESTAMP DESC
FETCH FIRST 20 ROWS ONLY"

Forensics: Check specific time window

Investigate exactly what happened during a failure window:

netapi ise dc query "
SELECT USERNAME, IDENTITY_STORE, FAILURE_REASON, PASSED,
       TO_CHAR(TIMESTAMP_TIMEZONE, 'HH24:MI:SS') as TIME
FROM RADIUS_AUTHENTICATIONS
WHERE TIMESTAMP_TIMEZONE BETWEEN
      TO_TIMESTAMP('2026-02-08 19:28:00', 'YYYY-MM-DD HH24:MI:SS')
  AND TO_TIMESTAMP('2026-02-08 19:30:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY TIMESTAMP_TIMEZONE"

Adjust the timestamps to your incident window.

Clear stuck session on switch

If ISE shows no recent logs but client is getting EAP-Failure, bounce the port:

netapi ios config "interface g1/0/X" "shutdown" && sleep 2 && netapi ios config "interface g1/0/X" "no shutdown"

Compare with working client

netapi ise dc query "
SELECT USERNAME, POLICY_SET_NAME, AUTHORIZATION_RULE
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%modestus-razer%'
  AND PASSED = 'Pass'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 1
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 3 ROWS ONLY"

Check identity groups for all clients

netapi ise dc query "
SELECT USERNAME, IDENTITY_GROUP
FROM RADIUS_AUTHENTICATIONS
WHERE USERNAME LIKE '%modestus%'
  AND PASSED = 'Pass'
  AND TIMESTAMP_TIMEZONE > SYSDATE - 7
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 10 ROWS ONLY"

Working clients should show Workstation or MGMT_DEVICES identity group.

Check NetworkManager 802.1X config

sudo nmcli -s connection show "Wired-802.1X-Vault" | grep 802-1x

Key settings:

  • 802-1x.private-key-password-flags: 4 (not required) - No password prompt

  • All cert paths should exist

Rollback

If authentication fails:

sudo nmcli connection delete "Wired-802.1X-Vault"
sudo nmcli connection up "Wired-802.1X"

See Also