Phase 8b: 802.1X EAP-TLS

Phase 8b: 802.1X EAP-TLS Authentication

This section migrates the P16g from iPSK (DOMUS-IoT) to certificate-based 802.1X EAP-TLS (DOMUS-Secure) for both wired and wireless. After this, the P16g authenticates like the Razer — mutual TLS with Vault-issued certificates, no shared keys.

Prerequisites

  • Vault cluster running and unsealed (vault-01/vault-02)

  • Vault PKI intermediate (pki_int) with domus-client role configured

  • ISE trusts DOMUS-ISSUING-CA in its certificate store

  • ISE policy set configured for EAP-TLS

  • WiFi backend switched from iwd to wpa_supplicant

Verify Razer’s Working Config (Reference)

Before configuring the P16g, confirm what the Razer uses — this is the target state.

# From the Razer — check the cert that ISE trusts
sudo openssl x509 -in /etc/ssl/certs/modestus-razer-eaptls.pem -noout -subject -issuer
Actual output (modestus-razer)
subject=O=Domus-Infrastructure, OU=Domus-Admins, CN=modestus-razer.inside.domusdigitalis.dev
issuer=CN=DOMUS-ISSUING-CA
# Verify the nmcli connections
nmcli c s | grep -E "Domus-Wired|Domus-WiFi"
Actual output (modestus-razer)
Domus-Wired-Mgmt-VLAN100      3f9fefb8-4da8-4a45-9c86-971b20afee42  ethernet  enp130s0
Domus-WiFi-Data-VLAN10       23672874-9cc0-48cf-9c72-d0e0dee53a7a  wifi      wlan0

Load Vault Credentials

ds d000 dev/vault
vault status
Expected output (Vault healthy)
Seal Type       shamir
Initialized     true
Sealed          false
HA Enabled      true
Active Node     https://vault-02.inside.domusdigitalis.dev:8200

Issue Certificate from Vault PKI

Full Workflow

# 1. Load Vault credentials FIRST — without this, vault defaults to localhost:8200
ds d000 dev/vault
# 2. Set the TARGET hostname
#    Local issuance — auto-detect from hostnamectl:
HOSTNAME=$(hostnamectl --static)

#    Remote issuance (e.g., issuing P16g cert from the Razer) — set explicitly:
#    HOSTNAME="modestus-p16g"

echo "Issuing cert for: ${HOSTNAME}.inside.domusdigitalis.dev"
# 3. Issue certificate — tee saves full JSON AND pipes summary to screen
#    Full JSON → /tmp/${HOSTNAME}-vault-cert.json (cert + key for extraction)
#    Summary  → /tmp/${HOSTNAME}-vault-summary.json (safe to view later)
vault write -format=json pki_int/issue/domus-client \
    common_name="${HOSTNAME}.inside.domusdigitalis.dev" \
    ttl=8760h \
    | tee /tmp/${HOSTNAME}-vault-cert.json \
    | jq '{common_name: .data.common_name, serial: .data.serial_number, expiration: .data.expiration}' \
    > /tmp/${HOSTNAME}-vault-summary.json
# 4. View the summary (doesn't expose private key)
cat /tmp/${HOSTNAME}-vault-summary.json | jq .

Extract Certificate Components with jq

$HOSTNAME should still be set from the issuance step above. Verify before extracting:
echo "Extracting for: $HOSTNAME"
# If empty, set it again: HOSTNAME="modestus-p16g"
# Client certificate
jq -r '.data.certificate' /tmp/${HOSTNAME}-vault-cert.json >| /tmp/${HOSTNAME}-eaptls.pem
# Private key
jq -r '.data.private_key' /tmp/${HOSTNAME}-vault-cert.json >| /tmp/${HOSTNAME}-eaptls.key
# CA chain (issuing CA + root CA)
jq -r '.data.ca_chain[]' /tmp/${HOSTNAME}-vault-cert.json >| /tmp/DOMUS-CA-CHAIN.pem
# Issuing CA only (optional — chain file is usually sufficient)
jq -r '.data.issuing_ca' /tmp/${HOSTNAME}-vault-cert.json >| /tmp/DOMUS-ISSUING-CA.pem

Verify Issued Certificate

# Subject, issuer, and validity dates
openssl x509 -in /tmp/${HOSTNAME}-eaptls.pem -noout -subject -issuer -dates
# CRITICAL: Must include "TLS Web Client Authentication" — without it, ISE silently rejects
openssl x509 -in /tmp/${HOSTNAME}-eaptls.pem -noout -text | grep -A1 "Extended Key Usage"
# Serial number (save this — needed for revocation)
jq -r '.data.serial_number' /tmp/${HOSTNAME}-vault-cert.json
# Expiration as human-readable date
jq -r '.data.expiration' /tmp/${HOSTNAME}-vault-cert.json | xargs -I {} date -d @{}

Verify Key Matches Certificate

# Both MD5 hashes MUST match — mismatched key/cert is a common cause of "TLS handshake failed"
CERT_MOD=$(openssl x509 -in /tmp/${HOSTNAME}-eaptls.pem -noout -modulus | md5sum | cut -d' ' -f1)
KEY_MOD=$(openssl rsa -in /tmp/${HOSTNAME}-eaptls.key -noout -modulus 2>/dev/null | md5sum | cut -d' ' -f1)
echo "Cert: $CERT_MOD"
echo "Key:  $KEY_MOD"
[[ "$CERT_MOD" == "$KEY_MOD" ]] && echo "MATCH" || echo "MISMATCH — DO NOT PROCEED"

Install Certificates

# HOSTNAME should still be set from above — verify: echo $HOSTNAME
# If empty: HOSTNAME="modestus-p16g"
# Install client cert
sudo cp /tmp/${HOSTNAME}-eaptls.pem /etc/ssl/certs/
sudo chmod 644 /etc/ssl/certs/${HOSTNAME}-eaptls.pem
# Install private key (root-only read)
sudo cp /tmp/${HOSTNAME}-eaptls.key /etc/ssl/private/
sudo chmod 600 /etc/ssl/private/${HOSTNAME}-eaptls.key
sudo chown root:root /etc/ssl/private/${HOSTNAME}-eaptls.key
# Install CA chain
sudo cp /tmp/DOMUS-CA-CHAIN.pem /etc/ssl/certs/
sudo chmod 644 /etc/ssl/certs/DOMUS-CA-CHAIN.pem
# Verify chain validates
openssl verify -CAfile /etc/ssl/certs/DOMUS-CA-CHAIN.pem /etc/ssl/certs/${HOSTNAME}-eaptls.pem
# Cleanup temp files
rm -f /tmp/${HOSTNAME}-vault-cert.json /tmp/${HOSTNAME}-eaptls.* /tmp/DOMUS-*.pem

Configure WiFi Backend (wpa_supplicant)

Enterprise 802.1X requires wpa_supplicant, not iwd. Arch defaults to iwd — fix this before creating connections.

sudo mkdir -p /etc/NetworkManager/conf.d
echo -e "[device]\nwifi.backend=wpa_supplicant" | sudo tee /etc/NetworkManager/conf.d/wifi_backend.conf
# Disable iwd completely
sudo systemctl stop iwd 2>/dev/null
sudo systemctl disable iwd 2>/dev/null
sudo systemctl mask iwd
# Enable wpa_supplicant
sudo systemctl enable wpa_supplicant
sudo systemctl start wpa_supplicant
sudo systemctl restart NetworkManager

Wired Management Access (Open Mode)

The wired 802.1X profile requires an administrator certificate with OU=Domus-Admins for ISE to assign VLAN 100. Before configuring 802.1X, you need management subnet access to verify the Vault PKI role and re-issue the cert if needed. This connection also serves as a backup path to infrastructure.

Switch ports are normally in 802.1X closed mode. Configure the target port for open mode before proceeding.
# Find the wired interface
WIRED_IF=$(ip -o link show | awk -F': ' '/state UP/ && !/lo|wlan|docker|br-|veth/ {print $2; exit}')
echo "Wired interface: $WIRED_IF"
# Scan for a free IP on the management subnet (no DHCP — static only)
for ip in 200 201 202 203 204 205; do
    ping -c 1 -W 1 10.50.1.$ip >/dev/null 2>&1 && \
    echo "10.50.1.$ip IN USE" || echo "10.50.1.$ip FREE"
done
# Create open-mode wired connection with static IP
# Replace <FREE_IP> with a verified free address from the scan above
sudo nmcli connection add \
    type ethernet \
    con-name "Domus-Wired-Open" \
    ifname "$WIRED_IF" \
    ipv4.method manual \
    ipv4.addresses "10.50.1.<FREE_IP>/24" \
    ipv4.gateway "10.50.1.1" \
    ipv4.dns "10.50.1.90" \
    connection.autoconnect no
nmcli connection up "Domus-Wired-Open"
# Verify management subnet access — Vault should respond
curl -sk https://vault-01.inside.domusdigitalis.dev:8200/v1/sys/health | jq '.sealed'

Verify Vault PKI Role (OU Configuration)

The ISE authorization policy assigns VLAN 100 when Certificate.Subject-OU == "Domus-Admins". Check whether the Vault role includes this OU.

ds d000 dev/vault
vault read pki_int/roles/domus-client

Look for organization and ou fields in the output. If they are empty, the cert was issued without OU — which is why the P16g cert only has a CN:

# Current (missing OU):
subject=CN=modestus-p16g.inside.domusdigitalis.dev

# Required (for VLAN 100 assignment):
subject=O=Domus-Infrastructure, OU=Domus-Admins, CN=modestus-p16g.inside.domusdigitalis.dev

If the role lacks organization/ou, update the role with the full field set. vault write on a role replaces the entire configuration — every field not specified resets to its default. Include all required fields:

vault write pki_int/roles/domus-client \
    allowed_domains="inside.domusdigitalis.dev" \
    allow_subdomains=true \
    allow_ip_sans=true \
    enforce_hostnames=true \
    require_cn=true \
    client_flag=true \
    server_flag=true \
    ext_key_usage="ClientAuth" \
    key_usage="DigitalSignature,KeyEncipherment" \
    key_type="rsa" \
    key_bits=2048 \
    max_ttl="8760h" \
    organization="Domus-Infrastructure" \
    ou="Domus-Admins" \
    signature_bits=256 \
    no_store=false

vault write replaces, it does not merge. Omitting a field resets it to the default. Verify the full role after updating:

vault read pki_int/roles/domus-client

Confirm allowed_domains, ext_key_usage, organization, ou, and max_ttl are all correct before issuing certs.

Then re-issue the cert using the workflow in Issue Certificate from Vault PKI above, and reinstall it before proceeding to the 802.1X wired profile.

Create Wired 802.1X Connection

Once the P16g cert has OU=Domus-Admins, configure the wired 802.1X profile with a static IP for VLAN 100 (MANAGEMENT_VLAN). No DHCP — the management subnet uses static addressing by design.

# HOSTNAME should still be set from above — verify: echo $HOSTNAME
# If empty: HOSTNAME="modestus-p16g"

# Find the wired interface name
WIRED_IF=$(ip -o link show | awk -F': ' '/state UP/ && !/lo|wlan/ {print $2; exit}')
echo "Wired interface: $WIRED_IF"
sudo nmcli connection add \
    type ethernet \
    con-name "Domus-Wired-Mgmt-VLAN100" \
    ifname "$WIRED_IF" \
    802-1x.eap tls \
    802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-CA-CHAIN.pem \
    802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
    802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
    802-1x.private-key-password-flags 4 \
    802-1x.password-flags 4 \
    ipv4.method manual \
    ipv4.addresses "10.50.1.<STATIC_IP>/24" \
    ipv4.gateway "10.50.1.1" \
    ipv4.dns "10.50.1.90" \
    connection.autoconnect yes
nmcli connection up "Domus-Wired-Mgmt-VLAN100"

private-key-password-flags 4 AND password-flags 4 — BOTH are CRITICAL for passwordless EAP-TLS. Without them, NetworkManager prompts for secrets that don’t exist: "Secrets were required, but not provided." private-key-password-flags covers the key passphrase; password-flags covers the EAP password. Both must be set to 4 (not required).

identity-flags — removed. Not supported on newer NetworkManager versions (1.48+). Identity is stored in the connection file by default.

Static IP — MANAGEMENT_VLAN (10.50.1.0/24) has no DHCP. Replace <STATIC_IP> with the same IP used for Domus-Wired-Open, or a different reserved address. ISE assigns VLAN 100 via RADIUS (Tunnel-Private-Group-ID), and the static IP must be on that subnet.

Once 802.1X is verified, remove the open-mode connection:

nmcli connection delete "Domus-Wired-Open"

Create WiFi 802.1X Connection

# HOSTNAME should still be set from above — verify: echo $HOSTNAME
# If empty: HOSTNAME="modestus-p16g"
sudo nmcli connection add \
    type wifi \
    con-name "Domus-WiFi-Data-VLAN10" \
    ifname wlan0 \
    ssid "Domus-Secure" \
    wifi-sec.key-mgmt wpa-eap \
    802-1x.eap tls \
    802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-CA-CHAIN.pem \
    802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
    802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
    802-1x.private-key-password-flags 4 \
    802-1x.password-flags 4 \
    connection.autoconnect yes

Do NOT bounce the WiFi connection from an SSH session over that same WiFi.

nmcli connection down kills the WiFi link — your SSH session dies instantly and connection up never runs. You’re locked out until you walk to the machine.

Option A: Run locally — open Kitty on the P16g desktop:

nmcli connection down "Domus-WiFi-Data-VLAN10" && nmcli connection up "Domus-WiFi-Data-VLAN10"

Option B: From SSH — write a script and run it with nohup so it survives the disconnect:

echo 'sleep 2 && nmcli connection down "Domus-WiFi-Data-VLAN10" && nmcli connection up "Domus-WiFi-Data-VLAN10"' > /tmp/bounce-wifi.sh
chmod +x /tmp/bounce-wifi.sh
nohup /tmp/bounce-wifi.sh &

Then wait 30 seconds and SSH back in with the new IP (DHCP may reassign).

Option C: Tee the command to a file, run locally:

echo 'nmcli connection down "Domus-WiFi-Data-VLAN10" && nmcli connection up "Domus-WiFi-Data-VLAN10"' > /tmp/bounce-wifi.sh
chmod +x /tmp/bounce-wifi.sh
# Then run /tmp/bounce-wifi.sh from the P16g's local terminal
Do NOT use identity-flags for WiFi — causes "invalid property" error. WiFi stores identity in the connection file by default.

Create WiFi Admin Connection (VLAN 100)

The default Domus-WiFi-Data-VLAN10 uses DHCP and lands on DATA VLAN (10.50.10.x) — intended for regular work. For infrastructure admin access over WiFi, create a separate connection with a static IP on the management subnet.

Same cert, same SSID, different IP config. Switch between them depending on what you need.

Set hostname
HOSTNAME=$(hostnamectl --static)
Create the WiFi admin connection with static IP for VLAN 100
sudo nmcli connection add \
    type wifi \
    con-name "Domus-WiFi-Mgmt-VLAN100" \
    ifname wlan0 \
    ssid "Domus-Secure" \
    wifi-sec.key-mgmt wpa-eap \
    802-1x.eap tls \
    802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-CA-CHAIN.pem \
    802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
    802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
    802-1x.private-key-password-flags 4 \
    802-1x.password-flags 4 \
    wifi.cloned-mac-address permanent \
    ipv4.method manual \
    ipv4.addresses "10.50.1.204/24" \
    ipv4.gateway "10.50.1.1" \
    ipv4.dns "10.50.1.90,10.50.1.91" \
    connection.autoconnect no

autoconnect no — the admin WiFi profile should not auto-activate. Use Domus-WiFi-Data-VLAN10 (DATA VLAN, DHCP) for daily work. Switch to Domus-WiFi-Mgmt-VLAN100 (MGMT VLAN, static) only when you need infrastructure access over WiFi.

wifi.cloned-mac-address permanent — disables MAC randomization. Required for EAP-TLS — ISE tracks the real MAC for session management.

Verify ISE Rule Coverage

The Domus_Cert_Admin_P16g rule in Domus_8021X matches on CERTIFICATE:Subject - Common Name contains "p16g" — this covers both wired and WiFi since both use the same cert. Verify the WiFi MAC is not in the rejected list before activating.

Check WiFi MAC auth history
netapi ise dc auth-history e0:d5:5d:6c:e1:66 --hours 24
Check rejected endpoints
netapi ise get-rejected-endpoints

Switching Between Profiles

Regular work — DATA VLAN (DHCP, internet + DNS)
nmcli connection up "Domus-WiFi-Data-VLAN10"
Infrastructure admin — MGMT VLAN 100 (static, full access)
nmcli connection up "Domus-WiFi-Mgmt-VLAN100"
Check which is active and what IP you have
nmcli connection show --active | grep -E "Domus-WiFi"
ip -4 addr show wlan0 | awk '/inet / {print $2}'

WiFi Profile Summary

Field Domus-WiFi-Data-VLAN10 Domus-WiFi-Mgmt-VLAN100

Purpose

Daily work

Infrastructure admin

VLAN

DATA (10.50.10.x)

MGMT 100 (10.50.1.x)

IP Method

DHCP

Static (10.50.1.204/24)

Autoconnect

Yes

No

ISE Profile

Domus_Secure_Profile

Domus_Admin_Profile

DACL

None

DACL_ADMIN_FULL

Do NOT bounce the WiFi admin connection from an SSH session over that same WiFi. Same risk as the regular WiFi — connection down kills your SSH instantly. Run locally or use the nohup pattern documented above.

Verify All Connections

nmcli connection show --active | grep -E "Domus-Wired|Domus-WiFi"
# Verify 802.1X settings — clean summary of non-default fields
# Usage: replace connection name as needed
nmcli c s "Domus-Wired-Mgmt-VLAN100" | awk -F: '
BEGIN { prev = "" }
/^(connection\.|802-1x\.|802-3-ethernet\.|802-11-wireless|wifi|ipv4\.)/ && !/proxy|ipv6/ {
    val = ""
    for (i = 2; i <= NF; i++) { if (i > 2) val = val ":"; val = val $i }
    gsub(/^[ \t]+/, "", val)
    if (val == "" || val == "--" || val == "no" || val == "yes" || \
        val == "unknown" || val == "default" || val == "auto" || \
        val == "infrastructure" || val ~ /^0/ || val ~ /^-1/ || \
        val == "<hidden>" || \
        $1 ~ /phase[12]/ || $1 ~ /password-raw/ || $1 ~ /pin/ || \
        $1 ~ /wep-key/ || $1 ~ /leap-/ || $1 ~ /psk/ || \
        $1 ~ /\.timestamp$/ || $1 ~ /seen-bssids/ || \
        $1 ~ /\.uuid$/ || $1 ~ /deprecated/ || $1 ~ /may-fail/ || \
        $1 ~ /\.optional$/) next
    split($1, p, "."); s = p[1]
    if (s != prev) { if (prev != "") print ""; printf "=== %s ===\n", toupper(s); prev = s }
    printf "  %-45s %s\n", $1, val
}'
# Check IP — should be on DOMUS-Secure VLAN, not IoT VLAN
ip -4 addr show | awk '/inet / && !/127.0.0.1/ {print $NF, $2}'

ISE Verification (from Razer)

ds d000 dev/network
WIRED_MAC="a8:2b:dd:8f:23:e6"
WIFI_MAC="e0:d5:5d:6c:e1:66"
# Wired — should show dot1x/EAP-TLS, not mab
netapi ise mnt session $WIRED_MAC
netapi ise dc auth-history $WIRED_MAC --hours 1
# WiFi — should show dot1x/EAP-TLS
netapi ise mnt session $WIFI_MAC
netapi ise dc auth-history $WIFI_MAC --hours 1
# Raw SQL — confirm method changed from mab to dot1x
netapi ise dc query "
SELECT mac_address, authentication_method, passed, timestamp
FROM radius_authentications
WHERE mac_address IN ('$WIRED_MAC', '$WIFI_MAC')
  AND timestamp > SYSDATE - 1
ORDER BY timestamp DESC
"

Expected Results

Field Wired (Admin) WiFi (Data) WiFi (Admin)

Connection

Domus-Wired-Mgmt-VLAN100

Domus-WiFi-Data-VLAN10

Domus-WiFi-Mgmt-VLAN100

Auth Method

dot1x

dot1x

dot1x

Protocol

EAP-TLS

EAP-TLS

EAP-TLS

Identity

modestus-p16g.inside.domusdigitalis.dev

modestus-p16g.inside.domusdigitalis.dev

modestus-p16g.inside.domusdigitalis.dev

Cert Issuer

DOMUS-ISSUING-CA

DOMUS-ISSUING-CA

DOMUS-ISSUING-CA

NAD

Home-3560CX-01 (10.50.1.10)

Home-9800-WLC (10.50.1.40)

Home-9800-WLC (10.50.1.40)

Policy Set

Domus_8021X

Domus_8021X

Domus_8021X

ISE Rule

Domus_Cert_Admin_P16g

Domus_Cert_Users_Exact

Domus_Cert_Admin_P16g

AuthZ Profile

Domus_Admin_Profile

Domus_Secure_Profile

Domus_Admin_Profile

VLAN

100 (MANAGEMENT_VLAN)

10 (DATA_VLAN, DHCP)

100 (MANAGEMENT_VLAN)

DACL

DACL_ADMIN_FULL

None

DACL_ADMIN_FULL

IP Method

Static (10.50.1.203/24)

DHCP

Static (10.50.1.204/24)

Autoconnect

Yes

Yes

No

Remove iPSK Connection

Once EAP-TLS is verified on WiFi:

nmcli connection delete "DOMUS-IoT"
# Verify only EAP-TLS connections remain
nmcli connection show | grep -E "Domus|EAP"

Troubleshooting

# Live logs — NetworkManager + wpa_supplicant
journalctl -u NetworkManager -t wpa_supplicant -f | grep -iE "eap|tls|auth|fail"
# Certificate chain verification
openssl verify -CAfile /etc/ssl/certs/DOMUS-CA-CHAIN.pem /etc/ssl/certs/$(hostname)-eaptls.pem
# Key/cert match (both hashes must be identical)
openssl x509 -in /etc/ssl/certs/$(hostname)-eaptls.pem -noout -modulus | md5sum
sudo openssl rsa -in /etc/ssl/private/$(hostname)-eaptls.key -noout -modulus | md5sum
# Bounce connection
nmcli connection down "Domus-WiFi-Data-VLAN10" && nmcli connection up "Domus-WiFi-Data-VLAN10"
Issue Fix

"Secrets required but not provided"

Missing private-key-password-flags 4 AND/OR password-flags 4. Both must be set to 4 (not required) for passwordless EAP-TLS. Fix: nmcli connection modify "<name>" 802-1x.private-key-password-flags 4 802-1x.password-flags 4

"TLS handshake failed"

CA chain incomplete — ensure DOMUS-CA-CHAIN.pem contains both ISSUING-CA and ROOT-CA

"Authentication rejected"

Certificate CN doesn’t match ISE identity — verify with openssl x509 -noout -subject

"invalid property identity-flags"

Remove identity-flags — not supported on NetworkManager 1.48+. Identity is stored in the connection file by default.

Interface disappeared after disabling iwd

Reload WiFi driver: sudo modprobe -r iwlmvm mac80211 iwlwifi && sudo modprobe iwlwifi

Wrong VLAN assigned

Check ISE authorization policy order — netapi ise get-authz-rules "Corp WIFI"

DOT1X-5-FAIL on switch, all auth attempts fail

Check ISE rejected endpoints: netapi ise get-rejected-endpoints. If MAC is listed, release it: netapi ise release-rejected <MAC>. Endpoints rejected during early failed attempts stay blocked even after config is fixed.

NM logs show missing ca_cert and identity in supplicant config

Delete and recreate the connection — NM sometimes fails to pass all 802-1x fields to wpa_supplicant. Verify the .nmconnection file directly: sudo cat /etc/NetworkManager/system-connections/<name>.nmconnection

Vault vault write on PKI role wiped all fields

vault write on a role replaces the entire config. Always include every field. Verify with vault read pki_int/roles/domus-client immediately after writing.

Static IP unreachable after 802.1X auth

ISE may assign a different VLAN than expected. If static IP is on MGMT subnet (10.50.1.x) but ISE assigns DATA VLAN (10.50.10.x), the IP is on the wrong subnet. Switch to DHCP temporarily or fix the ISE authorization policy first.

Wired 802.1X Troubleshooting Session (Apr 14, 2026)

Complete command sequence from the P16g wired 802.1X bring-up. Included for reference and portfolio documentation.

1. Vault Unseal (Both Nodes)

# Load credentials
ds d000 dev/vault
vault status   # Showed: Sealed true

# Unseal vault-01
vault operator unseal   # Key 1
vault operator unseal   # Key 2
# Result: Sealed false, Active Node: vault-02

# vault-02 was still sealed — each Raft node unseals independently
export VAULT_ADDR="https://vault-02.inside.domusdigitalis.dev:8200"
vault operator unseal   # Key 1
vault operator unseal   # Key 2
# Result: Sealed false

2. Vault PKI Role Fix

# Check role — organization and ou were empty []
vault read pki_int/roles/domus-client

# Fix: vault write REPLACES entire role — must include ALL fields
vault write pki_int/roles/domus-client \
    allowed_domains="inside.domusdigitalis.dev" \
    allow_subdomains=true \
    allow_ip_sans=true \
    enforce_hostnames=true \
    require_cn=true \
    client_flag=true \
    server_flag=true \
    ext_key_usage="ClientAuth" \
    key_usage="DigitalSignature,KeyEncipherment" \
    key_type="rsa" \
    key_bits=2048 \
    max_ttl="8760h" \
    organization="Domus-Infrastructure" \
    ou="Domus-Admins" \
    signature_bits=256 \
    no_store=false

# Verify — organization [Domus-Infrastructure], ou [Domus-Admins]
vault read pki_int/roles/domus-client

3. Certificate Re-Issuance

HOSTNAME=$(hostnamectl --static)
echo "Issuing cert for: ${HOSTNAME}.inside.domusdigitalis.dev"

vault write -format=json pki_int/issue/domus-client \
    common_name="${HOSTNAME}.inside.domusdigitalis.dev" \
    ttl=8760h \
    | tee /tmp/${HOSTNAME}-vault-cert.json \
    | jq '{common_name: .data.common_name, serial: .data.serial_number, expiration: .data.expiration}' \
    > /tmp/${HOSTNAME}-vault-summary.json

# Verify OU present
jq -r '.data.certificate' /tmp/${HOSTNAME}-vault-cert.json \
    | openssl x509 -noout -subject
# Output: subject=O=Domus-Infrastructure, OU=Domus-Admins, CN=modestus-p16g.inside.domusdigitalis.dev

# Verify EKU
jq -r '.data.certificate' /tmp/${HOSTNAME}-vault-cert.json \
    | openssl x509 -noout -text | grep -A1 "Extended Key Usage"
# Output: TLS Web Server Authentication, TLS Web Client Authentication

# Extract, install, verify chain (see full workflow above)

4. ISE Rejected Endpoint

# Discovery — MAC in rejected list from earlier failed auth attempts
netapi ise get-rejected-endpoints
# Output: A8:2B:DD:8F:23:E6 | EndPoint

# Release
netapi ise release-rejected A8:2B:DD:8F:23:E6
# Output: ✓ Released rejected endpoint: A8:2B:DD:8F:23:E6

5. Wired 802.1X Connection

HOSTNAME=$(hostnamectl --static)
WIRED_IF="enp134s0"

sudo nmcli connection add \
    type ethernet \
    con-name "Domus-Wired-Mgmt-VLAN100" \
    ifname "$WIRED_IF" \
    802-1x.eap tls \
    802-1x.identity "${HOSTNAME}.inside.domusdigitalis.dev" \
    802-1x.ca-cert /etc/ssl/certs/DOMUS-CA-CHAIN.pem \
    802-1x.client-cert /etc/ssl/certs/${HOSTNAME}-eaptls.pem \
    802-1x.private-key /etc/ssl/private/${HOSTNAME}-eaptls.key \
    802-1x.private-key-password-flags 4 \
    802-1x.password-flags 4 \
    ipv4.method manual \
    ipv4.addresses "10.50.1.203/24" \
    ipv4.gateway "10.50.1.1" \
    ipv4.dns "10.50.1.90" \
    connection.autoconnect yes

nmcli connection up "Domus-Wired-Mgmt-VLAN100"

6. Switch Verification

LAB-3560CX-01#sh access-sess int g1/0/4 d
          MAC Address:  a82b.dd8f.23e6
         IPv4 Address:  10.50.1.203
            User-Name:  modestus-p16g.inside.domusdigitalis.dev
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
       Session timeout:  28800s (server)
       Current Policy:  PMAP_DefaultWiredDot1xClosedAuth_1X_MAB
       Vlan Group:  Vlan: 10       <-- Expected: 100 (MGMT)

Method status list:
      dot1x              Authc Success
      mab                Stopped

7. Connection Profile Audit

# Clean summary of non-default fields (reusable for any connection)
nmcli c s "Domus-Wired-Mgmt-VLAN100" | awk -F: '
BEGIN { prev = "" }
/^(connection\.|802-1x\.|802-3-ethernet\.|802-11-wireless|wifi|ipv4\.)/ && !/proxy|ipv6/ {
    val = ""
    for (i = 2; i <= NF; i++) { if (i > 2) val = val ":"; val = val $i }
    gsub(/^[ \t]+/, "", val)
    if (val == "" || val == "--" || val == "no" || val == "yes" || \
        val == "unknown" || val == "default" || val == "auto" || \
        val == "infrastructure" || val ~ /^0/ || val ~ /^-1/ || \
        val == "<hidden>" || \
        $1 ~ /phase[12]/ || $1 ~ /password-raw/ || $1 ~ /pin/ || \
        $1 ~ /wep-key/ || $1 ~ /leap-/ || $1 ~ /psk/ || \
        $1 ~ /\.timestamp$/ || $1 ~ /seen-bssids/ || \
        $1 ~ /\.uuid$/ || $1 ~ /deprecated/ || $1 ~ /may-fail/ || \
        $1 ~ /\.optional$/) next
    split($1, p, "."); s = p[1]
    if (s != prev) { if (prev != "") print ""; printf "=== %s ===\n", toupper(s); prev = s }
    printf "  %-45s %s\n", $1, val
}'

# Ground truth — the actual connection file
sudo cat /etc/NetworkManager/system-connections/Domus-Wired-Mgmt-VLAN100.nmconnection

8. ISE Policy Investigation

# List policy sets — find the correct name
netapi ise -f json get-policy-sets | jq '.[] | {name, id, state}'
Actual output
{"name": "Domus_8021X", "id": "056a2880-5821-465f-adb2-90c32de0b06f", "state": "enabled"}
{"name": "Domus_MAB",   "id": "76bba980-befd-45d4-9c2a-4be81ac47f8c", "state": "enabled"}
{"name": "Default",     "id": "3a7f1206-d371-42fe-a845-cd3460535b6e", "state": "enabled"}
# Check the authorization profile that should assign VLAN 100
netapi ise -f json get-authz-profile "Domus_Admin_Profile" | jq '{name, vlan, daclName, reauth}'
Actual output
{
  "name": "Domus_Admin_Profile",
  "vlan": {"nameID": "MANAGEMENT_VLAN", "tagID": 1},
  "daclName": "DACL_ADMIN_FULL",
  "reauth": {"timer": 28800, "connectivity": "RADIUS_REQUEST", "reauthType": "TIMER"}
}
# Full authorization rules for the 802.1X policy set
netapi ise -f json get-authz-rules "Domus_8021X" | jq '.[] | {
    name: .rule.name,
    rank: .rule.rank,
    hits: .rule.hitCounts,
    profile: .profile[0],
    conditions: [.rule.condition | recurse(.children[]?) | select(.attributeName?) | {(.attributeName): .attributeValue}]
}'
Actual output — all 12 rules in evaluation order
Rank  Rule                        Profile                Hits  Conditions
────  ──────────────────────────  ─────────────────────  ────  ──────────────────────────────────────────────
  0   Domus_Cert_Admins           Domus_Admin_Profile      12  EAP-TLS + O contains "Infrastructure"
                                                                + (CN contains "razer" OR CN contains "aw")
  1   Domus_Cert_Users_Exact      Domus_Secure_Profile      5  EAP-TLS + O == "Domus-Infrastructure"
  2   Domus_Cert_Research_Exact   Domus_Research_Profile    0  EAP-TLS + O == "Research"
  3   Domus_BYOD_Devices          Domus_Secure_Profile      7  EAP-TLS + CN contains "byod"
  4   Domus_Cert_Infra_Specific   Domus_Admin_Profile       0  EAP-TLS + O == "Domus-Infrastructure"
  5   Domus_Cert_Research_Specific Domus_Research_Profile   0  EAP-TLS + O == "Domus-Research"
  6   Domus_TEAP_Machine_Only     Domus_Secure_Profile      0  TEAP + User failed, machine succeeded
  7   Domus_TEAP_Chaining         Domus_Secure_Profile      0  TEAP + User and machine both succeeded
  8   Domus_Cert_Users            Domus_Secure_Profile      0  EAP-TLS + O contains "Digitalis"
  9   Domus_Cert_Research         Domus_Research_Profile    0  EAP-TLS + O contains "Domus"
 10   EAP_TLS_Permit              PermitAccess              4  EAP-TLS (any)
 11   Default                     DenyAccess                0  (unconditional)

9. Root Cause Analysis — Wrong VLAN Assignment

Problem: P16g cert has O=Domus-Infrastructure, OU=Domus-Admins but ISE assigns VLAN 10 instead of VLAN 100.

Rule evaluation trace for P16g cert:

  1. Rule 0 (Domus_Cert_Admins) — checks EAP-TLS ✓, O contains "Infrastructure" ✓, then requires CN contains "razer" OR "aw". P16g CN is modestus-p16gMISS. Falls through.

  2. Rule 1 (Domus_Cert_Users_Exact) — checks EAP-TLS ✓, O == "Domus-Infrastructure" ✓ — HIT. Assigns Domus_Secure_Profile → VLAN 10 (DATA).

  3. Rule 4 (Domus_Cert_Infra_Specific) — identical condition to Rule 1, but assigns Domus_Admin_Profile → VLAN 100. Never reached — Rule 1 shadows it.

Design flaw: Rule 0 uses per-host CN matching (razer, aw) instead of certificate OU. When a new workstation (P16g) is added, it doesn’t match the hardcoded hostname list and falls through to the generic user rule.

Correct approach: Replace the CN-based OR block in Rule 0 with CERTIFICATE:Subject - Organizational Unit contains "Admins". This matches all admin workstations by OU — no per-host entries needed. This is the architecture documented in the WRKLOG-2026-03-07 ISE authorization policy logic:

IF Certificate.Subject-OU == "Domus-Admins"
   AND Certificate.Issuer == "DOMUS-ISSUING-CA"
THEN
   VLAN = 100 (MGMT)
   dACL = DACL_ADMIN_FULL

Fix: Update Rule 0 (Domus_Cert_Admins) via netapi — replace the CN OR block with OU-based matching. Rule 4 (Domus_Cert_Infra_Specific) can then be removed as redundant.

10. ISE Authorization Rule Creation — P16g Admin Access

Temporary per-host rule to grant P16g admin VLAN access. Will be consolidated into OU-based matching later.

Target rule:

  • Name: Domus_Cert_Admin_P16g

  • Policy set: Domus_8021X (ID: 056a2880-5821-465f-adb2-90c32de0b06f)

  • Profile: Domus_Admin_Profile (VLAN 100 + DACL_ADMIN_FULL)

  • Condition: EAP-TLS AND CERTIFICATE:Subject - Common Name contains p16g

  • Rank: 0 (first evaluated — before the Razer rule)

Method 1: netapi
# Dry run first — verify payload without creating
netapi ise add-authz-rule "Domus_8021X" "Domus_Cert_Admin_P16g" "Domus_Admin_Profile" \
    --and "Network Access:EapAuthentication:EAP-TLS" \
    --and "CERTIFICATE:Subject - Common Name:contains:p16g" \
    --rank 0 \
    --dry-run
# Create the rule
netapi ise add-authz-rule "Domus_8021X" "Domus_Cert_Admin_P16g" "Domus_Admin_Profile" \
    --and "Network Access:EapAuthentication:EAP-TLS" \
    --and "CERTIFICATE:Subject - Common Name:contains:p16g" \
    --rank 0
# Verify — should appear at rank 0
netapi ise -f json get-authz-rules "Domus_8021X" | jq '.[] | select(.rule.name == "Domus_Cert_Admin_P16g") | {
    name: .rule.name,
    rank: .rule.rank,
    profile: .profile[0],
    conditions: [.rule.condition | recurse(.children[]?) | select(.attributeName?) | {(.attributeName): .attributeValue}]
}'
Method 2: curl (equivalent API call)

ISE OpenAPI endpoint for authorization rules within a policy set.

# Variables — load from dsec or set manually
ds d000 dev/network
POLICY_SET_ID="056a2880-5821-465f-adb2-90c32de0b06f"
# Create authorization rule via ISE OpenAPI
curl -sk -u "${ISE_ERS_USER}:${ISE_ERS_PASSWORD}" \
    "https://${ISE_PAN_FQDN}/api/v1/policy/network-access/policy-set/${POLICY_SET_ID}/authorization" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -X POST \
    -d '{
      "rule": {
        "name": "Domus_Cert_Admin_P16g",
        "rank": 0,
        "state": "enabled",
        "condition": {
          "conditionType": "ConditionAndBlock",
          "isNegate": false,
          "children": [
            {
              "conditionType": "ConditionAttributes",
              "isNegate": false,
              "dictionaryName": "Network Access",
              "attributeName": "EapAuthentication",
              "operator": "equals",
              "attributeValue": "EAP-TLS"
            },
            {
              "conditionType": "ConditionAttributes",
              "isNegate": false,
              "dictionaryName": "CERTIFICATE",
              "attributeName": "Subject - Common Name",
              "operator": "contains",
              "attributeValue": "p16g"
            }
          ]
        }
      },
      "profile": ["Domus_Admin_Profile"]
    }' | jq '.'
# Verify via curl — list all authz rules and filter
curl -sk -u "${ISE_ERS_USER}:${ISE_ERS_PASSWORD}" \
    "https://${ISE_PAN_FQDN}/api/v1/policy/network-access/policy-set/${POLICY_SET_ID}/authorization" \
    -H "Accept: application/json" \
    | jq '.response[] | select(.rule.name | test("P16g")) | {
        name: .rule.name,
        rank: .rule.rank,
        state: .rule.state,
        profile: .profile,
        conditions: [.rule.condition.children[] | {dict: .dictionaryName, attr: .attributeName, op: .operator, val: .attributeValue}]
    }'
jq Reference — ISE Authorization Rule Queries
# Compact summary of all rules (name, rank, profile, hit count)
netapi ise -f json get-authz-rules "Domus_8021X" | jq -r '
    ["RANK","NAME","PROFILE","HITS"],
    (.[] | [.rule.rank, .rule.name, (.profile[0] // "none"), .rule.hitCounts]) | @tsv
' | column -ts $'\t'
# Find which rule a specific cert would match (filter by condition value)
netapi ise -f json get-authz-rules "Domus_8021X" | jq '.[] | select(
    .rule.condition | recurse(.children[]?) | select(.attributeValue? | test("p16g"; "i"))
) | {name: .rule.name, profile: .profile[0]}'
# All rules that assign Domus_Admin_Profile
netapi ise -f json get-authz-rules "Domus_8021X" | jq '.[] | select(
    .profile[] | test("Admin")
) | {name: .rule.name, rank: .rule.rank, hits: .rule.hitCounts}'
# Extract all certificate-based conditions across rules
netapi ise -f json get-authz-rules "Domus_8021X" | jq '.[] | {
    rule: .rule.name,
    cert_conditions: [.rule.condition | recurse(.children[]?) | select(.dictionaryName? == "CERTIFICATE") | {(.attributeName): .attributeValue}]
} | select(.cert_conditions | length > 0)'
Verification After Rule Creation
# 1. Switch to EAP-TLS wired connection with static IP
nmcli connection up "Domus-Wired-Mgmt-VLAN100"
# 2. Check switch — should show VLAN 100, not VLAN 10
# On switch: show access-sessions interface Gi1/0/4 detail
# 3. Verify ISE auth from another machine
ds d000 dev/network
WIRED_MAC="a8:2b:dd:8f:23:e6"
netapi ise dc auth-history $WIRED_MAC --hours 1
netapi ise mnt session $WIRED_MAC
# 4. Test infrastructure reachability from VLAN 100
ping -c 2 10.50.1.1    # Gateway (VyOS VIP)
ping -c 2 10.50.1.70   # NAS
ping -c 2 10.50.1.60   # Vault
ping -c 2 10.50.1.20   # ISE
Future Consolidation

Once verified, consolidate all admin rules into a single OU-based rule:

# Target state: one rule for all admin certs (replaces per-host CN rules)
netapi ise add-authz-rule "Domus_8021X" "Domus_Cert_Admins_OU" "Domus_Admin_Profile" \
    --and "Network Access:EapAuthentication:EAP-TLS" \
    --and "CERTIFICATE:Subject - Organizational Unit:contains:Admins" \
    --rank 0

# Then remove per-host rules:
# netapi ise delete-authz-rule "Domus_8021X" "Domus_Cert_Admin_P16g"
# netapi ise delete-authz-rule "Domus_8021X" "Domus_Cert_Admins"

Outcome

802.1X EAP-TLS authenticated successfully on wired (dot1x: Authc Success). ISE assigned VLAN 10 (DATA) instead of expected VLAN 100 (MGMT) due to authorization rule Domus_Cert_Admins using per-host CN matching instead of OU-based matching. Static IP 10.50.1.203/24 on MGMT subnet unreachable when placed on DATA VLAN. Next step: Create P16g-specific authorization rule, verify VLAN 100 assignment, then consolidate to OU-based matching.