RCA-2026-03-13-001: WiFi EAP-TLS DHCP Failure
Executive Summary
WiFi connection to Domus-Secure SSID fails after successful EAP-TLS authentication. The 802.1X authentication completes, WPA key negotiation succeeds, but IP configuration fails. Root cause: Static IP 10.50.1.200 configured on WiFi connection is already in use by another device (MAC 3C:EC:EF:43:50:42). Additionally, MAC address randomization was enabled, causing ISE session tracking issues.
Timeline
| Time | Event |
|---|---|
2026-03-13 10:54:45 |
EAP-TLS authentication started on wlan0 |
2026-03-13 10:54:46 |
EAP-TLS authentication completed successfully (all certs validated) |
2026-03-13 10:55:06 |
WPA key negotiation completed, CTRL-EVENT-CONNECTED |
2026-03-13 10:55:06 |
(Expected: DHCP DISCOVER sent, DHCP OFFER received) |
2026-03-13 10:55:07 |
CTRL-EVENT-DISCONNECTED reason=250 (DHCP timeout) |
2026-03-13 10:55:07+ |
Retry loop: reconnect attempts, AP adds client to ignore list |
2026-03-13 10:55:41 |
SSID temporarily disabled after multiple failures |
Problem Statement
Symptoms
-
nmcli conn up Domus-WiFi-EAP-TLSfails with "IP configuration could not be reserved" -
EAP-TLS authentication succeeds (verified in wpa_supplicant logs)
-
WPA handshake completes successfully
-
No IP address assigned
-
Connection drops ~1 second after association
-
AP adds client to ignore list due to rapid reconnection attempts
Expected Behavior
After successful EAP-TLS authentication and WPA handshake:
-
Client sends DHCP DISCOVER
-
DHCP server (pfSense) responds with DHCP OFFER
-
Client sends DHCP REQUEST
-
Server responds with DHCP ACK
-
Client configured with IP, gateway, DNS
Actual Behavior
-
EAP-TLS succeeds ✓
-
WPA handshake completes ✓
-
DHCP DISCOVER sent (presumed)
-
No DHCP response received
-
NetworkManager times out after ~1 second
-
Connection dropped with reason=250
Root Cause
5 Whys Analysis
| Why # | Question and Answer |
|---|---|
1 |
Why did the connection fail? |
2 |
Why couldn’t NetworkManager configure the IP? |
3 |
Why was the IP already in use? |
4 |
Why was WiFi configured with a static IP? |
5 |
Why did initial diagnosis point to DHCP? |
Hypotheses (Ordered by Likelihood)
| # | Hypothesis | Verification Method |
|---|---|---|
1 |
VyOS DHCP service down or unresponsive |
SSH to vyos-01/02, check |
2 |
WiFi VLAN 10 (DATA) not trunked properly |
Check VyOS eth1 VLAN subinterfaces, verify AP trunk |
3 |
ISE not assigning VLAN 10 correctly |
|
4 |
AP not passing DHCP to correct VLAN |
Check WLC client details, VLAN assignment |
5 |
DHCP pool exhausted on 10.50.10.0/24 |
Check lease count vs pool size (.100-.199 = 100 addresses) |
6 |
Client MAC not receiving offers |
tcpdump on VyOS to see if DISCOVER arrives |
Root Cause Statement
|
Static IP conflict + MAC randomization
|
Contributing Factors
| Factor | Description | Preventable? |
|---|---|---|
Short DHCP timeout |
NetworkManager default DHCP timeout may be too aggressive |
Yes - increase ipv4.dhcp-timeout |
No fallback connection |
No WPA2-PSK fallback for when EAP-TLS infra fails |
Yes - create hotspot connection |
Wired worked |
Wired EAP-TLS working rules out cert issues, ISE auth |
N/A (diagnostic) |
Impact
Severity
| Metric | Value |
|---|---|
Severity |
P2 - Degraded (wired fallback available) |
Duration |
Ongoing until resolved |
Users/Systems Affected |
1 (modestus-razer WiFi) |
Data Loss |
None |
Business Impact
-
Reduced mobility (must use wired connection)
-
Cannot test WiFi-dependent scenarios
-
Blocks validation of WiFi EAP-TLS deployment
Resolution
Step 1: Identify Conflicting Device
# Find what device owns 3C:EC:EF:43:50:42
# Ping to populate ARP, then check
ping -c 1 -W 1 10.50.1.200
ip neigh | grep "10.50.1.200"
# Result: 3c:ec:ef:43:50:42 (Dell OUI)
Step 1b: Find Free IP in MGMT Range
# Scan for free IPs (no DHCP on MGMT, must use static)
for ip in 201 202 203 204 205; do
ping -c 1 -W 1 10.50.1.$ip >/dev/null 2>&1 && \
echo "10.50.1.$ip IN USE" || echo "10.50.1.$ip FREE"
done
# Result: 10.50.1.201 FREE
Step 2: Choose a Fix
Option A: Use Different Static IP (RECOMMENDED)
# Update WiFi connection to 10.50.1.201 (verified free)
nmcli c mod Domus-WiFi-EAP-TLS ipv4.addresses "10.50.1.201/24"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.gateway "10.50.1.1"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.dns "10.50.1.90,10.50.1.91"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.method manual
Option B: Free Up 10.50.1.200
# If 3C:EC:EF:43:50:42 is your wired interface or another device you control
# Change that device's IP, then WiFi can use 10.50.1.200
Option C: Switch to DHCP (DATA VLAN)
# If MGMT VLAN access not required, use DHCP on DATA VLAN
nmcli c mod Domus-WiFi-EAP-TLS ipv4.method auto
nmcli c mod Domus-WiFi-EAP-TLS ipv4.addresses ""
nmcli c mod Domus-WiFi-EAP-TLS ipv4.gateway ""
nmcli c mod Domus-WiFi-EAP-TLS ipv4.dns ""
Step 3: Fix MAC Randomization
# Disable MAC randomization (required for EAP-TLS)
nmcli c mod Domus-WiFi-EAP-TLS wifi.cloned-mac-address permanent
# Verify
nmcli c show Domus-WiFi-EAP-TLS | grep -i mac
Step 4: Reconnect
nmcli c down Domus-WiFi-EAP-TLS
nmcli c up Domus-WiFi-EAP-TLS
Step 5: Verify Success
# Check IP assigned
ip -4 -o addr show wlan0 | awk '{print $4}'
# Check connectivity
ping -c 3 10.50.1.1
# Verify ISE session (use real MAC now)
netapi ise mnt sessions -d
Diagnostic Commands Used
# View current WiFi config
nmcli c show Domus-WiFi-EAP-TLS | grep -E 'ipv4|mac'
# Get WiFi MAC
nmcli -g GENERAL.HWADDR d show wlan0
# Check NetworkManager logs for IP conflict
journalctl -u NetworkManager --since "10 minutes ago" | grep -E 'wlan0|cannot be configured|already in use'
# Check ISE active sessions
netapi ise mnt sessions -d
# Check ISE auth logs for specific MAC
netapi ise mnt auth-logs <MAC>
# Check VyOS DHCP leases
ssh vyos@10.50.1.2 "show dhcp server leases"
# Check VyOS ARP for IP ownership
ssh vyos@10.50.1.2 "show arp | grep 10.50.1.200"
Immediate Actions (Completed)
-
[x] Documented issue in worklog
-
[x] Created RCA document
-
[x] Identified root cause: IP conflict + MAC randomization
-
[x] Applied fix: Changed static IP to 10.50.1.201, disabled MAC randomization
-
[x] Verified: Connection successfully activated
Verification
After applying fix:
# Confirm connected
nmcli c s -a | grep Domus-WiFi
# Confirm IP assigned
ip -4 addr show wlan0
# Confirm routing
ip route | grep wlan0
# Confirm DNS resolution
dig @10.50.1.90 ise-01.inside.domusdigitalis.dev +short
Preventive Measures
Short-term (This week)
| Action | Owner | Status |
|---|---|---|
Create WPA2-PSK fallback connection (mobile hotspot) |
Evan |
[ ] Pending |
Increase DHCP timeout on WiFi connection |
Evan |
[ ] Pending |
Document WiFi troubleshooting in nmcli.adoc |
Evan |
[ ] Pending |
Long-term (This quarter)
| Action | Owner | Status |
|---|---|---|
Add DHCP monitoring to pfSense |
Evan |
[ ] Pending |
Document WiFi VLAN architecture |
Evan |
[ ] Pending |
Detection
How was it detected?
-
Manual observation - connection failed on
nmcli c up
Detection Gap
Could add monitoring for:
-
DHCP lease assignment success rate
-
WiFi client association/disassociation events
-
802.1X authentication success vs IP assignment success
Lessons Learned
What went well
-
EAP-TLS debugging skills - quickly identified auth was NOT the issue
-
Wired fallback available - work not blocked
-
Log analysis identified DHCP timeout as root cause
What could be improved
-
Need WPA2-PSK fallback for infrastructure failures
-
DHCP timeout default too short
-
No documented troubleshooting runbook for WiFi issues
Key Takeaways
|
Appendix: WPA2-PSK Fallback Connection
When EAP-TLS infrastructure is down, use mobile hotspot:
# CREATE WPA2-PSK CONNECTION (one-time setup)
nmcli c add type wifi con-name "Mobile-Hotspot" \
ssid "YourHotspotSSID" \
wifi-sec.key-mgmt wpa-psk \
wifi-sec.psk "YourPassword" \
ipv4.method auto \
connection.autoconnect no
# CONNECT WHEN NEEDED
nmcli c up Mobile-Hotspot
# DISCONNECT
nmcli c down Mobile-Hotspot
WiFi Operations Reference
This section provides a comprehensive nmcli command reference for WiFi management. Useful for sharing with others or quick lookup.
Profile Management
Create Connections
# WPA2-PSK (Personal/Hotspot) - Most Common
nmcli c add type wifi con-name "Connection-Name" \
ssid "Network-SSID" \
wifi-sec.key-mgmt wpa-psk \
wifi-sec.psk "YourPassword" \
ipv4.method auto \
connection.autoconnect no
# WPA3-Personal (newer, more secure)
nmcli c add type wifi con-name "WPA3-Home" \
ssid "WPA3-Network" \
wifi-sec.key-mgmt sae \
wifi-sec.psk "YourPassword"
# EAP-TLS (802.1X Enterprise)
nmcli c add type wifi con-name "Corp-WiFi" \
ssid "Corporate-SSID" \
wifi-sec.key-mgmt wpa-eap \
802-1x.eap tls \
802-1x.identity "user@domain.com" \
802-1x.client-cert "/etc/ssl/certs/user.crt" \
802-1x.private-key "/etc/ssl/private/user.key" \
802-1x.ca-cert "/etc/ssl/certs/ca.crt" \
wifi.cloned-mac-address permanent
# EAP-PEAP/MSCHAPv2 (username/password enterprise)
nmcli c add type wifi con-name "PEAP-WiFi" \
ssid "Enterprise-SSID" \
wifi-sec.key-mgmt wpa-eap \
802-1x.eap peap \
802-1x.phase2-auth mschapv2 \
802-1x.identity "username" \
802-1x.password "password" \
802-1x.ca-cert "/etc/ssl/certs/ca.crt"
Modify Connections
# Change IP configuration (static)
nmcli c mod "Connection" ipv4.method manual
nmcli c mod "Connection" ipv4.addresses "10.50.1.201/24"
nmcli c mod "Connection" ipv4.gateway "10.50.1.1"
nmcli c mod "Connection" ipv4.dns "10.50.1.90,10.50.1.91"
# Switch to DHCP
nmcli c mod "Connection" ipv4.method auto
nmcli c mod "Connection" ipv4.addresses ""
nmcli c mod "Connection" ipv4.gateway ""
nmcli c mod "Connection" ipv4.dns ""
# Fix MAC randomization (required for EAP-TLS)
nmcli c mod "Connection" wifi.cloned-mac-address permanent
# Enable MAC randomization (privacy for public networks)
nmcli c mod "Connection" wifi.cloned-mac-address random
# Increase DHCP timeout (default is 45s)
nmcli c mod "Connection" ipv4.dhcp-timeout 120
# Set connection priority (higher = preferred)
nmcli c mod "Connection" connection.autoconnect-priority 100
# Set route metric (lower = preferred for routing)
nmcli c mod "Connection" ipv4.route-metric 50
Delete and Export
# Delete connection
nmcli c delete "Connection-Name"
# Export connection to file
nmcli c export "Connection-Name" > connection.nmconnection
# Import connection from file
sudo cp connection.nmconnection /etc/NetworkManager/system-connections/
sudo chmod 600 /etc/NetworkManager/system-connections/connection.nmconnection
sudo nmcli c reload
Connection Lifecycle
# Connect
nmcli c up "Connection-Name"
# Disconnect
nmcli c down "Connection-Name"
# Disconnect and reconnect (reset)
nmcli c down "Connection-Name" && nmcli c up "Connection-Name"
# Enable autoconnect
nmcli c mod "Connection" connection.autoconnect yes
# Disable autoconnect
nmcli c mod "Connection" connection.autoconnect no
Diagnostics and Status
List and Scan
# List all saved connections
nmcli c show
# List active connections only
nmcli c show --active
# Show connection details
nmcli c show "Connection-Name"
# Show WiFi-specific settings
nmcli c show "Connection-Name" | grep -E 'wifi|802-1x|ipv4'
# Scan for available networks
nmcli d wifi rescan && nmcli d wifi list
# Show WiFi device status
nmcli d show wlan0
# Get WiFi interface MAC address
nmcli -g GENERAL.HWADDR d show wlan0
Logs and Troubleshooting
# NetworkManager logs (last 5 minutes)
journalctl -u NetworkManager --since "5 minutes ago"
# Filter for WiFi interface
journalctl -u NetworkManager --since "10 min ago" | grep -E 'wlan0|wifi'
# Look for IP conflicts
journalctl -u NetworkManager --since "10 min ago" | grep -E 'cannot be configured|already in use'
# wpa_supplicant logs (EAP authentication)
journalctl -u wpa_supplicant --since "10 min ago"
# Filter for auth events
journalctl -u wpa_supplicant --since "10 min ago" | grep -E 'CTRL-EVENT|EAP|associated'
# Real-time log monitoring
journalctl -u NetworkManager -f
# Check DNS resolution
dig @10.50.1.90 google.com +short
# Verify IP and routing
ip -4 addr show wlan0
ip route | grep wlan0
Advanced Operations
Network Priority and Metrics
# Set WiFi as backup to wired (higher metric = lower priority)
nmcli c mod "WiFi-Connection" ipv4.route-metric 200
nmcli c mod "Wired-Connection" ipv4.route-metric 100
# Set connection priority (for autoconnect selection)
nmcli c mod "Preferred-WiFi" connection.autoconnect-priority 100
nmcli c mod "Backup-WiFi" connection.autoconnect-priority 50
Hidden Networks
# Connect to hidden SSID
nmcli c add type wifi con-name "Hidden-Network" \
ssid "Hidden-SSID" \
wifi.hidden yes \
wifi-sec.key-mgmt wpa-psk \
wifi-sec.psk "password"
Batch Operations with AWK
# List all WiFi connections
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" {print $1}'
# Delete all WiFi connections matching pattern
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" && $1~/Guest/ {print $1}' | \
xargs -I {} nmcli c delete "{}"
# Export all WiFi connections
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" {print $1}' | \
while read conn; do nmcli c export "$conn" > "${conn}.nmconnection"; done
# List connections with their UUIDs
nmcli -t -f NAME,UUID,TYPE c show | awk -F: '$3=="802-11-wireless" {printf "%-30s %s\n", $1, $2}'
# Find connections using specific SSID
nmcli -t -f NAME,802-11-wireless.ssid c show | awk -F: '$2=="Target-SSID" {print $1}'
Signal Strength Monitoring
# One-shot signal check
nmcli -f SSID,SIGNAL d wifi list | awk 'NR>1 {print $1, $2"%"}' | sort -t'%' -k2 -nr
# Continuous monitoring (every 2 seconds)
watch -n 2 'nmcli -f SSID,SIGNAL,BARS d wifi list'
# Log signal strength over time
while true; do
nmcli -t -f SSID,SIGNAL d wifi list | grep "^YourSSID:" | \
awk -F: -v ts="$(date +%H:%M:%S)" '{print ts, $2"%"}'
sleep 10
done >> wifi-signal.log
Quick Reference Card
| Task | Command |
|---|---|
List saved connections |
|
List available WiFi |
|
Connect to saved profile |
|
Disconnect |
|
Show connection details |
|
Modify connection |
|
Delete connection |
|
Force WiFi rescan |
|
View NetworkManager logs |
|
Get interface MAC |
|
Fix MAC randomization |
|
Metadata
| Field | Value |
|---|---|
RCA ID |
RCA-2026-03-13-001 |
Author |
Evan Rosado |
Date Created |
2026-03-13 |
Last Updated |
2026-03-14 |
Status |
Resolved |
Review Date |
2026-04-12 (30 days from incident) |