RCA-2026-03-13-001: Fix Applied

Resolution

Step 1: Identify Conflicting Device

# Find what device owns 3C:EC:EF:43:50:42
# Ping to populate ARP, then check
ping -c 1 -W 1 10.50.1.200
ip neigh | grep "10.50.1.200"

# Result: 3c:ec:ef:43:50:42 (Dell OUI)

Step 1b: Find Free IP in MGMT Range

# Scan for free IPs (no DHCP on MGMT, must use static)
for ip in 201 202 203 204 205; do
    ping -c 1 -W 1 10.50.1.$ip >/dev/null 2>&1 && \
    echo "10.50.1.$ip IN USE" || echo "10.50.1.$ip FREE"
done

# Result: 10.50.1.201 FREE

Step 2: Choose a Fix

Option A: Use Different Static IP (RECOMMENDED)

# Update WiFi connection to 10.50.1.201 (verified free)
nmcli c mod Domus-WiFi-EAP-TLS ipv4.addresses "10.50.1.201/24"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.gateway "10.50.1.1"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.dns "10.50.1.90,10.50.1.91"
nmcli c mod Domus-WiFi-EAP-TLS ipv4.method manual

Option B: Free Up 10.50.1.200

# If 3C:EC:EF:43:50:42 is your wired interface or another device you control
# Change that device's IP, then WiFi can use 10.50.1.200

Option C: Switch to DHCP (DATA VLAN)

# If MGMT VLAN access not required, use DHCP on DATA VLAN
nmcli c mod Domus-WiFi-EAP-TLS ipv4.method auto
nmcli c mod Domus-WiFi-EAP-TLS ipv4.addresses ""
nmcli c mod Domus-WiFi-EAP-TLS ipv4.gateway ""
nmcli c mod Domus-WiFi-EAP-TLS ipv4.dns ""

Step 3: Fix MAC Randomization

# Disable MAC randomization (required for EAP-TLS)
nmcli c mod Domus-WiFi-EAP-TLS wifi.cloned-mac-address permanent

# Verify
nmcli c show Domus-WiFi-EAP-TLS | grep -i mac

Step 4: Reconnect

nmcli c down Domus-WiFi-EAP-TLS
nmcli c up Domus-WiFi-EAP-TLS

Step 5: Verify Success

# Check IP assigned
ip -4 -o addr show wlan0 | awk '{print $4}'

# Check connectivity
ping -c 3 10.50.1.1

# Verify ISE session (use real MAC now)
netapi ise mnt sessions -d

Diagnostic Commands Used

# View current WiFi config
nmcli c show Domus-WiFi-EAP-TLS | grep -E 'ipv4|mac'

# Get WiFi MAC
nmcli -g GENERAL.HWADDR d show wlan0

# Check NetworkManager logs for IP conflict
journalctl -u NetworkManager --since "10 minutes ago" | grep -E 'wlan0|cannot be configured|already in use'

# Check ISE active sessions
netapi ise mnt sessions -d

# Check ISE auth logs for specific MAC
netapi ise mnt auth-logs <MAC>

# Check VyOS DHCP leases
ssh vyos@10.50.1.2 "show dhcp server leases"

# Check VyOS ARP for IP ownership
ssh vyos@10.50.1.2 "show arp | grep 10.50.1.200"

Immediate Actions (Completed)

  1. [x] Documented issue in worklog

  2. [x] Created RCA document

  3. [x] Identified root cause: IP conflict + MAC randomization

  4. [x] Applied fix: Changed static IP to 10.50.1.201, disabled MAC randomization

  5. [x] Verified: Connection successfully activated

Verification

After applying fix:

# Confirm connected
nmcli c s -a | grep Domus-WiFi

# Confirm IP assigned
ip -4 addr show wlan0

# Confirm routing
ip route | grep wlan0

# Confirm DNS resolution
dig @10.50.1.90 ise-01.inside.domusdigitalis.dev +short

Appendix: WPA2-PSK Fallback Connection

When EAP-TLS infrastructure is down, use mobile hotspot:

# CREATE WPA2-PSK CONNECTION (one-time setup)
nmcli c add type wifi con-name "Mobile-Hotspot" \
  ssid "YourHotspotSSID" \
  wifi-sec.key-mgmt wpa-psk \
  wifi-sec.psk "YourPassword" \
  ipv4.method auto \
  connection.autoconnect no

# CONNECT WHEN NEEDED
nmcli c up Mobile-Hotspot

# DISCONNECT
nmcli c down Mobile-Hotspot

WiFi Operations Reference

This section provides a comprehensive nmcli command reference for WiFi management. Useful for sharing with others or quick lookup.

Workflow Diagram

nmcli WiFi Workflow

Profile Management

Create Connections

# WPA2-PSK (Personal/Hotspot) - Most Common
nmcli c add type wifi con-name "Connection-Name" \
  ssid "Network-SSID" \
  wifi-sec.key-mgmt wpa-psk \
  wifi-sec.psk "YourPassword" \
  ipv4.method auto \
  connection.autoconnect no

# WPA3-Personal (newer, more secure)
nmcli c add type wifi con-name "WPA3-Home" \
  ssid "WPA3-Network" \
  wifi-sec.key-mgmt sae \
  wifi-sec.psk "YourPassword"

# EAP-TLS (802.1X Enterprise)
nmcli c add type wifi con-name "Corp-WiFi" \
  ssid "Corporate-SSID" \
  wifi-sec.key-mgmt wpa-eap \
  802-1x.eap tls \
  802-1x.identity "user@domain.com" \
  802-1x.client-cert "/etc/ssl/certs/user.crt" \
  802-1x.private-key "/etc/ssl/private/user.key" \
  802-1x.ca-cert "/etc/ssl/certs/ca.crt" \
  wifi.cloned-mac-address permanent

# EAP-PEAP/MSCHAPv2 (username/password enterprise)
nmcli c add type wifi con-name "PEAP-WiFi" \
  ssid "Enterprise-SSID" \
  wifi-sec.key-mgmt wpa-eap \
  802-1x.eap peap \
  802-1x.phase2-auth mschapv2 \
  802-1x.identity "username" \
  802-1x.password "password" \
  802-1x.ca-cert "/etc/ssl/certs/ca.crt"

Modify Connections

# Change IP configuration (static)
nmcli c mod "Connection" ipv4.method manual
nmcli c mod "Connection" ipv4.addresses "10.50.1.201/24"
nmcli c mod "Connection" ipv4.gateway "10.50.1.1"
nmcli c mod "Connection" ipv4.dns "10.50.1.90,10.50.1.91"

# Switch to DHCP
nmcli c mod "Connection" ipv4.method auto
nmcli c mod "Connection" ipv4.addresses ""
nmcli c mod "Connection" ipv4.gateway ""
nmcli c mod "Connection" ipv4.dns ""

# Fix MAC randomization (required for EAP-TLS)
nmcli c mod "Connection" wifi.cloned-mac-address permanent

# Enable MAC randomization (privacy for public networks)
nmcli c mod "Connection" wifi.cloned-mac-address random

# Increase DHCP timeout (default is 45s)
nmcli c mod "Connection" ipv4.dhcp-timeout 120

# Set connection priority (higher = preferred)
nmcli c mod "Connection" connection.autoconnect-priority 100

# Set route metric (lower = preferred for routing)
nmcli c mod "Connection" ipv4.route-metric 50

Delete and Export

# Delete connection
nmcli c delete "Connection-Name"

# Export connection to file
nmcli c export "Connection-Name" > connection.nmconnection

# Import connection from file
sudo cp connection.nmconnection /etc/NetworkManager/system-connections/
sudo chmod 600 /etc/NetworkManager/system-connections/connection.nmconnection
sudo nmcli c reload

Connection Lifecycle

# Connect
nmcli c up "Connection-Name"

# Disconnect
nmcli c down "Connection-Name"

# Disconnect and reconnect (reset)
nmcli c down "Connection-Name" && nmcli c up "Connection-Name"

# Enable autoconnect
nmcli c mod "Connection" connection.autoconnect yes

# Disable autoconnect
nmcli c mod "Connection" connection.autoconnect no

Diagnostics and Status

List and Scan

# List all saved connections
nmcli c show

# List active connections only
nmcli c show --active

# Show connection details
nmcli c show "Connection-Name"

# Show WiFi-specific settings
nmcli c show "Connection-Name" | grep -E 'wifi|802-1x|ipv4'

# Scan for available networks
nmcli d wifi rescan && nmcli d wifi list

# Show WiFi device status
nmcli d show wlan0

# Get WiFi interface MAC address
nmcli -g GENERAL.HWADDR d show wlan0

Logs and Troubleshooting

# NetworkManager logs (last 5 minutes)
journalctl -u NetworkManager --since "5 minutes ago"

# Filter for WiFi interface
journalctl -u NetworkManager --since "10 min ago" | grep -E 'wlan0|wifi'

# Look for IP conflicts
journalctl -u NetworkManager --since "10 min ago" | grep -E 'cannot be configured|already in use'

# wpa_supplicant logs (EAP authentication)
journalctl -u wpa_supplicant --since "10 min ago"

# Filter for auth events
journalctl -u wpa_supplicant --since "10 min ago" | grep -E 'CTRL-EVENT|EAP|associated'

# Real-time log monitoring
journalctl -u NetworkManager -f

# Check DNS resolution
dig @10.50.1.90 google.com +short

# Verify IP and routing
ip -4 addr show wlan0
ip route | grep wlan0

Advanced Operations

Network Priority and Metrics

# Set WiFi as backup to wired (higher metric = lower priority)
nmcli c mod "WiFi-Connection" ipv4.route-metric 200
nmcli c mod "Wired-Connection" ipv4.route-metric 100

# Set connection priority (for autoconnect selection)
nmcli c mod "Preferred-WiFi" connection.autoconnect-priority 100
nmcli c mod "Backup-WiFi" connection.autoconnect-priority 50

Hidden Networks

# Connect to hidden SSID
nmcli c add type wifi con-name "Hidden-Network" \
  ssid "Hidden-SSID" \
  wifi.hidden yes \
  wifi-sec.key-mgmt wpa-psk \
  wifi-sec.psk "password"

Batch Operations with AWK

# List all WiFi connections
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" {print $1}'

# Delete all WiFi connections matching pattern
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" && $1~/Guest/ {print $1}' | \
  xargs -I {} nmcli c delete "{}"

# Export all WiFi connections
nmcli -t -f NAME,TYPE c show | awk -F: '$2=="802-11-wireless" {print $1}' | \
  while read conn; do nmcli c export "$conn" > "${conn}.nmconnection"; done

# List connections with their UUIDs
nmcli -t -f NAME,UUID,TYPE c show | awk -F: '$3=="802-11-wireless" {printf "%-30s %s\n", $1, $2}'

# Find connections using specific SSID
nmcli -t -f NAME,802-11-wireless.ssid c show | awk -F: '$2=="Target-SSID" {print $1}'

Signal Strength Monitoring

# One-shot signal check
nmcli -f SSID,SIGNAL d wifi list | awk 'NR>1 {print $1, $2"%"}' | sort -t'%' -k2 -nr

# Continuous monitoring (every 2 seconds)
watch -n 2 'nmcli -f SSID,SIGNAL,BARS d wifi list'

# Log signal strength over time
while true; do
  nmcli -t -f SSID,SIGNAL d wifi list | grep "^YourSSID:" | \
    awk -F: -v ts="$(date +%H:%M:%S)" '{print ts, $2"%"}'
  sleep 10
done >> wifi-signal.log

Quick Reference Card

Task Command

List saved connections

nmcli c show

List available WiFi

nmcli d wifi list

Connect to saved profile

nmcli c up "Name"

Disconnect

nmcli c down "Name"

Show connection details

nmcli c show "Name"

Modify connection

nmcli c mod "Name" key value

Delete connection

nmcli c delete "Name"

Force WiFi rescan

nmcli d wifi rescan

View NetworkManager logs

journalctl -u NetworkManager --since "5 min ago"

Get interface MAC

nmcli -g GENERAL.HWADDR d show wlan0

Fix MAC randomization

nmcli c mod "Name" wifi.cloned-mac-address permanent