NetworkManager Troubleshooting
Connection States
NetworkManager connections can be in one of these states:
| State | Meaning | Action Required |
|---|---|---|
Active |
Connection is up and running |
None - working correctly |
Activating |
Connection is in progress (authenticating) |
Wait or check authentication logs |
Inactive |
Connection defined but not active |
Use |
Unavailable |
Device not ready or carrier lost |
Check cable/interface status |
Not defined |
Connection profile doesn’t exist |
Create connection profile first |
Common Issues
Wrong Connection Active
Symptoms
$ sudo nmcli connection down Wired-802.1X
Error: 'Wired-802.1X' is not an active connection.
$ nmcli connection show --active
NAME UUID TYPE DEVICE
Wired connection 1 a715eca9-234a-3223-8143-8526abc819b1 ethernet enp130s0 ← Wrong!
HomeRF 56d2bc52-4812-4503-9021-6c404c77a6e1 wifi wlan0
Problem: The interface enp130s0 is using a different connection profile ("Wired connection 1") instead of your 802.1X profile ("Wired-802.1X").
Resolution
Step 1: Down the active connection
# Use the connection NAME or UUID from 'nmcli connection show --active'
sudo nmcli connection down "Wired connection 1"
# Or use UUID
sudo nmcli connection down a715eca9-234a-3223-8143-8526abc819b1
Step 2: Bring up 802.1X connection
sudo nmcli connection up Wired-802.1X
Step 3: Verify correct connection active
nmcli connection show --active | grep enp130s0
# Should show: Wired-802.1X ... ethernet enp130s0
Prevent Auto-Connect Conflicts
If "Wired connection 1" keeps auto-connecting:
# Disable auto-connect on the unwanted profile
sudo nmcli connection modify "Wired connection 1" connection.autoconnect no
# Or delete it entirely if not needed
sudo nmcli connection delete "Wired connection 1"
# Ensure 802.1X profile auto-connects
sudo nmcli connection modify Wired-802.1X connection.autoconnect yes
sudo nmcli connection modify Wired-802.1X connection.autoconnect-priority 100
No Active Connection After Boot
Diagnosis
# Check if connection exists
nmcli connection show Wired-802.1X
# Check interface status
nmcli device status
# Check if interface has carrier (cable plugged in)
cat /sys/class/net/enp130s0/carrier
# Should return: 1
# Check auto-connect setting
nmcli connection show Wired-802.1X | grep autoconnect
Authentication Failure
Symptoms
$ sudo nmcli connection up Wired-802.1X
Error: Connection activation failed: (7) Secrets were required, but not provided.
Or connection activates but drops immediately.
Diagnosis
# Check connection details (look for certificate/key paths)
nmcli connection show Wired-802.1X
# Verify certificate files exist and have correct permissions
ls -l /etc/ssl/private/modestus-razer.key # Should be 600
ls -l /etc/ssl/certs/modestus-razer.pem # Should be 644
# Check NetworkManager logs
sudo journalctl -u NetworkManager -f
# In another terminal, try to connect:
# sudo nmcli connection up Wired-802.1X
Common Authentication Issues
1. Private key password not set or wrong
# Set private key password in connection
sudo nmcli connection modify Wired-802.1X \
802-1x.private-key-password "YourKeyPassword"
# Set password flags to 0 (save password)
sudo nmcli connection modify Wired-802.1X \
802-1x.private-key-password-flags 0
2. Certificate/key mismatch
# Verify certificate and key match
openssl x509 -noout -modulus -in /etc/ssl/certs/modestus-razer.pem | openssl md5
openssl rsa -noout -modulus -in /etc/ssl/private/modestus-razer.key | openssl md5
# MD5 hashes should match
3. CA certificate not trusted
# Verify CA cert path in connection
nmcli connection show Wired-802.1X | grep ca-cert
# Check CA cert exists
ls -l /etc/ssl/certs/HOME-ROOT-CA.pem
Interface Not Managed by NetworkManager
Symptoms
$ nmcli device status
DEVICE TYPE STATE CONNECTION
enp130s0 ethernet unmanaged -- ← Unmanaged!
Resolution
Check if disabled in config:
# Check NetworkManager config
grep -r "unmanaged-devices" /etc/NetworkManager/
# If found, edit the config file
sudo vi /etc/NetworkManager/NetworkManager.conf
# Remove or comment out:
# [keyfile]
# unmanaged-devices=interface-name:enp130s0
# Restart NetworkManager
sudo systemctl restart NetworkManager
Set device as managed:
sudo nmcli device set enp130s0 managed yes
Connection Exists But Won’t Activate
Symptoms
$ sudo nmcli connection up Wired-802.1X
Error: Connection activation failed: No suitable device found for this connection.
Diagnosis
# Check connection binding
nmcli connection show Wired-802.1X | grep interface-name
# Check available devices
nmcli device status
# Check MAC address restrictions
nmcli connection show Wired-802.1X | grep mac-address
Resolution
If interface name mismatch:
# Update connection to use correct interface
sudo nmcli connection modify Wired-802.1X connection.interface-name enp130s0
If MAC address locked to different interface:
# Clear MAC address restriction
sudo nmcli connection modify Wired-802.1X ethernet.mac-address ""
# Or set to correct MAC
sudo nmcli connection modify Wired-802.1X ethernet.mac-address 98:bb:1e:1f:a7:13
Deep 802.1X Troubleshooting
Check Switch Port Authentication Status
Symptoms
Connection appears active but network access doesn’t work, or connection shows as "activating" indefinitely.
Switch-Side Verification
# Check authentication session on switch port
netapi ios exec "show authentication sessions interface GigabitEthernet1/0/5"
# Detailed authentication status
netapi ios exec "show access-session interface GigabitEthernet1/0/5 details"
Example output showing failed authentication:
$ netapi ios exec "show access-s int g1/0/5 d"
Interface: GigabitEthernet1/0/5
MAC Address: 98bb.1e1f.a713
IPv4 Address: 10.50.40.104
User-Name: modestus-razer.{domain}
Status: Unauthorized ← Authentication failed!
Domain: DATA
Oper host mode: multi-auth
Common Session ID: 0A32010A00000605C5D04B6B
Current Policy: PMAP_DefaultWiredDot1xClosedAuth_1X_MAB
Method status list:
Method State
dot1x Authc Failed ← EAP-TLS authentication failed
mab Stopped
Key fields:
| Field | Meaning |
|---|---|
|
Authentication failed - no network access |
|
Authentication succeeded - full access |
|
EAP-TLS authentication problem (cert/key issue) |
|
802.1X authentication successful |
|
MAC Authentication Bypass not attempted |
If Status Shows "Unauthorized"
This means the switch received authentication failure from ISE.
Check ISE side:
# Source credentials
source <(DSEC_SECURITY_MODE=permissive ~/.secrets/bin/dsec source d000 dev/network)
# Check authentication logs
netapi ise mnt auth-logs 98:bb:1e:1f:a7:13
# Check for failed authentications
netapi ise mnt failed --limit 10
# Get detailed failure reason
netapi ise dc recent --limit 5
Common causes:
-
Certificate expired or not yet valid
-
Private key password wrong or not set
-
Certificate/key mismatch
-
ISE doesn’t trust CA certificate
-
Certificate doesn’t match identity (hostname mismatch)
Network Connectivity Testing
Quick Port Connectivity Test
# Test if RADIUS server (ISE) is reachable
timeout 5 nc -zv 10.50.1.21 1812 2>&1 # RADIUS auth
timeout 5 nc -zv 10.50.1.21 1813 2>&1 # RADIUS accounting
# Test switch management
timeout 5 nc -zv 10.50.1.10 22 2>&1 # SSH to switch
# Test gateway
timeout 5 nc -zv 10.50.10.1 22 2>&1
# Test DNS
timeout 5 nc -zv 10.50.1.1 53 2>&1
Comprehensive Connectivity Matrix
#!/bin/bash
# network-connectivity-test.sh
declare -A hosts=(
["Gateway"]="10.50.10.1"
["Switch"]="10.50.1.10"
["ISE"]="10.50.1.21"
["DC"]="10.50.1.50"
["pfSense"]="10.50.1.1"
)
echo "=== Network Connectivity Test ==="
echo ""
for name in "${!hosts[@]}"; do
ip="${hosts[$name]}"
printf "%-20s %-15s " "$name" "$ip"
if timeout 2 ping -c 1 "$ip" &>/dev/null; then
echo -e "\033[0;32m✓ ICMP OK\033[0m"
else
echo -e "\033[0;31m✗ ICMP FAILED\033[0m"
fi
done
echo ""
echo "=== RADIUS Connectivity ==="
printf "%-35s " "ISE RADIUS Auth (UDP/1812)"
if timeout 2 nc -zu 10.50.1.21 1812 2>&1 | grep -q succeeded; then
echo -e "\033[0;32m✓ OK\033[0m"
else
echo -e "\033[0;31m✗ FAILED\033[0m"
fi
printf "%-35s " "ISE RADIUS Acct (UDP/1813)"
if timeout 2 nc -zu 10.50.1.21 1813 2>&1 | grep -q succeeded; then
echo -e "\033[0;32m✓ OK\033[0m"
else
echo -e "\033[0;31m✗ FAILED\033[0m"
fi
EAP-TLS Packet Capture
For deep authentication troubleshooting:
# Capture 802.1X EAPOL packets
sudo tcpdump -i enp130s0 -w /tmp/eapol.pcap 'ether proto 0x888e'
# In another terminal, trigger authentication
sudo nmcli connection down Wired-802.1X
sudo nmcli connection up Wired-802.1X
# Stop capture (Ctrl+C) and analyze
wireshark /tmp/eapol.pcap
What to look for in Wireshark:
-
EAPOL Start- Client initiates -
EAP Request/Identity- Switch requests identity -
EAP Response/Identity- Client sends identity -
EAP Request/TLS- TLS handshake begins -
EAP TLS: Certificate- Client sends certificate -
EAP Success- Authentication succeeded -
EAP Failure- Authentication failed (check ISE logs for reason)
Complete Diagnostic Script
#!/bin/bash
# nm-diagnostic.sh - Complete NetworkManager 802.1X diagnostics
CONN_NAME="Wired-802.1X"
INTERFACE="enp130s0"
echo "=== NetworkManager 802.1X Diagnostics ==="
echo ""
echo "1. Connection Status"
echo "--------------------"
nmcli connection show --active | grep -E "NAME|$INTERFACE|$CONN_NAME" || echo " No active connection on $INTERFACE"
echo ""
echo "2. Device Status"
echo "----------------"
nmcli device status | grep -E "DEVICE|$INTERFACE"
echo ""
echo "3. Carrier Status (cable plugged in?)"
echo "--------------------------------------"
if [[ -f /sys/class/net/$INTERFACE/carrier ]]; then
carrier=$(cat /sys/class/net/$INTERFACE/carrier)
if [[ "$carrier" == "1" ]]; then
echo " ✓ Carrier detected (cable connected)"
else
echo " ✗ No carrier (cable unplugged or interface down)"
fi
else
echo " ✗ Interface $INTERFACE not found"
fi
echo ""
echo "4. Connection Profile Details"
echo "------------------------------"
if nmcli connection show "$CONN_NAME" &>/dev/null; then
echo " Connection exists"
echo " Auto-connect: $(nmcli connection show "$CONN_NAME" | grep 'connection.autoconnect:' | awk '{print $2}')"
echo " Interface: $(nmcli connection show "$CONN_NAME" | grep 'connection.interface-name' | awk '{print $2}')"
else
echo " ✗ Connection '$CONN_NAME' not found"
fi
echo ""
echo "5. Certificate Files"
echo "--------------------"
for file in /etc/ssl/private/modestus-razer.key \
/etc/ssl/certs/modestus-razer.pem \
/etc/ssl/certs/HOME-ROOT-CA.pem; do
if [[ -f "$file" ]]; then
perms=$(stat -c "%a" "$file")
echo " ✓ $file (perms: $perms)"
else
echo " ✗ $file (missing)"
fi
done
echo ""
echo "6. Recent NetworkManager Logs"
echo "------------------------------"
sudo journalctl -u NetworkManager -n 20 --no-pager
echo ""
echo "=== Troubleshooting Commands ==="
echo ""
echo "If wrong connection active:"
echo " sudo nmcli connection down 'Wired connection 1'"
echo " sudo nmcli connection up '$CONN_NAME'"
echo ""
echo "If authentication fails:"
echo " sudo journalctl -u NetworkManager -f"
echo " # Then in another terminal:"
echo " sudo nmcli connection up '$CONN_NAME'"
echo ""
echo "If no connection:"
echo " sudo nmcli connection modify '$CONN_NAME' connection.autoconnect yes"
echo " sudo nmcli connection up '$CONN_NAME'"
Quick Reference
Essential Commands
# Show all connections
nmcli connection show
# Show active connections only
nmcli connection show --active
# Show device status
nmcli device status
# Up/Down connections
sudo nmcli connection up Wired-802.1X
sudo nmcli connection down Wired-802.1X
# Force down active connection on interface
sudo nmcli connection down "Wired connection 1"
# Show connection details
nmcli connection show Wired-802.1X
# Edit connection interactively
sudo nmcli connection edit Wired-802.1X
# Modify connection property
sudo nmcli connection modify Wired-802.1X \
connection.autoconnect yes
# Reload all connection profiles
sudo nmcli connection reload
# Restart NetworkManager
sudo systemctl restart NetworkManager
Real-World Experiences
PKI Migration: CA Certificate Chain Validation Failure
Scenario
After migrating from legacy PKI (HOME-ROOT-CA) to enterprise PKI (DOMUS-ROOT-CA), Linux workstations failed 802.1X authentication despite ISE successfully validating client certificates.
Environment
-
ISE: ise-02.inside.domusdigitalis.dev (ISE 3.3)
-
PKI: HashiCorp Vault (DOMUS-ROOT-CA → DOMUS-ISSUING-CA)
-
Clients: Arch Linux with NetworkManager/wpa_supplicant
Symptoms
TLS: Certificate verification failed, error 19 (self-signed certificate in certificate chain)
depth 2 for '/C=US/O=Domus Digitalis/OU=Enterprise PKI/CN=DOMUS-ROOT-CA'
err='self-signed certificate in certificate chain'
SSL: SSL3 alert: write (local SSL3 detected an error):fatal:unknown CA
OpenSSL: openssl_handshake - SSL_connect error:0A000086:SSL routines::certificate verify failed
Key observation: Error 19 indicates the client does not trust the Root CA that signed ISE’s server certificate.
Root Cause Analysis
In EAP-TLS mutual authentication, both parties must trust each other’s certificate chain:
| Direction | Server validates | Client validates |
|---|---|---|
ISE → Client |
Client certificate signed by trusted CA |
Working (ISE had new CA in trust store) |
Client → ISE |
N/A |
ISE server certificate signed by trusted CA |
The Problem: NetworkManager connection profile still referenced the old CA certificate path:
nmcli connection show Wired-802.1X | grep "802-1x.ca-cert"
# Output: 802-1x.ca-cert: /etc/ssl/certs/HOME-ROOT-CA.pem ← OLD CA
Resolution
# Download CA from Vault (PEM format)
ssh certmgr-01 "vault read -field=certificate pki/cert/ca" > /tmp/DOMUS-ROOT-CA.crt
# Install to system trust store (Arch Linux)
sudo cp /tmp/DOMUS-ROOT-CA.crt /etc/ca-certificates/trust-source/anchors/DOMUS-ROOT-CA.pem
sudo update-ca-trust
# Verify installation
trust list | grep -i domus
# Update CA certificate path
sudo nmcli connection modify Wired-802.1X 802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem
# Reconnect
nmcli connection up Wired-802.1X
CTRL-EVENT-EAP-PEER-CERT depth=2 subject='CN=DOMUS-ROOT-CA'
CTRL-EVENT-EAP-PEER-CERT depth=1 subject='CN=DOMUS-ISSUING-CA'
CTRL-EVENT-EAP-PEER-CERT depth=0 subject='CN=ise-02.inside.domusdigitalis.dev'
CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
Secondary Issue: Client Certificate from Old PKI
If the client certificate was also issued by the old PKI, ISE will reject it even after updating the CA on the client.
openssl x509 -in /etc/ssl/certs/client-eaptls.pem -noout -issuer
# Output: issuer=CN=HOME-ROOT-CA ← OLD PKI
# On certmgr-01 (Vault server)
vault write -format=json pki_int/issue/domus-client \
common_name="<hostname>.inside.domusdigitalis.dev" \
ttl="8760h" > /tmp/<hostname>.json
# Extract certificate and key
jq -r '.data.certificate' /tmp/<hostname>.json > /tmp/<hostname>-eaptls.pem
jq -r '.data.private_key' /tmp/<hostname>.json > /tmp/<hostname>-eaptls.key
# Deploy to client
scp /tmp/<hostname>-eaptls.* <client>:/tmp/
ssh <client> "sudo cp /tmp/<hostname>-eaptls.pem /etc/ssl/certs/"
ssh <client> "sudo cp /tmp/<hostname>-eaptls.key /etc/ssl/private/"
ssh <client> "sudo chmod 644 /etc/ssl/certs/<hostname>-eaptls.pem"
ssh <client> "sudo chmod 600 /etc/ssl/private/<hostname>-eaptls.key"
# Update NetworkManager connection
ssh <client> "sudo nmcli connection modify Wired-802.1X \
802-1x.client-cert /etc/ssl/certs/<hostname>-eaptls.pem \
802-1x.private-key /etc/ssl/private/<hostname>-eaptls.key"
Key Lessons
|
PKI Migration Checklist for 802.1X Clients
Common Mistakes
|
Switch-Side Verification
After successful authentication, verify on switch:
netapi ios exec "show access-session interface GigabitEthernet1/0/5 details"
Interface: GigabitEthernet1/0/5
MAC Address: 98bb.1e1f.a713
IPv4 Address: 10.50.10.130
User-Name: <hostname>.inside.domusdigitalis.dev
Status: Authorized
Domain: DATA
Method status list:
Method State
dot1x Authc Success ← EAP-TLS successful
mab Stopped
Intermittent Connection Drops: Dual DHCP Client Conflict
Symptoms
# Interface shows TWO IP addresses
$ ip addr show wlan0
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 14:f6:d8:7b:31:80 brd ff:ff:ff:ff:ff:ff
inet 10.50.10.121/24 brd 10.50.10.255 scope global dynamic noprefixroute wlan0
valid_lft 6517sec preferred_lft 6517sec
inet 10.50.10.135/24 brd 10.50.10.255 scope global secondary dynamic noprefixroute wlan0 ← SECOND IP
valid_lft 6521sec preferred_lft 5621sec
User Experience:
-
Connection works initially after boot
-
Drops internet connectivity intermittently (every 10-30 minutes)
-
Running
sudo nmcli connection down/up Domus-Securetemporarily fixes it -
Problem recurs after next DHCP lease renewal
Diagnosis
Check for multiple DHCP clients:
# Check for dhcpcd processes
ps aux | grep dhcpcd | grep -v grep
# Expected: Nothing (if using NetworkManager)
# Problem: Shows dhcpcd running on wlan0
Example conflict output:
$ ps aux | grep dhcpcd
dhcpcd 1834 0.0 0.0 9108 4640 ? Ss 19:05 0:00 dhcpcd: wlan0 [ip4] [ip6]
root 1835 0.0 0.0 8964 2600 ? S 19:05 0:00 dhcpcd: [privileged proxy] wlan0
dhcpcd 76495 0.0 0.0 8964 2340 ? S 22:38 0:00 dhcpcd: [BPF ARP] wlan0 10.50.10.135
dhcpcd 76539 0.0 0.0 8964 2276 ? S 22:38 0:00 dhcpcd: [BOOTP proxy] 10.50.10.135
Check dhcpcd service status:
systemctl status dhcpcd@wlan0
# Problem: Shows "active (running)" and "enabled"
Root Cause
Both dhcpcd and NetworkManager are running DHCP clients on the same interface:
| DHCP Client | IP Leased | Behavior |
|---|---|---|
NetworkManager (internal) |
10.50.10.121 |
Requests DHCP, manages routing |
dhcpcd@wlan0.service |
10.50.10.135 |
Also requests DHCP, conflicts with NetworkManager |
Result |
Dual IPs, routing conflicts |
Connection drops when leases renew/conflict |
Why This Happens:
Arch Linux ships with dhcpcd enabled by default. When NetworkManager is installed on top, both services try to manage the same interface, causing:
-
Duplicate DHCP requests to the same DHCP server
-
Conflicting default routes
-
Race conditions during lease renewals
-
Intermittent connectivity loss
Resolution
Step 1: Stop dhcpcd
sudo systemctl stop dhcpcd@wlan0
Step 2: Disable dhcpcd (prevent restart on boot)
sudo systemctl disable dhcpcd@wlan0
Step 3: Mask dhcpcd (prevent any auto-start)
sudo systemctl mask dhcpcd
Step 4: Kill any remaining dhcpcd processes
sudo pkill -9 dhcpcd
Step 5: Remove secondary IP address manually
sudo ip addr del 10.50.10.135/24 dev wlan0
Step 6: Restart NetworkManager connection
sudo nmcli connection down Domus-Secure
sleep 2
sudo nmcli connection up Domus-Secure
Verification
Check for single IP:
ip addr show wlan0 | grep "inet "
# Should show ONLY: inet 10.50.10.121/24
Check for dhcpcd processes:
ps aux | grep dhcpcd | grep -v grep
# Should return nothing
Verify dhcpcd is masked:
systemctl status dhcpcd@wlan0
# Should show: "masked (Reason: Unit dhcpcd.service is masked.)"
Expected Result
$ ip addr show wlan0
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 14:f6:d8:7b:31:80 brd ff:ff:ff:ff:ff:ff
inet 10.50.10.121/24 brd 10.50.10.255 scope global dynamic noprefixroute wlan0 ← ONLY ONE IP
valid_lft 7143sec preferred_lft 7143sec
inet6 fe80::5ff8:e039:a27d:1bb1/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Connection should now be stable without intermittent drops.
If Still Dropping After Fix
If connection continues to drop after removing dhcpcd, investigate other causes:
Check NetworkManager logs:
journalctl -u NetworkManager -f
Look for:
-
EAP-TLS authentication failures (certificate issues)
-
ISE CoA (Change of Authorization) / reauthentication messages
-
DHCP lease renewal failures
-
Driver issues (check
dmesgfor wireless driver errors)
Check ISE session (if using 802.1X):
# If using netapi
netapi ise mnt session <MAC_ADDRESS>
# Check for CoA/reauthentication events
netapi ise dc auth-history <MAC_ADDRESS> --limit 20
Test without 802.1X:
# Temporarily connect to open network to isolate 802.1X vs networking issue
sudo nmcli device wifi connect "Open-Network"
# If stable on open network → 802.1X/ISE issue
# If still dropping → driver/hardware issue
Prevention
For fresh Arch Linux installs using NetworkManager:
# During initial setup, mask dhcpcd before enabling NetworkManager
sudo systemctl mask dhcpcd
sudo systemctl enable NetworkManager
sudo systemctl start NetworkManager
Check for conflicting network services:
# List all networking-related services
systemctl list-units --type=service --state=active | grep -E 'network|dhcp|wpa'
# Only NetworkManager should be active for WiFi/Ethernet management
Common conflicting services:
-
dhcpcd.serviceordhcpcd@<interface>.service -
systemd-networkd.service -
netctlprofiles -
Manual
wpa_supplicantinstances
Best practice: Choose one network manager (NetworkManager) and disable all others.