RCA-2026-03-09-001: NetworkManager EAP-TLS Static IP

Executive Summary

The Domus-Wired-EAP-TLS connection on modestus-razer lost its static IP configuration (10.50.1.106/24) and had only a single DNS server configured, breaking DNS HA. Root cause: connection was configured with ipv4.method auto (DHCP) instead of ipv4.method manual.

Timeline

Time Event

2026-03-09 ~10:00

Noticed resolv.conf only had single DNS (10.50.1.90)

2026-03-09 ~10:05

Investigated nmcli, found Domus-Wired-EAP-TLS down

2026-03-09 ~10:10

Identified missing static IP configuration

2026-03-09 ~10:15

Applied fix with nmcli

2026-03-09 ~10:16

Verified connectivity and DNS HA

Problem Statement

Symptoms

  • cat /etc/resolv.conf showed only nameserver 10.50.1.90 (missing bind-02)

  • Domus-Wired-EAP-TLS connection was down

  • No static IP assigned to EAP-TLS interface

Expected Behavior

  • Static IP: 10.50.1.106/24

  • Gateway: 10.50.1.1 (VyOS VIP)

  • DNS: 10.50.1.90 AND 10.50.1.91 for HA

Root Cause

5 Whys Analysis

Why # Question and Answer

1

Why was DNS HA broken?
Because: resolv.conf only had one nameserver

2

Why only one nameserver?
Because: NetworkManager connection only had one DNS configured

3

Why only one DNS?
Because: Static IP configuration wasn’t properly saved

4

Why wasn’t static config preserved?
Because: No runbook exists for EAP-TLS connection creation

5

Why no runbook?
Because: EAP-TLS setup was done ad-hoc without documentation

Root Cause Statement: The EAP-TLS connection was missing static IP and dual DNS configuration, likely reset during system update or never fully configured.

Resolution

nmcli con mod "Domus-Wired-EAP-TLS" \
  ipv4.method manual \
  ipv4.addresses "10.50.1.106/24" \
  ipv4.gateway "10.50.1.1" \
  ipv4.dns "10.50.1.90,10.50.1.91"

nmcli con up "Domus-Wired-EAP-TLS"

Verification

ip -4 addr show enp130s0 | awk '/inet/{print $2}'
# Output: 10.50.1.106/24

cat /etc/resolv.conf
# Output: nameserver 10.50.1.90, nameserver 10.50.1.91

CLI Mastery: nmcli Patterns

Connection Inspection

# List all connections with status
nmcli -t -f NAME,UUID,TYPE,DEVICE,STATE con show

# Show specific connection
nmcli con show "Domus-Wired-EAP-TLS"

# Filter to 802.1X and IPv4 settings
nmcli con show "Domus-Wired-EAP-TLS" | grep -E "802-1x|ipv4\."

# Show only 802.1X settings
nmcli -g 802-1x con show "Domus-Wired-EAP-TLS"

EAP-TLS Connection Creation

nmcli con add \
  type ethernet \
  con-name "Domus-Wired-EAP-TLS" \
  ifname enp130s0 \
  ipv4.method manual \
  ipv4.addresses "10.50.1.106/24" \
  ipv4.gateway "10.50.1.1" \
  ipv4.dns "10.50.1.90,10.50.1.91" \
  802-1x.eap tls \
  802-1x.identity "modestus-razer.inside.domusdigitalis.dev" \
  802-1x.ca-cert "/etc/ssl/certs/DOMUS-ROOT-CA.pem" \
  802-1x.client-cert "/etc/ssl/certs/modestus-razer-eaptls.pem" \
  802-1x.private-key "/etc/ssl/private/modestus-razer-eaptls.key" \
  802-1x.private-key-password-flags 4

Modify Existing Connection

# Change IP address
nmcli con mod "Domus-Wired-EAP-TLS" ipv4.addresses "10.50.1.107/24"

# Add second DNS (append with +)
nmcli con mod "Domus-Wired-EAP-TLS" +ipv4.dns "10.50.1.91"

# Replace all DNS
nmcli con mod "Domus-Wired-EAP-TLS" ipv4.dns "10.50.1.90,10.50.1.91"

# Set autoconnect priority (higher = preferred)
nmcli con mod "Domus-Wired-EAP-TLS" connection.autoconnect-priority 100

Debugging EAP-TLS

# Enable debug logging
sudo nmcli general logging level DEBUG domains ALL

# Watch for connection changes
nmcli monitor

# Check wpa_supplicant logs
journalctl -u wpa_supplicant -f --since "5 minutes ago"

# EAP status via wpa_cli
wpa_cli -i enp130s0 status | grep -E "EAP|wpa_state|key_mgmt"

# Return to normal logging
sudo nmcli general logging level INFO domains DEFAULT

CLI Mastery: Certificate Inspection

# Get certificate paths from connection
CERT=$(nmcli -g 802-1x.client-cert con show "Domus-Wired-EAP-TLS")
KEY=$(nmcli -g 802-1x.private-key con show "Domus-Wired-EAP-TLS")
CA=$(nmcli -g 802-1x.ca-cert con show "Domus-Wired-EAP-TLS")

# Certificate expiry check
openssl x509 -in "$CERT" -noout -dates

# Days until expiry (30-day warning)
openssl x509 -in "$CERT" -noout -checkend $((86400*30)) && \
  echo "Valid for 30+ days" || echo "EXPIRES WITHIN 30 DAYS"

# Validate certificate chain
openssl verify -CAfile "$CA" "$CERT"

# Check key matches certificate
openssl x509 -in "$CERT" -noout -modulus | openssl md5
openssl rsa -in "$KEY" -noout -modulus | openssl md5
# Both should output same hash

CLI Mastery: ip Command

# Show all IPv4 addresses with interface
ip -4 -o addr show | awk '{print $2, $4}'

# Show default gateway
ip route show default | awk '{print $3}'

# Show route to specific destination
ip route get 10.50.1.1

# ARP table
ip neigh show | awk '{print $1, $5}' | column -t

# Full network status one-liner
echo "IP: $(ip -4 -o addr show enp130s0 | awk '{print $4}')" && \
echo "GW: $(ip route show default | awk '{print $3}')" && \
echo "DNS: $(grep nameserver /etc/resolv.conf | awk '{print $2}' | tr '\n' ' ')"

CLI Mastery: Network Diagnostics

# Connectivity check
for h in 10.50.1.1 10.50.1.90 8.8.8.8; do
  ping -c1 -W1 $h &>/dev/null && echo "$h: OK" || echo "$h: FAIL"
done

# DNS resolution test (both servers)
for dns in 10.50.1.90 10.50.1.91; do
  echo -n "$dns: "
  dig +short vault-01.inside.domusdigitalis.dev @$dns
done

# Interface errors and drops
ip -s link show enp130s0 | awk '/RX:|TX:|errors/{print}'

# Listen for EAP traffic
sudo tcpdump -i enp130s0 -c 10 'ether proto 0x888e'

Current Configuration

Setting Value

Connection Name

Domus-Wired-EAP-TLS

Interface

enp130s0

IP Address

10.50.1.106/24

Gateway

10.50.1.1

DNS Servers

10.50.1.90, 10.50.1.91

EAP Method

TLS

Identity

modestus-razer.inside.domusdigitalis.dev

CA Certificate

/etc/ssl/certs/DOMUS-ROOT-CA.pem

Client Certificate

/etc/ssl/certs/modestus-razer-eaptls.pem

Private Key

/etc/ssl/private/modestus-razer-eaptls.key

Key Lessons

  1. Always configure dual DNS for HA (bind-01 + bind-02)

  2. Use ipv4.method manual for infrastructure workstations

  3. Document EAP-TLS parameters - certificates, identity, etc.

  4. 802-1x.private-key-password-flags 4 means "not required" (unencrypted key)

Metadata

Field Value

RCA ID

RCA-2026-03-09-001

Author

Evan Rosado

Date Created

2026-03-09

Status

Final

Category

Network / 802.1X Authentication