RCA-2026-02-001: Linux Bridge VLAN Persistence

Executive Summary

VMs on kvm-02 intermittently lost network connectivity after VM restarts or host reboots. Root cause: Linux bridge vlan add commands are ephemeral - vnet interfaces are recreated without VLAN tags on VM lifecycle events. Resolution: Production libvirt hook that configures VLANs and PVID at VM start using MAC-based vnet matching.

Timeline

Date Event

2026-02-XX

WLC-02 loses management connectivity after restart

2026-02-XX

Investigation: PVID on vnet11 is 1 instead of 100

2026-02-XX

Root cause identified: bridge vlan commands non-persistent

2026-02-XX

Initial libvirt hook deployed (fragile sleep 3)

2026-02-XX

Race condition discovered during simultaneous VM starts

2026-03-XX

Production hook deployed with MAC-based vnet matching

Impact

  • WLC-02 management plane unreachable

  • VyOS control plane disrupted

  • ISE, BIND, Keycloak management interfaces down

  • All VMs requiring PVID 100 affected after restart

Problem Statement

Symptoms

  • VMs respond to ping on VLAN-tagged interfaces but not management (untagged)

  • bridge vlan show dev vnetN shows PVID 1 instead of PVID 100

  • Missing VLAN tags on newly created vnet interfaces

  • Silent failure - no errors logged

Expected Behavior

VMs on br-mgmt with untagged management traffic (10.50.1.x/24) require:

  • All trunk VLANs: 10, 20, 30, 40, 100, 110, 120

  • PVID 100 for untagged ingress traffic

Affected VMs

VM Bridge PVID Required Reason

vyos-01, vyos-02

br-mgmt

PVID 100

eth0 = MGMT untagged (10.50.1.x)

9800-WLC-01, 9800-WLC-02

br-mgmt

PVID 100

Native VLAN 100 (management untagged)

ise-01, ise-02

br-mgmt

PVID 100

eth0 = MGMT untagged

bind-01, bind-02

br-mgmt

PVID 100

eth0 = MGMT untagged

home-dc01, home-dc02

br-mgmt

PVID 100

eth0 = MGMT untagged

keycloak-01, keycloak-02

br-mgmt

PVID 100

eth0 = MGMT untagged

vault-01, vault-02, vault-03

br-mgmt

PVID 100

eth0 = MGMT untagged

ipa-01, ipa-02

br-mgmt

PVID 100

eth0 = MGMT untagged

k3s-master-, k3s-worker-

br-mgmt

PVID 100

eth0 = MGMT untagged

Other VMs

br-mgmt

PVID 1 (default)

Tagged VLANs only

Metadata

Field Value

RCA ID

RCA-2026-02-001

Author

Evan Rosado

Date Created

2026-02

Status

Final

Category

Virtualization / Networking