RCA-2026-02-001: Prevention

Key Lessons

  1. Bridge VLAN commands are ephemeral - Libvirt hooks required for persistence

  2. PVID determines untagged traffic destination - Critical for management VLANs

  3. Never use virsh in hooks - Causes deadlock with libvirtd

  4. MAC-based vnet matching - Avoids race conditions during concurrent VM starts

  5. Poll instead of sleep - More reliable than fixed delays

Prevention Checklist

Adding New VMs

  • Determine if VM needs PVID 100 (management on 10.50.1.x)

  • Add VM name to PVID100_VMS list in hook

  • Restart libvirtd to reload hook

  • Start VM and verify VLAN configuration

Troubleshooting

  • Check journalctl -t "libvirt-hook[<vm-name>]"

  • Verify vnet exists: virsh domiflist <vm-name>

  • Check bridge attachment: ip link show master br-mgmt

  • Verify VLAN state: bridge vlan show dev vnetN

Evolution History

Version Date Changes

v1.0

2026-02

Initial hook with sleep 3 and naive vnet enumeration

v2.0

2026-03

MAC-based vnet matching, polling with timeout, error tracking

Related Documentation