VyOS Firewall Deployment

Migrate from pfSense to VyOS for CLI-native, Linux-based firewall management. This runbook covers complete infrastructure audit, deployment, and cutover with zero gaps.

1. Executive Summary

Item Value

Current Firewall

pfSense (FreeBSD-based, GUI-first)

Target Firewall

VyOS (Debian-based, CLI-native)

Deployment Model

vyos-02 on kvm-02 first (parallel), then vyos-01 + VRRP HA

Downtime

< 5 minutes during cutover (with instant rollback capability)

2. Why VyOS

Aspect pfSense VyOS

Interface

GUI-first, WebUI required

CLI-native, SSH/API only

Base OS

FreeBSD (limited ecosystem)

Debian Bookworm (full apt access)

Automation

xmlrpc hacks, limited API

HTTP API, Ansible, NETCONF/YANG

Observability

Packages required

Native: node_exporter, wazuh-agent, suricata

Firewall

pf (FreeBSD)

nftables (Linux, eBPF/XDP capable)

Config Management

XML backup

/config/config.boot - git trackable, diff-able

Learning Value

FreeBSD-specific

Transfers to RHCSA, kernel studies, enterprise Linux

3. Architecture

3.1. Historical: pfSense Topology (Decommissioned 2026-03-07)

pfSense was decommissioned on 2026-03-07.

3.2. Current VyOS HA Topology

VyOS HA Topology

3.3. Final VLAN Design

VLAN Name Subnet Gateway Purpose

Infrastructure VLANs (servers/services)

100

INFRA

10.50.1.0/24

.1 (VIP)

Network hardware, hypervisors, k3s nodes (MetalLB: .130-.140)

110

SECURITY

10.50.110.0/24

.1

Crown jewels: Vault, ISE, secrets management

120

SERVICES

10.50.120.0/24

.1

VMs: Keycloak, Gitea, FreeIPA, BIND, etc.

Client VLANs (endpoints only)

10

DATA

10.50.10.0/24

.1

Corporate wired/wireless devices

20

VOICE

10.50.20.0/24

.1

VoIP phones (future)

30

GUEST

10.50.30.0/24

.1

Guest wireless, internet-only

40

IOT

10.50.40.0/24

.1

IoT devices, limited access

999

CRITICAL_AUTH

(no gateway)

-

802.1X failure quarantine

Infrastructure Segmentation:

  • VLAN 100 (INFRA) - Physical switches, hypervisors, BMC/IPMI, k3s cluster. Access from admin workstations only.

  • VLAN 110 (SECURITY) - Vault PKI, ISE RADIUS, secrets. Most restrictive - only speaks to services that NEED it.

  • VLAN 120 (SERVICES) - General infrastructure VMs. Can talk to SECURITY tier via explicit rules.

Migration: See Phase 18 for VM migration to VLAN 110/120 after VyOS HA is stable.

3.4. Switch Trunk Configuration (Reference)

! Te1/0/1 → kvm-02 (vyos-02)
! Te1/0/2 → kvm-01 (vyos-01)
interface TenGigabitEthernet1/0/1
 description TRUNK-TO-SUPERMICRO-KVM-02
 switchport trunk allowed vlan 10,20,30,40,100,110,120,999
 switchport trunk native vlan 100
 switchport mode trunk
!
interface TenGigabitEthernet1/0/2
 description TRUNK-TO-SUPERMICRO-KVM-01
 switchport trunk allowed vlan 10,20,30,40,100,110,120,999
 switchport trunk native vlan 100
 switchport mode trunk

3.5. Kubernetes Cluster (6-Node HA in MGMT VLAN)

Node IP Hypervisor Role

k3s-master-01

10.50.1.120

kvm-01

Control plane (active)

k3s-master-02

10.50.1.121

kvm-02

Control plane (planned)

k3s-master-03

10.50.1.122

kvm-02

Control plane (planned)

k3s-worker-01

10.50.1.123

kvm-01

Workloads (planned)

k3s-worker-02

10.50.1.124

kvm-02

Workloads (planned)

k3s-worker-03

10.50.1.125

kvm-02

Workloads (planned)

3.6. Kubernetes LoadBalancer Services (BGP Advertised)

Service IP Ports

Traefik Ingress

10.50.1.130

80, 443

Wazuh Indexer

10.50.1.131

9200

Wazuh Dashboard

10.50.1.132

443

Wazuh Workers

10.50.1.133

1514

Wazuh Manager

10.50.1.134

55000, 1515, 514/udp

LoadBalancer IPs (10.50.1.128/28) are advertised via Cilium BGP to VyOS. See Phase 17.

4. Prerequisites

Complete ALL items before starting deployment:

  • kvm-02 operational (22/22 health checks) - COMPLETE 2026-03-01

  • NAS NFS access configured for kvm-02

  • VyOS ISO downloaded (see Phase 1)

  • pfSense config backup exported (GUI → Diagnostics → Backup)

  • Maintenance window scheduled (cutover needs < 5 min)

  • Console access to kvm-01 and kvm-02 (for emergency recovery)

  • This runbook printed or accessible offline

5. Session Variables

DNS is the source of truth. Hostnames resolve via BIND. Use hostnames directly in commands - no need to store IPs in variables.

5.1. Required Variables

# Target Node - change to vyos-01 for second deployment
VYOS_NODE="vyos-02"
VYOS_PRIORITY="100"       # 100=BACKUP, 200=MASTER

# Hypervisor derived from node (vyos-02 → kvm-02, vyos-01 → kvm-01)
KVM_HOST="${VYOS_NODE/vyos/kvm}"

echo "Deploying: $VYOS_NODE (priority $VYOS_PRIORITY) on $KVM_HOST"
Expected Output
Deploying: vyos-02 (priority 100) on kvm-02

That’s it. Two variables set manually, one derived. DNS resolves hostnames directly - no IP variables needed.

5.2. Verify DNS Works

# If this works, DNS is configured correctly
ssh kvm-02 "hostname"
Expected Output
kvm-02

If SSH by hostname fails, verify PRE-0 DNS records were created in pfSense.

6. Deployment Sequence

Order Phase Action Hypervisor

1

Phase 1-13

Deploy vyos-02, configure zones, services, validation

kvm-02

2

Phase 14

Cutover: pfSense DOWN, vyos-02 takes gateway IPs

kvm-02

3

Phase 15

Deploy vyos-01 + VRRP HA (after 48-72h stable vyos-02)

kvm-01

4

Phase 16-17

k3s firewall rules + Cilium BGP peering

Both

Why vyos-02 on kvm-02 FIRST?

  • kvm-02 is freshly deployed and available NOW

  • kvm-01 still runs pfSense (cannot deploy vyos-01 until pfSense is decommissioned)

  • vyos-02 runs in parallel with pfSense during validation

  • After cutover, kvm-01’s pfSense VM is shut down, freeing resources for vyos-01


7. Phase 1: Deploy VM on Hypervisor

7.1. 1.0 Pre-Validation

Complete these checks BEFORE proceeding. Failures here will cause deployment issues.

PRE-0: Create DNS records in pfSense (REQUIRED before deployment)

DNS records must exist BEFORE deploying VyOS. Use netapi pfsense dns add:

dsource d000 dev/network
netapi pfsense dns add -h vyos-01 -d inside.domusdigitalis.dev -i 10.50.1.2 --descr "VyOS HA Master"
netapi pfsense dns add -h vyos-02 -d inside.domusdigitalis.dev -i 10.50.1.3 --descr "VyOS HA Backup"
Verify DNS records created
netapi pfsense dns list | grep -i vyos
Expected output
vyos-01.inside.domusdigitalis.dev  10.50.1.2   VyOS HA Master
vyos-02.inside.domusdigitalis.dev  10.50.1.3   VyOS HA Backup
pfSense provides immediate resolution. BIND is the authoritative source - update both.
PRE-0b: Add BIND forward zone records
ssh bind-01
sudo vi /var/named/inside.domusdigitalis.dev.zone

1. Increment SOA serial (format YYYYMMDDNN)

2. Add A records (in Network Devices section, after switches):

; VyOS Routers (.2-.3)
vyos-01         IN  A       10.50.1.2
vyos-02         IN  A       10.50.1.3

3. Save and validate:

sudo named-checkzone inside.domusdigitalis.dev /var/named/inside.domusdigitalis.dev.zone

4. Reload zone:

sudo rndc reload inside.domusdigitalis.dev
PRE-0c: Add BIND reverse zone records
sudo vi /var/named/10.50.1.rev

1. Increment SOA serial

2. Add PTR records (NO leading whitespace!):

2               IN      PTR     vyos-01.inside.domusdigitalis.dev.
3               IN      PTR     vyos-02.inside.domusdigitalis.dev.

3. Validate and reload:

sudo named-checkzone 1.50.10.in-addr.arpa /var/named/10.50.1.rev
sudo rndc reload 1.50.10.in-addr.arpa
PRE-0d: Verify DNS resolution
dig +short vyos-01.inside.domusdigitalis.dev @10.50.1.90
dig +short vyos-02.inside.domusdigitalis.dev @10.50.1.90
Expected output
10.50.1.2
10.50.1.3
PRE-1: Verify session variables are set
echo "Deploying: $VYOS_NODE on $KVM_HOST"
Direct (on hypervisor)
hostname && uptime
Remote (from workstation)
ssh "$KVM_HOST" "hostname && uptime"
Direct
ip link show type bridge | awk '/^[0-9]+:/{print $2}' | tr -d ':'
Remote
ssh "$KVM_HOST" "ip link show type bridge | awk '/^[0-9]+:/{print \$2}' | tr -d ':'"
Table 1. Required Bridges by Hypervisor
Hypervisor Required Bridges Notes

kvm-01

br-wan + br-mgmt

WAN bridge (eno7). LAN trunk on br-mgmt (VLAN filtering + libvirt hook).

kvm-02

br-wan (or ixl0 passthrough) + br-mgmt

WAN via PCI passthrough (ixl0 10GbE). LAN trunk on br-mgmt.

SYMMETRIC HA Architecture: Both nodes have identical dual-interface configuration. Each hypervisor has dedicated 10GbE NIC (ixl0) for WAN passthrough to ISP modem. True HA failover for both WAN and LAN traffic.

Direct
df -h /var/lib/libvirt/images | awk 'NR==2{print "Available: " $4}'
Remote
ssh "$KVM_HOST" "df -h /var/lib/libvirt/images | awk 'NR==2{print \"Available: \" \$4}'"
Direct
sudo virsh list --all | grep -q "$VYOS_NODE" && echo 'WARNING: VM exists!' || echo 'OK: VM does not exist'
Remote
ssh "$KVM_HOST" "sudo virsh list --all | grep -q '$VYOS_NODE' && echo 'WARNING: VM exists!' || echo 'OK: VM does not exist'"

7.2. 1.1 Download VyOS ISO

VyOS offers rolling (nightly) and LTS releases. Rolling is free and stable for home enterprise.

Command Format: Each command shows TWO options:

  • Direct - Run when SSH’d into the hypervisor

  • Remote - Run from workstation via SSH

Direct (on hypervisor)
VYOS_LATEST=$(curl -sL https://api.github.com/repositories/674742659/releases | jq -r '.[0].tag_name') && echo "Latest: $VYOS_LATEST"
Remote (from workstation)
ssh "$KVM_HOST" "VYOS_LATEST=\$(curl -sL https://api.github.com/repositories/674742659/releases | jq -r '.[0].tag_name') && echo \"Latest: \$VYOS_LATEST\""
Direct (on hypervisor)
curl -Lo /var/lib/libvirt/images/vyos-rolling-latest.iso \
  "https://github.com/vyos/vyos-nightly-build/releases/download/${VYOS_LATEST}/vyos-${VYOS_LATEST}-generic-amd64.iso"
Remote (from workstation)
ssh "$KVM_HOST" "curl -Lo /var/lib/libvirt/images/vyos-rolling-latest.iso \
  'https://github.com/vyos/vyos-nightly-build/releases/download/${VYOS_LATEST}/vyos-${VYOS_LATEST}-generic-amd64.iso'"
Direct
ls -lh /var/lib/libvirt/images/vyos*.iso
Remote
ssh "$KVM_HOST" "ls -lh /var/lib/libvirt/images/vyos*.iso"

7.3. 1.2 Create VM Disk

Direct
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/${VYOS_NODE}.qcow2 10G
Remote
ssh "$KVM_HOST" "sudo qemu-img create -f qcow2 /var/lib/libvirt/images/${VYOS_NODE}.qcow2 10G"
Direct
ls -lh /var/lib/libvirt/images/${VYOS_NODE}.qcow2
Remote
ssh "$KVM_HOST" "ls -lh /var/lib/libvirt/images/${VYOS_NODE}.qcow2"

7.4. 1.3 Create VM

For vyos-02 on kvm-02 (dual interface - WAN + LAN):

Direct (on kvm-02)
sudo virt-install \
  --name "$VYOS_NODE" \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/var/lib/libvirt/images/${VYOS_NODE}.qcow2,format=qcow2,bus=virtio \
  --cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
  --os-variant debian11 \
  --network bridge=br-wan,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole
Remote (from workstation)
ssh "$KVM_HOST" "sudo virt-install \
  --name '$VYOS_NODE' \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/var/lib/libvirt/images/${VYOS_NODE}.qcow2,format=qcow2,bus=virtio \
  --cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
  --os-variant debian11 \
  --network bridge=br-wan,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole"

--extra-args only works with --location (network install), not --cdrom. Connect via VNC or serial console after VM starts:

# Option 1: VNC (get port number)
sudo virsh vncdisplay "$VYOS_NODE"

# Option 2: Serial console (after VyOS boots)
sudo virsh console "$VYOS_NODE"

For vyos-01 on kvm-01 (dual interface - WAN + LAN, deploy after cutover):

kvm-01 Storage Architecture: Root disk is only 14GB. VM images go to onboard SSD:

  • kvm-01: /mnt/onboard-ssd/libvirt/images/ (978GB SSD)

  • kvm-02: /var/lib/libvirt/images/ (1.5TB NVMe LVM)

Create storage directory (kvm-01 only)
sudo mkdir -p /mnt/onboard-ssd/libvirt/images
Direct (on kvm-01)
sudo virt-install \
  --name "$VYOS_NODE" \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/mnt/onboard-ssd/libvirt/images/${VYOS_NODE}.qcow2,size=10,format=qcow2,bus=virtio \
  --cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
  --os-variant debian11 \
  --network bridge=br-wan,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole
Remote (from workstation)
ssh "$KVM_HOST" "sudo virt-install \
  --name '$VYOS_NODE' \
  --memory 2048 \
  --vcpus 2 \
  --disk path=/mnt/onboard-ssd/libvirt/images/${VYOS_NODE}.qcow2,size=10,format=qcow2,bus=virtio \
  --cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
  --os-variant debian11 \
  --network bridge=br-wan,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole"

Symmetric HA Deployment Architecture:

  • vyos-02 (kvm-02): Dual interface → br-wan (eth0 = WAN 10GbE) + br-mgmt (eth1 = LAN trunk)

  • vyos-01 (kvm-01): Dual interface → br-wan (eth0 = WAN 10GbE) + br-mgmt (eth1 = LAN trunk)

Both nodes have identical capability. vyos-02 deploys first as BACKUP (priority 100). After cutover (Phase 14), vyos-01 deploys and becomes MASTER (priority 200). Full WAN + LAN failover with conntrack-sync for stateful session preservation.

7.5. 1.4 Install VyOS to Disk

After VM boots to live ISO, connect via console:

Direct
sudo virsh console "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh console $VYOS_NODE"

Login: vyos / vyos (live environment)

install image
Table 2. Install Prompts
Prompt Answer

Would you like to continue? [y/N]

y

What would you like to name this image?

Press Enter (accept default version)

Please enter a password for the "vyos" user

your strong password

Please confirm password for the "vyos" user

repeat password

What console should be used by default? (K: KVM, S: Serial)?

S (Serial for virsh console)

Which one should I install to? (Default: /dev/vda)

Press Enter (accept /dev/vda)

After install completes:

reboot

Detach from console: Ctrl+]

7.6. 1.5 Start VM and Verify

After reboot, VM will be shut off. Start it:

Direct
sudo virsh start "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh start '$VYOS_NODE'"

Connect to console:

Direct
sudo virsh console "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh console $VYOS_NODE"

Login with vyos and the password you set during install.

7.7. 1.6 Post-Validation

Direct
sudo virsh list --all | grep "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh list --all | grep '$VYOS_NODE'"
Expected Output
 1     vyos-02    running
Direct
sudo virsh domiflist "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh domiflist '$VYOS_NODE'"
Expected Output (vyos-02 - dual interface on kvm-02)
 Interface   Type     Source    Model    MAC
 ----------------------------------------------------------------
 vnet0       bridge   br-wan    virtio   52:54:00:xx:xx:xx
 vnet1       bridge   br-mgmt   virtio   52:54:00:xx:xx:xx
Expected Output (vyos-01 - dual interface on kvm-01)
 Interface   Type     Source    Model    MAC
 ----------------------------------------------------------------
 vnet0       bridge   br-wan    virtio   52:54:00:xx:xx:xx
 vnet1       bridge   br-mgmt   virtio   52:54:00:xx:xx:xx
POST-2b: Verify VLAN configuration on LAN interface (CRITICAL)

The libvirt hook should automatically configure VLANs on the br-mgmt vnet interface. If it didn’t trigger, apply manually:

# Check current VLAN state (replace vnetX with actual interface from POST-2)
sudo bridge vlan show dev vnet1
Expected (if hook worked)
port              vlan-id
vnet1             10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
If only VLAN 1 shown, apply manually
# Add all VLANs
for vid in 10 20 30 40 100 110 120; do
  sudo bridge vlan add vid $vid dev vnet1
done

# Remove default PVID 1, set PVID 100 for MGMT
sudo bridge vlan del vid 1 dev vnet1 pvid untagged
sudo bridge vlan add vid 100 dev vnet1 pvid untagged

# Verify
sudo bridge vlan show dev vnet1

Why PVID 100? VyOS eth0/eth1 has 10.50.1.2/24 or 10.50.1.3/24 configured directly (untagged). PVID determines which VLAN receives untagged frames. Must be 100 to match MGMT VLAN.

Direct
sudo virsh domblklist "$VYOS_NODE"
Remote
ssh "$KVM_HOST" "sudo virsh domblklist '$VYOS_NODE'"

Phase 1 Complete. VM is running. Proceed to Phase 2 for configuration.


8. Phase 2: Interface and VLAN Configuration

8.1. 2.0 Pre-Validation

8.1.1. PRE-2.0a: Hypervisor Bridge Verification (on kvm-02)

Run these commands on the hypervisor before configuring VyOS interfaces.

All connections with status pivot
nmcli -t conn show | awk -F: '{status[$4]++; print $1, "→", $4} END{print "---"; for(s in status) print s": "status[s]}'
Physical NICs with speed/state
nmcli -t -f DEVICE,TYPE,STATE,CONNECTION device | awk -F: '$2=="ethernet"{printf "%-12s %-8s %s\n", $1, $3, $4}'
Bridge slave relationships (CRITICAL - shows what’s in br-wan/br-mgmt)
nmcli -t conn show | awk -F: '{print $1}' | while read c; do
  master=$(nmcli -g connection.master conn show "$c" 2>/dev/null)
  [[ -n "$master" ]] && echo "$c → $master"
done
All bridges with their slaves
for br in $(ip link show type bridge | awk -F': ' '{print $2}'); do
  echo "=== $br ==="
  ip link show master $br | awk -F': ' '/^[0-9]/{print "  "$2}'
done
Quick check: What physical NIC is in each bridge
echo "br-mgmt: $(ip link show master br-mgmt 2>/dev/null | awk -F': ' '/^[0-9].*eno/{print $2}')"
echo "br-wan:  $(ip link show master br-wan 2>/dev/null | awk -F': ' '/^[0-9].*eno/{print $2}')"
Verify VM has both interfaces
sudo virsh domiflist "$VYOS_NODE" | awk 'NR>2 && NF{printf "%-8s → %s\n", $1, $3}'
Expected output (kvm-02)
br-mgmt: eno8
br-wan:  eno7
vnet1    → br-mgmt
vnet2    → br-wan

8.1.2. PRE-2.0b: Bridge VLAN Configuration (CRITICAL)

KVM bridge VLAN filtering requires VLANs on the VM’s vnet interface.

Without this, VLAN-tagged traffic (DHCP, DNS, etc.) is silently dropped even if the bridge and physical trunk have the VLANs.

Check current VLAN assignments
sudo bridge vlan show
Expected: vnet interfaces should have ALL VLANs (not just VLAN 1)
port              vlan-id
eno8              1 PVID Egress Untagged
                  10
                  20
                  30
                  40
                  100
                  110
                  120
br-mgmt           1 PVID Egress Untagged
                  10
                  20
                  30
                  40
                  100
                  110
                  120
vnet1             1 PVID Egress Untagged   <-- PROBLEM: Missing VLANs!
                  10                        <-- NEED: All VLANs like br-mgmt
                  20
                  30
                  40
                  100
                  110
                  120
Add ALL VLANs to vyos-02’s LAN interface AND set PVID 100
# Add all VLANs
for vid in 10 20 30 40 100 110 120; do
  sudo bridge vlan add vid $vid dev vnet1
done

# CRITICAL: Set PVID 100 for MGMT (vyos eth0 = 10.50.1.x untagged)
sudo bridge vlan add vid 100 dev vnet1 pvid untagged

Why PVID 100?

VyOS eth0 has 10.50.1.3/24 (MGMT) configured directly - not as eth0.100. This means MGMT traffic is UNTAGGED.

The bridge PVID determines which VLAN receives untagged frames. Default PVID 1 sends traffic to wrong VLAN. Must be PVID 100 to match MGMT VLAN.

Verify VLANs added
sudo bridge vlan show dev vnet1

bridge vlan add is NOT persistent! On reboot, vnet1 loses all VLANs.

Make persistent via libvirt hook or NetworkManager. See KVM Bridge VLAN Persistence.

Quick test: DHCP from VyOS should now work
# From workstation on VyOS-managed VLAN (e.g., VLAN 40/IOT)
ip -4 addr show $INTERFACE | awk '/inet/{print $2}'
# Expected: 10.50.40.x (from vyos-02 DHCP pool)

8.1.3. PRE-2.0c: Session Variables

# PRE-1: Verify session variables
echo "Configuring: $VYOS_NODE"
echo "Hypervisor: $KVM_HOST"

8.1.4. PRE-2.0c: Connect to VyOS

# PRE-2: Connect to VyOS console
ssh "$KVM_HOST" "sudo virsh console $VYOS_NODE --force"
# Login: vyos / <password set during install>
# PRE-3: Verify we're on the correct node (inside VyOS)
show version | head -3

Enter configuration mode:

configure

8.2. 2.1 System Basics (vyos-02)

These commands run inside VyOS configuration mode on vyos-02.
# System identity - vyos-02
set system host-name vyos-02
set system domain-name inside.domusdigitalis.dev
set system time-zone America/Los_Angeles

# DNS servers
set system name-server 10.50.1.90
set system name-server 10.50.1.91

# Console access for virsh console
set system console device ttyS0 speed 115200

# NTP - sync to external, serve to internal
set service ntp server time.cloudflare.com
set service ntp server time.google.com
set service ntp listen-address 10.50.1.3
set service ntp listen-address 10.50.1.1

commit

8.3. 2.1b System Basics (vyos-01)

Run these on vyos-01 after vyos-02 is validated (Phase 15).
# System identity - vyos-01
set system host-name vyos-01
set system domain-name inside.domusdigitalis.dev
set system time-zone America/Los_Angeles

# DNS servers
set system name-server 10.50.1.90
set system name-server 10.50.1.91

# Console access for virsh console
set system console device ttyS0 speed 115200

# NTP - sync to external, serve to internal
set service ntp server time.cloudflare.com
set service ntp server time.google.com
set service ntp listen-address 10.50.1.2
set service ntp listen-address 10.50.1.1

commit

8.4. 2.2 Interface Configuration (vyos-02 on kvm-02)

Interface mappings differ between nodes! See antora.yml for attributes.

Node WAN Iface LAN Iface Reason Attribute Reference

vyos-01

eth0

eth1

Original design (WAN first, LAN second)

vyos-01-wan-iface, vyos-01-lan-iface

vyos-02

eth1

eth0

Hot-added WAN (LAN first, WAN second)

vyos-02-wan-iface, vyos-02-lan-iface

Interface mapping for vyos-02:

VyOS Bridge Purpose Notes

eth0

br-mgmt

LAN-TRUNK

Trunk to Catalyst switch (VLANs)

eth1

br-wan

WAN

DHCP from ISP modem via eno7 (1GbE)

Why this order? VM created with br-mgmt first, br-wan hot-added second. Interface order is determined by attachment sequence, not virt-install flags.

# LAN trunk interface (eth0)
set interfaces ethernet eth0 description 'LAN-TRUNK'

# INFRA - Native VLAN 100 (untagged on switch trunk)
# Use vyos-02's real IP (10.50.1.3), NOT the VIP (VIP is for VRRP later)
set interfaces ethernet eth0 address 10.50.1.3/24

# SECURITY - VLAN 110 (crown jewels: Vault, ISE)
set interfaces ethernet eth0 vif 110 address 10.50.110.1/24
set interfaces ethernet eth0 vif 110 description 'SECURITY'

# SERVICES - VLAN 120 (general VMs: Keycloak, Gitea, etc.)
set interfaces ethernet eth0 vif 120 address 10.50.120.1/24
set interfaces ethernet eth0 vif 120 description 'SERVICES'

# DATA - VLAN 10
set interfaces ethernet eth0 vif 10 address 10.50.10.1/24
set interfaces ethernet eth0 vif 10 description 'DATA'

# VOICE - VLAN 20
set interfaces ethernet eth0 vif 20 address 10.50.20.1/24
set interfaces ethernet eth0 vif 20 description 'VOICE'

# GUEST - VLAN 30
set interfaces ethernet eth0 vif 30 address 10.50.30.1/24
set interfaces ethernet eth0 vif 30 description 'GUEST'

# IOT - VLAN 40
set interfaces ethernet eth0 vif 40 address 10.50.40.1/24
set interfaces ethernet eth0 vif 40 description 'IOT'

# WAN - DHCP from ISP modem (eth1)
set interfaces ethernet eth1 description 'WAN'
set interfaces ethernet eth1 address dhcp
set interfaces ethernet eth1 ipv6 address no-default-link-local

commit
POST-2.2 Validation: Verify interface configuration
run show interfaces ethernet
Expected output
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             {vyos-02-ip}/24                   u/u  LAN-TRUNK
eth0.10          {gateway-data}/24                 u/u  DATA
eth0.20          {gateway-voice}/24                u/u  VOICE
eth0.30          {gateway-guest}/24                u/u  GUEST
eth0.40          {gateway-iot}/24                  u/u  IOT
eth0.110         {gateway-security}/24             u/u  SECURITY
eth0.120         {gateway-services}/24             u/u  SERVICES
eth1             <dhcp-assigned>                   u/u  WAN
Detailed Output (click to expand)
show interfaces
 ethernet eth0 {
     address 10.50.1.3/24
     description LAN-TRUNK
     hw-id 52:54:00:31:eb:e3
     offload {
         gro
         gso
         sg
         tso
     }
     vif 10 {
         address 10.50.10.1/24
         description DATA
     }
     vif 20 {
         address 10.50.20.1/24
         description VOICE
     }
     vif 30 {
         address 10.50.30.1/24
         description GUEST
     }
     vif 40 {
         address 10.50.40.1/24
         description IOT
     }
     vif 110 {
         address 10.50.110.1/24
         description SECURITY
     }
     vif 120 {
         address 10.50.120.1/24
         description SERVICES
     }
 }
 ethernet eth1 {
     address dhcp
     description WAN
 }
 loopback lo {
 }

SYMMETRIC HA: Both vyos-01 and vyos-02 have identical interface configuration. Each has dedicated 10GbE WAN to ISP modem for true active/standby failover.

VLAN 110/120 are READY - VMs migrate to these VLANs in Phase 18 after VyOS HA is verified.

8.5. 2.3 Interface Configuration (vyos-01 on kvm-01)

Dual interface deployment - eth0 is WAN, eth1 is LAN trunk.

# WAN - DHCP from ISP modem (eth0)
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 ipv6 address no-default-link-local

# LAN trunk interface (eth1)
set interfaces ethernet eth1 description 'LAN-TRUNK'

# INFRA - Native VLAN 100
# Use vyos-01's real IP (10.50.1.2), NOT the VIP (VIP is for VRRP later)
set interfaces ethernet eth1 address 10.50.1.2/24

# SECURITY - VLAN 110 (crown jewels: Vault, ISE)
set interfaces ethernet eth1 vif 110 address 10.50.110.1/24
set interfaces ethernet eth1 vif 110 description 'SECURITY'

# SERVICES - VLAN 120 (general VMs: Keycloak, Gitea, etc.)
set interfaces ethernet eth1 vif 120 address 10.50.120.1/24
set interfaces ethernet eth1 vif 120 description 'SERVICES'

# DATA - VLAN 10
set interfaces ethernet eth1 vif 10 address 10.50.10.1/24
set interfaces ethernet eth1 vif 10 description 'DATA'

# VOICE - VLAN 20
set interfaces ethernet eth1 vif 20 address 10.50.20.1/24
set interfaces ethernet eth1 vif 20 description 'VOICE'

# GUEST - VLAN 30
set interfaces ethernet eth1 vif 30 address 10.50.30.1/24
set interfaces ethernet eth1 vif 30 description 'GUEST'

# IOT - VLAN 40
set interfaces ethernet eth1 vif 40 address 10.50.40.1/24
set interfaces ethernet eth1 vif 40 description 'IOT'

8.6. 2.4 Loopback (for services)

set interfaces loopback lo

8.7. 2.5 Commit and Save

# Show pending changes before commit
compare

# Commit changes
commit

# Save to config.boot
save

8.8. 2.6 Post-Validation

# POST-1: Verify all interfaces are up
run show interfaces
Expected Output (for vyos-02 - dual interface on kvm-02)
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface         IP Address          S/L  Description
---------         ----------          ---  -----------
eth0              192.168.1.x/24      u/u  WAN
eth1              10.50.1.3/24     u/u  LAN-TRUNK
eth1.10           10.50.10.1/24       u/u  DATA
eth1.20           10.50.20.1/24       u/u  VOICE
eth1.30           10.50.30.1/24       u/u  GUEST
eth1.40           10.50.40.1/24       u/u  IOT
eth1.110          10.50.110.1/24      u/u  SECURITY
eth1.120          10.50.120.1/24      u/u  SERVICES
lo                127.0.0.1/8         u/u
Expected Output (for vyos-01 - dual interface on kvm-01)
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface         IP Address          S/L  Description
---------         ----------          ---  -----------
eth0              192.168.1.x/24      u/u  WAN
eth1              10.50.1.2/24     u/u  LAN-TRUNK
eth1.10           10.50.10.1/24       u/u  DATA
eth1.20           10.50.20.1/24       u/u  VOICE
eth1.30           10.50.30.1/24       u/u  GUEST
eth1.40           10.50.40.1/24       u/u  IOT
eth1.110          10.50.110.1/24      u/u  SECURITY
eth1.120          10.50.120.1/24      u/u  SERVICES
lo                127.0.0.1/8         u/u
# POST-2: Verify LAN interface has correct IP (eth1 is LAN trunk)
run show interfaces ethernet eth1
# POST-3: Verify DNS resolution works (via existing pfSense gateway)
run ping -c 2 google.com
# POST-4: Verify hostname is set correctly
run show host name

Phase 2 Complete. Interfaces configured. Proceed to Phase 3 for firewall zones.

Both vyos-01 and vyos-02 have WAN interfaces (eth0). However, until cutover (Phase 14), internet access still routes via pfSense. VyOS WAN gets DHCP from modem but isn’t the default gateway yet.


9. Phase 3: Zone-Based Firewall

VyOS uses zone-based firewall with nftables backend. Define zones, then policies between zones.

9.1. 3.0 Pre-Validation

# PRE-1: Verify Phase 2 completed (interfaces exist)
show interfaces

# Expected: eth0 (WAN), eth1 (LAN trunk), eth1.10/20/30/40 (VLANs)
# PRE-2: Verify no existing zone config (clean slate)
show firewall zone

# Expected: empty or error (no zones defined yet)
# PRE-3: Document current firewall state (baseline)
show firewall summary

9.2. 3.1 Define Firewall Zones

Interface mappings differ per node!

Node WAN Zone Interface LAN Zone Interfaces

vyos-01

eth0

eth1, eth1.10, eth1.20, etc.

vyos-02

eth1

eth0, eth0.10, eth0.20, etc.

Commands below are for vyos-02. For vyos-01, swap eth0↔eth1.

For vyos-02 (eth0=LAN, eth1=WAN):

VyOS 1.4+ Syntax: Zone membership uses member interface, not just interface.

configure

# WAN Zone (untrusted) - eth1 for vyos-02
set firewall zone WAN member interface eth1
set firewall zone WAN default-action drop
set firewall zone WAN description 'Untrusted Internet'

# MGMT Zone (high trust) - eth0 for vyos-02
set firewall zone MGMT member interface eth0
set firewall zone MGMT default-action drop
set firewall zone MGMT description 'Management and Infrastructure'

# DATA Zone (medium trust)
set firewall zone DATA member interface eth0.10
set firewall zone DATA default-action drop
set firewall zone DATA description 'Corporate Devices'

# VOICE Zone (medium trust)
set firewall zone VOICE member interface eth0.20
set firewall zone VOICE default-action drop
set firewall zone VOICE description 'VoIP Phones'

# GUEST Zone (low trust)
set firewall zone GUEST member interface eth0.30
set firewall zone GUEST default-action drop
set firewall zone GUEST description 'Guest Wireless - Internet Only'

# IOT Zone (low trust)
set firewall zone IOT member interface eth0.40
set firewall zone IOT default-action drop
set firewall zone IOT description 'IoT Devices - Limited Access'

# SECURITY Zone (crown jewels)
set firewall zone SECURITY member interface eth0.110
set firewall zone SECURITY default-action drop
set firewall zone SECURITY description 'Crown Jewels - Vault, ISE'

# SERVICES Zone (infrastructure VMs)
set firewall zone SERVICES member interface eth0.120
set firewall zone SERVICES default-action drop
set firewall zone SERVICES description 'Infrastructure VMs'

# LOCAL Zone (firewall itself)
set firewall zone LOCAL local-zone
set firewall zone LOCAL default-action drop
set firewall zone LOCAL description 'Firewall Services'

commit
save
POST: Verify all 9 zones created
run show firewall zone | awk 'NR>2 && NF>=2 {printf "%-10s → %s\n", $1, $2}'
Expected output
DATA       → eth0.10
GUEST      → eth0.30
IOT        → eth0.40
LOCAL      → LOCAL
MGMT       → eth0
SECURITY   → eth0.110
SERVICES   → eth0.120
VOICE      → eth0.20
WAN        → eth1
Detailed Output (click to expand)
show firewall zone
 zone DATA {
     default-action drop
     description "Corporate Devices"
     member {
         interface eth0.10
     }
 }
 zone GUEST {
     default-action drop
     description "Guest Wireless - Internet Only"
     member {
         interface eth0.30
     }
 }
 zone IOT {
     default-action drop
     description "IoT Devices - Limited Access"
     member {
         interface eth0.40
     }
 }
 zone LOCAL {
     default-action drop
     description "Firewall Services"
     local-zone
 }
 zone MGMT {
     default-action drop
     description "Management and Infrastructure"
     member {
         interface eth0
     }
 }
 zone SECURITY {
     default-action drop
     description "Crown Jewels - Vault, ISE"
     member {
         interface eth0.110
     }
 }
 zone SERVICES {
     default-action drop
     description "Infrastructure VMs"
     member {
         interface eth0.120
     }
 }
 zone VOICE {
     default-action drop
     description "VoIP Phones"
     member {
         interface eth0.20
     }
 }
 zone WAN {
     default-action drop
     description "Untrusted Internet"
     member {
         interface eth1
     }
 }

9.3. 3.2 Firewall Groups (Address Lists)

These commands use session variables for consistency. Verify variables are set before proceeding.
# PRE: Verify k3s variables are set
echo "k3s Masters: $K3S_MASTER_01, $K3S_MASTER_02, $K3S_MASTER_03"
echo "k3s Workers: $K3S_WORKER_01, $K3S_WORKER_02, $K3S_WORKER_03"
echo "DNS: $DNS_PRIMARY, $DNS_SECONDARY"
echo "Wazuh: $WAZUH_MANAGER"
# RFC1918 Private Networks
set firewall group network-group RFC1918 network 10.0.0.0/8
set firewall group network-group RFC1918 network 172.16.0.0/12
set firewall group network-group RFC1918 network 192.168.0.0/16

# Internal Networks (all VLANs) - for blanket rules
set firewall group network-group INTERNAL network 10.50.0.0/16

# Per-VLAN Network Groups (for granular NAT/firewall control)
set firewall group network-group NET_INFRA network 10.50.1.0/24
set firewall group network-group NET_INFRA description 'VLAN 100 - Infrastructure'
set firewall group network-group NET_DATA network 10.50.10.0/24
set firewall group network-group NET_DATA description 'VLAN 10 - Corporate Data'
set firewall group network-group NET_VOICE network 10.50.20.0/24
set firewall group network-group NET_VOICE description 'VLAN 20 - VoIP'
set firewall group network-group NET_GUEST network 10.50.30.0/24
set firewall group network-group NET_GUEST description 'VLAN 30 - Guest WiFi'
set firewall group network-group NET_IOT network 10.50.40.0/24
set firewall group network-group NET_IOT description 'VLAN 40 - IoT Devices'
set firewall group network-group NET_SECURITY network 10.50.110.0/24
set firewall group network-group NET_SECURITY description 'VLAN 110 - Crown Jewels'
set firewall group network-group NET_SERVICES network 10.50.120.0/24
set firewall group network-group NET_SERVICES description 'VLAN 120 - Infrastructure VMs'

# Wazuh Manager (all agents need access)
set firewall group address-group WAZUH_MANAGER address 10.50.1.134

# DNS Servers
set firewall group address-group DNS_SERVERS address 10.50.1.90
set firewall group address-group DNS_SERVERS address 10.50.1.91

# K8s Services (LoadBalancer VIPs)
set firewall group network-group K8S_SERVICES network 10.50.1.130/32
set firewall group network-group K8S_SERVICES network 10.50.1.131/32
set firewall group network-group K8S_SERVICES network 10.50.1.132/32
set firewall group network-group K8S_SERVICES network 10.50.1.133/32
set firewall group network-group K8S_SERVICES network 10.50.1.134/32

# k3s Cluster Nodes (6-node HA cluster)
set firewall group address-group K3S_NODES address 10.50.1.120
set firewall group address-group K3S_NODES address 10.50.1.121
set firewall group address-group K3S_NODES address 10.50.1.122
set firewall group address-group K3S_NODES address 10.50.1.123
set firewall group address-group K3S_NODES address 10.50.1.124
set firewall group address-group K3S_NODES address 10.50.1.125

# k3s Control Plane (masters only - for etcd)
set firewall group address-group K3S_MASTERS address 10.50.1.120
set firewall group address-group K3S_MASTERS address 10.50.1.121
set firewall group address-group K3S_MASTERS address 10.50.1.122

# ADMINS - Privileged workstations (SSH, HTTPS management access)
# Purpose: Prevent lateral movement - only these IPs can SSH to infrastructure
set firewall group address-group ADMINS address 10.50.10.106
set firewall group address-group ADMINS address 10.50.10.107
set firewall group address-group ADMINS address 10.50.10.108
set firewall group address-group ADMINS description 'Admin workstations with privileged access'

# =============================================================================
# CRITICAL INFRASTRUCTURE ADDRESS GROUPS
# =============================================================================

# --- SECURITY / CROWN JEWELS ---

# ISE Cluster (802.1X, TACACS+, RADIUS)
set firewall group address-group ISE_NODES address 10.50.1.20
set firewall group address-group ISE_NODES address 10.50.1.21
set firewall group address-group ISE_NODES description 'Cisco ISE cluster - 802.1X/RADIUS/TACACS+'

# Vault Cluster (PKI, SSH CA, Secrets)
set firewall group address-group VAULT_NODES address 10.50.1.60
set firewall group address-group VAULT_NODES address 10.50.1.61
set firewall group address-group VAULT_NODES address 10.50.1.62
set firewall group address-group VAULT_NODES description 'HashiCorp Vault HA cluster'

# iPSK Manager (wireless PSK management)
set firewall group address-group IPSK_NODES address 10.50.1.30
set firewall group address-group IPSK_NODES address 10.50.1.31
set firewall group address-group IPSK_NODES description 'iPSK Manager cluster'

# --- IDENTITY ---

# Active Directory Domain Controllers
set firewall group address-group AD_DCS address 10.50.1.50
set firewall group address-group AD_DCS address 10.50.1.51
set firewall group address-group AD_DCS description 'Windows AD Domain Controllers'

# Keycloak IdP Cluster
set firewall group address-group KEYCLOAK_NODES address 10.50.1.80
set firewall group address-group KEYCLOAK_NODES address 10.50.1.81
set firewall group address-group KEYCLOAK_NODES description 'Keycloak Identity Provider cluster'

# FreeIPA Cluster
set firewall group address-group IPA_NODES address 10.50.1.100
set firewall group address-group IPA_NODES address 10.50.1.101
set firewall group address-group IPA_NODES description 'FreeIPA Identity Management cluster'

# --- NETWORK INFRASTRUCTURE ---

# Wireless LAN Controllers
set firewall group address-group WLC_NODES address 10.50.1.40
set firewall group address-group WLC_NODES address 10.50.1.41
set firewall group address-group WLC_NODES description 'Cisco 9800 WLC cluster'

# Switches (management access)
set firewall group address-group SWITCHES address 10.50.1.11
set firewall group address-group SWITCHES address 10.50.1.10
set firewall group address-group SWITCHES description 'Cisco Catalyst switches'

# VyOS Routers (HA pair)
set firewall group address-group VYOS_NODES address 10.50.1.2
set firewall group address-group VYOS_NODES address 10.50.1.3
set firewall group address-group VYOS_NODES description 'VyOS router HA pair'

# pfSense (legacy/migration)
set firewall group address-group PFSENSE address 10.50.1.1
set firewall group address-group PFSENSE description 'pfSense firewall (migration target)'

# --- STORAGE ---

# NAS Cluster
set firewall group address-group NAS_NODES address 10.50.1.70
set firewall group address-group NAS_NODES address 10.50.1.71
set firewall group address-group NAS_NODES description 'Synology NAS cluster'

# Git Server
set firewall group address-group GITEA address 10.50.1.72
set firewall group address-group GITEA description 'Gitea git server'

# Object Storage
set firewall group address-group MINIO address 10.50.1.73
set firewall group address-group MINIO description 'MinIO S3-compatible storage'

# --- COMPUTE / HYPERVISORS ---

# KVM Hypervisors
set firewall group address-group KVM_HOSTS address 10.50.1.110
set firewall group address-group KVM_HOSTS address 10.50.1.111
set firewall group address-group KVM_HOSTS description 'KVM/libvirt hypervisors'

# IPMI/BMC (out-of-band management)
set firewall group address-group IPMI_NODES address 10.50.1.200
set firewall group address-group IPMI_NODES address 10.50.1.201
set firewall group address-group IPMI_NODES description 'IPMI/BMC out-of-band management'

# --- OBSERVABILITY ---

# Zabbix Monitoring
set firewall group address-group ZABBIX address 10.50.1.135
set firewall group address-group ZABBIX description 'Zabbix monitoring server'

# Wazuh SIEM (already have WAZUH_MANAGER, add full cluster)
set firewall group address-group WAZUH_NODES address 10.50.1.131
set firewall group address-group WAZUH_NODES address 10.50.1.132
set firewall group address-group WAZUH_NODES address 10.50.1.134
set firewall group address-group WAZUH_NODES address 10.50.1.133
set firewall group address-group WAZUH_NODES description 'Wazuh SIEM cluster VIPs'

# =============================================================================
# PORT GROUPS
# =============================================================================

# --- COMMON SERVICES ---

# SSH/Management
set firewall group port-group SSH port 22
set firewall group port-group HTTPS port 443
set firewall group port-group HTTP port 80
set firewall group port-group MGMT_WEB port 80
set firewall group port-group MGMT_WEB port 443

# DNS (TCP and UDP - use with protocol specification in rules)
set firewall group port-group DNS_PORTS port 53

# NTP
set firewall group port-group NTP port 123

# --- AUTHENTICATION & IDENTITY ---

# RADIUS (ISE authentication)
set firewall group port-group RADIUS port 1812
set firewall group port-group RADIUS port 1813
set firewall group port-group RADIUS port 1645
set firewall group port-group RADIUS port 1646

# TACACS+ (ISE device admin)
set firewall group port-group TACACS port 49

# ISE Services (ERS API, Admin, pxGrid)
set firewall group port-group ISE_SERVICES port 443
set firewall group port-group ISE_SERVICES port 9060
set firewall group port-group ISE_SERVICES port 8910

# Active Directory (Kerberos, LDAP, DNS, GC)
set firewall group port-group AD_PORTS port 88
set firewall group port-group AD_PORTS port 389
set firewall group port-group AD_PORTS port 636
set firewall group port-group AD_PORTS port 3268
set firewall group port-group AD_PORTS port 3269
set firewall group port-group AD_PORTS port 464
set firewall group port-group AD_PORTS port 53
set firewall group port-group AD_PORTS port 445
set firewall group port-group AD_PORTS port 135

# Keycloak (OIDC/SAML)
set firewall group port-group KEYCLOAK port 8080
set firewall group port-group KEYCLOAK port 8443

# --- PKI & SECRETS ---

# Vault (API and Raft cluster)
set firewall group port-group VAULT port 8200
set firewall group port-group VAULT_CLUSTER port 8201

# --- STORAGE ---

# NFS (NAS mounts)
set firewall group port-group NFS port 111
set firewall group port-group NFS port 2049
set firewall group port-group NFS port 20048

# SMB/CIFS (Windows shares, Synology)
set firewall group port-group SMB port 445
set firewall group port-group SMB port 139

# iSCSI
set firewall group port-group ISCSI port 3260

# Synology DSM
set firewall group port-group SYNOLOGY port 5000
set firewall group port-group SYNOLOGY port 5001

# MinIO (S3 API)
set firewall group port-group MINIO port 9000
set firewall group port-group MINIO port 9001

# Gitea
set firewall group port-group GITEA port 3000
set firewall group port-group GITEA port 22

# --- WIRELESS ---

# CAPWAP (AP to WLC)
set firewall group port-group CAPWAP port 5246
set firewall group port-group CAPWAP port 5247

# --- MONITORING & SIEM ---

# Wazuh (syslog, agent, API)
set firewall group port-group WAZUH_PORTS port 514
set firewall group port-group WAZUH_PORTS port 1514
set firewall group port-group WAZUH_PORTS port 1515
set firewall group port-group WAZUH_PORTS port 55000

# Wazuh Dashboard/Indexer
set firewall group port-group WAZUH_WEB port 443
set firewall group port-group WAZUH_WEB port 9200

# Zabbix
set firewall group port-group ZABBIX port 10050
set firewall group port-group ZABBIX port 10051

# Prometheus/Grafana
set firewall group port-group PROMETHEUS port 9090
set firewall group port-group GRAFANA port 3000

# SNMP
set firewall group port-group SNMP port 161
set firewall group port-group SNMP port 162

# Syslog (UDP/TCP)
set firewall group port-group SYSLOG port 514

# --- KUBERNETES ---

# k3s Cluster Ports
set firewall group port-group K3S_API port 6443
set firewall group port-group K3S_ETCD port 2379-2380
set firewall group port-group K3S_KUBELET port 10250
set firewall group port-group K3S_VXLAN port 8472

# Cilium Ports
set firewall group port-group CILIUM_HEALTH port 4240
set firewall group port-group CILIUM_HUBBLE port 4244

# MetalLB (if using L2 mode with memberlist)
set firewall group port-group METALLB port 7946

# NodePort Range
set firewall group port-group NODEPORT port 30000-32767

# --- ROUTING ---

# BGP
set firewall group port-group BGP port 179

# VRRP (protocol 112, not TCP/UDP - handled separately in rules)
# POST-3.2: Verify ALL firewall groups created
run show firewall group
POST-3.2 Validation: Confirm ADMINS group has correct IPs
# Verify ADMINS group members (should show 3 IPs)
run show firewall group address-group ADMINS

# Expected: 10.50.10.106, 10.50.10.107, 10.50.10.108
POST-3.2 Validation: Count all groups (address + port + network)
# Count address groups (expected: 22)
# ADMINS, AD_DCS, DNS_SERVERS, GITEA, IPA_NODES, IPMI_NODES, IPSK_NODES, ISE_NODES,
# K3S_MASTERS, K3S_NODES, KEYCLOAK_NODES, KVM_HOSTS, MINIO, NAS_NODES, PFSENSE,
# SWITCHES, VAULT_NODES, VYOS_NODES, WAZUH_MANAGER, WAZUH_NODES, WLC_NODES, ZABBIX
run show firewall group | awk '/address-group/{count++} END{print "Address groups:", count}'

# Count port groups (expected: 31)
# SSH, HTTPS, HTTP, MGMT_WEB, DNS_PORTS, NTP, RADIUS, TACACS, ISE_SERVICES, AD_PORTS,
# KEYCLOAK, VAULT, VAULT_CLUSTER, NFS, SMB, ISCSI, SYNOLOGY, MINIO, GITEA, CAPWAP,
# WAZUH_PORTS, WAZUH_WEB, ZABBIX, PROMETHEUS, GRAFANA, SNMP, SYSLOG, K3S_API,
# K3S_ETCD, K3S_KUBELET, K3S_VXLAN, CILIUM_HEALTH, CILIUM_HUBBLE, METALLB, NODEPORT, BGP
run show firewall group | awk '/port-group/{count++} END{print "Port groups:", count}'

# Count network groups (expected: 10)
# RFC1918, INTERNAL, NET_INFRA, NET_DATA, NET_VOICE, NET_GUEST, NET_IOT,
# NET_SECURITY, NET_SERVICES, K8S_SERVICES
run show firewall group | awk '/network-group/{count++} END{print "Network groups:", count}'

If ADMINS group is missing or has wrong IPs, SSH from your workstations will be BLOCKED after firewall is applied. Fix before proceeding.

9.4. 3.3 Base Firewall Rules

Create reusable rule sets:

# Allow established/related (used by all zones)
set firewall ipv4 name ALLOW_ESTABLISHED default-action drop
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 action accept
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 state established
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 state related
set firewall ipv4 name ALLOW_ESTABLISHED rule 20 action drop
set firewall ipv4 name ALLOW_ESTABLISHED rule 20 state invalid

9.5. 3.4 WAN Inbound Rules

# WAN to LOCAL (firewall services)
set firewall ipv4 name WAN_LOCAL default-action drop
set firewall ipv4 name WAN_LOCAL rule 10 action accept
set firewall ipv4 name WAN_LOCAL rule 10 state established
set firewall ipv4 name WAN_LOCAL rule 10 state related
set firewall ipv4 name WAN_LOCAL rule 20 action accept
set firewall ipv4 name WAN_LOCAL rule 20 protocol icmp
set firewall ipv4 name WAN_LOCAL rule 20 icmp type-name echo-request
set firewall ipv4 name WAN_LOCAL rule 20 limit rate 5/second

# LOCAL to WAN (router outbound - updates, NTP, DNS queries, etc.)
# CRITICAL: Without this, vyos cannot reach internet!
set firewall ipv4 name LOCAL_WAN default-action accept
set firewall ipv4 name LOCAL_WAN description 'Router outbound to WAN'

# LOCAL to MGMT (router to infrastructure - BIND, ISE, Vault, etc.)
# CRITICAL: Without this, vyos cannot reach internal services!
set firewall ipv4 name LOCAL_MGMT default-action accept
set firewall ipv4 name LOCAL_MGMT description 'Router to MGMT infrastructure'

# WAN to any internal (drop all unsolicited)
set firewall ipv4 name WAN_IN default-action drop
set firewall ipv4 name WAN_IN rule 10 action accept
set firewall ipv4 name WAN_IN rule 10 state established
set firewall ipv4 name WAN_IN rule 10 state related

9.6. 3.5 LAN to WAN Rules (Outbound)

# MGMT to WAN (full access)
set firewall ipv4 name MGMT_WAN default-action accept

# DATA to WAN (full access)
set firewall ipv4 name DATA_WAN default-action accept

# VOICE to WAN (limited - SIP/RTP for cloud PBX if needed)
set firewall ipv4 name VOICE_WAN default-action accept

# GUEST to WAN (full internet access)
set firewall ipv4 name GUEST_WAN default-action accept

# IOT to WAN (limited - only specific services)
set firewall ipv4 name IOT_WAN default-action drop
set firewall ipv4 name IOT_WAN rule 10 action accept
set firewall ipv4 name IOT_WAN rule 10 state established
set firewall ipv4 name IOT_WAN rule 10 state related
set firewall ipv4 name IOT_WAN rule 20 action accept
set firewall ipv4 name IOT_WAN rule 20 protocol tcp
set firewall ipv4 name IOT_WAN rule 20 destination port 80,443
set firewall ipv4 name IOT_WAN rule 20 description 'Allow HTTP/HTTPS'
set firewall ipv4 name IOT_WAN rule 30 action accept
set firewall ipv4 name IOT_WAN rule 30 protocol udp
set firewall ipv4 name IOT_WAN rule 30 destination port 123
set firewall ipv4 name IOT_WAN rule 30 description 'Allow NTP'
set firewall ipv4 name IOT_WAN rule 60 action accept
set firewall ipv4 name IOT_WAN rule 60 protocol udp
set firewall ipv4 name IOT_WAN rule 60 destination port 5246
set firewall ipv4 name IOT_WAN rule 60 description 'Allow CAPWAP control (OEAP)'
set firewall ipv4 name IOT_WAN rule 70 action accept
set firewall ipv4 name IOT_WAN rule 70 protocol udp
set firewall ipv4 name IOT_WAN rule 70 destination port 5247
set firewall ipv4 name IOT_WAN rule 70 description 'Allow CAPWAP data (OEAP)'

9.7. 3.6 Inter-VLAN Rules

# MGMT to DATA (allow - admin access)
set firewall ipv4 name MGMT_DATA default-action accept

# MGMT to all internal zones (admin access)
set firewall ipv4 name MGMT_VOICE default-action accept
set firewall ipv4 name MGMT_GUEST default-action accept
set firewall ipv4 name MGMT_IOT default-action accept

# DATA to MGMT (allow access to infrastructure services)
set firewall ipv4 name DATA_MGMT default-action drop
set firewall ipv4 name DATA_MGMT rule 10 action accept
set firewall ipv4 name DATA_MGMT rule 10 state established
set firewall ipv4 name DATA_MGMT rule 10 state related
set firewall ipv4 name DATA_MGMT rule 20 action accept
set firewall ipv4 name DATA_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name DATA_MGMT rule 20 destination group address-group DNS_SERVERS
set firewall ipv4 name DATA_MGMT rule 20 destination group port-group DNS_PORTS
set firewall ipv4 name DATA_MGMT rule 20 description 'Allow DNS'
set firewall ipv4 name DATA_MGMT rule 30 action accept
set firewall ipv4 name DATA_MGMT rule 30 protocol tcp_udp
set firewall ipv4 name DATA_MGMT rule 30 destination group address-group WAZUH_MANAGER
set firewall ipv4 name DATA_MGMT rule 30 destination group port-group WAZUH_PORTS
set firewall ipv4 name DATA_MGMT rule 30 description 'Allow Wazuh agent'
set firewall ipv4 name DATA_MGMT rule 40 action accept
set firewall ipv4 name DATA_MGMT rule 40 destination group network-group K8S_SERVICES
set firewall ipv4 name DATA_MGMT rule 40 destination port 80,443
set firewall ipv4 name DATA_MGMT rule 40 protocol tcp
set firewall ipv4 name DATA_MGMT rule 40 description 'Allow K8s services'

# VOICE to MGMT (limited - DHCP, DNS, NTP, TFTP for phones)
set firewall ipv4 name VOICE_MGMT default-action drop
set firewall ipv4 name VOICE_MGMT rule 10 action accept
set firewall ipv4 name VOICE_MGMT rule 10 state established
set firewall ipv4 name VOICE_MGMT rule 10 state related
set firewall ipv4 name VOICE_MGMT rule 20 action accept
set firewall ipv4 name VOICE_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name VOICE_MGMT rule 20 destination group address-group DNS_SERVERS
set firewall ipv4 name VOICE_MGMT rule 20 destination group port-group DNS_PORTS

# GUEST to internal (BLOCK - internet only)
set firewall ipv4 name GUEST_MGMT default-action drop
set firewall ipv4 name GUEST_DATA default-action drop
set firewall ipv4 name GUEST_VOICE default-action drop
set firewall ipv4 name GUEST_IOT default-action drop

# IOT to internal (BLOCK - limited exceptions)
set firewall ipv4 name IOT_MGMT default-action drop
set firewall ipv4 name IOT_MGMT rule 10 action accept
set firewall ipv4 name IOT_MGMT rule 10 state established
set firewall ipv4 name IOT_MGMT rule 10 state related
set firewall ipv4 name IOT_MGMT rule 20 action accept
set firewall ipv4 name IOT_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name IOT_MGMT rule 20 destination group address-group WAZUH_MANAGER
set firewall ipv4 name IOT_MGMT rule 20 destination group port-group WAZUH_PORTS
set firewall ipv4 name IOT_MGMT rule 20 description 'Allow Wazuh agent'

set firewall ipv4 name IOT_DATA default-action drop
set firewall ipv4 name IOT_VOICE default-action drop
set firewall ipv4 name IOT_GUEST default-action drop

# SECURITY zone rules (crown jewels - Vault, ISE)
# MGMT → SECURITY (admin access)
set firewall ipv4 name MGMT_SECURITY default-action accept

# DATA → SECURITY (allow access to Vault API, ISE services)
set firewall ipv4 name DATA_SECURITY default-action drop
set firewall ipv4 name DATA_SECURITY rule 10 action accept
set firewall ipv4 name DATA_SECURITY rule 10 state established
set firewall ipv4 name DATA_SECURITY rule 10 state related
set firewall ipv4 name DATA_SECURITY rule 20 action accept
set firewall ipv4 name DATA_SECURITY rule 20 protocol tcp
set firewall ipv4 name DATA_SECURITY rule 20 destination port 8200
set firewall ipv4 name DATA_SECURITY rule 20 description 'Vault API'
set firewall ipv4 name DATA_SECURITY rule 30 action accept
set firewall ipv4 name DATA_SECURITY rule 30 protocol tcp
set firewall ipv4 name DATA_SECURITY rule 30 destination port 1812,1813
set firewall ipv4 name DATA_SECURITY rule 30 description 'RADIUS auth/acct'

# SERVICES → SECURITY (allow services to reach Vault/ISE)
set firewall ipv4 name SERVICES_SECURITY default-action drop
set firewall ipv4 name SERVICES_SECURITY rule 10 action accept
set firewall ipv4 name SERVICES_SECURITY rule 10 state established
set firewall ipv4 name SERVICES_SECURITY rule 10 state related
set firewall ipv4 name SERVICES_SECURITY rule 20 action accept
set firewall ipv4 name SERVICES_SECURITY rule 20 protocol tcp
set firewall ipv4 name SERVICES_SECURITY rule 20 destination port 8200
set firewall ipv4 name SERVICES_SECURITY rule 20 description 'Vault API'

# SECURITY → SERVICES (allow Vault/ISE to reach LDAP, DNS, etc.)
set firewall ipv4 name SECURITY_SERVICES default-action drop
set firewall ipv4 name SECURITY_SERVICES rule 10 action accept
set firewall ipv4 name SECURITY_SERVICES rule 10 state established
set firewall ipv4 name SECURITY_SERVICES rule 10 state related
set firewall ipv4 name SECURITY_SERVICES rule 20 action accept
set firewall ipv4 name SECURITY_SERVICES rule 20 protocol tcp
set firewall ipv4 name SECURITY_SERVICES rule 20 destination port 389,636
set firewall ipv4 name SECURITY_SERVICES rule 20 description 'LDAP/LDAPS'

# SERVICES zone rules
# MGMT → SERVICES (admin access)
set firewall ipv4 name MGMT_SERVICES default-action accept

# DATA → SERVICES (allow access to Keycloak, FreeIPA, etc.)
set firewall ipv4 name DATA_SERVICES default-action drop
set firewall ipv4 name DATA_SERVICES rule 10 action accept
set firewall ipv4 name DATA_SERVICES rule 10 state established
set firewall ipv4 name DATA_SERVICES rule 10 state related
set firewall ipv4 name DATA_SERVICES rule 20 action accept
set firewall ipv4 name DATA_SERVICES rule 20 protocol tcp
set firewall ipv4 name DATA_SERVICES rule 20 destination port 80,443
set firewall ipv4 name DATA_SERVICES rule 20 description 'Web services'
set firewall ipv4 name DATA_SERVICES rule 30 action accept
set firewall ipv4 name DATA_SERVICES rule 30 protocol tcp
set firewall ipv4 name DATA_SERVICES rule 30 destination port 389,636
set firewall ipv4 name DATA_SERVICES rule 30 description 'LDAP/LDAPS'

9.8. 3.7 LOCAL Zone Rules (Firewall Services)

# Allow all zones to reach firewall for DHCP, DNS, SSH
set firewall ipv4 name MGMT_LOCAL default-action drop
set firewall ipv4 name MGMT_LOCAL rule 10 action accept
set firewall ipv4 name MGMT_LOCAL rule 10 state established
set firewall ipv4 name MGMT_LOCAL rule 10 state related
set firewall ipv4 name MGMT_LOCAL rule 20 action accept
set firewall ipv4 name MGMT_LOCAL rule 20 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 20 destination port 22
set firewall ipv4 name MGMT_LOCAL rule 20 description 'SSH'
set firewall ipv4 name MGMT_LOCAL rule 30 action accept
set firewall ipv4 name MGMT_LOCAL rule 30 protocol udp
set firewall ipv4 name MGMT_LOCAL rule 30 destination port 67
set firewall ipv4 name MGMT_LOCAL rule 30 description 'DHCP'
set firewall ipv4 name MGMT_LOCAL rule 40 action accept
set firewall ipv4 name MGMT_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name MGMT_LOCAL rule 40 destination port 53
set firewall ipv4 name MGMT_LOCAL rule 40 description 'DNS'
set firewall ipv4 name MGMT_LOCAL rule 50 action accept
set firewall ipv4 name MGMT_LOCAL rule 50 protocol icmp
set firewall ipv4 name MGMT_LOCAL rule 50 description 'ICMP'
set firewall ipv4 name MGMT_LOCAL rule 60 action accept
set firewall ipv4 name MGMT_LOCAL rule 60 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 60 destination port 443
set firewall ipv4 name MGMT_LOCAL rule 60 description 'HTTPS API'

# BGP from k3s Cilium (for LoadBalancer IP advertisement)
set firewall ipv4 name MGMT_LOCAL rule 70 action accept
set firewall ipv4 name MGMT_LOCAL rule 70 source group address-group K3S_NODES
set firewall ipv4 name MGMT_LOCAL rule 70 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 70 destination group port-group BGP
set firewall ipv4 name MGMT_LOCAL rule 70 description 'BGP from Cilium'

# Copy similar rules for other zones (DATA, VOICE, GUEST, IOT)
# SSH restricted to ADMINS group only (prevents lateral movement)
set firewall ipv4 name DATA_LOCAL default-action drop
set firewall ipv4 name DATA_LOCAL rule 10 action accept
set firewall ipv4 name DATA_LOCAL rule 10 state established
set firewall ipv4 name DATA_LOCAL rule 10 state related
set firewall ipv4 name DATA_LOCAL rule 20 action accept
set firewall ipv4 name DATA_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name DATA_LOCAL rule 20 protocol tcp
set firewall ipv4 name DATA_LOCAL rule 20 destination port 22
set firewall ipv4 name DATA_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name DATA_LOCAL rule 25 action accept
set firewall ipv4 name DATA_LOCAL rule 25 source group address-group ADMINS
set firewall ipv4 name DATA_LOCAL rule 25 protocol tcp
set firewall ipv4 name DATA_LOCAL rule 25 destination port 443
set firewall ipv4 name DATA_LOCAL rule 25 description 'HTTPS API from ADMINS only'
set firewall ipv4 name DATA_LOCAL rule 30 action accept
set firewall ipv4 name DATA_LOCAL rule 30 protocol udp
set firewall ipv4 name DATA_LOCAL rule 30 destination port 67
set firewall ipv4 name DATA_LOCAL rule 40 action accept
set firewall ipv4 name DATA_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name DATA_LOCAL rule 40 destination port 53
set firewall ipv4 name DATA_LOCAL rule 50 action accept
set firewall ipv4 name DATA_LOCAL rule 50 protocol icmp

set firewall ipv4 name VOICE_LOCAL default-action drop
set firewall ipv4 name VOICE_LOCAL rule 10 action accept
set firewall ipv4 name VOICE_LOCAL rule 10 state established
set firewall ipv4 name VOICE_LOCAL rule 10 state related
set firewall ipv4 name VOICE_LOCAL rule 30 action accept
set firewall ipv4 name VOICE_LOCAL rule 30 protocol udp
set firewall ipv4 name VOICE_LOCAL rule 30 destination port 67
set firewall ipv4 name VOICE_LOCAL rule 40 action accept
set firewall ipv4 name VOICE_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name VOICE_LOCAL rule 40 destination port 53

set firewall ipv4 name GUEST_LOCAL default-action drop
set firewall ipv4 name GUEST_LOCAL rule 10 action accept
set firewall ipv4 name GUEST_LOCAL rule 10 state established
set firewall ipv4 name GUEST_LOCAL rule 10 state related
set firewall ipv4 name GUEST_LOCAL rule 30 action accept
set firewall ipv4 name GUEST_LOCAL rule 30 protocol udp
set firewall ipv4 name GUEST_LOCAL rule 30 destination port 67
set firewall ipv4 name GUEST_LOCAL rule 40 action accept
set firewall ipv4 name GUEST_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name GUEST_LOCAL rule 40 destination port 53

set firewall ipv4 name IOT_LOCAL default-action drop
set firewall ipv4 name IOT_LOCAL rule 10 action accept
set firewall ipv4 name IOT_LOCAL rule 10 state established
set firewall ipv4 name IOT_LOCAL rule 10 state related
set firewall ipv4 name IOT_LOCAL rule 30 action accept
set firewall ipv4 name IOT_LOCAL rule 30 protocol udp
set firewall ipv4 name IOT_LOCAL rule 30 destination port 67
set firewall ipv4 name IOT_LOCAL rule 40 action accept
set firewall ipv4 name IOT_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name IOT_LOCAL rule 40 destination port 53

# SECURITY_LOCAL (Vault, ISE access to firewall services)
set firewall ipv4 name SECURITY_LOCAL default-action drop
set firewall ipv4 name SECURITY_LOCAL rule 10 action accept
set firewall ipv4 name SECURITY_LOCAL rule 10 state established
set firewall ipv4 name SECURITY_LOCAL rule 10 state related
set firewall ipv4 name SECURITY_LOCAL rule 20 action accept
set firewall ipv4 name SECURITY_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name SECURITY_LOCAL rule 20 protocol tcp
set firewall ipv4 name SECURITY_LOCAL rule 20 destination port 22
set firewall ipv4 name SECURITY_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name SECURITY_LOCAL rule 30 action accept
set firewall ipv4 name SECURITY_LOCAL rule 30 protocol udp
set firewall ipv4 name SECURITY_LOCAL rule 30 destination port 67
set firewall ipv4 name SECURITY_LOCAL rule 40 action accept
set firewall ipv4 name SECURITY_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name SECURITY_LOCAL rule 40 destination port 53
set firewall ipv4 name SECURITY_LOCAL rule 50 action accept
set firewall ipv4 name SECURITY_LOCAL rule 50 protocol icmp

# SERVICES_LOCAL (FreeIPA, Keycloak, BIND access to firewall services)
set firewall ipv4 name SERVICES_LOCAL default-action drop
set firewall ipv4 name SERVICES_LOCAL rule 10 action accept
set firewall ipv4 name SERVICES_LOCAL rule 10 state established
set firewall ipv4 name SERVICES_LOCAL rule 10 state related
set firewall ipv4 name SERVICES_LOCAL rule 20 action accept
set firewall ipv4 name SERVICES_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name SERVICES_LOCAL rule 20 protocol tcp
set firewall ipv4 name SERVICES_LOCAL rule 20 destination port 22
set firewall ipv4 name SERVICES_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name SERVICES_LOCAL rule 30 action accept
set firewall ipv4 name SERVICES_LOCAL rule 30 protocol udp
set firewall ipv4 name SERVICES_LOCAL rule 30 destination port 67
set firewall ipv4 name SERVICES_LOCAL rule 40 action accept
set firewall ipv4 name SERVICES_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name SERVICES_LOCAL rule 40 destination port 53
set firewall ipv4 name SERVICES_LOCAL rule 50 action accept
set firewall ipv4 name SERVICES_LOCAL rule 50 protocol icmp
POST-3.7 Validation: Verify *_LOCAL SSH rules with ADMINS restriction
# Check that SSH (rule 20) has source ADMINS restriction
# MGMT_LOCAL rule 20 may NOT have ADMINS (allows Ansible from any MGMT host)
# DATA, SECURITY, SERVICES should have ADMINS restriction
for zone in DATA SECURITY SERVICES; do
  echo -n "${zone}_LOCAL rule 20 (SSH): "
  show firewall ipv4 name ${zone}_LOCAL rule 20 | grep -q "ADMINS" && echo "✓ ADMINS restricted" || echo "✗ UNRESTRICTED - FIX THIS"
done
POST-3.7 Validation: Confirm untrusted zones have NO SSH (VOICE, GUEST, IOT)
# These zones should NOT have port 22 rules
for zone in VOICE GUEST IOT; do
  echo -n "${zone}_LOCAL: "
  show firewall ipv4 name ${zone}_LOCAL | grep -q "dport 22" && echo "✗ HAS SSH - FIX THIS" || echo "✓ No SSH (secure)"
done
POST-3.7 Validation: Verify DNS rules have protocol tcp_udp
# All *_LOCAL rule 40 (DNS) must have protocol tcp_udp
for zone in MGMT DATA VOICE GUEST IOT SECURITY SERVICES; do
  echo -n "${zone}_LOCAL rule 40: "
  show firewall ipv4 name ${zone}_LOCAL rule 40 | grep -q "tcp_udp" && echo "✓ tcp_udp" || echo "✗ MISSING"
done
POST-3.7 Validation: Verify port-group rules have protocol tcp_udp
# Rules using port-group must specify protocol
for rule in "DATA_MGMT 20" "DATA_MGMT 30" "VOICE_MGMT 20" "IOT_MGMT 20"; do
  name=$(echo $rule | cut -d' ' -f1)
  num=$(echo $rule | cut -d' ' -f2)
  echo -n "$name rule $num: "
  show firewall ipv4 name $name rule $num | grep -q "tcp_udp" && echo "✓ tcp_udp" || echo "✗ MISSING"
done

Security verification (9 zones):

  • MGMT_LOCAL has SSH (rule 20) - for infrastructure automation (Ansible)

  • DATA_LOCAL has SSH (rule 20) with source ADMINS - your workstations ONLY

  • SECURITY_LOCAL has SSH (rule 20) with source ADMINS - Vault/ISE admin access

  • SERVICES_LOCAL has SSH (rule 20) with source ADMINS - FreeIPA/Keycloak admin

  • VOICE_LOCAL, GUEST_LOCAL, IOT_LOCAL - NO SSH rules (blocked)

If any *_LOCAL rule 20 is missing source group address-group ADMINS, ANY device in that zone can SSH to the firewall. Fix immediately.

9.9. 3.8 Apply Zone Policies

# WAN zone
set firewall zone WAN from MGMT firewall name MGMT_WAN
set firewall zone WAN from DATA firewall name DATA_WAN
set firewall zone WAN from VOICE firewall name VOICE_WAN
set firewall zone WAN from GUEST firewall name GUEST_WAN
set firewall zone WAN from IOT firewall name IOT_WAN
set firewall zone WAN from LOCAL firewall name LOCAL_WAN

# MGMT zone
set firewall zone MGMT from WAN firewall name WAN_IN
set firewall zone MGMT from DATA firewall name DATA_MGMT
set firewall zone MGMT from VOICE firewall name VOICE_MGMT
set firewall zone MGMT from GUEST firewall name GUEST_MGMT
set firewall zone MGMT from IOT firewall name IOT_MGMT
set firewall zone MGMT from LOCAL firewall name LOCAL_MGMT

# DATA zone
set firewall zone DATA from WAN firewall name WAN_IN
set firewall zone DATA from MGMT firewall name MGMT_DATA
set firewall zone DATA from GUEST firewall name GUEST_DATA
set firewall zone DATA from IOT firewall name IOT_DATA

# VOICE zone
set firewall zone VOICE from WAN firewall name WAN_IN
set firewall zone VOICE from MGMT firewall name MGMT_VOICE

# GUEST zone
set firewall zone GUEST from WAN firewall name WAN_IN
set firewall zone GUEST from MGMT firewall name MGMT_GUEST

# IOT zone
set firewall zone IOT from WAN firewall name WAN_IN
set firewall zone IOT from MGMT firewall name MGMT_IOT

# SECURITY zone (crown jewels - Vault, ISE)
set firewall zone SECURITY from WAN firewall name WAN_IN
set firewall zone SECURITY from MGMT firewall name MGMT_SECURITY
set firewall zone SECURITY from DATA firewall name DATA_SECURITY
set firewall zone SECURITY from SERVICES firewall name SERVICES_SECURITY

# SERVICES zone (infrastructure VMs)
set firewall zone SERVICES from WAN firewall name WAN_IN
set firewall zone SERVICES from MGMT firewall name MGMT_SERVICES
set firewall zone SERVICES from DATA firewall name DATA_SERVICES
set firewall zone SERVICES from SECURITY firewall name SECURITY_SERVICES

# LOCAL zone (firewall itself)
set firewall zone LOCAL from WAN firewall name WAN_LOCAL
set firewall zone LOCAL from MGMT firewall name MGMT_LOCAL
set firewall zone LOCAL from DATA firewall name DATA_LOCAL
set firewall zone LOCAL from VOICE firewall name VOICE_LOCAL
set firewall zone LOCAL from GUEST firewall name GUEST_LOCAL
set firewall zone LOCAL from IOT firewall name IOT_LOCAL
set firewall zone LOCAL from SECURITY firewall name SECURITY_LOCAL
set firewall zone LOCAL from SERVICES firewall name SERVICES_LOCAL

9.10. 3.9 Commit Firewall

commit
save

9.11. 3.10 Post-Validation

POST-1: Verify all 9 zones created (awk summary)
run show firewall zone | awk 'NR>2 && NF>=2 {printf "%-12s → %s\n", $1, $2}'
Expected output
DATA         → eth0.10
GUEST        → eth0.30
IOT          → eth0.40
LOCAL        → LOCAL
MGMT         → eth0
SECURITY     → eth0.110
SERVICES     → eth0.120
VOICE        → eth0.20
WAN          → eth1
POST-2: Zone-to-zone policy matrix
# View zone policies (VyOS table format shows zone, interface, from-zone, firewall)
run show firewall zone | head -50
POST-3: Firewall rule count per ruleset (awk pivot table)
# Count rules per firewall ruleset
# VyOS format: ipv4 Firewall "name RULESET_NAME"
run show firewall | awk -F'"' '
  /ipv4 Firewall "name/ {
    split($2, a, " ")
    name=a[2]
  }
  /^[0-9]+|^default/ && name != "" {
    rules[name]++
  }
  END {
    printf "%-25s %s\n", "FIREWALL_RULESET", "RULES"
    printf "%-25s %s\n", "----------------", "-----"
    for (n in rules) printf "%-25s %d\n", n, rules[n] | "sort"
  }
'
POST-4: Verify nftables backend loaded
sudo nft list ruleset | awk '/chain/ {print}' | head -20
POST-5: JSON export for audit (jq)
# Export full config as JSON and extract zone info
run show configuration json | jq '.firewall.zone | keys'
POST-6: Zone membership verification (jq deep inspect)
# Show each zone's member interfaces
run show configuration json | jq -r '
  .firewall.zone | to_entries[] |
  "\(.key): \(.value.member.interface // "LOCAL")"
'
POST-7: xargs batch verification of critical rules
# Verify ADMINS source restriction on SSH for trusted zones
for z in DATA SECURITY SERVICES; do
  show firewall ipv4 name ${z}_LOCAL | grep -q "ADMINS" && echo "✓ ${z}_LOCAL: ADMINS restricted" || echo "✗ ${z}_LOCAL: MISSING ADMINS - SECURITY HOLE"
done
POST-8: Confirm untrusted zones blocked from SSH
# These zones should NOT have SSH (port 22) rules
for z in VOICE GUEST IOT; do
  show firewall ipv4 name ${z}_LOCAL | grep -q "dport 22" && echo "✗ ${z}_LOCAL: HAS SSH - FIX IMMEDIATELY" || echo "✓ ${z}_LOCAL: No SSH (secure)"
done
POST-9: Full security audit one-liner (awk + xargs)
# Combined security check - single command
{
  echo "=== ZONE COUNT ==="
  run show firewall zone | awk 'NR>2 && NF>=2' | wc -l | xargs -I{} echo "Zones: {} (expected: 9)"
  echo ""
  echo "=== ADMINS GROUP ==="
  run show firewall group address-group ADMINS | awk '/address/ {print "  " $2}'
  echo ""
  echo "=== SSH SECURITY ==="
  for z in DATA SECURITY SERVICES; do
    run show firewall ipv4 name ${z}_LOCAL | grep -q "ADMINS" && echo "✓ ${z}_LOCAL: ADMINS" || echo "✗ ${z}_LOCAL: OPEN"
  done
  for z in VOICE GUEST IOT; do
    run show firewall ipv4 name ${z}_LOCAL | grep -q "port 22" && echo "✗ ${z}_LOCAL: SSH FOUND" || echo "✓ ${z}_LOCAL: No SSH"
  done
}
POST-10: Verify LOCAL_WAN (router internet connectivity)
# CRITICAL: Without LOCAL_WAN, router cannot reach internet
# Verify zone policy exists
run show firewall zone-policy | grep -A1 "WAN" | grep -q "LOCAL" && echo "✓ LOCAL_WAN zone policy exists" || echo "✗ MISSING: set firewall zone WAN from LOCAL firewall name LOCAL_WAN"

# Test actual connectivity
ping -c 2 8.8.8.8 >/dev/null 2>&1 && echo "✓ Router can reach internet" || echo "✗ Router cannot reach internet - check LOCAL_WAN"
POST-11: Verify LOCAL_MGMT (router to infrastructure connectivity)
# CRITICAL: Without LOCAL_MGMT, router cannot reach BIND, ISE, Vault, etc.
# Verify zone policy exists
run show firewall zone-policy | awk '/^MGMT/,/^[A-Z]/' | grep -q "LOCAL" && echo "✓ LOCAL_MGMT zone policy exists" || echo "✗ MISSING: set firewall zone MGMT from LOCAL firewall name LOCAL_MGMT"

# Test actual connectivity to BIND
ping -c 2 10.50.1.90 >/dev/null 2>&1 && echo "✓ Router can reach BIND (10.50.1.90)" || echo "✗ Router cannot reach BIND - check LOCAL_MGMT"

Phase 3 Complete Checklist (9 zones):

  • POST-1: Shows 9 zones (WAN, MGMT, DATA, VOICE, GUEST, IOT, SECURITY, SERVICES, LOCAL)

  • POST-5: jq shows all 9 zone keys

  • POST-7: All trusted zones show "ADMINS restricted"

  • POST-8: All untrusted zones show "No SSH (secure)"

  • POST-9: Full audit passes with no "✗" marks

  • POST-10: Router can reach internet (LOCAL_WAN working)

  • POST-11: Router can reach BIND (LOCAL_MGMT working)

If any check fails, do NOT proceed. Fix firewall before continuing.


10. Phase 4: NAT Configuration

10.1. 4.0 Pre-Validation

# PRE-1: Verify Phase 3 zones exist
show firewall zone | grep -E "WAN|MGMT|DATA"

# Expected: Zones listed
# PRE-2: Document current NAT state
show nat source rules
show nat destination rules

# Expected: Empty or minimal rules

10.2. 4.1 Source NAT (Masquerade) - Enterprise Per-VLAN

Interface mapping reminder:

  • vyos-02: eth0=LAN (br-mgmt), eth1=WAN (br-wan)

  • vyos-01: eth0=WAN, eth1=LAN (original design)

Adjust outbound-interface accordingly. Commands below are for vyos-02 using eth1.

configure

# =============================================================================
# ENTERPRISE NAT: Per-VLAN Source NAT Rules
# Each VLAN gets its own rule for:
# - Granular logging and accounting
# - Per-VLAN outbound policies (future: bandwidth limits, QoS)
# - Selective NAT (e.g., block IoT from internet)
# =============================================================================

# Rule 100: INFRA (VLAN 100) - Infrastructure management traffic
set nat source rule 100 outbound-interface name eth1
set nat source rule 100 source group network-group NET_INFRA
set nat source rule 100 translation address masquerade
set nat source rule 100 description 'SNAT INFRA → WAN'

# Rule 110: DATA (VLAN 10) - Corporate devices
set nat source rule 110 outbound-interface name eth1
set nat source rule 110 source group network-group NET_DATA
set nat source rule 110 translation address masquerade
set nat source rule 110 description 'SNAT DATA → WAN'

# Rule 120: VOICE (VLAN 20) - VoIP (may need different treatment for SIP ALG)
set nat source rule 120 outbound-interface name eth1
set nat source rule 120 source group network-group NET_VOICE
set nat source rule 120 translation address masquerade
set nat source rule 120 description 'SNAT VOICE → WAN'

# Rule 130: GUEST (VLAN 30) - Guest WiFi
set nat source rule 130 outbound-interface name eth1
set nat source rule 130 source group network-group NET_GUEST
set nat source rule 130 translation address masquerade
set nat source rule 130 description 'SNAT GUEST → WAN'

# Rule 140: IOT (VLAN 40) - IoT devices
# NOTE: Enable only if IoT needs internet. Can be disabled for LAN-only IoT.
set nat source rule 140 outbound-interface name eth1
set nat source rule 140 source group network-group NET_IOT
set nat source rule 140 translation address masquerade
set nat source rule 140 description 'SNAT IOT → WAN'
# set nat source rule 140 disable  # Uncomment to block IoT internet

# Rule 150: SECURITY (VLAN 110) - Crown jewels (Vault, ISE)
# WARNING: Security tier typically should NOT need outbound internet
# Enable only for specific requirements (updates, CRL fetch, etc.)
set nat source rule 150 outbound-interface name eth1
set nat source rule 150 source group network-group NET_SECURITY
set nat source rule 150 translation address masquerade
set nat source rule 150 description 'SNAT SECURITY → WAN'
# set nat source rule 150 disable  # Uncomment to block (recommended)

# Rule 160: SERVICES (VLAN 120) - Infrastructure VMs
set nat source rule 160 outbound-interface name eth1
set nat source rule 160 source group network-group NET_SERVICES
set nat source rule 160 translation address masquerade
set nat source rule 160 description 'SNAT SERVICES → WAN'

commit
save
Verification (VyOS awk mastery)
# Verify all SNAT rules created
run show nat source rules | awk '/^[0-9]/{print "Rule "$1": "$NF}'

# Count rules per interface
run show nat source rules | awk '/outbound-interface/{iface[$NF]++} END{for(i in iface) print i": "iface[i]" rules"}'

# Show rules using network-groups (enterprise pattern)
run show nat source rules | awk '/network-group/{print}'

10.3. 4.2 Destination NAT (Port Forwards)

No DNAT required initially. Examples below for future reference.

# Example: Forward WAN:443 to internal web server
# NOTE: Use eth1 for vyos-02 (WAN interface)
# set nat destination rule 10 inbound-interface name eth1
# set nat destination rule 10 destination port 443
# set nat destination rule 10 protocol tcp
# set nat destination rule 10 translation address 10.50.1.x
# set nat destination rule 10 translation port 443
# set nat destination rule 10 description 'HTTPS to internal server'

# Example: Forward WAN:8443 to ISE Admin GUI
# set nat destination rule 20 inbound-interface name eth1
# set nat destination rule 20 destination port 8443
# set nat destination rule 20 protocol tcp
# set nat destination rule 20 translation address 10.50.1.20
# set nat destination rule 20 translation port 8443
# set nat destination rule 20 description 'ISE Admin GUI'

10.4. 4.10 Post-Validation

# POST-1: Count SNAT rules (should be 7: rules 100-160)
run show nat source rule | awk '/^[0-9]+/{count++} END{print "Total SNAT rules:", count}'

# Expected: Total SNAT rules: 7
# POST-2: Formatted table - rule → network-group mapping
run show nat source rule | awk '/^[0-9]+/{printf "Rule %s → %s\n", $1, $2}'

# Expected:
# Rule 100 → @N_NET_INFRA
# Rule 110 → @N_NET_DATA
# Rule 120 → @N_NET_VOICE
# Rule 130 → @N_NET_GUEST
# Rule 140 → @N_NET_IOT
# Rule 150 → @N_NET_SECURITY
# Rule 160 → @N_NET_SERVICES
# POST-3: Pivot by interface (all should be eth1 for vyos-02)
run show nat source rule | awk '/^[0-9]+/{iface[$5]++} END{for(i in iface) print i": "iface[i]" rules"}'

# Expected: eth1: 7 rules
# POST-4: One-liner status check
run show nat source rule | awk '/^[0-9]+/{r++; g[$2]++} END{print r" rules,", length(g)" network-groups"}'

# Expected: 7 rules, 7 network-groups
# POST-5: Test outbound connectivity (from VyOS)
ping -c 2 8.8.8.8

# Expected: Replies from Google DNS
# POST-6: Verify nftables NAT chain (low-level)
sudo nft list table ip vyos_nat 2>/dev/null | awk '/chain POSTROUTING/,/^}/' | head -15

# Expected: Multiple masquerade rules matching network-groups
# POST-7: NAT translations (active flows) - will populate when traffic exists
run show nat source translations | awk 'NR>1{proto[$1]++} END{for(p in proto) print p": "proto[p]" flows"}'

# Expected: Initially empty, shows tcp/udp counts when traffic flows

11. Phase 5: DHCP Server

11.1. 5.0 Pre-Validation

# PRE: Verify no existing DHCP config
show service dhcp-server

# Expected: Configuration under specified path is empty
configure

11.2. 5.1 DHCP for DATA VLAN

# DATA VLAN DHCP Pool
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 subnet-id 10
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 range 0 start 10.50.10.100
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 range 0 stop 10.50.10.199
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option default-router 10.50.10.1
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 lease 86400

# DHCP Reservations - Admin Workstations (ADMINS firewall group)
# Purpose: Static IPs for privileged access - prevents lateral movement
# NOTE: static-mapping MUST be inside subnet path (VyOS 1.4 syntax)
# CRITICAL: VyOS uses 'mac' NOT 'mac-address' (unlike some other DHCP servers)
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-razer mac 98:BB:1E:1F:A7:13
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-razer ip-address 10.50.10.106
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-aw mac 14:F6:D8:7B:31:80
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-aw ip-address 10.50.10.107
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-p50 mac C8:5B:76:C6:59:62
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-p50 ip-address 10.50.10.108
POST-5.1 Validation: Verify DHCP reservations created
# List all static mappings with formatted output
run show configuration commands | grep static-mapping | awk -F'static-mapping ' '{print $2}' | awk '{printf "%-20s %s\n", $1, $2}'
POST-5.1 Validation: Confirm reservation count and details
# Count reservations (expected: 3 for ADMINS workstations)
run show configuration commands | grep -c static-mapping

# Verify specific MACs are mapped (case-insensitive)
run show configuration commands | grep -i "98:BB:1E:1F:A7:13"
run show configuration commands | grep -i "14:F6:D8:7B:31:80"
run show configuration commands | grep -i "C8:5B:76:C6:59:62"
POST-5.1 Validation: Check DHCP server status
# Verify DHCP server is configured for DATA network
run show dhcp server statistics

# Show current leases (will be empty until clients connect)
run show dhcp server leases

After cutover, workstations will get these static IPs via DHCP reservation. The ADMINS firewall group references these IPs - if reservations are wrong, SSH access breaks.

Quick MAC verification from workstation:

ip link show | awk '/ether/{print $2}'

11.3. 5.2 DHCP for VOICE VLAN

set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 subnet-id 20
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 range 0 start 10.50.20.100
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 range 0 stop 10.50.20.199
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option default-router 10.50.20.1
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 lease 86400
# VoIP-specific options (TFTP for phone configs, Option 150 for Cisco phones)
# set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option tftp-server-name <tftp-server>

11.4. 5.3 DHCP for GUEST VLAN

# Guest uses public DNS (not internal) - short lease
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 subnet-id 30
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 range 0 start 10.50.30.100
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 range 0 stop 10.50.30.250
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option default-router 10.50.30.1
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option name-server 1.1.1.1
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option name-server 8.8.8.8
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 lease 3600
Guest VLAN uses public DNS (Cloudflare/Google), not internal DNS. Short lease (1 hour) for security.

11.5. 5.4 DHCP for IOT VLAN

set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 subnet-id 40
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 range 0 start 10.50.40.100
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 range 0 stop 10.50.40.250
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option default-router 10.50.40.1
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 lease 86400

11.6. 5.5 Commit DHCP

commit
save

11.7. 5.6 Post-Validation

# POST-1: Verify DHCP server status
run show service dhcp-server
# POST-2: Verify pool configuration
run show dhcp server leases
MGMT VLAN (10.50.1.0/24) uses static IPs for infrastructure. No DHCP server needed.

12. Phase 6: DNS Architecture

12.1. 6.0 Pre-Validation

# PRE-1: Verify BIND servers are reachable from VyOS
ping -c 1 10.50.1.90
ping -c 1 10.50.1.91

# Expected: Both respond
# PRE-2: Verify BIND resolves internal zones
dig ise-01.inside.domusdigitalis.dev @10.50.1.90 +short

# Expected: 10.50.1.20
# PRE-3: Verify BIND resolves external domains
dig google.com @10.50.1.90 +short

# Expected: Google IPs returned
# PRE-4: Verify Phase 5 DHCP config references BIND
show configuration commands | grep name-server

# Expected: DHCP pools point to BIND servers

12.2. 6.1 Design Decision: BIND Handles All DNS

VyOS does NOT run DNS forwarding. BIND is the enterprise DNS infrastructure.

Component Role

BIND (bind-01, bind-02)

Authoritative for internal zones + recursive resolver for external

VyOS

Routes DNS traffic only (UDP/TCP 53)

DHCP

Hands out BIND IPs as DNS servers

Why this is better:

  • Single DNS infrastructure (BIND) - no duplicate caching/forwarding

  • BIND already configured as recursive resolver (verified: resolves google.com)

  • Simpler VyOS config - fewer services to manage

  • Enterprise pattern - firewalls route, DNS servers resolve

12.3. 6.2 Verify BIND Handles Both Internal and External

# Test internal resolution
dig ise-01.inside.domusdigitalis.dev @10.50.1.90 +short
# Test external resolution
dig google.com @10.50.1.90 +short
# Test secondary BIND
dig google.com @10.50.1.91 +short

12.4. 6.3 DHCP Hands Out BIND Servers

DHCP configuration (Phase 5) uses BIND IPs as DNS servers:

set service dhcp-server shared-network-name DATA subnet ... name-server 10.50.1.90
set service dhcp-server shared-network-name DATA subnet ... name-server 10.50.1.91

Clients receive 10.50.1.90 and 10.50.1.91 as DNS - they query BIND directly.

12.5. 6.4 Post-Validation (from client)

# After DHCP lease renewal, verify DNS servers
resolvectl status | grep -A2 "DNS Servers"
# Test resolution works
dig vault-01.inside.domusdigitalis.dev +short
dig github.com +short

Phase 6 Complete. DNS architecture verified. BIND handles all resolution, VyOS just routes.


13. Phase 7: Threat Intelligence (pfBlockerNG Replacement)

VyOS doesn’t have pfBlockerNG, but we can achieve similar functionality with firewall groups populated from threat feeds.

13.1. 7.0 Pre-Validation

# PRE-1: Verify Phase 6 complete (DNS working)
dig google.com +short

# Expected: Google IPs returned
# PRE-2: Verify internet connectivity (for feed downloads)
curl -sI https://www.spamhaus.org | head -1

# Expected: HTTP/2 200
# PRE-3: Verify firewall configuration exists (Phase 3)
show firewall ipv4

# Expected: WAN_IN and other firewall names listed
# PRE-4: Check scripts directory exists
ls -la /config/scripts/ 2>/dev/null || echo "Will be created"

13.2. 7.1 Create Threat Feed Script

SSH into VyOS and create update script:

sudo su -
cat > /config/scripts/update-threat-feeds.sh << 'EOF'
#!/bin/bash
# Update threat intelligence feeds for VyOS firewall
# Run via cron: 0 4 * * * /config/scripts/update-threat-feeds.sh

FEED_DIR="/config/threat-feeds"
mkdir -p $FEED_DIR

# Spamhaus DROP (Don't Route Or Peer)
curl -s https://www.spamhaus.org/drop/drop.txt | grep -v '^;' | awk '{print $1}' > $FEED_DIR/spamhaus-drop.txt

# Emerging Threats compromised IPs
curl -s https://rules.emergingthreats.net/blockrules/compromised-ips.txt | grep -v '^#' > $FEED_DIR/et-compromised.txt

# Abuse.ch Feodo Tracker (banking trojans C2)
curl -s https://feodotracker.abuse.ch/downloads/ipblocklist.txt | grep -v '^#' > $FEED_DIR/feodo.txt

# Combine all feeds
cat $FEED_DIR/*.txt | sort -u | grep -v '^$' > $FEED_DIR/combined-blocklist.txt

# Update VyOS firewall group
source /opt/vyatta/etc/functions/script-template
configure

# Clear existing entries
delete firewall group network-group THREAT_FEEDS

# Add networks from blocklist
while read -r ip; do
  if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(/[0-9]+)?$ ]]; then
    # Add /32 if no CIDR notation
    [[ $ip =~ / ]] || ip="$ip/32"
    run set firewall group network-group THREAT_FEEDS network "$ip"
  fi
done < $FEED_DIR/combined-blocklist.txt

commit
save
exit

echo "Threat feeds updated: $(wc -l < $FEED_DIR/combined-blocklist.txt) entries"
EOF

chmod +x /config/scripts/update-threat-feeds.sh

13.3. 7.2 Add Threat Feed Firewall Rules

configure

# Block inbound from threat feeds
set firewall ipv4 name WAN_IN rule 5 action drop
set firewall ipv4 name WAN_IN rule 5 source group network-group THREAT_FEEDS
set firewall ipv4 name WAN_IN rule 5 log
set firewall ipv4 name WAN_IN rule 5 description 'Block threat feeds inbound'

# Block outbound to threat feeds (C2 prevention)
set firewall ipv4 name MGMT_WAN rule 5 action drop
set firewall ipv4 name MGMT_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name MGMT_WAN rule 5 log
set firewall ipv4 name MGMT_WAN rule 5 description 'Block threat feeds outbound'

set firewall ipv4 name DATA_WAN rule 5 action drop
set firewall ipv4 name DATA_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name DATA_WAN rule 5 log

set firewall ipv4 name IOT_WAN rule 5 action drop
set firewall ipv4 name IOT_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name IOT_WAN rule 5 log

commit
save

13.4. 7.3 Schedule Threat Feed Updates

set system task-scheduler task update-threat-feeds executable path /config/scripts/update-threat-feeds.sh
set system task-scheduler task update-threat-feeds interval 1d

commit
save

13.5. 7.4 Run Initial Feed Update

sudo /config/scripts/update-threat-feeds.sh

13.6. 7.5 Post-Validation

# POST-1: Verify threat feed files downloaded
ls -la /config/threat-feeds/

# Expected: spamhaus-drop.txt, et-compromised.txt, feodo.txt, combined-blocklist.txt
# POST-2: Verify network-group populated
show firewall group network-group THREAT_FEEDS

# Expected: Multiple network entries (should be 1000+ IPs)
# POST-3: Verify firewall rules reference THREAT_FEEDS
show firewall ipv4 name WAN_IN rule 5

# Expected: source group network-group THREAT_FEEDS, action drop
# POST-4: Verify task scheduler configured
show system task-scheduler

# Expected: update-threat-feeds task with interval 1d
# POST-5: Count blocked IPs
wc -l /config/threat-feeds/combined-blocklist.txt

# Expected: 1000+ entries

14. Phase 8: Suricata IDS

VyOS supports Suricata as an IDS/IPS. This replaces Snort/Suricata packages on pfSense.

14.1. 8.0 Pre-Validation

# PRE-1: Verify Phase 7 complete (threat feeds working)
ls /config/threat-feeds/combined-blocklist.txt

# Expected: File exists
# PRE-2: Verify internet access for rule downloads
curl -sI https://rules.emergingthreats.net | head -1

# Expected: HTTP/2 200
# PRE-3: Check if Suricata already installed
which suricata || echo "Not installed (expected for fresh deploy)"
# PRE-4: Verify sufficient disk space
df -h /var/log/

# Expected: >5GB available for logs

14.2. 8.1 Install Suricata

sudo su -
apt update
apt install -y suricata suricata-update

14.3. 8.2 Configure Suricata

# Backup default config
cp /etc/suricata/suricata.yaml /etc/suricata/suricata.yaml.orig

# Edit config
cat > /etc/suricata/suricata.yaml << 'EOF'
%YAML 1.1
---
vars:
  address-groups:
    HOME_NET: "[10.50.0.0/16]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    DNS_SERVERS: "[{bind-ip},{bind-02-ip}]"

  port-groups:
    HTTP_PORTS: "80,443,8080"
    DNS_PORTS: "53"

default-log-dir: /var/log/suricata/

outputs:
  - eve-log:
      enabled: yes
      filetype: regular
      filename: eve.json
      types:
        - alert
        - http
        - dns
        - tls
        - files
        - smtp

af-packet:
  - interface: eth0
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
  - interface: eth1
    cluster-id: 98
    cluster-type: cluster_flow
    defrag: yes

rule-files:
  - suricata.rules

classification-file: /etc/suricata/classification.config
reference-config-file: /etc/suricata/reference.config
EOF

14.4. 8.3 Update Suricata Rules

suricata-update
suricata-update enable-source et/open
suricata-update

14.5. 8.4 Enable Suricata Service

systemctl enable suricata
systemctl start suricata
systemctl status suricata

14.6. 8.5 Integrate with Wazuh

Wazuh can ingest Suricata alerts. Add to Wazuh agent config:

cat >> /var/ossec/etc/ossec.conf << 'EOF'
<localfile>
  <log_format>json</log_format>
  <location>/var/log/suricata/eve.json</location>
</localfile>
EOF

systemctl restart wazuh-agent

14.7. 8.6 Post-Validation

# POST-1: Verify Suricata service running
systemctl status suricata | grep -E "Active:|loaded"

# Expected: active (running)
# POST-2: Verify Suricata listening on interfaces
suricata --build-info | grep -i "af-packet"

# Expected: af-packet support enabled
# POST-3: Verify rules loaded
suricata-update list-sources | grep -E "enabled|et/open"

# Expected: et/open source enabled
# POST-4: Verify eve.json logging
ls -la /var/log/suricata/eve.json

# Expected: File exists and growing
# POST-5: Test Suricata detection (safe EICAR-like test)
tail -f /var/log/suricata/eve.json &
curl -s http://testmyids.com
# Ctrl+C to stop tail

# Expected: Alert logged for test signature
# POST-6: Verify Wazuh integration
grep -i suricata /var/ossec/etc/ossec.conf

# Expected: localfile entry for eve.json

15. Phase 9: Monitoring Integration

15.1. 9.0 Pre-Validation

# PRE-1: Verify Phase 8 complete (Suricata running)
systemctl status suricata | grep "Active:"

# Expected: active (running)
# PRE-2: Verify connectivity to monitoring infrastructure
ping -c 1 10.50.1.134

# Expected: Reply from Wazuh manager
# PRE-3: Verify Prometheus endpoint reachable (from k3s cluster)
# Run this FROM k3s-master-01 after Phase 9 complete
# curl -s http://10.50.1.2:9100/metrics | head -5
echo "Will verify after node_exporter installed"
# PRE-4: Verify apt repository access
apt-cache policy prometheus-node-exporter

# Expected: Package available

15.2. 9.1 Install Prometheus Node Exporter

sudo su -
apt update
apt install -y prometheus-node-exporter

systemctl enable prometheus-node-exporter
systemctl start prometheus-node-exporter

# Verify
curl -s localhost:9100/metrics | head -5

15.3. 9.2 Install Wazuh Agent

# Add Wazuh repository
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" > /etc/apt/sources.list.d/wazuh.list

apt update
apt install -y wazuh-agent

# Configure manager address
sed -i "s/MANAGER_IP/{wazuh-manager-vip}/g" /var/ossec/etc/ossec.conf

systemctl daemon-reload
systemctl enable wazuh-agent
systemctl start wazuh-agent

15.4. 9.3 Enable SNMP (Optional)

configure

set service snmp community public authorization ro
set service snmp listen-address 10.50.1.2

commit
save

15.5. 9.4 Post-Validation

# POST-1: Verify node_exporter running
systemctl status prometheus-node-exporter | grep "Active:"

# Expected: active (running)
# POST-2: Verify node_exporter metrics available
curl -s http://localhost:9100/metrics | grep -E "^node_cpu|^node_memory" | head -5

# Expected: CPU and memory metrics
# POST-3: Verify Wazuh agent registered
sudo /var/ossec/bin/agent_control -l | head -5

# Expected: Agent listed (may take a minute)
# POST-4: Verify Wazuh agent connected
systemctl status wazuh-agent | grep "Active:"

# Expected: active (running)
# POST-5: Verify SNMP responding (if enabled)
snmpwalk -v2c -c public 10.50.1.2 system 2>/dev/null | head -3 || echo "SNMP not configured (optional)"

# Expected: System info if SNMP enabled
# POST-6: Verify from Prometheus (run from k3s-master-01)
# curl -s http://10.50.1.2:9100/metrics | wc -l

# Expected: Many lines of metrics

16. Phase 10: SSH Hardening and Access

16.1. 10.0 Pre-Validation

# PRE-1: Verify Phase 9 complete (monitoring working)
systemctl status prometheus-node-exporter | grep "Active:"

# Expected: active (running)
# PRE-2: Check current SSH config
show service ssh

# Expected: Default SSH config or empty
# PRE-3: Verify we have SSH key to add
echo "Verify you have user's public SSH key ready"
# Example: ssh-ed25519 AAAAC3... user@host

16.2. 10.1 Configure SSH

configure

# SSH service
set service ssh port 22
set service ssh listen-address 10.50.1.2
set service ssh disable-password-authentication

# Create admin user with SSH key
set system login user evan full-name 'Evan Rosado'
set system login user evan authentication public-keys modestus-razer type ssh-ed25519
set system login user evan authentication public-keys modestus-razer key 'AAAAC3NzaC1lZDI1NTE5AAAAILqgbqJQwk7SikO3mJPVX8/83ULOXLi6iB6G+tM0i9f7'

commit
save

16.3. 10.2 Verify SSH Access

# From workstation
ssh evan@10.50.1.2 "show version"

16.4. 10.3 Post-Validation

# POST-1: Verify SSH service config
show service ssh

# Expected: port 22, password auth disabled
# POST-2: Verify SSH listening
sudo ss -tlnp | grep ":22"

# Expected: Listening on 10.50.1.2:22
# POST-3: Verify user created
show system login user

# Expected: evan user listed
# POST-4: Test SSH from workstation (run from workstation)
ssh -o BatchMode=yes evan@10.50.1.2 "whoami"

# Expected: evan (key-based auth succeeds)
# POST-5: Verify password auth disabled (should fail)
ssh -o PreferredAuthentications=password evan@10.50.1.2 "whoami" 2>&1 | grep -i "permission denied"

# Expected: Permission denied (password auth disabled)

17. Phase 11: API Access

17.1. 11.0 Pre-Validation

# PRE-1: Verify Phase 10 complete (SSH working)
show service ssh

# Expected: SSH configured with key auth
# PRE-2: Check current HTTPS/API config
show service https

# Expected: Empty or minimal config
# PRE-3: Generate API key (run in operational mode)
run generate system api-key

# Expected: Generates a new API key - SAVE THIS!

17.2. 11.1 Enable HTTP API

configure

# Generate API key (run in operational mode first)
# run generate system api-key

set service https api keys id automation key 'YOUR_API_KEY_HERE'
set service https listen-address 10.50.1.2
set service https port 443

commit
save

17.3. 11.2 Test API

curl -k -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"op": "show", "path": ["interfaces"]}' \
  https://10.50.1.2/retrieve

17.4. 11.3 Post-Validation

# POST-1: Verify HTTPS service config
show service https

# Expected: API enabled on port 443
# POST-2: Verify HTTPS listening
sudo ss -tlnp | grep ":443"

# Expected: Listening on 10.50.1.2:443
# POST-3: Test API endpoint (from workstation)
curl -sk https://10.50.1.2/ | head -5

# Expected: VyOS API response or redirect
# POST-4: Test API authentication (replace YOUR_API_KEY)
curl -sk -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"op": "show", "path": ["version"]}' \
  https://10.50.1.2/retrieve | jq .

# Expected: VyOS version info in JSON
# POST-5: Verify API key stored
show configuration commands | grep "api keys"

# Expected: API key ID listed (not the key itself)

18. Phase 12: Git Config Tracking

18.1. 12.0 Pre-Validation

# PRE-1: Verify Phase 11 complete (API working)
show service https

# Expected: HTTPS/API configured
# PRE-2: Check if git already initialized
ls -la /config/.git 2>/dev/null || echo "Git not initialized (expected)"
# PRE-3: Verify config.boot exists
ls -la /config/config.boot

# Expected: File exists with current config

18.2. 12.1 Initialize Git Repository

sudo su -
cd /config
git init
git config user.email "vyos@inside.domusdigitalis.dev"
git config user.name "VyOS Config"
git add config.boot
git commit -m "Initial VyOS configuration"

18.3. 12.2 Post-Commit Hook (Optional)

cat > /config/.git/hooks/post-commit << 'EOF'
#!/bin/bash
# Push config to central git repo after each commit
# git push origin main
EOF
chmod +x /config/.git/hooks/post-commit

18.4. 12.3 Post-Validation

# POST-1: Verify git repo initialized
ls -la /config/.git/

# Expected: .git directory with objects, refs, etc.
# POST-2: Verify initial commit exists
cd /config && git log --oneline -1

# Expected: "Initial VyOS configuration" commit
# POST-3: Verify config.boot tracked
cd /config && git status

# Expected: clean working tree, config.boot tracked
# POST-4: Test change tracking (make minor config change)
configure
set system host-name vyos-01-test
commit
save
exit

cd /config && git status
# Expected: config.boot modified
# POST-5: Commit change and verify
cd /config && git add config.boot && git commit -m "Test: hostname change"
git log --oneline -2

# Expected: Two commits shown
# POST-6: Rollback test change
configure
set system host-name vyos-01
commit
save
exit

cd /config && git add config.boot && git commit -m "Revert: hostname back to vyos-01"

19. Phase 13: Pre-Cutover Testing

Complete ALL tests before scheduling cutover maintenance window.

19.1. 13.1 Connectivity Tests (from vyos-02)

# WAN connectivity
ping -c 3 8.8.8.8
ping -c 3 google.com

# DNS resolution
nslookup google.com 10.50.1.90
nslookup inside.domusdigitalis.dev 10.50.1.90

# Internal services
ping -c 3 10.50.1.70           # NAS
ping -c 3 10.50.1.120 # k3s
ping -c 3 10.50.1.90          # bind-01

19.2. 13.2 DHCP Tests

# Check DHCP server status
show service dhcp-server

# Check leases (will be empty until cutover)
show dhcp server leases

19.3. 13.3 Firewall Tests

# Show zone policy
show zone-policy

# Show firewall rules
show firewall

# Check nftables directly
sudo nft list ruleset | head -50

19.4. 13.4 Create Test Client

Connect a test device to DATA VLAN and verify:

  • Gets DHCP address from vyos-02 range (from 10.50.10.0/24)

  • Can ping gateway (10.50.10.1)

  • Can resolve DNS (internal and external)

  • Can reach internet

  • Cannot reach GUEST VLAN


20. Phase 14: Cutover from pfSense

This causes network disruption. Ensure:

  • Maintenance window scheduled

  • All tests passed

  • Rollback plan understood

  • Console access available

20.1. 14.1 Pre-Cutover Snapshot

# Snapshot pfSense
ssh kvm-01 "sudo virsh snapshot-create-as pfsense pre-vyos-cutover"

# Snapshot vyos-02
ssh kvm-02 "sudo virsh snapshot-create-as vyos-02 pre-cutover"

20.2. 14.2 Cutover Procedure

# 1. Shut down pfSense
ssh kvm-01 "sudo virsh shutdown pfsense"

# 2. Wait for clean shutdown (30 seconds)
sleep 30
ssh kvm-01 "sudo virsh list --all | grep pfsense"

# 3. Update vyos-02 to take over gateway IPs
ssh vyos-02
configure

# Change MGMT IP to gateway
delete interfaces ethernet eth1 address {vyos-01-ip}/24
set interfaces ethernet eth1 address {vyos-vip}/24

commit
save
exit

# 4. Verify from workstation
ping {vyos-vip}
ping 8.8.8.8

20.3. 14.3 Post-Cutover Validation

# From each VLAN, test:
# - Gateway ping
# - Internet access
# - DNS resolution
# - Access to K8s services (Wazuh dashboard, etc.)

# Check DHCP leases are being issued
ssh vyos-02 "show dhcp server leases"

# Check firewall logs
ssh vyos-02 "show log | match firewall"

20.4. 14.4 Rollback Procedure

If issues occur, rollback in < 2 minutes:

# 1. Shut down VyOS
ssh kvm-02 "sudo virsh shutdown vyos-02"

# 2. Restore pfSense
ssh kvm-01 "sudo virsh snapshot-revert pfsense pre-vyos-cutover"
ssh kvm-01 "sudo virsh start pfsense"

# 3. Verify
ping {vyos-vip}

21. Phase 15: Deploy vyos-01 and VRRP HA

After 48-72 hours of stable vyos-02 operation, deploy vyos-01 for VRRP HA.

21.1. 15.0 Pre-Validation

# PRE-1: Verify vyos-02 stable for 48-72 hours
ssh vyos-02 "show system uptime"

# Expected: >48 hours uptime
# PRE-2: Verify vyos-02 handling all traffic
ssh vyos-02 "show nat translations | wc -l"

# Expected: Active NAT translations (traffic flowing)
# PRE-3: Verify kvm-01 ready for vyos-01 deployment
ssh kvm-01 "sudo virsh list --all | grep vyos"

# Expected: No vyos VMs yet (vyos-02 is on kvm-02)
# PRE-4: Verify VIP is not in use
ping -c 1 10.50.1.1

# Expected: No response (VIP will be created by VRRP)

21.2. 15.1 Deploy vyos-01 on kvm-01

Repeat Phase 1-11 for vyos-01 with these differences:

  • Hostname: vyos-01

  • MGMT IP: 10.50.1.3/24 (real IP, not VIP)

21.3. 15.2 Firewall Rules for VRRP

VRRP uses protocol 112 (not a port number). Both INBOUND and OUTBOUND rules are required.

  • Inbound (to router): *_LOCAL firewalls need rule 15 to receive VRRP

  • Outbound (from router): LOCAL_* firewalls need rule 5 to send VRRP

  • Zone config: GUEST, VOICE, IOT zones need from LOCAL firewall assignment

Add VRRP inbound rules to all *_LOCAL firewalls:

configure

# DATA_LOCAL, GUEST_LOCAL, IOT_LOCAL, MGMT_LOCAL, SECURITY_LOCAL, SERVICES_LOCAL, VOICE_LOCAL
# Add rule 15 to each (after rule 10 established/related)
set firewall ipv4 name DATA_LOCAL rule 15 action 'accept'
set firewall ipv4 name DATA_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name DATA_LOCAL rule 15 protocol '112'

set firewall ipv4 name GUEST_LOCAL rule 15 action 'accept'
set firewall ipv4 name GUEST_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name GUEST_LOCAL rule 15 protocol '112'

set firewall ipv4 name IOT_LOCAL rule 15 action 'accept'
set firewall ipv4 name IOT_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name IOT_LOCAL rule 15 protocol '112'

set firewall ipv4 name MGMT_LOCAL rule 15 action 'accept'
set firewall ipv4 name MGMT_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name MGMT_LOCAL rule 15 protocol '112'

set firewall ipv4 name SECURITY_LOCAL rule 15 action 'accept'
set firewall ipv4 name SECURITY_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name SECURITY_LOCAL rule 15 protocol '112'

set firewall ipv4 name SERVICES_LOCAL rule 15 action 'accept'
set firewall ipv4 name SERVICES_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name SERVICES_LOCAL rule 15 protocol '112'

set firewall ipv4 name VOICE_LOCAL rule 15 action 'accept'
set firewall ipv4 name VOICE_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name VOICE_LOCAL rule 15 protocol '112'

commit
save

Add VRRP outbound rules to LOCAL_* firewalls:

configure

# LOCAL_IOT already exists - add rule 5
set firewall ipv4 name LOCAL_IOT rule 5 action 'accept'
set firewall ipv4 name LOCAL_IOT rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_IOT rule 5 protocol '112'

# Create LOCAL_GUEST and LOCAL_VOICE with VRRP rule
set firewall ipv4 name LOCAL_GUEST default-action 'accept'
set firewall ipv4 name LOCAL_GUEST description 'Router outbound to GUEST'
set firewall ipv4 name LOCAL_GUEST rule 5 action 'accept'
set firewall ipv4 name LOCAL_GUEST rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_GUEST rule 5 protocol '112'

set firewall ipv4 name LOCAL_VOICE default-action 'accept'
set firewall ipv4 name LOCAL_VOICE description 'Router outbound to VOICE'
set firewall ipv4 name LOCAL_VOICE rule 5 action 'accept'
set firewall ipv4 name LOCAL_VOICE rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_VOICE rule 5 protocol '112'

commit
save

Add zone "from LOCAL" assignments:

configure

# These zones need outbound firewall from LOCAL zone
set firewall zone GUEST from LOCAL firewall name 'LOCAL_GUEST'
set firewall zone IOT from LOCAL firewall name 'LOCAL_IOT'
set firewall zone VOICE from LOCAL firewall name 'LOCAL_VOICE'

commit
save

21.4. 15.3 Configure VRRP on Both Nodes

Interface mapping differs between nodes:

  • vyos-01: eth0=WAN, eth1=LAN (VIFs on eth1.X)

  • vyos-02: eth0=LAN, eth1=WAN (VIFs on eth0.X)

VIP addresses are .1 (gateway IPs). Real interface IPs:

  • vyos-01: .2 (e.g., 10.50.1.2)

  • vyos-02: .3 (e.g., 10.50.1.3)

VyOS 2026.03 syntax: Use address not virtual-address.

VRIDs: 10=MGMT, 11=DATA, 12=VOICE, 13=GUEST, 14=IOT, 15=SECURITY, 16=SERVICES

On vyos-01 (MASTER - priority 200, uses eth1 for LAN):

configure

# VRRP for MGMT interface - VIP is .1, interface IP is .2
set high-availability vrrp group MGMT interface 'eth1'
set high-availability vrrp group MGMT priority '200'
set high-availability vrrp group MGMT address '10.50.1.1/24'
set high-availability vrrp group MGMT vrid '10'
set high-availability vrrp group MGMT preempt-delay '30'

# VRRP for DATA
set high-availability vrrp group DATA interface 'eth1.10'
set high-availability vrrp group DATA priority '200'
set high-availability vrrp group DATA address '10.50.10.1/24'
set high-availability vrrp group DATA vrid '11'
set high-availability vrrp group DATA preempt-delay '30'

# VRRP for VOICE
set high-availability vrrp group VOICE interface 'eth1.20'
set high-availability vrrp group VOICE priority '200'
set high-availability vrrp group VOICE address '10.50.20.1/24'
set high-availability vrrp group VOICE vrid '12'
set high-availability vrrp group VOICE preempt-delay '30'

# VRRP for GUEST
set high-availability vrrp group GUEST interface 'eth1.30'
set high-availability vrrp group GUEST priority '200'
set high-availability vrrp group GUEST address '10.50.30.1/24'
set high-availability vrrp group GUEST vrid '13'
set high-availability vrrp group GUEST preempt-delay '30'

# VRRP for IOT
set high-availability vrrp group IOT interface 'eth1.40'
set high-availability vrrp group IOT priority '200'
set high-availability vrrp group IOT address '10.50.40.1/24'
set high-availability vrrp group IOT vrid '14'
set high-availability vrrp group IOT preempt-delay '30'

# VRRP for SECURITY
set high-availability vrrp group SECURITY interface 'eth1.110'
set high-availability vrrp group SECURITY priority '200'
set high-availability vrrp group SECURITY address '10.50.110.1/24'
set high-availability vrrp group SECURITY vrid '15'
set high-availability vrrp group SECURITY preempt-delay '30'

# VRRP for SERVICES
set high-availability vrrp group SERVICES interface 'eth1.120'
set high-availability vrrp group SERVICES priority '200'
set high-availability vrrp group SERVICES address '10.50.120.1/24'
set high-availability vrrp group SERVICES vrid '16'
set high-availability vrrp group SERVICES preempt-delay '30'

commit
save

On vyos-02 (BACKUP - priority 100, uses eth0 for LAN):

configure

# VRRP for MGMT interface - VIP is .1, interface IP is .3
set high-availability vrrp group MGMT interface 'eth0'
set high-availability vrrp group MGMT priority '100'
set high-availability vrrp group MGMT address '10.50.1.1/24'
set high-availability vrrp group MGMT vrid '10'
set high-availability vrrp group MGMT preempt-delay '30'

# VRRP for DATA
set high-availability vrrp group DATA interface 'eth0.10'
set high-availability vrrp group DATA priority '100'
set high-availability vrrp group DATA address '10.50.10.1/24'
set high-availability vrrp group DATA vrid '11'
set high-availability vrrp group DATA preempt-delay '30'

# VRRP for VOICE
set high-availability vrrp group VOICE interface 'eth0.20'
set high-availability vrrp group VOICE priority '100'
set high-availability vrrp group VOICE address '10.50.20.1/24'
set high-availability vrrp group VOICE vrid '12'
set high-availability vrrp group VOICE preempt-delay '30'

# VRRP for GUEST
set high-availability vrrp group GUEST interface 'eth0.30'
set high-availability vrrp group GUEST priority '100'
set high-availability vrrp group GUEST address '10.50.30.1/24'
set high-availability vrrp group GUEST vrid '13'
set high-availability vrrp group GUEST preempt-delay '30'

# VRRP for IOT
set high-availability vrrp group IOT interface 'eth0.40'
set high-availability vrrp group IOT priority '100'
set high-availability vrrp group IOT address '10.50.40.1/24'
set high-availability vrrp group IOT vrid '14'
set high-availability vrrp group IOT preempt-delay '30'

# VRRP for SECURITY
set high-availability vrrp group SECURITY interface 'eth0.110'
set high-availability vrrp group SECURITY priority '100'
set high-availability vrrp group SECURITY address '10.50.110.1/24'
set high-availability vrrp group SECURITY vrid '15'
set high-availability vrrp group SECURITY preempt-delay '30'

# VRRP for SERVICES
set high-availability vrrp group SERVICES interface 'eth0.120'
set high-availability vrrp group SERVICES priority '100'
set high-availability vrrp group SERVICES address '10.50.120.1/24'
set high-availability vrrp group SERVICES vrid '16'
set high-availability vrrp group SERVICES preempt-delay '30'

commit
save

21.5. 15.4 Create WAN Health-Check Script (Both Nodes)

KVM virtual NICs do NOT detect physical link state. Even when the WAN cable is unplugged, the interface shows state UP. Interface tracking (track interface eth0) will NOT trigger failover.

Solution: Use a health-check script that pings the upstream gateway. When pings fail, the sync-group transitions all VRRP groups to FAULT state.

On BOTH vyos-01 and vyos-02:

sudo tee /config/scripts/check-wan.sh << 'EOF'
#!/bin/bash
# Health check for VRRP - ping upstream gateway
# Exit 0 = healthy, Exit 1 = failed
ping -c 2 -W 2 192.168.1.254 > /dev/null 2>&1
EOF
sudo chmod +x /config/scripts/check-wan.sh

Test the script:

/config/scripts/check-wan.sh && echo "WAN OK" || echo "WAN FAILED"

21.6. 15.5 Configure Sync-Group with Health-Check (Both Nodes)

Sync-group ensures ALL VRRP groups failover together. Without it, individual groups could be in different states (split-brain).

Health-check MUST be on the sync-group, NOT on individual groups. VyOS rejects commits if health-check is on a group that’s also in a sync-group.

On BOTH vyos-01 and vyos-02:

configure

# Create sync-group - all VRRP groups fail together
set high-availability vrrp sync-group GATEWAY member MGMT
set high-availability vrrp sync-group GATEWAY member DATA
set high-availability vrrp sync-group GATEWAY member VOICE
set high-availability vrrp sync-group GATEWAY member GUEST
set high-availability vrrp sync-group GATEWAY member IOT
set high-availability vrrp sync-group GATEWAY member SECURITY
set high-availability vrrp sync-group GATEWAY member SERVICES

# Health-check on sync-group (NOT individual groups)
set high-availability vrrp sync-group GATEWAY health-check script /config/scripts/check-wan.sh
set high-availability vrrp sync-group GATEWAY health-check interval 5
set high-availability vrrp sync-group GATEWAY health-check failure-count 2

commit
save

Verify sync-group:

show configuration commands | grep sync-group

21.7. 15.6 Configure Conntrack Sync (Stateful Failover)

# On both vyos-01 and vyos-02
configure

set high-availability conntrack-sync accept-protocol tcp,udp,icmp
set high-availability conntrack-sync interface eth1
set high-availability conntrack-sync failover-mechanism vrrp sync-group GATEWAY
set high-availability conntrack-sync mcast-group 225.0.0.50

commit
save

21.8. 15.4 Verify VRRP

# On vyos-01
show vrrp

# Expected: MASTER for all groups

# On vyos-02
show vrrp

# Expected: BACKUP for all groups

21.9. 15.7 Test Failover

Test by pulling WAN cable, NOT by shutting down VM.

Pulling the WAN cable simulates real-world ISP failure. The health-check script will detect the upstream gateway is unreachable and trigger failover.

Expected behavior: - Health-check fails after 2 consecutive failures (10 seconds) - All VRRP groups transition to FAULT (sync-group) - vyos-02 becomes MASTER - 1 ping dropped during failover (acceptable) - Reconnect WAN → vyos-01 preempts back after 30 seconds

# Start continuous ping from phone/laptop on WiFi or wired
ping 8.8.8.8

# On vyos-01 - pull WAN cable (or disable interface in emergency)
# configure
# set interfaces ethernet eth0 disable
# commit

# Watch VRRP state on vyos-01 - should go to FAULT
show vrrp

# Expected: ALL groups show FAULT
# On vyos-02 - verify it became MASTER
show vrrp

# Expected: ALL groups show MASTER
# Reconnect WAN cable (or re-enable interface)
# configure
# delete interfaces ethernet eth0 disable
# commit

# Wait 30 seconds (preempt-delay)
# vyos-01 should preempt back to MASTER

show vrrp

# Expected: ALL groups show MASTER on vyos-01

21.10. 15.8 Post-Validation

# POST-1: Verify VRRP status on vyos-01
ssh vyos-01 "show vrrp"

# Expected: MASTER for all 7 groups (MGMT, DATA, VOICE, GUEST, IOT, SECURITY, SERVICES)
# POST-2: Verify VRRP status on vyos-02
ssh vyos-02 "show vrrp"

# Expected: BACKUP for all 7 groups
# POST-3: Verify VIP responds
ping -c 3 10.50.1.1

# Expected: Replies from VIP (10.50.1.1)
# POST-4: Verify sync-group configured on both nodes
ssh vyos-01 "show configuration commands | grep sync-group"
ssh vyos-02 "show configuration commands | grep sync-group"

# Expected: GATEWAY sync group with all 7 members and health-check script
# POST-5: Verify health-check script exists on both nodes
ssh vyos-01 "ls -la /config/scripts/check-wan.sh"
ssh vyos-02 "ls -la /config/scripts/check-wan.sh"

# Expected: Executable script on both nodes
# POST-6: Verify conntrack sync (if configured)
ssh vyos-01 "show conntrack-sync status"
ssh vyos-02 "show conntrack-sync status"

# Expected: Both show sync active
# POST-7: Verify preempt-delay configured
ssh vyos-01 "show configuration commands | grep preempt-delay"

# Expected: preempt-delay 30 on all groups
# POST-8: Verify both nodes healthy after failover test
ssh vyos-01 "show vrrp"
ssh vyos-02 "show vrrp"

# Expected: vyos-01 MASTER, vyos-02 BACKUP for all groups

22. Phase 16: Multi-Node k3s Cluster Support

22.1. 16.0 Pre-Validation

# PRE-1: Verify Phase 15 complete (VRRP working)
ssh vyos-01 "show vrrp"
ssh vyos-02 "show vrrp"

# Expected: vyos-01 MASTER, vyos-02 BACKUP
# PRE-2: Verify k3s cluster exists and healthy
kubectl get nodes

# Expected: k3s-master-01 Ready (single node initially)
# PRE-3: Verify Cilium running
cilium status

# Expected: Cilium OK, operator Ready
# PRE-4: Check current firewall groups (should not exist yet)
show firewall group address-group

# Expected: May be empty or missing K3S_NODES group

The k3s cluster is a 6-node HA deployment across kvm-01 and kvm-02:

  • Masters: k3s-master-01/02/03 (10.50.1.120-10.50.1.122) - embedded etcd

  • Workers: k3s-worker-01/02/03 (10.50.1.123-10.50.1.125)

  • CNI: Cilium with VXLAN overlay (10.42.0.0/16)

  • LoadBalancer: Cilium BGP advertises 10.50.1.128/28

All nodes communicate via the MGMT VLAN (10.50.1.0/24). These firewall rules enable node-to-node traffic.

22.2. 16.1 k3s Inter-Node Communication Rules

Create firewall rules to allow cluster traffic between k3s nodes:

configure

# VXLAN overlay (Cilium tunnel mode) - UDP 8472 between all nodes
set firewall ipv4 name MGMT_MGMT default-action drop
set firewall ipv4 name MGMT_MGMT rule 10 action accept
set firewall ipv4 name MGMT_MGMT rule 10 state established
set firewall ipv4 name MGMT_MGMT rule 10 state related

set firewall ipv4 name MGMT_MGMT rule 80 action accept
set firewall ipv4 name MGMT_MGMT rule 80 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 80 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 80 protocol udp
set firewall ipv4 name MGMT_MGMT rule 80 destination group port-group K3S_VXLAN
set firewall ipv4 name MGMT_MGMT rule 80 description 'k3s Cilium VXLAN overlay'

# etcd cluster (control plane only) - TCP 2379-2380
set firewall ipv4 name MGMT_MGMT rule 81 action accept
set firewall ipv4 name MGMT_MGMT rule 81 source group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 81 destination group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 81 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 81 destination group port-group K3S_ETCD
set firewall ipv4 name MGMT_MGMT rule 81 description 'k3s etcd cluster'

# Kubelet API - TCP 10250 between all nodes
set firewall ipv4 name MGMT_MGMT rule 82 action accept
set firewall ipv4 name MGMT_MGMT rule 82 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 82 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 82 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 82 destination group port-group K3S_KUBELET
set firewall ipv4 name MGMT_MGMT rule 82 description 'k3s kubelet API'

# Cilium health checks - TCP 4240
set firewall ipv4 name MGMT_MGMT rule 83 action accept
set firewall ipv4 name MGMT_MGMT rule 83 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 83 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 83 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 83 destination group port-group CILIUM_HEALTH
set firewall ipv4 name MGMT_MGMT rule 83 description 'Cilium health checks'

# Hubble Relay - TCP 4244 (optional - for observability)
set firewall ipv4 name MGMT_MGMT rule 84 action accept
set firewall ipv4 name MGMT_MGMT rule 84 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 84 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 84 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 84 destination group port-group CILIUM_HUBBLE
set firewall ipv4 name MGMT_MGMT rule 84 description 'Cilium Hubble Relay'

# k3s API server - TCP 6443 (all nodes need to reach masters)
set firewall ipv4 name MGMT_MGMT rule 85 action accept
set firewall ipv4 name MGMT_MGMT rule 85 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 85 destination group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 85 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 85 destination group port-group K3S_API
set firewall ipv4 name MGMT_MGMT rule 85 description 'k3s API server'

# Apply zone policy for intra-MGMT traffic
set firewall zone MGMT from MGMT firewall name MGMT_MGMT

commit
save

22.3. 16.2 Verify k3s Cluster Connectivity

After applying rules, test from any k3s node:

# From k3s-master-01, ping other nodes
ping -c 1 10.50.1.121  # master-02
ping -c 1 10.50.1.122  # master-03
ping -c 1 10.50.1.123  # worker-01
ping -c 1 10.50.1.124  # worker-02
ping -c 1 10.50.1.125  # worker-03

# Verify Cilium health
cilium status

# Check etcd cluster health
kubectl -n kube-system exec -it etcd-k3s-master-01 -- etcdctl member list

22.4. 16.3 Post-Validation

# POST-1: Verify address groups created
show firewall group address-group K3S_NODES
show firewall group address-group K3S_MASTERS

# Expected: IPs for all k3s nodes listed
# POST-2: Verify port groups created
show firewall group port-group K3S_VXLAN
show firewall group port-group K3S_ETCD
show firewall group port-group K3S_KUBELET

# Expected: Ports listed (8472, 2379-2380, 10250, etc.)
# POST-3: Verify MGMT_MGMT firewall rules
show firewall ipv4 name MGMT_MGMT rule 80
show firewall ipv4 name MGMT_MGMT rule 81
show firewall ipv4 name MGMT_MGMT rule 82

# Expected: k3s cluster rules with correct groups
# POST-4: Verify zone policy applied
show zone-policy zone MGMT

# Expected: from-zone MGMT firewall name MGMT_MGMT
# POST-5: Test VXLAN port connectivity between k3s nodes
# From k3s-master-01:
nc -zvu 10.50.1.121 8472

# Expected: Connection succeeded (or open in UDP)
# POST-6: Verify Cilium connectivity
cilium connectivity test --single-node

# Expected: All tests pass
# POST-7: Verify etcd cluster (if multi-master)
kubectl -n kube-system exec -it etcd-k3s-master-01 -- etcdctl endpoint health

# Expected: is healthy: successfully committed proposal

23. Phase 17: Cilium BGP Control Plane

23.1. 17.0 Pre-Validation

# PRE-1: Verify Phase 16 complete (k3s cluster communication working)
cilium connectivity test --single-node

# Expected: All tests pass
# PRE-2: Verify k3s cluster healthy
kubectl get nodes

# Expected: All nodes Ready
# PRE-3: Check current BGP config (should not exist)
show protocols bgp

# Expected: Empty or "BGP not configured"
# PRE-4: Verify MetalLB currently handling LoadBalancer (will be replaced)
kubectl get svc -A | grep LoadBalancer

# Expected: Services have external IPs from MetalLB
# PRE-5: Document current LoadBalancer IPs (for comparison after migration)
kubectl get svc -A -o wide | grep LoadBalancer | awk '{print $1, $2, $5}'
VyOS + k3s Cilium BGP Architecture

BGP replaces MetalLB L2 mode for LoadBalancer services.

Benefits: - Enterprise-grade routing protocol (CCIE/CCNP learning value) - Fast convergence on failover (BGP timers vs ARP timeout) - ECMP support for load distribution - Native Cilium integration (no separate MetalLB deployment)

Architecture:

VyOS (AS 65000) ◄───BGP───► Cilium (AS 65001) on each k3s node
                    │
                    ▼
        Routes for {lb-pool-cidr} installed in VyOS RIB

23.2. 17.1 VyOS BGP Configuration (vyos-02)

Run on vyos-02. Router-ID must be unique per node.
configure

# VyOS router ASN (65000 = infrastructure, 65001 = k3s)
set protocols bgp system-as 65000
set protocols bgp parameters router-id 10.50.1.3

# BGP neighbor - Cilium on each k3s node (6-node HA cluster)
set protocols bgp neighbor 10.50.1.120 remote-as 65001
set protocols bgp neighbor 10.50.1.120 description 'k3s-master-01 Cilium'
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.121 remote-as 65001
set protocols bgp neighbor 10.50.1.121 description 'k3s-master-02 Cilium'
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.122 remote-as 65001
set protocols bgp neighbor 10.50.1.122 description 'k3s-master-03 Cilium'
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.123 remote-as 65001
set protocols bgp neighbor 10.50.1.123 description 'k3s-worker-01 Cilium'
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.124 remote-as 65001
set protocols bgp neighbor 10.50.1.124 description 'k3s-worker-02 Cilium'
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.125 remote-as 65001
set protocols bgp neighbor 10.50.1.125 description 'k3s-worker-03 Cilium'
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast

# Accept only LoadBalancer pool routes (filter for safety)
set policy prefix-list K3S_LB_POOL rule 10 action permit
set policy prefix-list K3S_LB_POOL rule 10 prefix 10.50.1.128/28
set policy prefix-list K3S_LB_POOL rule 10 le 32

set policy route-map CILIUM_IMPORT rule 10 action permit
set policy route-map CILIUM_IMPORT rule 10 match ip address prefix-list K3S_LB_POOL

# Apply import policy to all Cilium neighbors
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast route-map import CILIUM_IMPORT

commit
save

23.3. 17.1b VyOS BGP Configuration (vyos-01)

Run on vyos-01 after vyos-02 is validated. Router-ID must be unique per node.
configure

# VyOS router ASN (65000 = infrastructure, 65001 = k3s)
set protocols bgp system-as 65000
set protocols bgp parameters router-id 10.50.1.2

# BGP neighbor - Cilium on each k3s node (6-node HA cluster)
set protocols bgp neighbor 10.50.1.120 remote-as 65001
set protocols bgp neighbor 10.50.1.120 description 'k3s-master-01 Cilium'
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.121 remote-as 65001
set protocols bgp neighbor 10.50.1.121 description 'k3s-master-02 Cilium'
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.122 remote-as 65001
set protocols bgp neighbor 10.50.1.122 description 'k3s-master-03 Cilium'
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.123 remote-as 65001
set protocols bgp neighbor 10.50.1.123 description 'k3s-worker-01 Cilium'
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.124 remote-as 65001
set protocols bgp neighbor 10.50.1.124 description 'k3s-worker-02 Cilium'
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast

set protocols bgp neighbor 10.50.1.125 remote-as 65001
set protocols bgp neighbor 10.50.1.125 description 'k3s-worker-03 Cilium'
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast

# Accept only LoadBalancer pool routes (filter for safety)
set policy prefix-list K3S_LB_POOL rule 10 action permit
set policy prefix-list K3S_LB_POOL rule 10 prefix 10.50.1.128/28
set policy prefix-list K3S_LB_POOL rule 10 le 32

set policy route-map CILIUM_IMPORT rule 10 action permit
set policy route-map CILIUM_IMPORT rule 10 match ip address prefix-list K3S_LB_POOL

# Apply import policy to all Cilium neighbors
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast route-map import CILIUM_IMPORT

commit
save

23.4. 17.2 Verify BGP Neighbors (VyOS)

# Show BGP summary
show ip bgp summary

# Expected: 6 neighbors in Established state (once Cilium is configured)

# Show BGP neighbors detail
show ip bgp neighbors

# Show received routes
show ip bgp

23.5. 17.3 Cilium BGP Configuration (k3s)

Update cilium-values.yaml to enable BGP Control Plane:

# /tmp/cilium-values.yaml
cluster:
  name: domus-k3s

k8sServiceHost: 127.0.0.1
k8sServicePort: 6443

kubeProxyReplacement: true
routingMode: tunnel
tunnelProtocol: vxlan

operator:
  replicas: 1

hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: false

# Enable BGP Control Plane
bgpControlPlane:
  enabled: true

Apply with Helm:

helm upgrade cilium cilium/cilium --version 1.16.5 \
  --namespace kube-system \
  -f /tmp/cilium-values.yaml

23.6. 17.4 Create Cilium BGP Resources

CiliumBGPPeeringPolicy - defines BGP peering with VyOS:

Uses session variables for VyOS IPs. Verify variables before applying.
# PRE: Verify VyOS IPs for Cilium peers
echo "VyOS-01: $\{VYOS_01_IP:-10.50.1.2}"
echo "VyOS-02: $\{VYOS_02_IP:-10.50.1.3}"
kubectl apply -f - << EOF
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: vyos-peering
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  virtualRouters:
  - localASN: 65001
    exportPodCIDR: false
    neighbors:
    - peerAddress: "10.50.1.2/32"
      peerASN: 65000
    - peerAddress: "10.50.1.3/32"
      peerASN: 65000
    serviceSelector:
      matchExpressions:
      - key: io.cilium/bgp-advertise
        operator: NotIn
        values: ["false"]
EOF

CiliumLoadBalancerIPPool - defines LoadBalancer IP range:

kubectl apply -f - << EOF
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lb-pool
spec:
  cidrs:
  - cidr: 10.50.1.128/28
EOF
CIDR 10.50.1.128/28 covers .128-.143 (16 IPs). Current services use .130-.134.

23.7. 17.5 Remove MetalLB (After BGP Verified)

Once Cilium BGP is working, remove MetalLB to avoid conflicts:

# Check current MetalLB resources
kubectl get all -n metallb-system

# Delete MetalLB
kubectl delete namespace metallb-system

# Verify LoadBalancer services still have external IPs
kubectl get svc -A | grep LoadBalancer

23.8. 17.6 Verify BGP Routes (VyOS)

After Cilium BGP is configured, verify routes are learned:

# Show BGP received routes
show ip bgp

# Expected output:
# Network          Next Hop         Metric  LocPrf  Path
# *> 10.50.1.130/32   10.50.1.123       0              65001 i
# *> 10.50.1.131/32   10.50.1.124       0              65001 i
# etc.

# Show routing table
show ip route bgp

# Verify reachability
ping 10.50.1.130       # Traefik Ingress
ping 10.50.1.134  # Wazuh Manager API

23.9. 17.7 Test BGP Failover

# From workstation, start continuous ping to a LoadBalancer IP
ping 10.50.1.134

# Identify which node is advertising the route
show ip bgp 10.50.1.134

# Stop Cilium on that node (simulates node failure)
# (Run on k3s node)
sudo systemctl stop k3s

# Observe:
# 1. BGP neighbor goes down
# 2. Route is withdrawn
# 3. Another node advertises the route
# 4. Traffic continues (1-2 packets lost during convergence)

# Restart k3s
sudo systemctl start k3s

23.10. 17.8 Post-Validation

# POST-1: Verify BGP sessions established on vyos-01
ssh vyos-01 "show ip bgp summary"

# Expected: 3-6 neighbors in "Established" state (k3s nodes)
# State/PfxRcd should show numbers (received prefixes), not "Connect" or "Active"
# POST-2: Verify BGP sessions on vyos-02
ssh vyos-02 "show ip bgp summary"

# Expected: Same neighbors established (VRRP backup receives same routes)
# POST-3: Verify routes received from k3s
ssh vyos-01 "show ip bgp"

# Expected: /32 routes for each LoadBalancer VIP
# Should see next-hops pointing to k3s node IPs
# POST-4: Verify routes in routing table
ssh vyos-01 "show ip route bgp"

# Expected: BGP routes marked with 'B' for LoadBalancer VIPs
# POST-5: Verify LoadBalancer services still accessible
curl -kIs https://10.50.1.132:443 | head -1
curl -kIs https://10.50.1.130:443 | head -1
curl -kIs https://10.50.1.130:443 | head -1

# Expected: HTTP/1.1 or HTTP/2 response headers (200, 301, 302 are all OK)
# POST-6: Verify Cilium BGP peering status
kubectl exec -n kube-system ds/cilium -- cilium bgp peers

# Expected: All peers show "Established" state
# POST-7: Verify Cilium is advertising routes
kubectl exec -n kube-system ds/cilium -- cilium bgp routes advertised

# Expected: Shows LoadBalancer VIP /32 routes being advertised
# POST-8: Verify MetalLB removed (no conflicts)
kubectl get pods -n metallb-system 2>&1 | grep -E "^No resources|metallb"

# Expected: "No resources found" (MetalLB namespace empty or deleted)

24. Phase 18: Deploy -02 Secondaries (HA Foundation)

Execute AFTER VyOS HA is stable and clients verified on new network.

Deploy -02 instances as secondaries FIRST: - Provides HA during -01 migration - Rollback path if migration fails - -02s stay in VLAN 100 initially

Order: 1. Deploy -02 secondaries (this phase) 2. Verify client connectivity 3. Migrate -01s to VLANs 110/120 (Phase 19)

24.1. 18.1 Deploy vault-02 (kvm-02)

See Vault HA Deployment for full runbook.

Quick summary:

# On kvm-02
sudo virt-install --name vault-02 \
  --vcpus 1 --memory 1024 \
  --disk /mnt/onboard-ssd/libvirt/images/vault-02.qcow2,size=20 \
  --os-variant rocky9 \
  --network bridge=virbr0 \
  --graphics none --console pty,target_type=serial \
  --cdrom /var/lib/libvirt/images/Rocky-9-GenericCloud.qcow2
Item Value

IP

10.50.1.61

Role

Vault Raft follower

Join to

vault-01 (10.50.1.60)

24.2. 18.2 Deploy bind-02 (kvm-02)

See BIND-02 Deployment for full runbook.

Item Value

IP

10.50.1.91

Role

Secondary DNS (zone transfer from bind-01)

Config

Slave zones, forward to bind-01

24.3. 18.3 Deploy ise-02 (kvm-02) - Optional

ISE HA requires licensing. Skip if single-node ISE is sufficient.

Item Value

IP

10.50.1.21

Role

ISE secondary PAN/PSN

Sync

Automatic from ise-01

24.4. 18.4 Client Verification Checklist

Before proceeding to Phase 19, verify ALL clients work:

# POST-1: Wired 802.1X authentication
netapi ise mnt sessions | awk '/Wired/ {print $1, $3}'

# POST-2: Wireless 802.1X authentication
netapi ise mnt sessions | awk '/Wireless/ {print $1, $3}'

# POST-3: DHCP leases from VyOS
ssh vyos-01 "show dhcp server leases"

# POST-4: DNS resolution (both bind-01 and bind-02)
dig @10.50.1.90 vault-01.inside.domusdigitalis.dev +short
dig @10.50.1.91 vault-01.inside.domusdigitalis.dev +short

# POST-5: Vault SSH CA still works
vault-ssh-sign
vault-ssh-test

# POST-6: Internet access through VyOS NAT
curl -s https://ifconfig.me

Do NOT proceed to Phase 19 until ALL checks pass.

If any check fails: - Troubleshoot with VyOS logs: show log | match <service> - Verify zone firewall rules: show firewall summary - Check VRRP state: show vrrp


25. Phase 19: Infrastructure Segmentation (VLAN 110/120 Migration)

Execute ONLY after Phase 18 secondaries are deployed and clients verified.

This phase migrates -01 VMs from flat VLAN 100 to segmented VLANs: - VLAN 110 (SECURITY): vault-01, ise-01, ipsk-mgr - VLAN 120 (SERVICES): keycloak-01, ipa-01, bind-01, gitea-01

The -02 secondaries provide continuity during migration.

25.1. 19.0 Pre-Validation

# PRE-1: Verify VyOS VLAN interfaces exist
ssh vyos-01 "show interfaces ethernet eth0 vif"
# Expected: vif 110 (SECURITY), vif 120 (SERVICES)
# PRE-2: Verify switch trunks include VLANs 110, 120
ssh c9300-01 "show vlan brief | include 110\|120"
# PRE-3: Document current VM IPs (rollback reference)
for vm in vault-01 ise-01 ipsk-mgr keycloak-01 ipa-01 bind-01 gitea-01; do
  echo "$vm: $(dig +short $vm.inside.domusdigitalis.dev)"
done

25.2. 19.1 Switch Trunk Preparation

On 3560-CX switch - add VLANs 110, 120 to trunk ports:

configure terminal

! Create VLANs
vlan 110
 name SECURITY
vlan 120
 name SERVICES

! Update trunk to kvm-01 (vyos-01)
interface TenGigabitEthernet1/0/2
 switchport trunk allowed vlan add 110,120

! Update trunk to kvm-02 (vyos-02)
interface TenGigabitEthernet1/0/1
 switchport trunk allowed vlan add 110,120

end
write memory
# POST: Verify VLANs added to trunks
show interfaces trunk | include Te1/0

25.3. 19.2 SECURITY VLAN Migration (vault-01, ise-01, ipsk-mgr)

25.3.1. 19.2.1 New IP Addresses

New IPs follow pattern: 10.50.{vlan-id}.{host-octet} - preserving host octet from current IP.
VM Old IP (VLAN 100) New IP (VLAN 110) Notes

vault-01

10.50.1.60

10.50.110.60

PKI CA - critical

ise-01

10.50.1.20

10.50.110.20

RADIUS - brief auth gap

ipsk-mgr

10.50.1.30

10.50.110.30

iPSK portal

25.3.2. 19.2.2 KVM VLAN Strategy

KVM VLAN Options:

  • Option A: Linux bridge per VLAN (br-vlan110, br-vlan120) - RECOMMENDED

  • Option B: Open vSwitch with VLAN tags

  • Option C: Configure VM guest to tag traffic (less clean)

Create bridge on kvm-01:

sudo nmcli conn add type bridge con-name br-vlan110 ifname br-vlan110
sudo nmcli conn add type vlan con-name vlan110 ifname eth0.110 dev eth0 id 110 master br-vlan110
sudo nmcli conn up vlan110
sudo nmcli conn up br-vlan110

25.3.3. 19.2.3 Migrate vault-01

# 1. SSH to kvm-01
ssh kvm-01
# 2. Edit vault-01 to use VLAN 110 bridge
sudo virsh edit vault-01
# Change: <source bridge='virbr0'/> → <source bridge='br-vlan110'/>
# 3. Restart vault-01
sudo virsh shutdown vault-01
sudo virsh start vault-01
# 4. Inside vault-01, configure new IP
ssh vault-01   # Still reachable via old IP until reboot
sudo nmcli conn modify eth0 ipv4.addresses 10.50.110.60/24
sudo nmcli conn modify eth0 ipv4.gateway 10.50.110.1
sudo nmcli conn up eth0

25.3.4. 19.2.4 Update DNS

# Update pfSense/VyOS DNS override
netapi pfsense dns delete -h vault-01
netapi pfsense dns add -h vault-01 -d inside.domusdigitalis.dev -i 10.50.110.60 --descr "Vault PKI (VLAN 110)"
# Update BIND zone if used
ssh bind-01 "sudo sed -i 's/10.50.1.60/10.50.110.60/' /etc/named/zones/db.inside.domusdigitalis.dev"
ssh bind-01 "sudo systemctl reload named"

25.3.5. 19.2.5 Verify vault-01

# From workstation
ping vault-01.inside.domusdigitalis.dev
# Verify Vault API
curl -k https://vault-01.inside.domusdigitalis.dev:8200/v1/sys/health | jq
# Test SSH CA signing
vault-ssh-sign

25.3.6. 19.2.6 Repeat for ise-01 and ipsk-mgr

Follow same pattern:

  1. Create bridge on KVM host if needed

  2. Edit VM to use VLAN bridge

  3. Configure new IP inside VM

  4. Update DNS

  5. Verify connectivity

ISE additional steps:

  • Update ISE network device definitions (switches, WLC point to ISE)

  • Verify RADIUS authentication still works

25.4. 19.3 SERVICES VLAN Migration (keycloak-01, ipa-01, bind-01, gitea-01)

25.4.1. 19.3.1 New IP Addresses

New IPs follow pattern: 10.50.{vlan-id}.{host-octet} - preserving host octet from current IP.
VM Old IP (VLAN 100) New IP (VLAN 120) Notes

keycloak-01

10.50.1.80

10.50.120.80

SAML/OIDC IdP

ipa-01

10.50.1.100

10.50.120.100

FreeIPA (Linux auth)

bind-01

10.50.1.90

10.50.120.90

Primary DNS

gitea-01

10.50.1.72

10.50.120.72

Git server

25.4.2. 19.3.2 DNS Migration Strategy

bind-01 is special - moving DNS server requires careful planning:

  1. Add bind-02 as secondary (stays in VLAN 100) FIRST

  2. Update all clients to use bind-02 as fallback

  3. Migrate bind-01 to VLAN 120

  4. Update DNS client configs

25.5. 19.4 VyOS Zone Firewall Updates

After migration, add inter-VLAN firewall rules:

configure
# Allow SERVICES (120) → SECURITY (110) for specific services
set firewall name SERVICES-to-SECURITY rule 10 action accept
set firewall name SERVICES-to-SECURITY rule 10 description 'Keycloak to ISE RADIUS'
set firewall name SERVICES-to-SECURITY rule 10 destination address 10.50.110.20
set firewall name SERVICES-to-SECURITY rule 10 destination port 1812,1813
set firewall name SERVICES-to-SECURITY rule 10 protocol udp
set firewall name SERVICES-to-SECURITY rule 20 action accept
set firewall name SERVICES-to-SECURITY rule 20 description 'All to Vault PKI'
set firewall name SERVICES-to-SECURITY rule 20 destination address 10.50.110.60
set firewall name SERVICES-to-SECURITY rule 20 destination port 8200
set firewall name SERVICES-to-SECURITY rule 20 protocol tcp
# Allow INFRA (100) → SECURITY (110) for admin access
set firewall name INFRA-to-SECURITY rule 10 action accept
set firewall name INFRA-to-SECURITY rule 10 description 'Admin SSH'
set firewall name INFRA-to-SECURITY rule 10 destination port 22
set firewall name INFRA-to-SECURITY rule 10 protocol tcp
set firewall name INFRA-to-SECURITY rule 20 action accept
set firewall name INFRA-to-SECURITY rule 20 description 'Admin HTTPS'
set firewall name INFRA-to-SECURITY rule 20 destination port 443
set firewall name INFRA-to-SECURITY rule 20 protocol tcp
# Apply to zones
set zone-policy zone SECURITY from SERVICES firewall name SERVICES-to-SECURITY
set zone-policy zone SECURITY from INFRA firewall name INFRA-to-SECURITY
commit
save

25.6. 19.5 Post-Migration Validation

# POST-1: Verify all VMs reachable
for vm in vault-01 ise-01 ipsk-mgr keycloak-01 ipa-01 bind-01 gitea-01; do
  echo -n "$vm: "
  ping -c1 -W2 $vm.inside.domusdigitalis.dev >/dev/null && echo "OK" || echo "FAIL"
done
# POST-2: Verify 802.1X authentication (ISE in VLAN 110)
# Connect a wired client and check ISE logs
netapi ise mnt sessions | head -5
# POST-3: Verify Vault SSH CA
vault-ssh-sign
vault-ssh-test
# POST-4: Verify DNS resolution
dig vault-01.inside.domusdigitalis.dev
dig ise-01.inside.domusdigitalis.dev

26. Phase 20: WLC HA (Optional)

9800-CL WLC HA Options:

  1. SSO (Stateful Switchover) - Active/Standby pair

    • 9800-WLC-01 (Active): 10.50.1.40

    • 9800-WLC-02 (Standby): 10.50.1.41

    • Requires second VM on kvm-02

    • Seamless client failover

  2. N+1 Redundancy - Single WLC with AP failover (current)

    • APs configured with primary/secondary controller

    • Clients reconnect on failure (~30s)

    • Simpler, adequate for home enterprise

Recommendation: N+1 is sufficient unless you need seamless roaming during failover.

For SSO HA deployment, create separate runbook: wlc-ha-deployment.adoc


27. Key Commands Reference

# Enter/exit config mode
configure
exit
# Show current config
show configuration
# Show pending changes
compare
# Commit and save
commit
save
# Rollback to previous config
rollback 1
commit
# Show interfaces
show interfaces
# Show firewall
show firewall
show firewall summary
show zone-policy
# Show VRRP status
show vrrp
show vrrp detail
# Show conntrack
show conntrack table
# Monitor traffic
monitor traffic interface eth0

Appendix A: CLI Mastery Reference

Advanced command patterns for network diagnostics, validation, and troubleshooting.

A.1. Network Interface Discovery

# All WiFi connections (nmcli parsing)
nmcli -t c s | awk -F: '$3=="wifi" {print $1}'
# Global IPv4 addresses (one-line for parsing)
ip -4 -o addr show scope global | awk '{print $2, $4}'
# Interface status pivot table
ip -o link show | awk -F': ' '{
  split($3, flags, ",")
  state = (index($3, "UP") ? "UP" : "DOWN")
  printf "%-12s %s\n", $2, state
}'
# Bridge members with state
for br in $(ip link show type bridge | awk -F': ' '{print $2}'); do
  echo "=== $br ==="
  ip link show master $br | awk -F': ' '/^[0-9]/{print "  "$2}'
done

A.2. Connection Analysis

# Top talkers by connection count
ss -tn | awk 'NR>1 {split($5,a,":"); ip[a[1]]++} END {for(i in ip) print ip[i], i}' | sort -rn | head -10
# Connections by state
ss -tan | awk 'NR>1 {state[$1]++} END {for(s in state) printf "%-12s %d\n", s, state[s]}'
# Listening ports with process
ss -tlnp | awk 'NR>1 {split($4,a,":"); print a[length(a)], $6}' | sort -n | uniq
# Established connections per remote IP
ss -tn state established | awk 'NR>1 {split($4,a,":"); print a[1]}' | sort | uniq -c | sort -rn

A.3. Parallel Operations

# Parallel subnet scan (SSH port)
echo {1..254} | tr ' ' '\n' | xargs -P 50 -I{} sh -c 'timeout 1 nc -z 10.50.1.{} 22 2>/dev/null && echo 10.50.1.{}'
# Parallel ping sweep
echo {1..254} | tr ' ' '\n' | xargs -P 50 -I{} sh -c 'ping -c1 -W1 10.50.1.{} >/dev/null 2>&1 && echo 10.50.1.{}'
# Parallel DNS lookup
echo "vault-01 ise-01 bind-01 nas-01 kvm-01 kvm-02" | tr ' ' '\n' | \
  xargs -P 10 -I{} sh -c 'echo -n "{}: "; dig +short {}.inside.domusdigitalis.dev'
# Parallel service check (multiple ports)
for port in 22 80 443 8200; do
  echo "=== Port $port ==="
  echo "vault-01 ise-01 bind-01" | tr ' ' '\n' | \
    xargs -P 10 -I{} sh -c "timeout 1 nc -z {}.inside.domusdigitalis.dev $port 2>/dev/null && echo '{}: OK' || echo '{}: FAIL'"
done

A.4. VyOS Show Commands with Parsing

# Zone summary (inside VyOS)
show firewall zone | awk 'NR>2 && NF>=2 {printf "%-12s → %s\n", $1, $2}'
# Interface status table
show interfaces | awk '/^eth|^lo/ {iface=$1; ip=$2} /u\/u|u\/D/ {print iface, ip, $0}'
# VRRP status (HA)
show vrrp | awk '/Group|State|Priority/ {print}'
# DHCP leases by pool
show dhcp server leases | awk 'NR>2 {pool[$5]++; print} END {print "---"; for(p in pool) print p": "pool[p]}'
# NAT translations active
show nat source translations | awk 'NR>1 {proto[$3]++} END {for(p in proto) print p": "proto[p]}'
# Conntrack table summary
show conntrack table | awk '{proto[$1]++} END {for(p in proto) printf "%-8s %d\n", p, proto[p]}'

A.5. Firewall Debugging

# Recent drops (via nftables)
sudo nft list ruleset | awk '/drop/ {found=1} found && /counter/ {print; found=0}'
# Zone policy matrix
show zone-policy | awk '/^[A-Z]/ {zone=$1} /from/ {print zone " ← " $2 ": " $4}'
# Firewall rule hit counts
show firewall | awk '/rule [0-9]+/ {rule=$2} /packets/ {print "Rule "rule": "$2" packets"}'

A.6. KVM/Libvirt Diagnostics

# VM status pivot
sudo virsh list --all | awk 'NR>2 && NF {state[$3]++; print} END {print "---"; for(s in state) print s": "state[s]}'
# VM interface to bridge mapping
for vm in $(sudo virsh list --name); do
  echo "=== $vm ==="
  sudo virsh domiflist $vm | awk 'NR>2 && NF {printf "  %s → %s\n", $1, $3}'
done
# VM resource usage
sudo virsh list --name | xargs -I{} sh -c 'echo "=== {} ==="; sudo virsh dominfo {} | awk "/CPU|Memory/"'
# Disk usage per VM
sudo virsh list --name | while read vm; do
  size=$(sudo virsh domblkinfo $vm vda 2>/dev/null | awk '/Capacity/{print $2/1024/1024/1024 " GB"}')
  echo "$vm: $size"
done

A.7. Infrastructure Validation

# Quick infrastructure health check
for h in vault-01 ise-01 bind-01 nas-01 home-dc01; do
  printf "%-12s " "$h"
  timeout 2 nc -z $h.inside.domusdigitalis.dev 22 2>/dev/null && echo "✓ SSH" || echo "✗ SSH"
done
# DNS resolution check (all critical hosts)
for h in vyos-01 vyos-02 vault-01 ise-01 bind-01 nas-01 kvm-01 kvm-02; do
  ip=$(dig +short $h.inside.domusdigitalis.dev)
  printf "%-12s %s\n" "$h" "${ip:-NXDOMAIN}"
done
# VLAN gateway reachability
for gw in 10.50.1.1 10.50.10.1 10.50.20.1 10.50.30.1 10.50.40.1; do
  printf "%-14s " "$gw"
  ping -c1 -W1 $gw >/dev/null 2>&1 && echo "✓" || echo "✗"
done
# Certificate expiry check (parallel)
echo "vault-01:8200 ise-01:443 nas-01:5001" | tr ' ' '\n' | \
  xargs -P 5 -I{} sh -c '
    host=$(echo {} | cut -d: -f1)
    port=$(echo {} | cut -d: -f2)
    exp=$(echo | openssl s_client -connect $host.inside.domusdigitalis.dev:$port 2>/dev/null | \
          openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
    printf "%-20s %s\n" "$host:$port" "${exp:-FAILED}"
  '

A.8. Log Analysis Patterns

# Syslog message type frequency
journalctl -p warning --since "1 hour ago" --no-pager | \
  awk '{print $5}' | sort | uniq -c | sort -rn | head -10
# Failed SSH attempts
journalctl -u sshd --since "1 hour ago" --no-pager | \
  awk '/Failed/ {for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn
# VyOS commit history
show system commit | awk '/^[0-9]/ {print $1, $2, $3, $5}'

A.9. One-Liners Quick Reference

Pattern Use Case

awk -F: '{print $1}'

Extract first field (colon-delimited)

awk 'NR>1'

Skip header row

awk '{a[$1]++} END {for(i in a) print a[i], i}'

Count occurrences (pivot)

sort | uniq -c | sort -rn

Frequency analysis

xargs -P 50 -I{}

Parallel execution (50 threads)

timeout 1 nc -z host port

Quick port check with timeout

ip -o / ss -t

One-line output (parseable)

tr ' ' '\n'

Convert space-delimited to lines

split($5,a,":")

Split field into array (awk)

printf "%-12s %s\n"

Formatted column output