VyOS Firewall Deployment
Migrate from pfSense to VyOS for CLI-native, Linux-based firewall management. This runbook covers complete infrastructure audit, deployment, and cutover with zero gaps.
1. Executive Summary
| Item | Value |
|---|---|
Current Firewall |
pfSense (FreeBSD-based, GUI-first) |
Target Firewall |
VyOS (Debian-based, CLI-native) |
Deployment Model |
vyos-02 on kvm-02 first (parallel), then vyos-01 + VRRP HA |
Downtime |
< 5 minutes during cutover (with instant rollback capability) |
2. Why VyOS
| Aspect | pfSense | VyOS |
|---|---|---|
Interface |
GUI-first, WebUI required |
CLI-native, SSH/API only |
Base OS |
FreeBSD (limited ecosystem) |
Debian Bookworm (full apt access) |
Automation |
xmlrpc hacks, limited API |
HTTP API, Ansible, NETCONF/YANG |
Observability |
Packages required |
Native: node_exporter, wazuh-agent, suricata |
Firewall |
pf (FreeBSD) |
nftables (Linux, eBPF/XDP capable) |
Config Management |
XML backup |
|
Learning Value |
FreeBSD-specific |
Transfers to RHCSA, kernel studies, enterprise Linux |
3. Architecture
3.1. Historical: pfSense Topology (Decommissioned 2026-03-07)
|
pfSense was decommissioned on 2026-03-07. |
3.3. Final VLAN Design
| VLAN | Name | Subnet | Gateway | Purpose |
|---|---|---|---|---|
Infrastructure VLANs (servers/services) |
||||
100 |
INFRA |
10.50.1.0/24 |
.1 (VIP) |
Network hardware, hypervisors, k3s nodes (MetalLB: .130-.140) |
110 |
SECURITY |
10.50.110.0/24 |
.1 |
Crown jewels: Vault, ISE, secrets management |
120 |
SERVICES |
10.50.120.0/24 |
.1 |
VMs: Keycloak, Gitea, FreeIPA, BIND, etc. |
Client VLANs (endpoints only) |
||||
10 |
DATA |
10.50.10.0/24 |
.1 |
Corporate wired/wireless devices |
20 |
VOICE |
10.50.20.0/24 |
.1 |
VoIP phones (future) |
30 |
GUEST |
10.50.30.0/24 |
.1 |
Guest wireless, internet-only |
40 |
IOT |
10.50.40.0/24 |
.1 |
IoT devices, limited access |
999 |
CRITICAL_AUTH |
(no gateway) |
- |
802.1X failure quarantine |
|
Infrastructure Segmentation:
Migration: See Phase 18 for VM migration to VLAN 110/120 after VyOS HA is stable. |
3.4. Switch Trunk Configuration (Reference)
! Te1/0/1 → kvm-02 (vyos-02)
! Te1/0/2 → kvm-01 (vyos-01)
interface TenGigabitEthernet1/0/1
description TRUNK-TO-SUPERMICRO-KVM-02
switchport trunk allowed vlan 10,20,30,40,100,110,120,999
switchport trunk native vlan 100
switchport mode trunk
!
interface TenGigabitEthernet1/0/2
description TRUNK-TO-SUPERMICRO-KVM-01
switchport trunk allowed vlan 10,20,30,40,100,110,120,999
switchport trunk native vlan 100
switchport mode trunk
3.5. Kubernetes Cluster (6-Node HA in MGMT VLAN)
| Node | IP | Hypervisor | Role |
|---|---|---|---|
k3s-master-01 |
10.50.1.120 |
kvm-01 |
Control plane (active) |
k3s-master-02 |
10.50.1.121 |
kvm-02 |
Control plane (planned) |
k3s-master-03 |
10.50.1.122 |
kvm-02 |
Control plane (planned) |
k3s-worker-01 |
10.50.1.123 |
kvm-01 |
Workloads (planned) |
k3s-worker-02 |
10.50.1.124 |
kvm-02 |
Workloads (planned) |
k3s-worker-03 |
10.50.1.125 |
kvm-02 |
Workloads (planned) |
3.6. Kubernetes LoadBalancer Services (BGP Advertised)
| Service | IP | Ports |
|---|---|---|
Traefik Ingress |
10.50.1.130 |
80, 443 |
Wazuh Indexer |
10.50.1.131 |
9200 |
Wazuh Dashboard |
10.50.1.132 |
443 |
Wazuh Workers |
10.50.1.133 |
1514 |
Wazuh Manager |
10.50.1.134 |
55000, 1515, 514/udp |
| LoadBalancer IPs (10.50.1.128/28) are advertised via Cilium BGP to VyOS. See Phase 17. |
4. Prerequisites
|
Complete ALL items before starting deployment:
|
5. Session Variables
|
DNS is the source of truth. Hostnames resolve via BIND. Use hostnames directly in commands - no need to store IPs in variables. |
5.1. Required Variables
# Target Node - change to vyos-01 for second deployment
VYOS_NODE="vyos-02"
VYOS_PRIORITY="100" # 100=BACKUP, 200=MASTER
# Hypervisor derived from node (vyos-02 → kvm-02, vyos-01 → kvm-01)
KVM_HOST="${VYOS_NODE/vyos/kvm}"
echo "Deploying: $VYOS_NODE (priority $VYOS_PRIORITY) on $KVM_HOST"
Deploying: vyos-02 (priority 100) on kvm-02
That’s it. Two variables set manually, one derived. DNS resolves hostnames directly - no IP variables needed.
6. Deployment Sequence
| Order | Phase | Action | Hypervisor |
|---|---|---|---|
1 |
Phase 1-13 |
Deploy vyos-02, configure zones, services, validation |
kvm-02 |
2 |
Phase 14 |
Cutover: pfSense DOWN, vyos-02 takes gateway IPs |
kvm-02 |
3 |
Phase 15 |
Deploy vyos-01 + VRRP HA (after 48-72h stable vyos-02) |
kvm-01 |
4 |
Phase 16-17 |
k3s firewall rules + Cilium BGP peering |
Both |
|
Why vyos-02 on kvm-02 FIRST?
|
7. Phase 1: Deploy VM on Hypervisor
7.1. 1.0 Pre-Validation
|
Complete these checks BEFORE proceeding. Failures here will cause deployment issues. |
DNS records must exist BEFORE deploying VyOS. Use netapi pfsense dns add:
dsource d000 dev/network
netapi pfsense dns add -h vyos-01 -d inside.domusdigitalis.dev -i 10.50.1.2 --descr "VyOS HA Master"
netapi pfsense dns add -h vyos-02 -d inside.domusdigitalis.dev -i 10.50.1.3 --descr "VyOS HA Backup"
netapi pfsense dns list | grep -i vyos
vyos-01.inside.domusdigitalis.dev 10.50.1.2 VyOS HA Master
vyos-02.inside.domusdigitalis.dev 10.50.1.3 VyOS HA Backup
| pfSense provides immediate resolution. BIND is the authoritative source - update both. |
ssh bind-01
sudo vi /var/named/inside.domusdigitalis.dev.zone
1. Increment SOA serial (format YYYYMMDDNN)
2. Add A records (in Network Devices section, after switches):
; VyOS Routers (.2-.3)
vyos-01 IN A 10.50.1.2
vyos-02 IN A 10.50.1.3
3. Save and validate:
sudo named-checkzone inside.domusdigitalis.dev /var/named/inside.domusdigitalis.dev.zone
4. Reload zone:
sudo rndc reload inside.domusdigitalis.dev
sudo vi /var/named/10.50.1.rev
1. Increment SOA serial
2. Add PTR records (NO leading whitespace!):
2 IN PTR vyos-01.inside.domusdigitalis.dev.
3 IN PTR vyos-02.inside.domusdigitalis.dev.
3. Validate and reload:
sudo named-checkzone 1.50.10.in-addr.arpa /var/named/10.50.1.rev
sudo rndc reload 1.50.10.in-addr.arpa
dig +short vyos-01.inside.domusdigitalis.dev @10.50.1.90
dig +short vyos-02.inside.domusdigitalis.dev @10.50.1.90
10.50.1.2
10.50.1.3
echo "Deploying: $VYOS_NODE on $KVM_HOST"
hostname && uptime
ssh "$KVM_HOST" "hostname && uptime"
ip link show type bridge | awk '/^[0-9]+:/{print $2}' | tr -d ':'
ssh "$KVM_HOST" "ip link show type bridge | awk '/^[0-9]+:/{print \$2}' | tr -d ':'"
| Hypervisor | Required Bridges | Notes |
|---|---|---|
kvm-01 |
|
WAN bridge (eno7). LAN trunk on br-mgmt (VLAN filtering + libvirt hook). |
kvm-02 |
|
WAN via PCI passthrough (ixl0 10GbE). LAN trunk on br-mgmt. |
|
SYMMETRIC HA Architecture: Both nodes have identical dual-interface configuration. Each hypervisor has dedicated 10GbE NIC (ixl0) for WAN passthrough to ISP modem. True HA failover for both WAN and LAN traffic. |
df -h /var/lib/libvirt/images | awk 'NR==2{print "Available: " $4}'
ssh "$KVM_HOST" "df -h /var/lib/libvirt/images | awk 'NR==2{print \"Available: \" \$4}'"
sudo virsh list --all | grep -q "$VYOS_NODE" && echo 'WARNING: VM exists!' || echo 'OK: VM does not exist'
ssh "$KVM_HOST" "sudo virsh list --all | grep -q '$VYOS_NODE' && echo 'WARNING: VM exists!' || echo 'OK: VM does not exist'"
7.2. 1.1 Download VyOS ISO
VyOS offers rolling (nightly) and LTS releases. Rolling is free and stable for home enterprise.
|
Command Format: Each command shows TWO options:
|
VYOS_LATEST=$(curl -sL https://api.github.com/repositories/674742659/releases | jq -r '.[0].tag_name') && echo "Latest: $VYOS_LATEST"
ssh "$KVM_HOST" "VYOS_LATEST=\$(curl -sL https://api.github.com/repositories/674742659/releases | jq -r '.[0].tag_name') && echo \"Latest: \$VYOS_LATEST\""
curl -Lo /var/lib/libvirt/images/vyos-rolling-latest.iso \
"https://github.com/vyos/vyos-nightly-build/releases/download/${VYOS_LATEST}/vyos-${VYOS_LATEST}-generic-amd64.iso"
ssh "$KVM_HOST" "curl -Lo /var/lib/libvirt/images/vyos-rolling-latest.iso \
'https://github.com/vyos/vyos-nightly-build/releases/download/${VYOS_LATEST}/vyos-${VYOS_LATEST}-generic-amd64.iso'"
ls -lh /var/lib/libvirt/images/vyos*.iso
ssh "$KVM_HOST" "ls -lh /var/lib/libvirt/images/vyos*.iso"
7.3. 1.2 Create VM Disk
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/${VYOS_NODE}.qcow2 10G
ssh "$KVM_HOST" "sudo qemu-img create -f qcow2 /var/lib/libvirt/images/${VYOS_NODE}.qcow2 10G"
ls -lh /var/lib/libvirt/images/${VYOS_NODE}.qcow2
ssh "$KVM_HOST" "ls -lh /var/lib/libvirt/images/${VYOS_NODE}.qcow2"
7.4. 1.3 Create VM
For vyos-02 on kvm-02 (dual interface - WAN + LAN):
sudo virt-install \
--name "$VYOS_NODE" \
--memory 2048 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/${VYOS_NODE}.qcow2,format=qcow2,bus=virtio \
--cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
--os-variant debian11 \
--network bridge=br-wan,model=virtio \
--network bridge=br-mgmt,model=virtio \
--graphics vnc,listen=0.0.0.0 \
--noautoconsole
ssh "$KVM_HOST" "sudo virt-install \
--name '$VYOS_NODE' \
--memory 2048 \
--vcpus 2 \
--disk path=/var/lib/libvirt/images/${VYOS_NODE}.qcow2,format=qcow2,bus=virtio \
--cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
--os-variant debian11 \
--network bridge=br-wan,model=virtio \
--network bridge=br-mgmt,model=virtio \
--graphics vnc,listen=0.0.0.0 \
--noautoconsole"
|
|
For vyos-01 on kvm-01 (dual interface - WAN + LAN, deploy after cutover):
|
kvm-01 Storage Architecture: Root disk is only 14GB. VM images go to onboard SSD:
|
sudo mkdir -p /mnt/onboard-ssd/libvirt/images
sudo virt-install \
--name "$VYOS_NODE" \
--memory 2048 \
--vcpus 2 \
--disk path=/mnt/onboard-ssd/libvirt/images/${VYOS_NODE}.qcow2,size=10,format=qcow2,bus=virtio \
--cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
--os-variant debian11 \
--network bridge=br-wan,model=virtio \
--network bridge=br-mgmt,model=virtio \
--graphics vnc,listen=0.0.0.0 \
--noautoconsole
ssh "$KVM_HOST" "sudo virt-install \
--name '$VYOS_NODE' \
--memory 2048 \
--vcpus 2 \
--disk path=/mnt/onboard-ssd/libvirt/images/${VYOS_NODE}.qcow2,size=10,format=qcow2,bus=virtio \
--cdrom /var/lib/libvirt/images/vyos-rolling-latest.iso \
--os-variant debian11 \
--network bridge=br-wan,model=virtio \
--network bridge=br-mgmt,model=virtio \
--graphics vnc,listen=0.0.0.0 \
--noautoconsole"
|
Symmetric HA Deployment Architecture:
Both nodes have identical capability. vyos-02 deploys first as BACKUP (priority 100). After cutover (Phase 14), vyos-01 deploys and becomes MASTER (priority 200). Full WAN + LAN failover with conntrack-sync for stateful session preservation. |
7.5. 1.4 Install VyOS to Disk
After VM boots to live ISO, connect via console:
sudo virsh console "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh console $VYOS_NODE"
Login: vyos / vyos (live environment)
install image
| Prompt | Answer |
|---|---|
Would you like to continue? [y/N] |
|
What would you like to name this image? |
Press Enter (accept default version) |
Please enter a password for the "vyos" user |
your strong password |
Please confirm password for the "vyos" user |
repeat password |
What console should be used by default? (K: KVM, S: Serial)? |
|
Which one should I install to? (Default: /dev/vda) |
Press Enter (accept |
After install completes:
reboot
Detach from console: Ctrl+]
7.6. 1.5 Start VM and Verify
After reboot, VM will be shut off. Start it:
sudo virsh start "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh start '$VYOS_NODE'"
Connect to console:
sudo virsh console "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh console $VYOS_NODE"
Login with vyos and the password you set during install.
7.7. 1.6 Post-Validation
sudo virsh list --all | grep "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh list --all | grep '$VYOS_NODE'"
1 vyos-02 running
sudo virsh domiflist "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh domiflist '$VYOS_NODE'"
Interface Type Source Model MAC ---------------------------------------------------------------- vnet0 bridge br-wan virtio 52:54:00:xx:xx:xx vnet1 bridge br-mgmt virtio 52:54:00:xx:xx:xx
Interface Type Source Model MAC ---------------------------------------------------------------- vnet0 bridge br-wan virtio 52:54:00:xx:xx:xx vnet1 bridge br-mgmt virtio 52:54:00:xx:xx:xx
The libvirt hook should automatically configure VLANs on the br-mgmt vnet interface. If it didn’t trigger, apply manually:
# Check current VLAN state (replace vnetX with actual interface from POST-2)
sudo bridge vlan show dev vnet1
port vlan-id
vnet1 10
20
30
40
100 PVID Egress Untagged
110
120
# Add all VLANs
for vid in 10 20 30 40 100 110 120; do
sudo bridge vlan add vid $vid dev vnet1
done
# Remove default PVID 1, set PVID 100 for MGMT
sudo bridge vlan del vid 1 dev vnet1 pvid untagged
sudo bridge vlan add vid 100 dev vnet1 pvid untagged
# Verify
sudo bridge vlan show dev vnet1
|
Why PVID 100? VyOS eth0/eth1 has 10.50.1.2/24 or 10.50.1.3/24 configured directly (untagged). PVID determines which VLAN receives untagged frames. Must be 100 to match MGMT VLAN. |
sudo virsh domblklist "$VYOS_NODE"
ssh "$KVM_HOST" "sudo virsh domblklist '$VYOS_NODE'"
|
Phase 1 Complete. VM is running. Proceed to Phase 2 for configuration. |
8. Phase 2: Interface and VLAN Configuration
8.1. 2.0 Pre-Validation
8.1.1. PRE-2.0a: Hypervisor Bridge Verification (on kvm-02)
Run these commands on the hypervisor before configuring VyOS interfaces.
nmcli -t conn show | awk -F: '{status[$4]++; print $1, "→", $4} END{print "---"; for(s in status) print s": "status[s]}'
nmcli -t -f DEVICE,TYPE,STATE,CONNECTION device | awk -F: '$2=="ethernet"{printf "%-12s %-8s %s\n", $1, $3, $4}'
nmcli -t conn show | awk -F: '{print $1}' | while read c; do
master=$(nmcli -g connection.master conn show "$c" 2>/dev/null)
[[ -n "$master" ]] && echo "$c → $master"
done
for br in $(ip link show type bridge | awk -F': ' '{print $2}'); do
echo "=== $br ==="
ip link show master $br | awk -F': ' '/^[0-9]/{print " "$2}'
done
echo "br-mgmt: $(ip link show master br-mgmt 2>/dev/null | awk -F': ' '/^[0-9].*eno/{print $2}')"
echo "br-wan: $(ip link show master br-wan 2>/dev/null | awk -F': ' '/^[0-9].*eno/{print $2}')"
sudo virsh domiflist "$VYOS_NODE" | awk 'NR>2 && NF{printf "%-8s → %s\n", $1, $3}'
br-mgmt: eno8 br-wan: eno7 vnet1 → br-mgmt vnet2 → br-wan
8.1.2. PRE-2.0b: Bridge VLAN Configuration (CRITICAL)
|
KVM bridge VLAN filtering requires VLANs on the VM’s vnet interface. Without this, VLAN-tagged traffic (DHCP, DNS, etc.) is silently dropped even if the bridge and physical trunk have the VLANs. |
sudo bridge vlan show
port vlan-id
eno8 1 PVID Egress Untagged
10
20
30
40
100
110
120
br-mgmt 1 PVID Egress Untagged
10
20
30
40
100
110
120
vnet1 1 PVID Egress Untagged <-- PROBLEM: Missing VLANs!
10 <-- NEED: All VLANs like br-mgmt
20
30
40
100
110
120
# Add all VLANs
for vid in 10 20 30 40 100 110 120; do
sudo bridge vlan add vid $vid dev vnet1
done
# CRITICAL: Set PVID 100 for MGMT (vyos eth0 = 10.50.1.x untagged)
sudo bridge vlan add vid 100 dev vnet1 pvid untagged
|
Why PVID 100? VyOS eth0 has 10.50.1.3/24 (MGMT) configured directly - not as eth0.100. This means MGMT traffic is UNTAGGED. The bridge PVID determines which VLAN receives untagged frames. Default PVID 1 sends traffic to wrong VLAN. Must be PVID 100 to match MGMT VLAN. |
sudo bridge vlan show dev vnet1
|
Make persistent via libvirt hook or NetworkManager. See KVM Bridge VLAN Persistence. |
# From workstation on VyOS-managed VLAN (e.g., VLAN 40/IOT)
ip -4 addr show $INTERFACE | awk '/inet/{print $2}'
# Expected: 10.50.40.x (from vyos-02 DHCP pool)
8.2. 2.1 System Basics (vyos-02)
| These commands run inside VyOS configuration mode on vyos-02. |
# System identity - vyos-02
set system host-name vyos-02
set system domain-name inside.domusdigitalis.dev
set system time-zone America/Los_Angeles
# DNS servers
set system name-server 10.50.1.90
set system name-server 10.50.1.91
# Console access for virsh console
set system console device ttyS0 speed 115200
# NTP - sync to external, serve to internal
set service ntp server time.cloudflare.com
set service ntp server time.google.com
set service ntp listen-address 10.50.1.3
set service ntp listen-address 10.50.1.1
commit
8.3. 2.1b System Basics (vyos-01)
| Run these on vyos-01 after vyos-02 is validated (Phase 15). |
# System identity - vyos-01
set system host-name vyos-01
set system domain-name inside.domusdigitalis.dev
set system time-zone America/Los_Angeles
# DNS servers
set system name-server 10.50.1.90
set system name-server 10.50.1.91
# Console access for virsh console
set system console device ttyS0 speed 115200
# NTP - sync to external, serve to internal
set service ntp server time.cloudflare.com
set service ntp server time.google.com
set service ntp listen-address 10.50.1.2
set service ntp listen-address 10.50.1.1
commit
8.4. 2.2 Interface Configuration (vyos-02 on kvm-02)
|
Interface mappings differ between nodes! See
|
Interface mapping for vyos-02:
| VyOS | Bridge | Purpose | Notes |
|---|---|---|---|
eth0 |
br-mgmt |
LAN-TRUNK |
Trunk to Catalyst switch (VLANs) |
eth1 |
br-wan |
WAN |
DHCP from ISP modem via eno7 (1GbE) |
|
Why this order? VM created with br-mgmt first, br-wan hot-added second. Interface order is determined by attachment sequence, not virt-install flags. |
# LAN trunk interface (eth0)
set interfaces ethernet eth0 description 'LAN-TRUNK'
# INFRA - Native VLAN 100 (untagged on switch trunk)
# Use vyos-02's real IP (10.50.1.3), NOT the VIP (VIP is for VRRP later)
set interfaces ethernet eth0 address 10.50.1.3/24
# SECURITY - VLAN 110 (crown jewels: Vault, ISE)
set interfaces ethernet eth0 vif 110 address 10.50.110.1/24
set interfaces ethernet eth0 vif 110 description 'SECURITY'
# SERVICES - VLAN 120 (general VMs: Keycloak, Gitea, etc.)
set interfaces ethernet eth0 vif 120 address 10.50.120.1/24
set interfaces ethernet eth0 vif 120 description 'SERVICES'
# DATA - VLAN 10
set interfaces ethernet eth0 vif 10 address 10.50.10.1/24
set interfaces ethernet eth0 vif 10 description 'DATA'
# VOICE - VLAN 20
set interfaces ethernet eth0 vif 20 address 10.50.20.1/24
set interfaces ethernet eth0 vif 20 description 'VOICE'
# GUEST - VLAN 30
set interfaces ethernet eth0 vif 30 address 10.50.30.1/24
set interfaces ethernet eth0 vif 30 description 'GUEST'
# IOT - VLAN 40
set interfaces ethernet eth0 vif 40 address 10.50.40.1/24
set interfaces ethernet eth0 vif 40 description 'IOT'
# WAN - DHCP from ISP modem (eth1)
set interfaces ethernet eth1 description 'WAN'
set interfaces ethernet eth1 address dhcp
set interfaces ethernet eth1 ipv6 address no-default-link-local
commit
run show interfaces ethernet
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 {vyos-02-ip}/24 u/u LAN-TRUNK
eth0.10 {gateway-data}/24 u/u DATA
eth0.20 {gateway-voice}/24 u/u VOICE
eth0.30 {gateway-guest}/24 u/u GUEST
eth0.40 {gateway-iot}/24 u/u IOT
eth0.110 {gateway-security}/24 u/u SECURITY
eth0.120 {gateway-services}/24 u/u SERVICES
eth1 <dhcp-assigned> u/u WAN
Detailed Output (click to expand)
show interfaces
ethernet eth0 {
address 10.50.1.3/24
description LAN-TRUNK
hw-id 52:54:00:31:eb:e3
offload {
gro
gso
sg
tso
}
vif 10 {
address 10.50.10.1/24
description DATA
}
vif 20 {
address 10.50.20.1/24
description VOICE
}
vif 30 {
address 10.50.30.1/24
description GUEST
}
vif 40 {
address 10.50.40.1/24
description IOT
}
vif 110 {
address 10.50.110.1/24
description SECURITY
}
vif 120 {
address 10.50.120.1/24
description SERVICES
}
}
ethernet eth1 {
address dhcp
description WAN
}
loopback lo {
}
|
SYMMETRIC HA: Both vyos-01 and vyos-02 have identical interface configuration. Each has dedicated 10GbE WAN to ISP modem for true active/standby failover. VLAN 110/120 are READY - VMs migrate to these VLANs in Phase 18 after VyOS HA is verified. |
8.5. 2.3 Interface Configuration (vyos-01 on kvm-01)
Dual interface deployment - eth0 is WAN, eth1 is LAN trunk.
# WAN - DHCP from ISP modem (eth0)
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 ipv6 address no-default-link-local
# LAN trunk interface (eth1)
set interfaces ethernet eth1 description 'LAN-TRUNK'
# INFRA - Native VLAN 100
# Use vyos-01's real IP (10.50.1.2), NOT the VIP (VIP is for VRRP later)
set interfaces ethernet eth1 address 10.50.1.2/24
# SECURITY - VLAN 110 (crown jewels: Vault, ISE)
set interfaces ethernet eth1 vif 110 address 10.50.110.1/24
set interfaces ethernet eth1 vif 110 description 'SECURITY'
# SERVICES - VLAN 120 (general VMs: Keycloak, Gitea, etc.)
set interfaces ethernet eth1 vif 120 address 10.50.120.1/24
set interfaces ethernet eth1 vif 120 description 'SERVICES'
# DATA - VLAN 10
set interfaces ethernet eth1 vif 10 address 10.50.10.1/24
set interfaces ethernet eth1 vif 10 description 'DATA'
# VOICE - VLAN 20
set interfaces ethernet eth1 vif 20 address 10.50.20.1/24
set interfaces ethernet eth1 vif 20 description 'VOICE'
# GUEST - VLAN 30
set interfaces ethernet eth1 vif 30 address 10.50.30.1/24
set interfaces ethernet eth1 vif 30 description 'GUEST'
# IOT - VLAN 40
set interfaces ethernet eth1 vif 40 address 10.50.40.1/24
set interfaces ethernet eth1 vif 40 description 'IOT'
8.7. 2.5 Commit and Save
# Show pending changes before commit
compare
# Commit changes
commit
# Save to config.boot
save
8.8. 2.6 Post-Validation
# POST-1: Verify all interfaces are up
run show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.1.x/24 u/u WAN
eth1 10.50.1.3/24 u/u LAN-TRUNK
eth1.10 10.50.10.1/24 u/u DATA
eth1.20 10.50.20.1/24 u/u VOICE
eth1.30 10.50.30.1/24 u/u GUEST
eth1.40 10.50.40.1/24 u/u IOT
eth1.110 10.50.110.1/24 u/u SECURITY
eth1.120 10.50.120.1/24 u/u SERVICES
lo 127.0.0.1/8 u/u
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.1.x/24 u/u WAN
eth1 10.50.1.2/24 u/u LAN-TRUNK
eth1.10 10.50.10.1/24 u/u DATA
eth1.20 10.50.20.1/24 u/u VOICE
eth1.30 10.50.30.1/24 u/u GUEST
eth1.40 10.50.40.1/24 u/u IOT
eth1.110 10.50.110.1/24 u/u SECURITY
eth1.120 10.50.120.1/24 u/u SERVICES
lo 127.0.0.1/8 u/u
# POST-2: Verify LAN interface has correct IP (eth1 is LAN trunk)
run show interfaces ethernet eth1
# POST-3: Verify DNS resolution works (via existing pfSense gateway)
run ping -c 2 google.com
# POST-4: Verify hostname is set correctly
run show host name
|
Phase 2 Complete. Interfaces configured. Proceed to Phase 3 for firewall zones. Both vyos-01 and vyos-02 have WAN interfaces (eth0). However, until cutover (Phase 14), internet access still routes via pfSense. VyOS WAN gets DHCP from modem but isn’t the default gateway yet. |
9. Phase 3: Zone-Based Firewall
VyOS uses zone-based firewall with nftables backend. Define zones, then policies between zones.
9.1. 3.0 Pre-Validation
# PRE-1: Verify Phase 2 completed (interfaces exist)
show interfaces
# Expected: eth0 (WAN), eth1 (LAN trunk), eth1.10/20/30/40 (VLANs)
# PRE-2: Verify no existing zone config (clean slate)
show firewall zone
# Expected: empty or error (no zones defined yet)
# PRE-3: Document current firewall state (baseline)
show firewall summary
9.2. 3.1 Define Firewall Zones
|
Interface mappings differ per node!
Commands below are for vyos-02. For vyos-01, swap eth0↔eth1. |
For vyos-02 (eth0=LAN, eth1=WAN):
|
VyOS 1.4+ Syntax: Zone membership uses |
configure
# WAN Zone (untrusted) - eth1 for vyos-02
set firewall zone WAN member interface eth1
set firewall zone WAN default-action drop
set firewall zone WAN description 'Untrusted Internet'
# MGMT Zone (high trust) - eth0 for vyos-02
set firewall zone MGMT member interface eth0
set firewall zone MGMT default-action drop
set firewall zone MGMT description 'Management and Infrastructure'
# DATA Zone (medium trust)
set firewall zone DATA member interface eth0.10
set firewall zone DATA default-action drop
set firewall zone DATA description 'Corporate Devices'
# VOICE Zone (medium trust)
set firewall zone VOICE member interface eth0.20
set firewall zone VOICE default-action drop
set firewall zone VOICE description 'VoIP Phones'
# GUEST Zone (low trust)
set firewall zone GUEST member interface eth0.30
set firewall zone GUEST default-action drop
set firewall zone GUEST description 'Guest Wireless - Internet Only'
# IOT Zone (low trust)
set firewall zone IOT member interface eth0.40
set firewall zone IOT default-action drop
set firewall zone IOT description 'IoT Devices - Limited Access'
# SECURITY Zone (crown jewels)
set firewall zone SECURITY member interface eth0.110
set firewall zone SECURITY default-action drop
set firewall zone SECURITY description 'Crown Jewels - Vault, ISE'
# SERVICES Zone (infrastructure VMs)
set firewall zone SERVICES member interface eth0.120
set firewall zone SERVICES default-action drop
set firewall zone SERVICES description 'Infrastructure VMs'
# LOCAL Zone (firewall itself)
set firewall zone LOCAL local-zone
set firewall zone LOCAL default-action drop
set firewall zone LOCAL description 'Firewall Services'
commit
save
run show firewall zone | awk 'NR>2 && NF>=2 {printf "%-10s → %s\n", $1, $2}'
DATA → eth0.10 GUEST → eth0.30 IOT → eth0.40 LOCAL → LOCAL MGMT → eth0 SECURITY → eth0.110 SERVICES → eth0.120 VOICE → eth0.20 WAN → eth1
Detailed Output (click to expand)
show firewall zone
zone DATA {
default-action drop
description "Corporate Devices"
member {
interface eth0.10
}
}
zone GUEST {
default-action drop
description "Guest Wireless - Internet Only"
member {
interface eth0.30
}
}
zone IOT {
default-action drop
description "IoT Devices - Limited Access"
member {
interface eth0.40
}
}
zone LOCAL {
default-action drop
description "Firewall Services"
local-zone
}
zone MGMT {
default-action drop
description "Management and Infrastructure"
member {
interface eth0
}
}
zone SECURITY {
default-action drop
description "Crown Jewels - Vault, ISE"
member {
interface eth0.110
}
}
zone SERVICES {
default-action drop
description "Infrastructure VMs"
member {
interface eth0.120
}
}
zone VOICE {
default-action drop
description "VoIP Phones"
member {
interface eth0.20
}
}
zone WAN {
default-action drop
description "Untrusted Internet"
member {
interface eth1
}
}
9.3. 3.2 Firewall Groups (Address Lists)
| These commands use session variables for consistency. Verify variables are set before proceeding. |
# PRE: Verify k3s variables are set
echo "k3s Masters: $K3S_MASTER_01, $K3S_MASTER_02, $K3S_MASTER_03"
echo "k3s Workers: $K3S_WORKER_01, $K3S_WORKER_02, $K3S_WORKER_03"
echo "DNS: $DNS_PRIMARY, $DNS_SECONDARY"
echo "Wazuh: $WAZUH_MANAGER"
# RFC1918 Private Networks
set firewall group network-group RFC1918 network 10.0.0.0/8
set firewall group network-group RFC1918 network 172.16.0.0/12
set firewall group network-group RFC1918 network 192.168.0.0/16
# Internal Networks (all VLANs) - for blanket rules
set firewall group network-group INTERNAL network 10.50.0.0/16
# Per-VLAN Network Groups (for granular NAT/firewall control)
set firewall group network-group NET_INFRA network 10.50.1.0/24
set firewall group network-group NET_INFRA description 'VLAN 100 - Infrastructure'
set firewall group network-group NET_DATA network 10.50.10.0/24
set firewall group network-group NET_DATA description 'VLAN 10 - Corporate Data'
set firewall group network-group NET_VOICE network 10.50.20.0/24
set firewall group network-group NET_VOICE description 'VLAN 20 - VoIP'
set firewall group network-group NET_GUEST network 10.50.30.0/24
set firewall group network-group NET_GUEST description 'VLAN 30 - Guest WiFi'
set firewall group network-group NET_IOT network 10.50.40.0/24
set firewall group network-group NET_IOT description 'VLAN 40 - IoT Devices'
set firewall group network-group NET_SECURITY network 10.50.110.0/24
set firewall group network-group NET_SECURITY description 'VLAN 110 - Crown Jewels'
set firewall group network-group NET_SERVICES network 10.50.120.0/24
set firewall group network-group NET_SERVICES description 'VLAN 120 - Infrastructure VMs'
# Wazuh Manager (all agents need access)
set firewall group address-group WAZUH_MANAGER address 10.50.1.134
# DNS Servers
set firewall group address-group DNS_SERVERS address 10.50.1.90
set firewall group address-group DNS_SERVERS address 10.50.1.91
# K8s Services (LoadBalancer VIPs)
set firewall group network-group K8S_SERVICES network 10.50.1.130/32
set firewall group network-group K8S_SERVICES network 10.50.1.131/32
set firewall group network-group K8S_SERVICES network 10.50.1.132/32
set firewall group network-group K8S_SERVICES network 10.50.1.133/32
set firewall group network-group K8S_SERVICES network 10.50.1.134/32
# k3s Cluster Nodes (6-node HA cluster)
set firewall group address-group K3S_NODES address 10.50.1.120
set firewall group address-group K3S_NODES address 10.50.1.121
set firewall group address-group K3S_NODES address 10.50.1.122
set firewall group address-group K3S_NODES address 10.50.1.123
set firewall group address-group K3S_NODES address 10.50.1.124
set firewall group address-group K3S_NODES address 10.50.1.125
# k3s Control Plane (masters only - for etcd)
set firewall group address-group K3S_MASTERS address 10.50.1.120
set firewall group address-group K3S_MASTERS address 10.50.1.121
set firewall group address-group K3S_MASTERS address 10.50.1.122
# ADMINS - Privileged workstations (SSH, HTTPS management access)
# Purpose: Prevent lateral movement - only these IPs can SSH to infrastructure
set firewall group address-group ADMINS address 10.50.10.106
set firewall group address-group ADMINS address 10.50.10.107
set firewall group address-group ADMINS address 10.50.10.108
set firewall group address-group ADMINS description 'Admin workstations with privileged access'
# =============================================================================
# CRITICAL INFRASTRUCTURE ADDRESS GROUPS
# =============================================================================
# --- SECURITY / CROWN JEWELS ---
# ISE Cluster (802.1X, TACACS+, RADIUS)
set firewall group address-group ISE_NODES address 10.50.1.20
set firewall group address-group ISE_NODES address 10.50.1.21
set firewall group address-group ISE_NODES description 'Cisco ISE cluster - 802.1X/RADIUS/TACACS+'
# Vault Cluster (PKI, SSH CA, Secrets)
set firewall group address-group VAULT_NODES address 10.50.1.60
set firewall group address-group VAULT_NODES address 10.50.1.61
set firewall group address-group VAULT_NODES address 10.50.1.62
set firewall group address-group VAULT_NODES description 'HashiCorp Vault HA cluster'
# iPSK Manager (wireless PSK management)
set firewall group address-group IPSK_NODES address 10.50.1.30
set firewall group address-group IPSK_NODES address 10.50.1.31
set firewall group address-group IPSK_NODES description 'iPSK Manager cluster'
# --- IDENTITY ---
# Active Directory Domain Controllers
set firewall group address-group AD_DCS address 10.50.1.50
set firewall group address-group AD_DCS address 10.50.1.51
set firewall group address-group AD_DCS description 'Windows AD Domain Controllers'
# Keycloak IdP Cluster
set firewall group address-group KEYCLOAK_NODES address 10.50.1.80
set firewall group address-group KEYCLOAK_NODES address 10.50.1.81
set firewall group address-group KEYCLOAK_NODES description 'Keycloak Identity Provider cluster'
# FreeIPA Cluster
set firewall group address-group IPA_NODES address 10.50.1.100
set firewall group address-group IPA_NODES address 10.50.1.101
set firewall group address-group IPA_NODES description 'FreeIPA Identity Management cluster'
# --- NETWORK INFRASTRUCTURE ---
# Wireless LAN Controllers
set firewall group address-group WLC_NODES address 10.50.1.40
set firewall group address-group WLC_NODES address 10.50.1.41
set firewall group address-group WLC_NODES description 'Cisco 9800 WLC cluster'
# Switches (management access)
set firewall group address-group SWITCHES address 10.50.1.11
set firewall group address-group SWITCHES address 10.50.1.10
set firewall group address-group SWITCHES description 'Cisco Catalyst switches'
# VyOS Routers (HA pair)
set firewall group address-group VYOS_NODES address 10.50.1.2
set firewall group address-group VYOS_NODES address 10.50.1.3
set firewall group address-group VYOS_NODES description 'VyOS router HA pair'
# pfSense (legacy/migration)
set firewall group address-group PFSENSE address 10.50.1.1
set firewall group address-group PFSENSE description 'pfSense firewall (migration target)'
# --- STORAGE ---
# NAS Cluster
set firewall group address-group NAS_NODES address 10.50.1.70
set firewall group address-group NAS_NODES address 10.50.1.71
set firewall group address-group NAS_NODES description 'Synology NAS cluster'
# Git Server
set firewall group address-group GITEA address 10.50.1.72
set firewall group address-group GITEA description 'Gitea git server'
# Object Storage
set firewall group address-group MINIO address 10.50.1.73
set firewall group address-group MINIO description 'MinIO S3-compatible storage'
# --- COMPUTE / HYPERVISORS ---
# KVM Hypervisors
set firewall group address-group KVM_HOSTS address 10.50.1.110
set firewall group address-group KVM_HOSTS address 10.50.1.111
set firewall group address-group KVM_HOSTS description 'KVM/libvirt hypervisors'
# IPMI/BMC (out-of-band management)
set firewall group address-group IPMI_NODES address 10.50.1.200
set firewall group address-group IPMI_NODES address 10.50.1.201
set firewall group address-group IPMI_NODES description 'IPMI/BMC out-of-band management'
# --- OBSERVABILITY ---
# Zabbix Monitoring
set firewall group address-group ZABBIX address 10.50.1.135
set firewall group address-group ZABBIX description 'Zabbix monitoring server'
# Wazuh SIEM (already have WAZUH_MANAGER, add full cluster)
set firewall group address-group WAZUH_NODES address 10.50.1.131
set firewall group address-group WAZUH_NODES address 10.50.1.132
set firewall group address-group WAZUH_NODES address 10.50.1.134
set firewall group address-group WAZUH_NODES address 10.50.1.133
set firewall group address-group WAZUH_NODES description 'Wazuh SIEM cluster VIPs'
# =============================================================================
# PORT GROUPS
# =============================================================================
# --- COMMON SERVICES ---
# SSH/Management
set firewall group port-group SSH port 22
set firewall group port-group HTTPS port 443
set firewall group port-group HTTP port 80
set firewall group port-group MGMT_WEB port 80
set firewall group port-group MGMT_WEB port 443
# DNS (TCP and UDP - use with protocol specification in rules)
set firewall group port-group DNS_PORTS port 53
# NTP
set firewall group port-group NTP port 123
# --- AUTHENTICATION & IDENTITY ---
# RADIUS (ISE authentication)
set firewall group port-group RADIUS port 1812
set firewall group port-group RADIUS port 1813
set firewall group port-group RADIUS port 1645
set firewall group port-group RADIUS port 1646
# TACACS+ (ISE device admin)
set firewall group port-group TACACS port 49
# ISE Services (ERS API, Admin, pxGrid)
set firewall group port-group ISE_SERVICES port 443
set firewall group port-group ISE_SERVICES port 9060
set firewall group port-group ISE_SERVICES port 8910
# Active Directory (Kerberos, LDAP, DNS, GC)
set firewall group port-group AD_PORTS port 88
set firewall group port-group AD_PORTS port 389
set firewall group port-group AD_PORTS port 636
set firewall group port-group AD_PORTS port 3268
set firewall group port-group AD_PORTS port 3269
set firewall group port-group AD_PORTS port 464
set firewall group port-group AD_PORTS port 53
set firewall group port-group AD_PORTS port 445
set firewall group port-group AD_PORTS port 135
# Keycloak (OIDC/SAML)
set firewall group port-group KEYCLOAK port 8080
set firewall group port-group KEYCLOAK port 8443
# --- PKI & SECRETS ---
# Vault (API and Raft cluster)
set firewall group port-group VAULT port 8200
set firewall group port-group VAULT_CLUSTER port 8201
# --- STORAGE ---
# NFS (NAS mounts)
set firewall group port-group NFS port 111
set firewall group port-group NFS port 2049
set firewall group port-group NFS port 20048
# SMB/CIFS (Windows shares, Synology)
set firewall group port-group SMB port 445
set firewall group port-group SMB port 139
# iSCSI
set firewall group port-group ISCSI port 3260
# Synology DSM
set firewall group port-group SYNOLOGY port 5000
set firewall group port-group SYNOLOGY port 5001
# MinIO (S3 API)
set firewall group port-group MINIO port 9000
set firewall group port-group MINIO port 9001
# Gitea
set firewall group port-group GITEA port 3000
set firewall group port-group GITEA port 22
# --- WIRELESS ---
# CAPWAP (AP to WLC)
set firewall group port-group CAPWAP port 5246
set firewall group port-group CAPWAP port 5247
# --- MONITORING & SIEM ---
# Wazuh (syslog, agent, API)
set firewall group port-group WAZUH_PORTS port 514
set firewall group port-group WAZUH_PORTS port 1514
set firewall group port-group WAZUH_PORTS port 1515
set firewall group port-group WAZUH_PORTS port 55000
# Wazuh Dashboard/Indexer
set firewall group port-group WAZUH_WEB port 443
set firewall group port-group WAZUH_WEB port 9200
# Zabbix
set firewall group port-group ZABBIX port 10050
set firewall group port-group ZABBIX port 10051
# Prometheus/Grafana
set firewall group port-group PROMETHEUS port 9090
set firewall group port-group GRAFANA port 3000
# SNMP
set firewall group port-group SNMP port 161
set firewall group port-group SNMP port 162
# Syslog (UDP/TCP)
set firewall group port-group SYSLOG port 514
# --- KUBERNETES ---
# k3s Cluster Ports
set firewall group port-group K3S_API port 6443
set firewall group port-group K3S_ETCD port 2379-2380
set firewall group port-group K3S_KUBELET port 10250
set firewall group port-group K3S_VXLAN port 8472
# Cilium Ports
set firewall group port-group CILIUM_HEALTH port 4240
set firewall group port-group CILIUM_HUBBLE port 4244
# MetalLB (if using L2 mode with memberlist)
set firewall group port-group METALLB port 7946
# NodePort Range
set firewall group port-group NODEPORT port 30000-32767
# --- ROUTING ---
# BGP
set firewall group port-group BGP port 179
# VRRP (protocol 112, not TCP/UDP - handled separately in rules)
# POST-3.2: Verify ALL firewall groups created
run show firewall group
# Verify ADMINS group members (should show 3 IPs)
run show firewall group address-group ADMINS
# Expected: 10.50.10.106, 10.50.10.107, 10.50.10.108
# Count address groups (expected: 22)
# ADMINS, AD_DCS, DNS_SERVERS, GITEA, IPA_NODES, IPMI_NODES, IPSK_NODES, ISE_NODES,
# K3S_MASTERS, K3S_NODES, KEYCLOAK_NODES, KVM_HOSTS, MINIO, NAS_NODES, PFSENSE,
# SWITCHES, VAULT_NODES, VYOS_NODES, WAZUH_MANAGER, WAZUH_NODES, WLC_NODES, ZABBIX
run show firewall group | awk '/address-group/{count++} END{print "Address groups:", count}'
# Count port groups (expected: 31)
# SSH, HTTPS, HTTP, MGMT_WEB, DNS_PORTS, NTP, RADIUS, TACACS, ISE_SERVICES, AD_PORTS,
# KEYCLOAK, VAULT, VAULT_CLUSTER, NFS, SMB, ISCSI, SYNOLOGY, MINIO, GITEA, CAPWAP,
# WAZUH_PORTS, WAZUH_WEB, ZABBIX, PROMETHEUS, GRAFANA, SNMP, SYSLOG, K3S_API,
# K3S_ETCD, K3S_KUBELET, K3S_VXLAN, CILIUM_HEALTH, CILIUM_HUBBLE, METALLB, NODEPORT, BGP
run show firewall group | awk '/port-group/{count++} END{print "Port groups:", count}'
# Count network groups (expected: 10)
# RFC1918, INTERNAL, NET_INFRA, NET_DATA, NET_VOICE, NET_GUEST, NET_IOT,
# NET_SECURITY, NET_SERVICES, K8S_SERVICES
run show firewall group | awk '/network-group/{count++} END{print "Network groups:", count}'
|
If ADMINS group is missing or has wrong IPs, SSH from your workstations will be BLOCKED after firewall is applied. Fix before proceeding. |
9.4. 3.3 Base Firewall Rules
Create reusable rule sets:
# Allow established/related (used by all zones)
set firewall ipv4 name ALLOW_ESTABLISHED default-action drop
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 action accept
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 state established
set firewall ipv4 name ALLOW_ESTABLISHED rule 10 state related
set firewall ipv4 name ALLOW_ESTABLISHED rule 20 action drop
set firewall ipv4 name ALLOW_ESTABLISHED rule 20 state invalid
9.5. 3.4 WAN Inbound Rules
# WAN to LOCAL (firewall services)
set firewall ipv4 name WAN_LOCAL default-action drop
set firewall ipv4 name WAN_LOCAL rule 10 action accept
set firewall ipv4 name WAN_LOCAL rule 10 state established
set firewall ipv4 name WAN_LOCAL rule 10 state related
set firewall ipv4 name WAN_LOCAL rule 20 action accept
set firewall ipv4 name WAN_LOCAL rule 20 protocol icmp
set firewall ipv4 name WAN_LOCAL rule 20 icmp type-name echo-request
set firewall ipv4 name WAN_LOCAL rule 20 limit rate 5/second
# LOCAL to WAN (router outbound - updates, NTP, DNS queries, etc.)
# CRITICAL: Without this, vyos cannot reach internet!
set firewall ipv4 name LOCAL_WAN default-action accept
set firewall ipv4 name LOCAL_WAN description 'Router outbound to WAN'
# LOCAL to MGMT (router to infrastructure - BIND, ISE, Vault, etc.)
# CRITICAL: Without this, vyos cannot reach internal services!
set firewall ipv4 name LOCAL_MGMT default-action accept
set firewall ipv4 name LOCAL_MGMT description 'Router to MGMT infrastructure'
# WAN to any internal (drop all unsolicited)
set firewall ipv4 name WAN_IN default-action drop
set firewall ipv4 name WAN_IN rule 10 action accept
set firewall ipv4 name WAN_IN rule 10 state established
set firewall ipv4 name WAN_IN rule 10 state related
9.6. 3.5 LAN to WAN Rules (Outbound)
# MGMT to WAN (full access)
set firewall ipv4 name MGMT_WAN default-action accept
# DATA to WAN (full access)
set firewall ipv4 name DATA_WAN default-action accept
# VOICE to WAN (limited - SIP/RTP for cloud PBX if needed)
set firewall ipv4 name VOICE_WAN default-action accept
# GUEST to WAN (full internet access)
set firewall ipv4 name GUEST_WAN default-action accept
# IOT to WAN (limited - only specific services)
set firewall ipv4 name IOT_WAN default-action drop
set firewall ipv4 name IOT_WAN rule 10 action accept
set firewall ipv4 name IOT_WAN rule 10 state established
set firewall ipv4 name IOT_WAN rule 10 state related
set firewall ipv4 name IOT_WAN rule 20 action accept
set firewall ipv4 name IOT_WAN rule 20 protocol tcp
set firewall ipv4 name IOT_WAN rule 20 destination port 80,443
set firewall ipv4 name IOT_WAN rule 20 description 'Allow HTTP/HTTPS'
set firewall ipv4 name IOT_WAN rule 30 action accept
set firewall ipv4 name IOT_WAN rule 30 protocol udp
set firewall ipv4 name IOT_WAN rule 30 destination port 123
set firewall ipv4 name IOT_WAN rule 30 description 'Allow NTP'
set firewall ipv4 name IOT_WAN rule 60 action accept
set firewall ipv4 name IOT_WAN rule 60 protocol udp
set firewall ipv4 name IOT_WAN rule 60 destination port 5246
set firewall ipv4 name IOT_WAN rule 60 description 'Allow CAPWAP control (OEAP)'
set firewall ipv4 name IOT_WAN rule 70 action accept
set firewall ipv4 name IOT_WAN rule 70 protocol udp
set firewall ipv4 name IOT_WAN rule 70 destination port 5247
set firewall ipv4 name IOT_WAN rule 70 description 'Allow CAPWAP data (OEAP)'
9.7. 3.6 Inter-VLAN Rules
# MGMT to DATA (allow - admin access)
set firewall ipv4 name MGMT_DATA default-action accept
# MGMT to all internal zones (admin access)
set firewall ipv4 name MGMT_VOICE default-action accept
set firewall ipv4 name MGMT_GUEST default-action accept
set firewall ipv4 name MGMT_IOT default-action accept
# DATA to MGMT (allow access to infrastructure services)
set firewall ipv4 name DATA_MGMT default-action drop
set firewall ipv4 name DATA_MGMT rule 10 action accept
set firewall ipv4 name DATA_MGMT rule 10 state established
set firewall ipv4 name DATA_MGMT rule 10 state related
set firewall ipv4 name DATA_MGMT rule 20 action accept
set firewall ipv4 name DATA_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name DATA_MGMT rule 20 destination group address-group DNS_SERVERS
set firewall ipv4 name DATA_MGMT rule 20 destination group port-group DNS_PORTS
set firewall ipv4 name DATA_MGMT rule 20 description 'Allow DNS'
set firewall ipv4 name DATA_MGMT rule 30 action accept
set firewall ipv4 name DATA_MGMT rule 30 protocol tcp_udp
set firewall ipv4 name DATA_MGMT rule 30 destination group address-group WAZUH_MANAGER
set firewall ipv4 name DATA_MGMT rule 30 destination group port-group WAZUH_PORTS
set firewall ipv4 name DATA_MGMT rule 30 description 'Allow Wazuh agent'
set firewall ipv4 name DATA_MGMT rule 40 action accept
set firewall ipv4 name DATA_MGMT rule 40 destination group network-group K8S_SERVICES
set firewall ipv4 name DATA_MGMT rule 40 destination port 80,443
set firewall ipv4 name DATA_MGMT rule 40 protocol tcp
set firewall ipv4 name DATA_MGMT rule 40 description 'Allow K8s services'
# VOICE to MGMT (limited - DHCP, DNS, NTP, TFTP for phones)
set firewall ipv4 name VOICE_MGMT default-action drop
set firewall ipv4 name VOICE_MGMT rule 10 action accept
set firewall ipv4 name VOICE_MGMT rule 10 state established
set firewall ipv4 name VOICE_MGMT rule 10 state related
set firewall ipv4 name VOICE_MGMT rule 20 action accept
set firewall ipv4 name VOICE_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name VOICE_MGMT rule 20 destination group address-group DNS_SERVERS
set firewall ipv4 name VOICE_MGMT rule 20 destination group port-group DNS_PORTS
# GUEST to internal (BLOCK - internet only)
set firewall ipv4 name GUEST_MGMT default-action drop
set firewall ipv4 name GUEST_DATA default-action drop
set firewall ipv4 name GUEST_VOICE default-action drop
set firewall ipv4 name GUEST_IOT default-action drop
# IOT to internal (BLOCK - limited exceptions)
set firewall ipv4 name IOT_MGMT default-action drop
set firewall ipv4 name IOT_MGMT rule 10 action accept
set firewall ipv4 name IOT_MGMT rule 10 state established
set firewall ipv4 name IOT_MGMT rule 10 state related
set firewall ipv4 name IOT_MGMT rule 20 action accept
set firewall ipv4 name IOT_MGMT rule 20 protocol tcp_udp
set firewall ipv4 name IOT_MGMT rule 20 destination group address-group WAZUH_MANAGER
set firewall ipv4 name IOT_MGMT rule 20 destination group port-group WAZUH_PORTS
set firewall ipv4 name IOT_MGMT rule 20 description 'Allow Wazuh agent'
set firewall ipv4 name IOT_DATA default-action drop
set firewall ipv4 name IOT_VOICE default-action drop
set firewall ipv4 name IOT_GUEST default-action drop
# SECURITY zone rules (crown jewels - Vault, ISE)
# MGMT → SECURITY (admin access)
set firewall ipv4 name MGMT_SECURITY default-action accept
# DATA → SECURITY (allow access to Vault API, ISE services)
set firewall ipv4 name DATA_SECURITY default-action drop
set firewall ipv4 name DATA_SECURITY rule 10 action accept
set firewall ipv4 name DATA_SECURITY rule 10 state established
set firewall ipv4 name DATA_SECURITY rule 10 state related
set firewall ipv4 name DATA_SECURITY rule 20 action accept
set firewall ipv4 name DATA_SECURITY rule 20 protocol tcp
set firewall ipv4 name DATA_SECURITY rule 20 destination port 8200
set firewall ipv4 name DATA_SECURITY rule 20 description 'Vault API'
set firewall ipv4 name DATA_SECURITY rule 30 action accept
set firewall ipv4 name DATA_SECURITY rule 30 protocol tcp
set firewall ipv4 name DATA_SECURITY rule 30 destination port 1812,1813
set firewall ipv4 name DATA_SECURITY rule 30 description 'RADIUS auth/acct'
# SERVICES → SECURITY (allow services to reach Vault/ISE)
set firewall ipv4 name SERVICES_SECURITY default-action drop
set firewall ipv4 name SERVICES_SECURITY rule 10 action accept
set firewall ipv4 name SERVICES_SECURITY rule 10 state established
set firewall ipv4 name SERVICES_SECURITY rule 10 state related
set firewall ipv4 name SERVICES_SECURITY rule 20 action accept
set firewall ipv4 name SERVICES_SECURITY rule 20 protocol tcp
set firewall ipv4 name SERVICES_SECURITY rule 20 destination port 8200
set firewall ipv4 name SERVICES_SECURITY rule 20 description 'Vault API'
# SECURITY → SERVICES (allow Vault/ISE to reach LDAP, DNS, etc.)
set firewall ipv4 name SECURITY_SERVICES default-action drop
set firewall ipv4 name SECURITY_SERVICES rule 10 action accept
set firewall ipv4 name SECURITY_SERVICES rule 10 state established
set firewall ipv4 name SECURITY_SERVICES rule 10 state related
set firewall ipv4 name SECURITY_SERVICES rule 20 action accept
set firewall ipv4 name SECURITY_SERVICES rule 20 protocol tcp
set firewall ipv4 name SECURITY_SERVICES rule 20 destination port 389,636
set firewall ipv4 name SECURITY_SERVICES rule 20 description 'LDAP/LDAPS'
# SERVICES zone rules
# MGMT → SERVICES (admin access)
set firewall ipv4 name MGMT_SERVICES default-action accept
# DATA → SERVICES (allow access to Keycloak, FreeIPA, etc.)
set firewall ipv4 name DATA_SERVICES default-action drop
set firewall ipv4 name DATA_SERVICES rule 10 action accept
set firewall ipv4 name DATA_SERVICES rule 10 state established
set firewall ipv4 name DATA_SERVICES rule 10 state related
set firewall ipv4 name DATA_SERVICES rule 20 action accept
set firewall ipv4 name DATA_SERVICES rule 20 protocol tcp
set firewall ipv4 name DATA_SERVICES rule 20 destination port 80,443
set firewall ipv4 name DATA_SERVICES rule 20 description 'Web services'
set firewall ipv4 name DATA_SERVICES rule 30 action accept
set firewall ipv4 name DATA_SERVICES rule 30 protocol tcp
set firewall ipv4 name DATA_SERVICES rule 30 destination port 389,636
set firewall ipv4 name DATA_SERVICES rule 30 description 'LDAP/LDAPS'
9.8. 3.7 LOCAL Zone Rules (Firewall Services)
# Allow all zones to reach firewall for DHCP, DNS, SSH
set firewall ipv4 name MGMT_LOCAL default-action drop
set firewall ipv4 name MGMT_LOCAL rule 10 action accept
set firewall ipv4 name MGMT_LOCAL rule 10 state established
set firewall ipv4 name MGMT_LOCAL rule 10 state related
set firewall ipv4 name MGMT_LOCAL rule 20 action accept
set firewall ipv4 name MGMT_LOCAL rule 20 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 20 destination port 22
set firewall ipv4 name MGMT_LOCAL rule 20 description 'SSH'
set firewall ipv4 name MGMT_LOCAL rule 30 action accept
set firewall ipv4 name MGMT_LOCAL rule 30 protocol udp
set firewall ipv4 name MGMT_LOCAL rule 30 destination port 67
set firewall ipv4 name MGMT_LOCAL rule 30 description 'DHCP'
set firewall ipv4 name MGMT_LOCAL rule 40 action accept
set firewall ipv4 name MGMT_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name MGMT_LOCAL rule 40 destination port 53
set firewall ipv4 name MGMT_LOCAL rule 40 description 'DNS'
set firewall ipv4 name MGMT_LOCAL rule 50 action accept
set firewall ipv4 name MGMT_LOCAL rule 50 protocol icmp
set firewall ipv4 name MGMT_LOCAL rule 50 description 'ICMP'
set firewall ipv4 name MGMT_LOCAL rule 60 action accept
set firewall ipv4 name MGMT_LOCAL rule 60 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 60 destination port 443
set firewall ipv4 name MGMT_LOCAL rule 60 description 'HTTPS API'
# BGP from k3s Cilium (for LoadBalancer IP advertisement)
set firewall ipv4 name MGMT_LOCAL rule 70 action accept
set firewall ipv4 name MGMT_LOCAL rule 70 source group address-group K3S_NODES
set firewall ipv4 name MGMT_LOCAL rule 70 protocol tcp
set firewall ipv4 name MGMT_LOCAL rule 70 destination group port-group BGP
set firewall ipv4 name MGMT_LOCAL rule 70 description 'BGP from Cilium'
# Copy similar rules for other zones (DATA, VOICE, GUEST, IOT)
# SSH restricted to ADMINS group only (prevents lateral movement)
set firewall ipv4 name DATA_LOCAL default-action drop
set firewall ipv4 name DATA_LOCAL rule 10 action accept
set firewall ipv4 name DATA_LOCAL rule 10 state established
set firewall ipv4 name DATA_LOCAL rule 10 state related
set firewall ipv4 name DATA_LOCAL rule 20 action accept
set firewall ipv4 name DATA_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name DATA_LOCAL rule 20 protocol tcp
set firewall ipv4 name DATA_LOCAL rule 20 destination port 22
set firewall ipv4 name DATA_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name DATA_LOCAL rule 25 action accept
set firewall ipv4 name DATA_LOCAL rule 25 source group address-group ADMINS
set firewall ipv4 name DATA_LOCAL rule 25 protocol tcp
set firewall ipv4 name DATA_LOCAL rule 25 destination port 443
set firewall ipv4 name DATA_LOCAL rule 25 description 'HTTPS API from ADMINS only'
set firewall ipv4 name DATA_LOCAL rule 30 action accept
set firewall ipv4 name DATA_LOCAL rule 30 protocol udp
set firewall ipv4 name DATA_LOCAL rule 30 destination port 67
set firewall ipv4 name DATA_LOCAL rule 40 action accept
set firewall ipv4 name DATA_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name DATA_LOCAL rule 40 destination port 53
set firewall ipv4 name DATA_LOCAL rule 50 action accept
set firewall ipv4 name DATA_LOCAL rule 50 protocol icmp
set firewall ipv4 name VOICE_LOCAL default-action drop
set firewall ipv4 name VOICE_LOCAL rule 10 action accept
set firewall ipv4 name VOICE_LOCAL rule 10 state established
set firewall ipv4 name VOICE_LOCAL rule 10 state related
set firewall ipv4 name VOICE_LOCAL rule 30 action accept
set firewall ipv4 name VOICE_LOCAL rule 30 protocol udp
set firewall ipv4 name VOICE_LOCAL rule 30 destination port 67
set firewall ipv4 name VOICE_LOCAL rule 40 action accept
set firewall ipv4 name VOICE_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name VOICE_LOCAL rule 40 destination port 53
set firewall ipv4 name GUEST_LOCAL default-action drop
set firewall ipv4 name GUEST_LOCAL rule 10 action accept
set firewall ipv4 name GUEST_LOCAL rule 10 state established
set firewall ipv4 name GUEST_LOCAL rule 10 state related
set firewall ipv4 name GUEST_LOCAL rule 30 action accept
set firewall ipv4 name GUEST_LOCAL rule 30 protocol udp
set firewall ipv4 name GUEST_LOCAL rule 30 destination port 67
set firewall ipv4 name GUEST_LOCAL rule 40 action accept
set firewall ipv4 name GUEST_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name GUEST_LOCAL rule 40 destination port 53
set firewall ipv4 name IOT_LOCAL default-action drop
set firewall ipv4 name IOT_LOCAL rule 10 action accept
set firewall ipv4 name IOT_LOCAL rule 10 state established
set firewall ipv4 name IOT_LOCAL rule 10 state related
set firewall ipv4 name IOT_LOCAL rule 30 action accept
set firewall ipv4 name IOT_LOCAL rule 30 protocol udp
set firewall ipv4 name IOT_LOCAL rule 30 destination port 67
set firewall ipv4 name IOT_LOCAL rule 40 action accept
set firewall ipv4 name IOT_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name IOT_LOCAL rule 40 destination port 53
# SECURITY_LOCAL (Vault, ISE access to firewall services)
set firewall ipv4 name SECURITY_LOCAL default-action drop
set firewall ipv4 name SECURITY_LOCAL rule 10 action accept
set firewall ipv4 name SECURITY_LOCAL rule 10 state established
set firewall ipv4 name SECURITY_LOCAL rule 10 state related
set firewall ipv4 name SECURITY_LOCAL rule 20 action accept
set firewall ipv4 name SECURITY_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name SECURITY_LOCAL rule 20 protocol tcp
set firewall ipv4 name SECURITY_LOCAL rule 20 destination port 22
set firewall ipv4 name SECURITY_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name SECURITY_LOCAL rule 30 action accept
set firewall ipv4 name SECURITY_LOCAL rule 30 protocol udp
set firewall ipv4 name SECURITY_LOCAL rule 30 destination port 67
set firewall ipv4 name SECURITY_LOCAL rule 40 action accept
set firewall ipv4 name SECURITY_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name SECURITY_LOCAL rule 40 destination port 53
set firewall ipv4 name SECURITY_LOCAL rule 50 action accept
set firewall ipv4 name SECURITY_LOCAL rule 50 protocol icmp
# SERVICES_LOCAL (FreeIPA, Keycloak, BIND access to firewall services)
set firewall ipv4 name SERVICES_LOCAL default-action drop
set firewall ipv4 name SERVICES_LOCAL rule 10 action accept
set firewall ipv4 name SERVICES_LOCAL rule 10 state established
set firewall ipv4 name SERVICES_LOCAL rule 10 state related
set firewall ipv4 name SERVICES_LOCAL rule 20 action accept
set firewall ipv4 name SERVICES_LOCAL rule 20 source group address-group ADMINS
set firewall ipv4 name SERVICES_LOCAL rule 20 protocol tcp
set firewall ipv4 name SERVICES_LOCAL rule 20 destination port 22
set firewall ipv4 name SERVICES_LOCAL rule 20 description 'SSH from ADMINS only'
set firewall ipv4 name SERVICES_LOCAL rule 30 action accept
set firewall ipv4 name SERVICES_LOCAL rule 30 protocol udp
set firewall ipv4 name SERVICES_LOCAL rule 30 destination port 67
set firewall ipv4 name SERVICES_LOCAL rule 40 action accept
set firewall ipv4 name SERVICES_LOCAL rule 40 protocol tcp_udp
set firewall ipv4 name SERVICES_LOCAL rule 40 destination port 53
set firewall ipv4 name SERVICES_LOCAL rule 50 action accept
set firewall ipv4 name SERVICES_LOCAL rule 50 protocol icmp
# Check that SSH (rule 20) has source ADMINS restriction
# MGMT_LOCAL rule 20 may NOT have ADMINS (allows Ansible from any MGMT host)
# DATA, SECURITY, SERVICES should have ADMINS restriction
for zone in DATA SECURITY SERVICES; do
echo -n "${zone}_LOCAL rule 20 (SSH): "
show firewall ipv4 name ${zone}_LOCAL rule 20 | grep -q "ADMINS" && echo "✓ ADMINS restricted" || echo "✗ UNRESTRICTED - FIX THIS"
done
# These zones should NOT have port 22 rules
for zone in VOICE GUEST IOT; do
echo -n "${zone}_LOCAL: "
show firewall ipv4 name ${zone}_LOCAL | grep -q "dport 22" && echo "✗ HAS SSH - FIX THIS" || echo "✓ No SSH (secure)"
done
# All *_LOCAL rule 40 (DNS) must have protocol tcp_udp
for zone in MGMT DATA VOICE GUEST IOT SECURITY SERVICES; do
echo -n "${zone}_LOCAL rule 40: "
show firewall ipv4 name ${zone}_LOCAL rule 40 | grep -q "tcp_udp" && echo "✓ tcp_udp" || echo "✗ MISSING"
done
# Rules using port-group must specify protocol
for rule in "DATA_MGMT 20" "DATA_MGMT 30" "VOICE_MGMT 20" "IOT_MGMT 20"; do
name=$(echo $rule | cut -d' ' -f1)
num=$(echo $rule | cut -d' ' -f2)
echo -n "$name rule $num: "
show firewall ipv4 name $name rule $num | grep -q "tcp_udp" && echo "✓ tcp_udp" || echo "✗ MISSING"
done
|
Security verification (9 zones):
If any *_LOCAL rule 20 is missing |
9.9. 3.8 Apply Zone Policies
# WAN zone
set firewall zone WAN from MGMT firewall name MGMT_WAN
set firewall zone WAN from DATA firewall name DATA_WAN
set firewall zone WAN from VOICE firewall name VOICE_WAN
set firewall zone WAN from GUEST firewall name GUEST_WAN
set firewall zone WAN from IOT firewall name IOT_WAN
set firewall zone WAN from LOCAL firewall name LOCAL_WAN
# MGMT zone
set firewall zone MGMT from WAN firewall name WAN_IN
set firewall zone MGMT from DATA firewall name DATA_MGMT
set firewall zone MGMT from VOICE firewall name VOICE_MGMT
set firewall zone MGMT from GUEST firewall name GUEST_MGMT
set firewall zone MGMT from IOT firewall name IOT_MGMT
set firewall zone MGMT from LOCAL firewall name LOCAL_MGMT
# DATA zone
set firewall zone DATA from WAN firewall name WAN_IN
set firewall zone DATA from MGMT firewall name MGMT_DATA
set firewall zone DATA from GUEST firewall name GUEST_DATA
set firewall zone DATA from IOT firewall name IOT_DATA
# VOICE zone
set firewall zone VOICE from WAN firewall name WAN_IN
set firewall zone VOICE from MGMT firewall name MGMT_VOICE
# GUEST zone
set firewall zone GUEST from WAN firewall name WAN_IN
set firewall zone GUEST from MGMT firewall name MGMT_GUEST
# IOT zone
set firewall zone IOT from WAN firewall name WAN_IN
set firewall zone IOT from MGMT firewall name MGMT_IOT
# SECURITY zone (crown jewels - Vault, ISE)
set firewall zone SECURITY from WAN firewall name WAN_IN
set firewall zone SECURITY from MGMT firewall name MGMT_SECURITY
set firewall zone SECURITY from DATA firewall name DATA_SECURITY
set firewall zone SECURITY from SERVICES firewall name SERVICES_SECURITY
# SERVICES zone (infrastructure VMs)
set firewall zone SERVICES from WAN firewall name WAN_IN
set firewall zone SERVICES from MGMT firewall name MGMT_SERVICES
set firewall zone SERVICES from DATA firewall name DATA_SERVICES
set firewall zone SERVICES from SECURITY firewall name SECURITY_SERVICES
# LOCAL zone (firewall itself)
set firewall zone LOCAL from WAN firewall name WAN_LOCAL
set firewall zone LOCAL from MGMT firewall name MGMT_LOCAL
set firewall zone LOCAL from DATA firewall name DATA_LOCAL
set firewall zone LOCAL from VOICE firewall name VOICE_LOCAL
set firewall zone LOCAL from GUEST firewall name GUEST_LOCAL
set firewall zone LOCAL from IOT firewall name IOT_LOCAL
set firewall zone LOCAL from SECURITY firewall name SECURITY_LOCAL
set firewall zone LOCAL from SERVICES firewall name SERVICES_LOCAL
9.11. 3.10 Post-Validation
run show firewall zone | awk 'NR>2 && NF>=2 {printf "%-12s → %s\n", $1, $2}'
DATA → eth0.10 GUEST → eth0.30 IOT → eth0.40 LOCAL → LOCAL MGMT → eth0 SECURITY → eth0.110 SERVICES → eth0.120 VOICE → eth0.20 WAN → eth1
# View zone policies (VyOS table format shows zone, interface, from-zone, firewall)
run show firewall zone | head -50
# Count rules per firewall ruleset
# VyOS format: ipv4 Firewall "name RULESET_NAME"
run show firewall | awk -F'"' '
/ipv4 Firewall "name/ {
split($2, a, " ")
name=a[2]
}
/^[0-9]+|^default/ && name != "" {
rules[name]++
}
END {
printf "%-25s %s\n", "FIREWALL_RULESET", "RULES"
printf "%-25s %s\n", "----------------", "-----"
for (n in rules) printf "%-25s %d\n", n, rules[n] | "sort"
}
'
sudo nft list ruleset | awk '/chain/ {print}' | head -20
# Export full config as JSON and extract zone info
run show configuration json | jq '.firewall.zone | keys'
# Show each zone's member interfaces
run show configuration json | jq -r '
.firewall.zone | to_entries[] |
"\(.key): \(.value.member.interface // "LOCAL")"
'
# Verify ADMINS source restriction on SSH for trusted zones
for z in DATA SECURITY SERVICES; do
show firewall ipv4 name ${z}_LOCAL | grep -q "ADMINS" && echo "✓ ${z}_LOCAL: ADMINS restricted" || echo "✗ ${z}_LOCAL: MISSING ADMINS - SECURITY HOLE"
done
# These zones should NOT have SSH (port 22) rules
for z in VOICE GUEST IOT; do
show firewall ipv4 name ${z}_LOCAL | grep -q "dport 22" && echo "✗ ${z}_LOCAL: HAS SSH - FIX IMMEDIATELY" || echo "✓ ${z}_LOCAL: No SSH (secure)"
done
# Combined security check - single command
{
echo "=== ZONE COUNT ==="
run show firewall zone | awk 'NR>2 && NF>=2' | wc -l | xargs -I{} echo "Zones: {} (expected: 9)"
echo ""
echo "=== ADMINS GROUP ==="
run show firewall group address-group ADMINS | awk '/address/ {print " " $2}'
echo ""
echo "=== SSH SECURITY ==="
for z in DATA SECURITY SERVICES; do
run show firewall ipv4 name ${z}_LOCAL | grep -q "ADMINS" && echo "✓ ${z}_LOCAL: ADMINS" || echo "✗ ${z}_LOCAL: OPEN"
done
for z in VOICE GUEST IOT; do
run show firewall ipv4 name ${z}_LOCAL | grep -q "port 22" && echo "✗ ${z}_LOCAL: SSH FOUND" || echo "✓ ${z}_LOCAL: No SSH"
done
}
# CRITICAL: Without LOCAL_WAN, router cannot reach internet
# Verify zone policy exists
run show firewall zone-policy | grep -A1 "WAN" | grep -q "LOCAL" && echo "✓ LOCAL_WAN zone policy exists" || echo "✗ MISSING: set firewall zone WAN from LOCAL firewall name LOCAL_WAN"
# Test actual connectivity
ping -c 2 8.8.8.8 >/dev/null 2>&1 && echo "✓ Router can reach internet" || echo "✗ Router cannot reach internet - check LOCAL_WAN"
# CRITICAL: Without LOCAL_MGMT, router cannot reach BIND, ISE, Vault, etc.
# Verify zone policy exists
run show firewall zone-policy | awk '/^MGMT/,/^[A-Z]/' | grep -q "LOCAL" && echo "✓ LOCAL_MGMT zone policy exists" || echo "✗ MISSING: set firewall zone MGMT from LOCAL firewall name LOCAL_MGMT"
# Test actual connectivity to BIND
ping -c 2 10.50.1.90 >/dev/null 2>&1 && echo "✓ Router can reach BIND (10.50.1.90)" || echo "✗ Router cannot reach BIND - check LOCAL_MGMT"
|
Phase 3 Complete Checklist (9 zones):
If any check fails, do NOT proceed. Fix firewall before continuing. |
10. Phase 4: NAT Configuration
10.1. 4.0 Pre-Validation
# PRE-1: Verify Phase 3 zones exist
show firewall zone | grep -E "WAN|MGMT|DATA"
# Expected: Zones listed
# PRE-2: Document current NAT state
show nat source rules
show nat destination rules
# Expected: Empty or minimal rules
10.2. 4.1 Source NAT (Masquerade) - Enterprise Per-VLAN
|
Interface mapping reminder:
Adjust |
configure
# =============================================================================
# ENTERPRISE NAT: Per-VLAN Source NAT Rules
# Each VLAN gets its own rule for:
# - Granular logging and accounting
# - Per-VLAN outbound policies (future: bandwidth limits, QoS)
# - Selective NAT (e.g., block IoT from internet)
# =============================================================================
# Rule 100: INFRA (VLAN 100) - Infrastructure management traffic
set nat source rule 100 outbound-interface name eth1
set nat source rule 100 source group network-group NET_INFRA
set nat source rule 100 translation address masquerade
set nat source rule 100 description 'SNAT INFRA → WAN'
# Rule 110: DATA (VLAN 10) - Corporate devices
set nat source rule 110 outbound-interface name eth1
set nat source rule 110 source group network-group NET_DATA
set nat source rule 110 translation address masquerade
set nat source rule 110 description 'SNAT DATA → WAN'
# Rule 120: VOICE (VLAN 20) - VoIP (may need different treatment for SIP ALG)
set nat source rule 120 outbound-interface name eth1
set nat source rule 120 source group network-group NET_VOICE
set nat source rule 120 translation address masquerade
set nat source rule 120 description 'SNAT VOICE → WAN'
# Rule 130: GUEST (VLAN 30) - Guest WiFi
set nat source rule 130 outbound-interface name eth1
set nat source rule 130 source group network-group NET_GUEST
set nat source rule 130 translation address masquerade
set nat source rule 130 description 'SNAT GUEST → WAN'
# Rule 140: IOT (VLAN 40) - IoT devices
# NOTE: Enable only if IoT needs internet. Can be disabled for LAN-only IoT.
set nat source rule 140 outbound-interface name eth1
set nat source rule 140 source group network-group NET_IOT
set nat source rule 140 translation address masquerade
set nat source rule 140 description 'SNAT IOT → WAN'
# set nat source rule 140 disable # Uncomment to block IoT internet
# Rule 150: SECURITY (VLAN 110) - Crown jewels (Vault, ISE)
# WARNING: Security tier typically should NOT need outbound internet
# Enable only for specific requirements (updates, CRL fetch, etc.)
set nat source rule 150 outbound-interface name eth1
set nat source rule 150 source group network-group NET_SECURITY
set nat source rule 150 translation address masquerade
set nat source rule 150 description 'SNAT SECURITY → WAN'
# set nat source rule 150 disable # Uncomment to block (recommended)
# Rule 160: SERVICES (VLAN 120) - Infrastructure VMs
set nat source rule 160 outbound-interface name eth1
set nat source rule 160 source group network-group NET_SERVICES
set nat source rule 160 translation address masquerade
set nat source rule 160 description 'SNAT SERVICES → WAN'
commit
save
# Verify all SNAT rules created
run show nat source rules | awk '/^[0-9]/{print "Rule "$1": "$NF}'
# Count rules per interface
run show nat source rules | awk '/outbound-interface/{iface[$NF]++} END{for(i in iface) print i": "iface[i]" rules"}'
# Show rules using network-groups (enterprise pattern)
run show nat source rules | awk '/network-group/{print}'
10.3. 4.2 Destination NAT (Port Forwards)
No DNAT required initially. Examples below for future reference.
# Example: Forward WAN:443 to internal web server
# NOTE: Use eth1 for vyos-02 (WAN interface)
# set nat destination rule 10 inbound-interface name eth1
# set nat destination rule 10 destination port 443
# set nat destination rule 10 protocol tcp
# set nat destination rule 10 translation address 10.50.1.x
# set nat destination rule 10 translation port 443
# set nat destination rule 10 description 'HTTPS to internal server'
# Example: Forward WAN:8443 to ISE Admin GUI
# set nat destination rule 20 inbound-interface name eth1
# set nat destination rule 20 destination port 8443
# set nat destination rule 20 protocol tcp
# set nat destination rule 20 translation address 10.50.1.20
# set nat destination rule 20 translation port 8443
# set nat destination rule 20 description 'ISE Admin GUI'
10.4. 4.10 Post-Validation
# POST-1: Count SNAT rules (should be 7: rules 100-160)
run show nat source rule | awk '/^[0-9]+/{count++} END{print "Total SNAT rules:", count}'
# Expected: Total SNAT rules: 7
# POST-2: Formatted table - rule → network-group mapping
run show nat source rule | awk '/^[0-9]+/{printf "Rule %s → %s\n", $1, $2}'
# Expected:
# Rule 100 → @N_NET_INFRA
# Rule 110 → @N_NET_DATA
# Rule 120 → @N_NET_VOICE
# Rule 130 → @N_NET_GUEST
# Rule 140 → @N_NET_IOT
# Rule 150 → @N_NET_SECURITY
# Rule 160 → @N_NET_SERVICES
# POST-3: Pivot by interface (all should be eth1 for vyos-02)
run show nat source rule | awk '/^[0-9]+/{iface[$5]++} END{for(i in iface) print i": "iface[i]" rules"}'
# Expected: eth1: 7 rules
# POST-4: One-liner status check
run show nat source rule | awk '/^[0-9]+/{r++; g[$2]++} END{print r" rules,", length(g)" network-groups"}'
# Expected: 7 rules, 7 network-groups
# POST-5: Test outbound connectivity (from VyOS)
ping -c 2 8.8.8.8
# Expected: Replies from Google DNS
# POST-6: Verify nftables NAT chain (low-level)
sudo nft list table ip vyos_nat 2>/dev/null | awk '/chain POSTROUTING/,/^}/' | head -15
# Expected: Multiple masquerade rules matching network-groups
# POST-7: NAT translations (active flows) - will populate when traffic exists
run show nat source translations | awk 'NR>1{proto[$1]++} END{for(p in proto) print p": "proto[p]" flows"}'
# Expected: Initially empty, shows tcp/udp counts when traffic flows
11. Phase 5: DHCP Server
11.1. 5.0 Pre-Validation
# PRE: Verify no existing DHCP config
show service dhcp-server
# Expected: Configuration under specified path is empty
configure
11.2. 5.1 DHCP for DATA VLAN
# DATA VLAN DHCP Pool
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 subnet-id 10
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 range 0 start 10.50.10.100
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 range 0 stop 10.50.10.199
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option default-router 10.50.10.1
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 lease 86400
# DHCP Reservations - Admin Workstations (ADMINS firewall group)
# Purpose: Static IPs for privileged access - prevents lateral movement
# NOTE: static-mapping MUST be inside subnet path (VyOS 1.4 syntax)
# CRITICAL: VyOS uses 'mac' NOT 'mac-address' (unlike some other DHCP servers)
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-razer mac 98:BB:1E:1F:A7:13
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-razer ip-address 10.50.10.106
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-aw mac 14:F6:D8:7B:31:80
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-aw ip-address 10.50.10.107
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-p50 mac C8:5B:76:C6:59:62
set service dhcp-server shared-network-name DATA subnet 10.50.10.0/24 static-mapping modestus-p50 ip-address 10.50.10.108
# List all static mappings with formatted output
run show configuration commands | grep static-mapping | awk -F'static-mapping ' '{print $2}' | awk '{printf "%-20s %s\n", $1, $2}'
# Count reservations (expected: 3 for ADMINS workstations)
run show configuration commands | grep -c static-mapping
# Verify specific MACs are mapped (case-insensitive)
run show configuration commands | grep -i "98:BB:1E:1F:A7:13"
run show configuration commands | grep -i "14:F6:D8:7B:31:80"
run show configuration commands | grep -i "C8:5B:76:C6:59:62"
# Verify DHCP server is configured for DATA network
run show dhcp server statistics
# Show current leases (will be empty until clients connect)
run show dhcp server leases
|
After cutover, workstations will get these static IPs via DHCP reservation. The ADMINS firewall group references these IPs - if reservations are wrong, SSH access breaks. Quick MAC verification from workstation:
|
11.3. 5.2 DHCP for VOICE VLAN
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 subnet-id 20
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 range 0 start 10.50.20.100
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 range 0 stop 10.50.20.199
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option default-router 10.50.20.1
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 lease 86400
# VoIP-specific options (TFTP for phone configs, Option 150 for Cisco phones)
# set service dhcp-server shared-network-name VOICE subnet 10.50.20.0/24 option tftp-server-name <tftp-server>
11.4. 5.3 DHCP for GUEST VLAN
# Guest uses public DNS (not internal) - short lease
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 subnet-id 30
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 range 0 start 10.50.30.100
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 range 0 stop 10.50.30.250
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option default-router 10.50.30.1
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option name-server 1.1.1.1
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 option name-server 8.8.8.8
set service dhcp-server shared-network-name GUEST subnet 10.50.30.0/24 lease 3600
| Guest VLAN uses public DNS (Cloudflare/Google), not internal DNS. Short lease (1 hour) for security. |
11.5. 5.4 DHCP for IOT VLAN
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 subnet-id 40
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 range 0 start 10.50.40.100
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 range 0 stop 10.50.40.250
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option default-router 10.50.40.1
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server 10.50.1.90
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server 10.50.1.91
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option domain-name inside.domusdigitalis.dev
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 lease 86400
12. Phase 6: DNS Architecture
12.1. 6.0 Pre-Validation
# PRE-1: Verify BIND servers are reachable from VyOS
ping -c 1 10.50.1.90
ping -c 1 10.50.1.91
# Expected: Both respond
# PRE-2: Verify BIND resolves internal zones
dig ise-01.inside.domusdigitalis.dev @10.50.1.90 +short
# Expected: 10.50.1.20
# PRE-3: Verify BIND resolves external domains
dig google.com @10.50.1.90 +short
# Expected: Google IPs returned
# PRE-4: Verify Phase 5 DHCP config references BIND
show configuration commands | grep name-server
# Expected: DHCP pools point to BIND servers
12.2. 6.1 Design Decision: BIND Handles All DNS
|
VyOS does NOT run DNS forwarding. BIND is the enterprise DNS infrastructure.
|
Why this is better:
-
Single DNS infrastructure (BIND) - no duplicate caching/forwarding
-
BIND already configured as recursive resolver (verified: resolves google.com)
-
Simpler VyOS config - fewer services to manage
-
Enterprise pattern - firewalls route, DNS servers resolve
12.3. 6.2 Verify BIND Handles Both Internal and External
# Test internal resolution
dig ise-01.inside.domusdigitalis.dev @10.50.1.90 +short
# Test external resolution
dig google.com @10.50.1.90 +short
# Test secondary BIND
dig google.com @10.50.1.91 +short
12.4. 6.3 DHCP Hands Out BIND Servers
DHCP configuration (Phase 5) uses BIND IPs as DNS servers:
set service dhcp-server shared-network-name DATA subnet ... name-server 10.50.1.90
set service dhcp-server shared-network-name DATA subnet ... name-server 10.50.1.91
Clients receive 10.50.1.90 and 10.50.1.91 as DNS - they query BIND directly.
12.5. 6.4 Post-Validation (from client)
# After DHCP lease renewal, verify DNS servers
resolvectl status | grep -A2 "DNS Servers"
# Test resolution works
dig vault-01.inside.domusdigitalis.dev +short
dig github.com +short
|
Phase 6 Complete. DNS architecture verified. BIND handles all resolution, VyOS just routes. |
13. Phase 7: Threat Intelligence (pfBlockerNG Replacement)
VyOS doesn’t have pfBlockerNG, but we can achieve similar functionality with firewall groups populated from threat feeds.
13.1. 7.0 Pre-Validation
# PRE-1: Verify Phase 6 complete (DNS working)
dig google.com +short
# Expected: Google IPs returned
# PRE-2: Verify internet connectivity (for feed downloads)
curl -sI https://www.spamhaus.org | head -1
# Expected: HTTP/2 200
# PRE-3: Verify firewall configuration exists (Phase 3)
show firewall ipv4
# Expected: WAN_IN and other firewall names listed
# PRE-4: Check scripts directory exists
ls -la /config/scripts/ 2>/dev/null || echo "Will be created"
13.2. 7.1 Create Threat Feed Script
SSH into VyOS and create update script:
sudo su -
cat > /config/scripts/update-threat-feeds.sh << 'EOF'
#!/bin/bash
# Update threat intelligence feeds for VyOS firewall
# Run via cron: 0 4 * * * /config/scripts/update-threat-feeds.sh
FEED_DIR="/config/threat-feeds"
mkdir -p $FEED_DIR
# Spamhaus DROP (Don't Route Or Peer)
curl -s https://www.spamhaus.org/drop/drop.txt | grep -v '^;' | awk '{print $1}' > $FEED_DIR/spamhaus-drop.txt
# Emerging Threats compromised IPs
curl -s https://rules.emergingthreats.net/blockrules/compromised-ips.txt | grep -v '^#' > $FEED_DIR/et-compromised.txt
# Abuse.ch Feodo Tracker (banking trojans C2)
curl -s https://feodotracker.abuse.ch/downloads/ipblocklist.txt | grep -v '^#' > $FEED_DIR/feodo.txt
# Combine all feeds
cat $FEED_DIR/*.txt | sort -u | grep -v '^$' > $FEED_DIR/combined-blocklist.txt
# Update VyOS firewall group
source /opt/vyatta/etc/functions/script-template
configure
# Clear existing entries
delete firewall group network-group THREAT_FEEDS
# Add networks from blocklist
while read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(/[0-9]+)?$ ]]; then
# Add /32 if no CIDR notation
[[ $ip =~ / ]] || ip="$ip/32"
run set firewall group network-group THREAT_FEEDS network "$ip"
fi
done < $FEED_DIR/combined-blocklist.txt
commit
save
exit
echo "Threat feeds updated: $(wc -l < $FEED_DIR/combined-blocklist.txt) entries"
EOF
chmod +x /config/scripts/update-threat-feeds.sh
13.3. 7.2 Add Threat Feed Firewall Rules
configure
# Block inbound from threat feeds
set firewall ipv4 name WAN_IN rule 5 action drop
set firewall ipv4 name WAN_IN rule 5 source group network-group THREAT_FEEDS
set firewall ipv4 name WAN_IN rule 5 log
set firewall ipv4 name WAN_IN rule 5 description 'Block threat feeds inbound'
# Block outbound to threat feeds (C2 prevention)
set firewall ipv4 name MGMT_WAN rule 5 action drop
set firewall ipv4 name MGMT_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name MGMT_WAN rule 5 log
set firewall ipv4 name MGMT_WAN rule 5 description 'Block threat feeds outbound'
set firewall ipv4 name DATA_WAN rule 5 action drop
set firewall ipv4 name DATA_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name DATA_WAN rule 5 log
set firewall ipv4 name IOT_WAN rule 5 action drop
set firewall ipv4 name IOT_WAN rule 5 destination group network-group THREAT_FEEDS
set firewall ipv4 name IOT_WAN rule 5 log
commit
save
13.4. 7.3 Schedule Threat Feed Updates
set system task-scheduler task update-threat-feeds executable path /config/scripts/update-threat-feeds.sh
set system task-scheduler task update-threat-feeds interval 1d
commit
save
13.6. 7.5 Post-Validation
# POST-1: Verify threat feed files downloaded
ls -la /config/threat-feeds/
# Expected: spamhaus-drop.txt, et-compromised.txt, feodo.txt, combined-blocklist.txt
# POST-2: Verify network-group populated
show firewall group network-group THREAT_FEEDS
# Expected: Multiple network entries (should be 1000+ IPs)
# POST-3: Verify firewall rules reference THREAT_FEEDS
show firewall ipv4 name WAN_IN rule 5
# Expected: source group network-group THREAT_FEEDS, action drop
# POST-4: Verify task scheduler configured
show system task-scheduler
# Expected: update-threat-feeds task with interval 1d
# POST-5: Count blocked IPs
wc -l /config/threat-feeds/combined-blocklist.txt
# Expected: 1000+ entries
14. Phase 8: Suricata IDS
VyOS supports Suricata as an IDS/IPS. This replaces Snort/Suricata packages on pfSense.
14.1. 8.0 Pre-Validation
# PRE-1: Verify Phase 7 complete (threat feeds working)
ls /config/threat-feeds/combined-blocklist.txt
# Expected: File exists
# PRE-2: Verify internet access for rule downloads
curl -sI https://rules.emergingthreats.net | head -1
# Expected: HTTP/2 200
# PRE-3: Check if Suricata already installed
which suricata || echo "Not installed (expected for fresh deploy)"
# PRE-4: Verify sufficient disk space
df -h /var/log/
# Expected: >5GB available for logs
14.3. 8.2 Configure Suricata
# Backup default config
cp /etc/suricata/suricata.yaml /etc/suricata/suricata.yaml.orig
# Edit config
cat > /etc/suricata/suricata.yaml << 'EOF'
%YAML 1.1
---
vars:
address-groups:
HOME_NET: "[10.50.0.0/16]"
EXTERNAL_NET: "!$HOME_NET"
HTTP_SERVERS: "$HOME_NET"
DNS_SERVERS: "[{bind-ip},{bind-02-ip}]"
port-groups:
HTTP_PORTS: "80,443,8080"
DNS_PORTS: "53"
default-log-dir: /var/log/suricata/
outputs:
- eve-log:
enabled: yes
filetype: regular
filename: eve.json
types:
- alert
- http
- dns
- tls
- files
- smtp
af-packet:
- interface: eth0
cluster-id: 99
cluster-type: cluster_flow
defrag: yes
- interface: eth1
cluster-id: 98
cluster-type: cluster_flow
defrag: yes
rule-files:
- suricata.rules
classification-file: /etc/suricata/classification.config
reference-config-file: /etc/suricata/reference.config
EOF
14.4. 8.3 Update Suricata Rules
suricata-update
suricata-update enable-source et/open
suricata-update
14.5. 8.4 Enable Suricata Service
systemctl enable suricata
systemctl start suricata
systemctl status suricata
14.6. 8.5 Integrate with Wazuh
Wazuh can ingest Suricata alerts. Add to Wazuh agent config:
cat >> /var/ossec/etc/ossec.conf << 'EOF'
<localfile>
<log_format>json</log_format>
<location>/var/log/suricata/eve.json</location>
</localfile>
EOF
systemctl restart wazuh-agent
14.7. 8.6 Post-Validation
# POST-1: Verify Suricata service running
systemctl status suricata | grep -E "Active:|loaded"
# Expected: active (running)
# POST-2: Verify Suricata listening on interfaces
suricata --build-info | grep -i "af-packet"
# Expected: af-packet support enabled
# POST-3: Verify rules loaded
suricata-update list-sources | grep -E "enabled|et/open"
# Expected: et/open source enabled
# POST-4: Verify eve.json logging
ls -la /var/log/suricata/eve.json
# Expected: File exists and growing
# POST-5: Test Suricata detection (safe EICAR-like test)
tail -f /var/log/suricata/eve.json &
curl -s http://testmyids.com
# Ctrl+C to stop tail
# Expected: Alert logged for test signature
# POST-6: Verify Wazuh integration
grep -i suricata /var/ossec/etc/ossec.conf
# Expected: localfile entry for eve.json
15. Phase 9: Monitoring Integration
15.1. 9.0 Pre-Validation
# PRE-1: Verify Phase 8 complete (Suricata running)
systemctl status suricata | grep "Active:"
# Expected: active (running)
# PRE-2: Verify connectivity to monitoring infrastructure
ping -c 1 10.50.1.134
# Expected: Reply from Wazuh manager
# PRE-3: Verify Prometheus endpoint reachable (from k3s cluster)
# Run this FROM k3s-master-01 after Phase 9 complete
# curl -s http://10.50.1.2:9100/metrics | head -5
echo "Will verify after node_exporter installed"
# PRE-4: Verify apt repository access
apt-cache policy prometheus-node-exporter
# Expected: Package available
15.2. 9.1 Install Prometheus Node Exporter
sudo su -
apt update
apt install -y prometheus-node-exporter
systemctl enable prometheus-node-exporter
systemctl start prometheus-node-exporter
# Verify
curl -s localhost:9100/metrics | head -5
15.3. 9.2 Install Wazuh Agent
# Add Wazuh repository
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --dearmor -o /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" > /etc/apt/sources.list.d/wazuh.list
apt update
apt install -y wazuh-agent
# Configure manager address
sed -i "s/MANAGER_IP/{wazuh-manager-vip}/g" /var/ossec/etc/ossec.conf
systemctl daemon-reload
systemctl enable wazuh-agent
systemctl start wazuh-agent
15.4. 9.3 Enable SNMP (Optional)
configure
set service snmp community public authorization ro
set service snmp listen-address 10.50.1.2
commit
save
15.5. 9.4 Post-Validation
# POST-1: Verify node_exporter running
systemctl status prometheus-node-exporter | grep "Active:"
# Expected: active (running)
# POST-2: Verify node_exporter metrics available
curl -s http://localhost:9100/metrics | grep -E "^node_cpu|^node_memory" | head -5
# Expected: CPU and memory metrics
# POST-3: Verify Wazuh agent registered
sudo /var/ossec/bin/agent_control -l | head -5
# Expected: Agent listed (may take a minute)
# POST-4: Verify Wazuh agent connected
systemctl status wazuh-agent | grep "Active:"
# Expected: active (running)
# POST-5: Verify SNMP responding (if enabled)
snmpwalk -v2c -c public 10.50.1.2 system 2>/dev/null | head -3 || echo "SNMP not configured (optional)"
# Expected: System info if SNMP enabled
# POST-6: Verify from Prometheus (run from k3s-master-01)
# curl -s http://10.50.1.2:9100/metrics | wc -l
# Expected: Many lines of metrics
16. Phase 10: SSH Hardening and Access
16.1. 10.0 Pre-Validation
# PRE-1: Verify Phase 9 complete (monitoring working)
systemctl status prometheus-node-exporter | grep "Active:"
# Expected: active (running)
# PRE-2: Check current SSH config
show service ssh
# Expected: Default SSH config or empty
# PRE-3: Verify we have SSH key to add
echo "Verify you have user's public SSH key ready"
# Example: ssh-ed25519 AAAAC3... user@host
16.2. 10.1 Configure SSH
configure
# SSH service
set service ssh port 22
set service ssh listen-address 10.50.1.2
set service ssh disable-password-authentication
# Create admin user with SSH key
set system login user evan full-name 'Evan Rosado'
set system login user evan authentication public-keys modestus-razer type ssh-ed25519
set system login user evan authentication public-keys modestus-razer key 'AAAAC3NzaC1lZDI1NTE5AAAAILqgbqJQwk7SikO3mJPVX8/83ULOXLi6iB6G+tM0i9f7'
commit
save
16.4. 10.3 Post-Validation
# POST-1: Verify SSH service config
show service ssh
# Expected: port 22, password auth disabled
# POST-2: Verify SSH listening
sudo ss -tlnp | grep ":22"
# Expected: Listening on 10.50.1.2:22
# POST-3: Verify user created
show system login user
# Expected: evan user listed
# POST-4: Test SSH from workstation (run from workstation)
ssh -o BatchMode=yes evan@10.50.1.2 "whoami"
# Expected: evan (key-based auth succeeds)
# POST-5: Verify password auth disabled (should fail)
ssh -o PreferredAuthentications=password evan@10.50.1.2 "whoami" 2>&1 | grep -i "permission denied"
# Expected: Permission denied (password auth disabled)
17. Phase 11: API Access
17.1. 11.0 Pre-Validation
# PRE-1: Verify Phase 10 complete (SSH working)
show service ssh
# Expected: SSH configured with key auth
# PRE-2: Check current HTTPS/API config
show service https
# Expected: Empty or minimal config
# PRE-3: Generate API key (run in operational mode)
run generate system api-key
# Expected: Generates a new API key - SAVE THIS!
17.2. 11.1 Enable HTTP API
configure
# Generate API key (run in operational mode first)
# run generate system api-key
set service https api keys id automation key 'YOUR_API_KEY_HERE'
set service https listen-address 10.50.1.2
set service https port 443
commit
save
17.3. 11.2 Test API
curl -k -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"op": "show", "path": ["interfaces"]}' \
https://10.50.1.2/retrieve
17.4. 11.3 Post-Validation
# POST-1: Verify HTTPS service config
show service https
# Expected: API enabled on port 443
# POST-2: Verify HTTPS listening
sudo ss -tlnp | grep ":443"
# Expected: Listening on 10.50.1.2:443
# POST-3: Test API endpoint (from workstation)
curl -sk https://10.50.1.2/ | head -5
# Expected: VyOS API response or redirect
# POST-4: Test API authentication (replace YOUR_API_KEY)
curl -sk -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"op": "show", "path": ["version"]}' \
https://10.50.1.2/retrieve | jq .
# Expected: VyOS version info in JSON
# POST-5: Verify API key stored
show configuration commands | grep "api keys"
# Expected: API key ID listed (not the key itself)
18. Phase 12: Git Config Tracking
18.1. 12.0 Pre-Validation
# PRE-1: Verify Phase 11 complete (API working)
show service https
# Expected: HTTPS/API configured
# PRE-2: Check if git already initialized
ls -la /config/.git 2>/dev/null || echo "Git not initialized (expected)"
# PRE-3: Verify config.boot exists
ls -la /config/config.boot
# Expected: File exists with current config
18.2. 12.1 Initialize Git Repository
sudo su -
cd /config
git init
git config user.email "vyos@inside.domusdigitalis.dev"
git config user.name "VyOS Config"
git add config.boot
git commit -m "Initial VyOS configuration"
18.3. 12.2 Post-Commit Hook (Optional)
cat > /config/.git/hooks/post-commit << 'EOF'
#!/bin/bash
# Push config to central git repo after each commit
# git push origin main
EOF
chmod +x /config/.git/hooks/post-commit
18.4. 12.3 Post-Validation
# POST-1: Verify git repo initialized
ls -la /config/.git/
# Expected: .git directory with objects, refs, etc.
# POST-2: Verify initial commit exists
cd /config && git log --oneline -1
# Expected: "Initial VyOS configuration" commit
# POST-3: Verify config.boot tracked
cd /config && git status
# Expected: clean working tree, config.boot tracked
# POST-4: Test change tracking (make minor config change)
configure
set system host-name vyos-01-test
commit
save
exit
cd /config && git status
# Expected: config.boot modified
# POST-5: Commit change and verify
cd /config && git add config.boot && git commit -m "Test: hostname change"
git log --oneline -2
# Expected: Two commits shown
# POST-6: Rollback test change
configure
set system host-name vyos-01
commit
save
exit
cd /config && git add config.boot && git commit -m "Revert: hostname back to vyos-01"
19. Phase 13: Pre-Cutover Testing
|
Complete ALL tests before scheduling cutover maintenance window. |
19.1. 13.1 Connectivity Tests (from vyos-02)
# WAN connectivity
ping -c 3 8.8.8.8
ping -c 3 google.com
# DNS resolution
nslookup google.com 10.50.1.90
nslookup inside.domusdigitalis.dev 10.50.1.90
# Internal services
ping -c 3 10.50.1.70 # NAS
ping -c 3 10.50.1.120 # k3s
ping -c 3 10.50.1.90 # bind-01
19.2. 13.2 DHCP Tests
# Check DHCP server status
show service dhcp-server
# Check leases (will be empty until cutover)
show dhcp server leases
20. Phase 14: Cutover from pfSense
|
This causes network disruption. Ensure:
|
20.1. 14.1 Pre-Cutover Snapshot
# Snapshot pfSense
ssh kvm-01 "sudo virsh snapshot-create-as pfsense pre-vyos-cutover"
# Snapshot vyos-02
ssh kvm-02 "sudo virsh snapshot-create-as vyos-02 pre-cutover"
20.2. 14.2 Cutover Procedure
# 1. Shut down pfSense
ssh kvm-01 "sudo virsh shutdown pfsense"
# 2. Wait for clean shutdown (30 seconds)
sleep 30
ssh kvm-01 "sudo virsh list --all | grep pfsense"
# 3. Update vyos-02 to take over gateway IPs
ssh vyos-02
configure
# Change MGMT IP to gateway
delete interfaces ethernet eth1 address {vyos-01-ip}/24
set interfaces ethernet eth1 address {vyos-vip}/24
commit
save
exit
# 4. Verify from workstation
ping {vyos-vip}
ping 8.8.8.8
20.3. 14.3 Post-Cutover Validation
# From each VLAN, test:
# - Gateway ping
# - Internet access
# - DNS resolution
# - Access to K8s services (Wazuh dashboard, etc.)
# Check DHCP leases are being issued
ssh vyos-02 "show dhcp server leases"
# Check firewall logs
ssh vyos-02 "show log | match firewall"
21. Phase 15: Deploy vyos-01 and VRRP HA
After 48-72 hours of stable vyos-02 operation, deploy vyos-01 for VRRP HA.
21.1. 15.0 Pre-Validation
# PRE-1: Verify vyos-02 stable for 48-72 hours
ssh vyos-02 "show system uptime"
# Expected: >48 hours uptime
# PRE-2: Verify vyos-02 handling all traffic
ssh vyos-02 "show nat translations | wc -l"
# Expected: Active NAT translations (traffic flowing)
# PRE-3: Verify kvm-01 ready for vyos-01 deployment
ssh kvm-01 "sudo virsh list --all | grep vyos"
# Expected: No vyos VMs yet (vyos-02 is on kvm-02)
# PRE-4: Verify VIP is not in use
ping -c 1 10.50.1.1
# Expected: No response (VIP will be created by VRRP)
21.2. 15.1 Deploy vyos-01 on kvm-01
Repeat Phase 1-11 for vyos-01 with these differences:
-
Hostname:
vyos-01 -
MGMT IP:
10.50.1.3/24(real IP, not VIP)
21.3. 15.2 Firewall Rules for VRRP
|
VRRP uses protocol 112 (not a port number). Both INBOUND and OUTBOUND rules are required.
|
Add VRRP inbound rules to all *_LOCAL firewalls:
configure
# DATA_LOCAL, GUEST_LOCAL, IOT_LOCAL, MGMT_LOCAL, SECURITY_LOCAL, SERVICES_LOCAL, VOICE_LOCAL
# Add rule 15 to each (after rule 10 established/related)
set firewall ipv4 name DATA_LOCAL rule 15 action 'accept'
set firewall ipv4 name DATA_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name DATA_LOCAL rule 15 protocol '112'
set firewall ipv4 name GUEST_LOCAL rule 15 action 'accept'
set firewall ipv4 name GUEST_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name GUEST_LOCAL rule 15 protocol '112'
set firewall ipv4 name IOT_LOCAL rule 15 action 'accept'
set firewall ipv4 name IOT_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name IOT_LOCAL rule 15 protocol '112'
set firewall ipv4 name MGMT_LOCAL rule 15 action 'accept'
set firewall ipv4 name MGMT_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name MGMT_LOCAL rule 15 protocol '112'
set firewall ipv4 name SECURITY_LOCAL rule 15 action 'accept'
set firewall ipv4 name SECURITY_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name SECURITY_LOCAL rule 15 protocol '112'
set firewall ipv4 name SERVICES_LOCAL rule 15 action 'accept'
set firewall ipv4 name SERVICES_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name SERVICES_LOCAL rule 15 protocol '112'
set firewall ipv4 name VOICE_LOCAL rule 15 action 'accept'
set firewall ipv4 name VOICE_LOCAL rule 15 description 'VRRP for HA'
set firewall ipv4 name VOICE_LOCAL rule 15 protocol '112'
commit
save
Add VRRP outbound rules to LOCAL_* firewalls:
configure
# LOCAL_IOT already exists - add rule 5
set firewall ipv4 name LOCAL_IOT rule 5 action 'accept'
set firewall ipv4 name LOCAL_IOT rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_IOT rule 5 protocol '112'
# Create LOCAL_GUEST and LOCAL_VOICE with VRRP rule
set firewall ipv4 name LOCAL_GUEST default-action 'accept'
set firewall ipv4 name LOCAL_GUEST description 'Router outbound to GUEST'
set firewall ipv4 name LOCAL_GUEST rule 5 action 'accept'
set firewall ipv4 name LOCAL_GUEST rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_GUEST rule 5 protocol '112'
set firewall ipv4 name LOCAL_VOICE default-action 'accept'
set firewall ipv4 name LOCAL_VOICE description 'Router outbound to VOICE'
set firewall ipv4 name LOCAL_VOICE rule 5 action 'accept'
set firewall ipv4 name LOCAL_VOICE rule 5 description 'VRRP for HA'
set firewall ipv4 name LOCAL_VOICE rule 5 protocol '112'
commit
save
Add zone "from LOCAL" assignments:
configure
# These zones need outbound firewall from LOCAL zone
set firewall zone GUEST from LOCAL firewall name 'LOCAL_GUEST'
set firewall zone IOT from LOCAL firewall name 'LOCAL_IOT'
set firewall zone VOICE from LOCAL firewall name 'LOCAL_VOICE'
commit
save
21.4. 15.3 Configure VRRP on Both Nodes
|
Interface mapping differs between nodes:
VIP addresses are .1 (gateway IPs). Real interface IPs:
VyOS 2026.03 syntax: Use VRIDs: 10=MGMT, 11=DATA, 12=VOICE, 13=GUEST, 14=IOT, 15=SECURITY, 16=SERVICES |
On vyos-01 (MASTER - priority 200, uses eth1 for LAN):
configure
# VRRP for MGMT interface - VIP is .1, interface IP is .2
set high-availability vrrp group MGMT interface 'eth1'
set high-availability vrrp group MGMT priority '200'
set high-availability vrrp group MGMT address '10.50.1.1/24'
set high-availability vrrp group MGMT vrid '10'
set high-availability vrrp group MGMT preempt-delay '30'
# VRRP for DATA
set high-availability vrrp group DATA interface 'eth1.10'
set high-availability vrrp group DATA priority '200'
set high-availability vrrp group DATA address '10.50.10.1/24'
set high-availability vrrp group DATA vrid '11'
set high-availability vrrp group DATA preempt-delay '30'
# VRRP for VOICE
set high-availability vrrp group VOICE interface 'eth1.20'
set high-availability vrrp group VOICE priority '200'
set high-availability vrrp group VOICE address '10.50.20.1/24'
set high-availability vrrp group VOICE vrid '12'
set high-availability vrrp group VOICE preempt-delay '30'
# VRRP for GUEST
set high-availability vrrp group GUEST interface 'eth1.30'
set high-availability vrrp group GUEST priority '200'
set high-availability vrrp group GUEST address '10.50.30.1/24'
set high-availability vrrp group GUEST vrid '13'
set high-availability vrrp group GUEST preempt-delay '30'
# VRRP for IOT
set high-availability vrrp group IOT interface 'eth1.40'
set high-availability vrrp group IOT priority '200'
set high-availability vrrp group IOT address '10.50.40.1/24'
set high-availability vrrp group IOT vrid '14'
set high-availability vrrp group IOT preempt-delay '30'
# VRRP for SECURITY
set high-availability vrrp group SECURITY interface 'eth1.110'
set high-availability vrrp group SECURITY priority '200'
set high-availability vrrp group SECURITY address '10.50.110.1/24'
set high-availability vrrp group SECURITY vrid '15'
set high-availability vrrp group SECURITY preempt-delay '30'
# VRRP for SERVICES
set high-availability vrrp group SERVICES interface 'eth1.120'
set high-availability vrrp group SERVICES priority '200'
set high-availability vrrp group SERVICES address '10.50.120.1/24'
set high-availability vrrp group SERVICES vrid '16'
set high-availability vrrp group SERVICES preempt-delay '30'
commit
save
On vyos-02 (BACKUP - priority 100, uses eth0 for LAN):
configure
# VRRP for MGMT interface - VIP is .1, interface IP is .3
set high-availability vrrp group MGMT interface 'eth0'
set high-availability vrrp group MGMT priority '100'
set high-availability vrrp group MGMT address '10.50.1.1/24'
set high-availability vrrp group MGMT vrid '10'
set high-availability vrrp group MGMT preempt-delay '30'
# VRRP for DATA
set high-availability vrrp group DATA interface 'eth0.10'
set high-availability vrrp group DATA priority '100'
set high-availability vrrp group DATA address '10.50.10.1/24'
set high-availability vrrp group DATA vrid '11'
set high-availability vrrp group DATA preempt-delay '30'
# VRRP for VOICE
set high-availability vrrp group VOICE interface 'eth0.20'
set high-availability vrrp group VOICE priority '100'
set high-availability vrrp group VOICE address '10.50.20.1/24'
set high-availability vrrp group VOICE vrid '12'
set high-availability vrrp group VOICE preempt-delay '30'
# VRRP for GUEST
set high-availability vrrp group GUEST interface 'eth0.30'
set high-availability vrrp group GUEST priority '100'
set high-availability vrrp group GUEST address '10.50.30.1/24'
set high-availability vrrp group GUEST vrid '13'
set high-availability vrrp group GUEST preempt-delay '30'
# VRRP for IOT
set high-availability vrrp group IOT interface 'eth0.40'
set high-availability vrrp group IOT priority '100'
set high-availability vrrp group IOT address '10.50.40.1/24'
set high-availability vrrp group IOT vrid '14'
set high-availability vrrp group IOT preempt-delay '30'
# VRRP for SECURITY
set high-availability vrrp group SECURITY interface 'eth0.110'
set high-availability vrrp group SECURITY priority '100'
set high-availability vrrp group SECURITY address '10.50.110.1/24'
set high-availability vrrp group SECURITY vrid '15'
set high-availability vrrp group SECURITY preempt-delay '30'
# VRRP for SERVICES
set high-availability vrrp group SERVICES interface 'eth0.120'
set high-availability vrrp group SERVICES priority '100'
set high-availability vrrp group SERVICES address '10.50.120.1/24'
set high-availability vrrp group SERVICES vrid '16'
set high-availability vrrp group SERVICES preempt-delay '30'
commit
save
21.5. 15.4 Create WAN Health-Check Script (Both Nodes)
|
KVM virtual NICs do NOT detect physical link state. Even when the WAN cable is unplugged,
the interface shows Solution: Use a health-check script that pings the upstream gateway. When pings fail, the sync-group transitions all VRRP groups to FAULT state. |
On BOTH vyos-01 and vyos-02:
sudo tee /config/scripts/check-wan.sh << 'EOF'
#!/bin/bash
# Health check for VRRP - ping upstream gateway
# Exit 0 = healthy, Exit 1 = failed
ping -c 2 -W 2 192.168.1.254 > /dev/null 2>&1
EOF
sudo chmod +x /config/scripts/check-wan.sh
Test the script:
/config/scripts/check-wan.sh && echo "WAN OK" || echo "WAN FAILED"
21.6. 15.5 Configure Sync-Group with Health-Check (Both Nodes)
|
Sync-group ensures ALL VRRP groups failover together. Without it, individual groups could be in different states (split-brain). Health-check MUST be on the sync-group, NOT on individual groups. VyOS rejects commits if health-check is on a group that’s also in a sync-group. |
On BOTH vyos-01 and vyos-02:
configure
# Create sync-group - all VRRP groups fail together
set high-availability vrrp sync-group GATEWAY member MGMT
set high-availability vrrp sync-group GATEWAY member DATA
set high-availability vrrp sync-group GATEWAY member VOICE
set high-availability vrrp sync-group GATEWAY member GUEST
set high-availability vrrp sync-group GATEWAY member IOT
set high-availability vrrp sync-group GATEWAY member SECURITY
set high-availability vrrp sync-group GATEWAY member SERVICES
# Health-check on sync-group (NOT individual groups)
set high-availability vrrp sync-group GATEWAY health-check script /config/scripts/check-wan.sh
set high-availability vrrp sync-group GATEWAY health-check interval 5
set high-availability vrrp sync-group GATEWAY health-check failure-count 2
commit
save
Verify sync-group:
show configuration commands | grep sync-group
21.7. 15.6 Configure Conntrack Sync (Stateful Failover)
# On both vyos-01 and vyos-02
configure
set high-availability conntrack-sync accept-protocol tcp,udp,icmp
set high-availability conntrack-sync interface eth1
set high-availability conntrack-sync failover-mechanism vrrp sync-group GATEWAY
set high-availability conntrack-sync mcast-group 225.0.0.50
commit
save
21.8. 15.4 Verify VRRP
# On vyos-01
show vrrp
# Expected: MASTER for all groups
# On vyos-02
show vrrp
# Expected: BACKUP for all groups
21.9. 15.7 Test Failover
|
Test by pulling WAN cable, NOT by shutting down VM. Pulling the WAN cable simulates real-world ISP failure. The health-check script will detect the upstream gateway is unreachable and trigger failover. Expected behavior: - Health-check fails after 2 consecutive failures (10 seconds) - All VRRP groups transition to FAULT (sync-group) - vyos-02 becomes MASTER - 1 ping dropped during failover (acceptable) - Reconnect WAN → vyos-01 preempts back after 30 seconds |
# Start continuous ping from phone/laptop on WiFi or wired
ping 8.8.8.8
# On vyos-01 - pull WAN cable (or disable interface in emergency)
# configure
# set interfaces ethernet eth0 disable
# commit
# Watch VRRP state on vyos-01 - should go to FAULT
show vrrp
# Expected: ALL groups show FAULT
# On vyos-02 - verify it became MASTER
show vrrp
# Expected: ALL groups show MASTER
# Reconnect WAN cable (or re-enable interface)
# configure
# delete interfaces ethernet eth0 disable
# commit
# Wait 30 seconds (preempt-delay)
# vyos-01 should preempt back to MASTER
show vrrp
# Expected: ALL groups show MASTER on vyos-01
21.10. 15.8 Post-Validation
# POST-1: Verify VRRP status on vyos-01
ssh vyos-01 "show vrrp"
# Expected: MASTER for all 7 groups (MGMT, DATA, VOICE, GUEST, IOT, SECURITY, SERVICES)
# POST-2: Verify VRRP status on vyos-02
ssh vyos-02 "show vrrp"
# Expected: BACKUP for all 7 groups
# POST-3: Verify VIP responds
ping -c 3 10.50.1.1
# Expected: Replies from VIP (10.50.1.1)
# POST-4: Verify sync-group configured on both nodes
ssh vyos-01 "show configuration commands | grep sync-group"
ssh vyos-02 "show configuration commands | grep sync-group"
# Expected: GATEWAY sync group with all 7 members and health-check script
# POST-5: Verify health-check script exists on both nodes
ssh vyos-01 "ls -la /config/scripts/check-wan.sh"
ssh vyos-02 "ls -la /config/scripts/check-wan.sh"
# Expected: Executable script on both nodes
# POST-6: Verify conntrack sync (if configured)
ssh vyos-01 "show conntrack-sync status"
ssh vyos-02 "show conntrack-sync status"
# Expected: Both show sync active
# POST-7: Verify preempt-delay configured
ssh vyos-01 "show configuration commands | grep preempt-delay"
# Expected: preempt-delay 30 on all groups
# POST-8: Verify both nodes healthy after failover test
ssh vyos-01 "show vrrp"
ssh vyos-02 "show vrrp"
# Expected: vyos-01 MASTER, vyos-02 BACKUP for all groups
22. Phase 16: Multi-Node k3s Cluster Support
22.1. 16.0 Pre-Validation
# PRE-1: Verify Phase 15 complete (VRRP working)
ssh vyos-01 "show vrrp"
ssh vyos-02 "show vrrp"
# Expected: vyos-01 MASTER, vyos-02 BACKUP
# PRE-2: Verify k3s cluster exists and healthy
kubectl get nodes
# Expected: k3s-master-01 Ready (single node initially)
# PRE-3: Verify Cilium running
cilium status
# Expected: Cilium OK, operator Ready
# PRE-4: Check current firewall groups (should not exist yet)
show firewall group address-group
# Expected: May be empty or missing K3S_NODES group
|
The k3s cluster is a 6-node HA deployment across kvm-01 and kvm-02:
All nodes communicate via the MGMT VLAN (10.50.1.0/24). These firewall rules enable node-to-node traffic. |
22.2. 16.1 k3s Inter-Node Communication Rules
Create firewall rules to allow cluster traffic between k3s nodes:
configure
# VXLAN overlay (Cilium tunnel mode) - UDP 8472 between all nodes
set firewall ipv4 name MGMT_MGMT default-action drop
set firewall ipv4 name MGMT_MGMT rule 10 action accept
set firewall ipv4 name MGMT_MGMT rule 10 state established
set firewall ipv4 name MGMT_MGMT rule 10 state related
set firewall ipv4 name MGMT_MGMT rule 80 action accept
set firewall ipv4 name MGMT_MGMT rule 80 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 80 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 80 protocol udp
set firewall ipv4 name MGMT_MGMT rule 80 destination group port-group K3S_VXLAN
set firewall ipv4 name MGMT_MGMT rule 80 description 'k3s Cilium VXLAN overlay'
# etcd cluster (control plane only) - TCP 2379-2380
set firewall ipv4 name MGMT_MGMT rule 81 action accept
set firewall ipv4 name MGMT_MGMT rule 81 source group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 81 destination group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 81 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 81 destination group port-group K3S_ETCD
set firewall ipv4 name MGMT_MGMT rule 81 description 'k3s etcd cluster'
# Kubelet API - TCP 10250 between all nodes
set firewall ipv4 name MGMT_MGMT rule 82 action accept
set firewall ipv4 name MGMT_MGMT rule 82 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 82 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 82 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 82 destination group port-group K3S_KUBELET
set firewall ipv4 name MGMT_MGMT rule 82 description 'k3s kubelet API'
# Cilium health checks - TCP 4240
set firewall ipv4 name MGMT_MGMT rule 83 action accept
set firewall ipv4 name MGMT_MGMT rule 83 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 83 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 83 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 83 destination group port-group CILIUM_HEALTH
set firewall ipv4 name MGMT_MGMT rule 83 description 'Cilium health checks'
# Hubble Relay - TCP 4244 (optional - for observability)
set firewall ipv4 name MGMT_MGMT rule 84 action accept
set firewall ipv4 name MGMT_MGMT rule 84 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 84 destination group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 84 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 84 destination group port-group CILIUM_HUBBLE
set firewall ipv4 name MGMT_MGMT rule 84 description 'Cilium Hubble Relay'
# k3s API server - TCP 6443 (all nodes need to reach masters)
set firewall ipv4 name MGMT_MGMT rule 85 action accept
set firewall ipv4 name MGMT_MGMT rule 85 source group address-group K3S_NODES
set firewall ipv4 name MGMT_MGMT rule 85 destination group address-group K3S_MASTERS
set firewall ipv4 name MGMT_MGMT rule 85 protocol tcp
set firewall ipv4 name MGMT_MGMT rule 85 destination group port-group K3S_API
set firewall ipv4 name MGMT_MGMT rule 85 description 'k3s API server'
# Apply zone policy for intra-MGMT traffic
set firewall zone MGMT from MGMT firewall name MGMT_MGMT
commit
save
22.3. 16.2 Verify k3s Cluster Connectivity
After applying rules, test from any k3s node:
# From k3s-master-01, ping other nodes
ping -c 1 10.50.1.121 # master-02
ping -c 1 10.50.1.122 # master-03
ping -c 1 10.50.1.123 # worker-01
ping -c 1 10.50.1.124 # worker-02
ping -c 1 10.50.1.125 # worker-03
# Verify Cilium health
cilium status
# Check etcd cluster health
kubectl -n kube-system exec -it etcd-k3s-master-01 -- etcdctl member list
22.4. 16.3 Post-Validation
# POST-1: Verify address groups created
show firewall group address-group K3S_NODES
show firewall group address-group K3S_MASTERS
# Expected: IPs for all k3s nodes listed
# POST-2: Verify port groups created
show firewall group port-group K3S_VXLAN
show firewall group port-group K3S_ETCD
show firewall group port-group K3S_KUBELET
# Expected: Ports listed (8472, 2379-2380, 10250, etc.)
# POST-3: Verify MGMT_MGMT firewall rules
show firewall ipv4 name MGMT_MGMT rule 80
show firewall ipv4 name MGMT_MGMT rule 81
show firewall ipv4 name MGMT_MGMT rule 82
# Expected: k3s cluster rules with correct groups
# POST-4: Verify zone policy applied
show zone-policy zone MGMT
# Expected: from-zone MGMT firewall name MGMT_MGMT
# POST-5: Test VXLAN port connectivity between k3s nodes
# From k3s-master-01:
nc -zvu 10.50.1.121 8472
# Expected: Connection succeeded (or open in UDP)
# POST-6: Verify Cilium connectivity
cilium connectivity test --single-node
# Expected: All tests pass
# POST-7: Verify etcd cluster (if multi-master)
kubectl -n kube-system exec -it etcd-k3s-master-01 -- etcdctl endpoint health
# Expected: is healthy: successfully committed proposal
23. Phase 17: Cilium BGP Control Plane
23.1. 17.0 Pre-Validation
# PRE-1: Verify Phase 16 complete (k3s cluster communication working)
cilium connectivity test --single-node
# Expected: All tests pass
# PRE-2: Verify k3s cluster healthy
kubectl get nodes
# Expected: All nodes Ready
# PRE-3: Check current BGP config (should not exist)
show protocols bgp
# Expected: Empty or "BGP not configured"
# PRE-4: Verify MetalLB currently handling LoadBalancer (will be replaced)
kubectl get svc -A | grep LoadBalancer
# Expected: Services have external IPs from MetalLB
# PRE-5: Document current LoadBalancer IPs (for comparison after migration)
kubectl get svc -A -o wide | grep LoadBalancer | awk '{print $1, $2, $5}'
|
BGP replaces MetalLB L2 mode for LoadBalancer services. Benefits: - Enterprise-grade routing protocol (CCIE/CCNP learning value) - Fast convergence on failover (BGP timers vs ARP timeout) - ECMP support for load distribution - Native Cilium integration (no separate MetalLB deployment) Architecture:
|
23.2. 17.1 VyOS BGP Configuration (vyos-02)
| Run on vyos-02. Router-ID must be unique per node. |
configure
# VyOS router ASN (65000 = infrastructure, 65001 = k3s)
set protocols bgp system-as 65000
set protocols bgp parameters router-id 10.50.1.3
# BGP neighbor - Cilium on each k3s node (6-node HA cluster)
set protocols bgp neighbor 10.50.1.120 remote-as 65001
set protocols bgp neighbor 10.50.1.120 description 'k3s-master-01 Cilium'
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.121 remote-as 65001
set protocols bgp neighbor 10.50.1.121 description 'k3s-master-02 Cilium'
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.122 remote-as 65001
set protocols bgp neighbor 10.50.1.122 description 'k3s-master-03 Cilium'
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.123 remote-as 65001
set protocols bgp neighbor 10.50.1.123 description 'k3s-worker-01 Cilium'
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.124 remote-as 65001
set protocols bgp neighbor 10.50.1.124 description 'k3s-worker-02 Cilium'
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.125 remote-as 65001
set protocols bgp neighbor 10.50.1.125 description 'k3s-worker-03 Cilium'
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast
# Accept only LoadBalancer pool routes (filter for safety)
set policy prefix-list K3S_LB_POOL rule 10 action permit
set policy prefix-list K3S_LB_POOL rule 10 prefix 10.50.1.128/28
set policy prefix-list K3S_LB_POOL rule 10 le 32
set policy route-map CILIUM_IMPORT rule 10 action permit
set policy route-map CILIUM_IMPORT rule 10 match ip address prefix-list K3S_LB_POOL
# Apply import policy to all Cilium neighbors
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast route-map import CILIUM_IMPORT
commit
save
23.3. 17.1b VyOS BGP Configuration (vyos-01)
| Run on vyos-01 after vyos-02 is validated. Router-ID must be unique per node. |
configure
# VyOS router ASN (65000 = infrastructure, 65001 = k3s)
set protocols bgp system-as 65000
set protocols bgp parameters router-id 10.50.1.2
# BGP neighbor - Cilium on each k3s node (6-node HA cluster)
set protocols bgp neighbor 10.50.1.120 remote-as 65001
set protocols bgp neighbor 10.50.1.120 description 'k3s-master-01 Cilium'
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.121 remote-as 65001
set protocols bgp neighbor 10.50.1.121 description 'k3s-master-02 Cilium'
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.122 remote-as 65001
set protocols bgp neighbor 10.50.1.122 description 'k3s-master-03 Cilium'
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.123 remote-as 65001
set protocols bgp neighbor 10.50.1.123 description 'k3s-worker-01 Cilium'
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.124 remote-as 65001
set protocols bgp neighbor 10.50.1.124 description 'k3s-worker-02 Cilium'
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast
set protocols bgp neighbor 10.50.1.125 remote-as 65001
set protocols bgp neighbor 10.50.1.125 description 'k3s-worker-03 Cilium'
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast
# Accept only LoadBalancer pool routes (filter for safety)
set policy prefix-list K3S_LB_POOL rule 10 action permit
set policy prefix-list K3S_LB_POOL rule 10 prefix 10.50.1.128/28
set policy prefix-list K3S_LB_POOL rule 10 le 32
set policy route-map CILIUM_IMPORT rule 10 action permit
set policy route-map CILIUM_IMPORT rule 10 match ip address prefix-list K3S_LB_POOL
# Apply import policy to all Cilium neighbors
set protocols bgp neighbor 10.50.1.120 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.121 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.122 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.123 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.124 address-family ipv4-unicast route-map import CILIUM_IMPORT
set protocols bgp neighbor 10.50.1.125 address-family ipv4-unicast route-map import CILIUM_IMPORT
commit
save
23.4. 17.2 Verify BGP Neighbors (VyOS)
# Show BGP summary
show ip bgp summary
# Expected: 6 neighbors in Established state (once Cilium is configured)
# Show BGP neighbors detail
show ip bgp neighbors
# Show received routes
show ip bgp
23.5. 17.3 Cilium BGP Configuration (k3s)
Update cilium-values.yaml to enable BGP Control Plane:
# /tmp/cilium-values.yaml
cluster:
name: domus-k3s
k8sServiceHost: 127.0.0.1
k8sServicePort: 6443
kubeProxyReplacement: true
routingMode: tunnel
tunnelProtocol: vxlan
operator:
replicas: 1
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: false
# Enable BGP Control Plane
bgpControlPlane:
enabled: true
Apply with Helm:
helm upgrade cilium cilium/cilium --version 1.16.5 \
--namespace kube-system \
-f /tmp/cilium-values.yaml
23.6. 17.4 Create Cilium BGP Resources
CiliumBGPPeeringPolicy - defines BGP peering with VyOS:
| Uses session variables for VyOS IPs. Verify variables before applying. |
# PRE: Verify VyOS IPs for Cilium peers
echo "VyOS-01: $\{VYOS_01_IP:-10.50.1.2}"
echo "VyOS-02: $\{VYOS_02_IP:-10.50.1.3}"
kubectl apply -f - << EOF
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
name: vyos-peering
spec:
nodeSelector:
matchLabels:
kubernetes.io/os: linux
virtualRouters:
- localASN: 65001
exportPodCIDR: false
neighbors:
- peerAddress: "10.50.1.2/32"
peerASN: 65000
- peerAddress: "10.50.1.3/32"
peerASN: 65000
serviceSelector:
matchExpressions:
- key: io.cilium/bgp-advertise
operator: NotIn
values: ["false"]
EOF
CiliumLoadBalancerIPPool - defines LoadBalancer IP range:
kubectl apply -f - << EOF
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: lb-pool
spec:
cidrs:
- cidr: 10.50.1.128/28
EOF
| CIDR 10.50.1.128/28 covers .128-.143 (16 IPs). Current services use .130-.134. |
23.7. 17.5 Remove MetalLB (After BGP Verified)
Once Cilium BGP is working, remove MetalLB to avoid conflicts:
# Check current MetalLB resources
kubectl get all -n metallb-system
# Delete MetalLB
kubectl delete namespace metallb-system
# Verify LoadBalancer services still have external IPs
kubectl get svc -A | grep LoadBalancer
23.8. 17.6 Verify BGP Routes (VyOS)
After Cilium BGP is configured, verify routes are learned:
# Show BGP received routes
show ip bgp
# Expected output:
# Network Next Hop Metric LocPrf Path
# *> 10.50.1.130/32 10.50.1.123 0 65001 i
# *> 10.50.1.131/32 10.50.1.124 0 65001 i
# etc.
# Show routing table
show ip route bgp
# Verify reachability
ping 10.50.1.130 # Traefik Ingress
ping 10.50.1.134 # Wazuh Manager API
23.9. 17.7 Test BGP Failover
# From workstation, start continuous ping to a LoadBalancer IP
ping 10.50.1.134
# Identify which node is advertising the route
show ip bgp 10.50.1.134
# Stop Cilium on that node (simulates node failure)
# (Run on k3s node)
sudo systemctl stop k3s
# Observe:
# 1. BGP neighbor goes down
# 2. Route is withdrawn
# 3. Another node advertises the route
# 4. Traffic continues (1-2 packets lost during convergence)
# Restart k3s
sudo systemctl start k3s
23.10. 17.8 Post-Validation
# POST-1: Verify BGP sessions established on vyos-01
ssh vyos-01 "show ip bgp summary"
# Expected: 3-6 neighbors in "Established" state (k3s nodes)
# State/PfxRcd should show numbers (received prefixes), not "Connect" or "Active"
# POST-2: Verify BGP sessions on vyos-02
ssh vyos-02 "show ip bgp summary"
# Expected: Same neighbors established (VRRP backup receives same routes)
# POST-3: Verify routes received from k3s
ssh vyos-01 "show ip bgp"
# Expected: /32 routes for each LoadBalancer VIP
# Should see next-hops pointing to k3s node IPs
# POST-4: Verify routes in routing table
ssh vyos-01 "show ip route bgp"
# Expected: BGP routes marked with 'B' for LoadBalancer VIPs
# POST-5: Verify LoadBalancer services still accessible
curl -kIs https://10.50.1.132:443 | head -1
curl -kIs https://10.50.1.130:443 | head -1
curl -kIs https://10.50.1.130:443 | head -1
# Expected: HTTP/1.1 or HTTP/2 response headers (200, 301, 302 are all OK)
# POST-6: Verify Cilium BGP peering status
kubectl exec -n kube-system ds/cilium -- cilium bgp peers
# Expected: All peers show "Established" state
# POST-7: Verify Cilium is advertising routes
kubectl exec -n kube-system ds/cilium -- cilium bgp routes advertised
# Expected: Shows LoadBalancer VIP /32 routes being advertised
# POST-8: Verify MetalLB removed (no conflicts)
kubectl get pods -n metallb-system 2>&1 | grep -E "^No resources|metallb"
# Expected: "No resources found" (MetalLB namespace empty or deleted)
24. Phase 18: Deploy -02 Secondaries (HA Foundation)
|
Execute AFTER VyOS HA is stable and clients verified on new network. Deploy -02 instances as secondaries FIRST: - Provides HA during -01 migration - Rollback path if migration fails - -02s stay in VLAN 100 initially Order: 1. Deploy -02 secondaries (this phase) 2. Verify client connectivity 3. Migrate -01s to VLANs 110/120 (Phase 19) |
24.1. 18.1 Deploy vault-02 (kvm-02)
See Vault HA Deployment for full runbook.
Quick summary:
# On kvm-02
sudo virt-install --name vault-02 \
--vcpus 1 --memory 1024 \
--disk /mnt/onboard-ssd/libvirt/images/vault-02.qcow2,size=20 \
--os-variant rocky9 \
--network bridge=virbr0 \
--graphics none --console pty,target_type=serial \
--cdrom /var/lib/libvirt/images/Rocky-9-GenericCloud.qcow2
| Item | Value |
|---|---|
IP |
10.50.1.61 |
Role |
Vault Raft follower |
Join to |
vault-01 (10.50.1.60) |
24.2. 18.2 Deploy bind-02 (kvm-02)
See BIND-02 Deployment for full runbook.
| Item | Value |
|---|---|
IP |
10.50.1.91 |
Role |
Secondary DNS (zone transfer from bind-01) |
Config |
Slave zones, forward to bind-01 |
24.3. 18.3 Deploy ise-02 (kvm-02) - Optional
|
ISE HA requires licensing. Skip if single-node ISE is sufficient. |
| Item | Value |
|---|---|
IP |
10.50.1.21 |
Role |
ISE secondary PAN/PSN |
Sync |
Automatic from ise-01 |
24.4. 18.4 Client Verification Checklist
Before proceeding to Phase 19, verify ALL clients work:
# POST-1: Wired 802.1X authentication
netapi ise mnt sessions | awk '/Wired/ {print $1, $3}'
# POST-2: Wireless 802.1X authentication
netapi ise mnt sessions | awk '/Wireless/ {print $1, $3}'
# POST-3: DHCP leases from VyOS
ssh vyos-01 "show dhcp server leases"
# POST-4: DNS resolution (both bind-01 and bind-02)
dig @10.50.1.90 vault-01.inside.domusdigitalis.dev +short
dig @10.50.1.91 vault-01.inside.domusdigitalis.dev +short
# POST-5: Vault SSH CA still works
vault-ssh-sign
vault-ssh-test
# POST-6: Internet access through VyOS NAT
curl -s https://ifconfig.me
|
Do NOT proceed to Phase 19 until ALL checks pass. If any check fails:
- Troubleshoot with VyOS logs: |
25. Phase 19: Infrastructure Segmentation (VLAN 110/120 Migration)
|
Execute ONLY after Phase 18 secondaries are deployed and clients verified. This phase migrates -01 VMs from flat VLAN 100 to segmented VLANs: - VLAN 110 (SECURITY): vault-01, ise-01, ipsk-mgr - VLAN 120 (SERVICES): keycloak-01, ipa-01, bind-01, gitea-01 The -02 secondaries provide continuity during migration. |
25.1. 19.0 Pre-Validation
# PRE-1: Verify VyOS VLAN interfaces exist
ssh vyos-01 "show interfaces ethernet eth0 vif"
# Expected: vif 110 (SECURITY), vif 120 (SERVICES)
# PRE-2: Verify switch trunks include VLANs 110, 120
ssh c9300-01 "show vlan brief | include 110\|120"
# PRE-3: Document current VM IPs (rollback reference)
for vm in vault-01 ise-01 ipsk-mgr keycloak-01 ipa-01 bind-01 gitea-01; do
echo "$vm: $(dig +short $vm.inside.domusdigitalis.dev)"
done
25.2. 19.1 Switch Trunk Preparation
On 3560-CX switch - add VLANs 110, 120 to trunk ports:
configure terminal
! Create VLANs
vlan 110
name SECURITY
vlan 120
name SERVICES
! Update trunk to kvm-01 (vyos-01)
interface TenGigabitEthernet1/0/2
switchport trunk allowed vlan add 110,120
! Update trunk to kvm-02 (vyos-02)
interface TenGigabitEthernet1/0/1
switchport trunk allowed vlan add 110,120
end
write memory
# POST: Verify VLANs added to trunks
show interfaces trunk | include Te1/0
25.3. 19.2 SECURITY VLAN Migration (vault-01, ise-01, ipsk-mgr)
25.3.1. 19.2.1 New IP Addresses
New IPs follow pattern: 10.50.{vlan-id}.{host-octet} - preserving host octet from current IP.
|
| VM | Old IP (VLAN 100) | New IP (VLAN 110) | Notes |
|---|---|---|---|
vault-01 |
10.50.1.60 |
10.50.110.60 |
PKI CA - critical |
ise-01 |
10.50.1.20 |
10.50.110.20 |
RADIUS - brief auth gap |
ipsk-mgr |
10.50.1.30 |
10.50.110.30 |
iPSK portal |
25.3.2. 19.2.2 KVM VLAN Strategy
|
KVM VLAN Options:
|
Create bridge on kvm-01:
sudo nmcli conn add type bridge con-name br-vlan110 ifname br-vlan110
sudo nmcli conn add type vlan con-name vlan110 ifname eth0.110 dev eth0 id 110 master br-vlan110
sudo nmcli conn up vlan110
sudo nmcli conn up br-vlan110
25.3.3. 19.2.3 Migrate vault-01
# 1. SSH to kvm-01
ssh kvm-01
# 2. Edit vault-01 to use VLAN 110 bridge
sudo virsh edit vault-01
# Change: <source bridge='virbr0'/> → <source bridge='br-vlan110'/>
# 3. Restart vault-01
sudo virsh shutdown vault-01
sudo virsh start vault-01
# 4. Inside vault-01, configure new IP
ssh vault-01 # Still reachable via old IP until reboot
sudo nmcli conn modify eth0 ipv4.addresses 10.50.110.60/24
sudo nmcli conn modify eth0 ipv4.gateway 10.50.110.1
sudo nmcli conn up eth0
25.3.4. 19.2.4 Update DNS
# Update pfSense/VyOS DNS override
netapi pfsense dns delete -h vault-01
netapi pfsense dns add -h vault-01 -d inside.domusdigitalis.dev -i 10.50.110.60 --descr "Vault PKI (VLAN 110)"
# Update BIND zone if used
ssh bind-01 "sudo sed -i 's/10.50.1.60/10.50.110.60/' /etc/named/zones/db.inside.domusdigitalis.dev"
ssh bind-01 "sudo systemctl reload named"
25.3.5. 19.2.5 Verify vault-01
# From workstation
ping vault-01.inside.domusdigitalis.dev
# Verify Vault API
curl -k https://vault-01.inside.domusdigitalis.dev:8200/v1/sys/health | jq
# Test SSH CA signing
vault-ssh-sign
25.3.6. 19.2.6 Repeat for ise-01 and ipsk-mgr
Follow same pattern:
-
Create bridge on KVM host if needed
-
Edit VM to use VLAN bridge
-
Configure new IP inside VM
-
Update DNS
-
Verify connectivity
ISE additional steps:
-
Update ISE network device definitions (switches, WLC point to ISE)
-
Verify RADIUS authentication still works
25.4. 19.3 SERVICES VLAN Migration (keycloak-01, ipa-01, bind-01, gitea-01)
25.4.1. 19.3.1 New IP Addresses
New IPs follow pattern: 10.50.{vlan-id}.{host-octet} - preserving host octet from current IP.
|
| VM | Old IP (VLAN 100) | New IP (VLAN 120) | Notes |
|---|---|---|---|
keycloak-01 |
10.50.1.80 |
10.50.120.80 |
SAML/OIDC IdP |
ipa-01 |
10.50.1.100 |
10.50.120.100 |
FreeIPA (Linux auth) |
bind-01 |
10.50.1.90 |
10.50.120.90 |
Primary DNS |
gitea-01 |
10.50.1.72 |
10.50.120.72 |
Git server |
25.5. 19.4 VyOS Zone Firewall Updates
After migration, add inter-VLAN firewall rules:
configure
# Allow SERVICES (120) → SECURITY (110) for specific services
set firewall name SERVICES-to-SECURITY rule 10 action accept
set firewall name SERVICES-to-SECURITY rule 10 description 'Keycloak to ISE RADIUS'
set firewall name SERVICES-to-SECURITY rule 10 destination address 10.50.110.20
set firewall name SERVICES-to-SECURITY rule 10 destination port 1812,1813
set firewall name SERVICES-to-SECURITY rule 10 protocol udp
set firewall name SERVICES-to-SECURITY rule 20 action accept
set firewall name SERVICES-to-SECURITY rule 20 description 'All to Vault PKI'
set firewall name SERVICES-to-SECURITY rule 20 destination address 10.50.110.60
set firewall name SERVICES-to-SECURITY rule 20 destination port 8200
set firewall name SERVICES-to-SECURITY rule 20 protocol tcp
# Allow INFRA (100) → SECURITY (110) for admin access
set firewall name INFRA-to-SECURITY rule 10 action accept
set firewall name INFRA-to-SECURITY rule 10 description 'Admin SSH'
set firewall name INFRA-to-SECURITY rule 10 destination port 22
set firewall name INFRA-to-SECURITY rule 10 protocol tcp
set firewall name INFRA-to-SECURITY rule 20 action accept
set firewall name INFRA-to-SECURITY rule 20 description 'Admin HTTPS'
set firewall name INFRA-to-SECURITY rule 20 destination port 443
set firewall name INFRA-to-SECURITY rule 20 protocol tcp
# Apply to zones
set zone-policy zone SECURITY from SERVICES firewall name SERVICES-to-SECURITY
set zone-policy zone SECURITY from INFRA firewall name INFRA-to-SECURITY
commit
save
25.6. 19.5 Post-Migration Validation
# POST-1: Verify all VMs reachable
for vm in vault-01 ise-01 ipsk-mgr keycloak-01 ipa-01 bind-01 gitea-01; do
echo -n "$vm: "
ping -c1 -W2 $vm.inside.domusdigitalis.dev >/dev/null && echo "OK" || echo "FAIL"
done
# POST-2: Verify 802.1X authentication (ISE in VLAN 110)
# Connect a wired client and check ISE logs
netapi ise mnt sessions | head -5
# POST-3: Verify Vault SSH CA
vault-ssh-sign
vault-ssh-test
# POST-4: Verify DNS resolution
dig vault-01.inside.domusdigitalis.dev
dig ise-01.inside.domusdigitalis.dev
26. Phase 20: WLC HA (Optional)
|
9800-CL WLC HA Options:
Recommendation: N+1 is sufficient unless you need seamless roaming during failover. |
For SSO HA deployment, create separate runbook: wlc-ha-deployment.adoc
27. Key Commands Reference
# Enter/exit config mode
configure
exit
# Show current config
show configuration
# Show pending changes
compare
# Commit and save
commit
save
# Rollback to previous config
rollback 1
commit
# Show interfaces
show interfaces
# Show firewall
show firewall
show firewall summary
show zone-policy
# Show VRRP status
show vrrp
show vrrp detail
# Show conntrack
show conntrack table
# Monitor traffic
monitor traffic interface eth0
Appendix A: CLI Mastery Reference
Advanced command patterns for network diagnostics, validation, and troubleshooting.
A.1. Network Interface Discovery
# All WiFi connections (nmcli parsing)
nmcli -t c s | awk -F: '$3=="wifi" {print $1}'
# Global IPv4 addresses (one-line for parsing)
ip -4 -o addr show scope global | awk '{print $2, $4}'
# Interface status pivot table
ip -o link show | awk -F': ' '{
split($3, flags, ",")
state = (index($3, "UP") ? "UP" : "DOWN")
printf "%-12s %s\n", $2, state
}'
# Bridge members with state
for br in $(ip link show type bridge | awk -F': ' '{print $2}'); do
echo "=== $br ==="
ip link show master $br | awk -F': ' '/^[0-9]/{print " "$2}'
done
A.2. Connection Analysis
# Top talkers by connection count
ss -tn | awk 'NR>1 {split($5,a,":"); ip[a[1]]++} END {for(i in ip) print ip[i], i}' | sort -rn | head -10
# Connections by state
ss -tan | awk 'NR>1 {state[$1]++} END {for(s in state) printf "%-12s %d\n", s, state[s]}'
# Listening ports with process
ss -tlnp | awk 'NR>1 {split($4,a,":"); print a[length(a)], $6}' | sort -n | uniq
# Established connections per remote IP
ss -tn state established | awk 'NR>1 {split($4,a,":"); print a[1]}' | sort | uniq -c | sort -rn
A.3. Parallel Operations
# Parallel subnet scan (SSH port)
echo {1..254} | tr ' ' '\n' | xargs -P 50 -I{} sh -c 'timeout 1 nc -z 10.50.1.{} 22 2>/dev/null && echo 10.50.1.{}'
# Parallel ping sweep
echo {1..254} | tr ' ' '\n' | xargs -P 50 -I{} sh -c 'ping -c1 -W1 10.50.1.{} >/dev/null 2>&1 && echo 10.50.1.{}'
# Parallel DNS lookup
echo "vault-01 ise-01 bind-01 nas-01 kvm-01 kvm-02" | tr ' ' '\n' | \
xargs -P 10 -I{} sh -c 'echo -n "{}: "; dig +short {}.inside.domusdigitalis.dev'
# Parallel service check (multiple ports)
for port in 22 80 443 8200; do
echo "=== Port $port ==="
echo "vault-01 ise-01 bind-01" | tr ' ' '\n' | \
xargs -P 10 -I{} sh -c "timeout 1 nc -z {}.inside.domusdigitalis.dev $port 2>/dev/null && echo '{}: OK' || echo '{}: FAIL'"
done
A.4. VyOS Show Commands with Parsing
# Zone summary (inside VyOS)
show firewall zone | awk 'NR>2 && NF>=2 {printf "%-12s → %s\n", $1, $2}'
# Interface status table
show interfaces | awk '/^eth|^lo/ {iface=$1; ip=$2} /u\/u|u\/D/ {print iface, ip, $0}'
# VRRP status (HA)
show vrrp | awk '/Group|State|Priority/ {print}'
# DHCP leases by pool
show dhcp server leases | awk 'NR>2 {pool[$5]++; print} END {print "---"; for(p in pool) print p": "pool[p]}'
# NAT translations active
show nat source translations | awk 'NR>1 {proto[$3]++} END {for(p in proto) print p": "proto[p]}'
# Conntrack table summary
show conntrack table | awk '{proto[$1]++} END {for(p in proto) printf "%-8s %d\n", p, proto[p]}'
A.5. Firewall Debugging
# Recent drops (via nftables)
sudo nft list ruleset | awk '/drop/ {found=1} found && /counter/ {print; found=0}'
# Zone policy matrix
show zone-policy | awk '/^[A-Z]/ {zone=$1} /from/ {print zone " ← " $2 ": " $4}'
# Firewall rule hit counts
show firewall | awk '/rule [0-9]+/ {rule=$2} /packets/ {print "Rule "rule": "$2" packets"}'
A.6. KVM/Libvirt Diagnostics
# VM status pivot
sudo virsh list --all | awk 'NR>2 && NF {state[$3]++; print} END {print "---"; for(s in state) print s": "state[s]}'
# VM interface to bridge mapping
for vm in $(sudo virsh list --name); do
echo "=== $vm ==="
sudo virsh domiflist $vm | awk 'NR>2 && NF {printf " %s → %s\n", $1, $3}'
done
# VM resource usage
sudo virsh list --name | xargs -I{} sh -c 'echo "=== {} ==="; sudo virsh dominfo {} | awk "/CPU|Memory/"'
# Disk usage per VM
sudo virsh list --name | while read vm; do
size=$(sudo virsh domblkinfo $vm vda 2>/dev/null | awk '/Capacity/{print $2/1024/1024/1024 " GB"}')
echo "$vm: $size"
done
A.7. Infrastructure Validation
# Quick infrastructure health check
for h in vault-01 ise-01 bind-01 nas-01 home-dc01; do
printf "%-12s " "$h"
timeout 2 nc -z $h.inside.domusdigitalis.dev 22 2>/dev/null && echo "✓ SSH" || echo "✗ SSH"
done
# DNS resolution check (all critical hosts)
for h in vyos-01 vyos-02 vault-01 ise-01 bind-01 nas-01 kvm-01 kvm-02; do
ip=$(dig +short $h.inside.domusdigitalis.dev)
printf "%-12s %s\n" "$h" "${ip:-NXDOMAIN}"
done
# VLAN gateway reachability
for gw in 10.50.1.1 10.50.10.1 10.50.20.1 10.50.30.1 10.50.40.1; do
printf "%-14s " "$gw"
ping -c1 -W1 $gw >/dev/null 2>&1 && echo "✓" || echo "✗"
done
# Certificate expiry check (parallel)
echo "vault-01:8200 ise-01:443 nas-01:5001" | tr ' ' '\n' | \
xargs -P 5 -I{} sh -c '
host=$(echo {} | cut -d: -f1)
port=$(echo {} | cut -d: -f2)
exp=$(echo | openssl s_client -connect $host.inside.domusdigitalis.dev:$port 2>/dev/null | \
openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
printf "%-20s %s\n" "$host:$port" "${exp:-FAILED}"
'
A.8. Log Analysis Patterns
# Syslog message type frequency
journalctl -p warning --since "1 hour ago" --no-pager | \
awk '{print $5}' | sort | uniq -c | sort -rn | head -10
# Failed SSH attempts
journalctl -u sshd --since "1 hour ago" --no-pager | \
awk '/Failed/ {for(i=1;i<=NF;i++) if($i=="from") print $(i+1)}' | sort | uniq -c | sort -rn
# VyOS commit history
show system commit | awk '/^[0-9]/ {print $1, $2, $3, $5}'
A.9. One-Liners Quick Reference
| Pattern | Use Case |
|---|---|
|
Extract first field (colon-delimited) |
|
Skip header row |
|
Count occurrences (pivot) |
|
Frequency analysis |
|
Parallel execution (50 threads) |
|
Quick port check with timeout |
|
One-line output (parseable) |
|
Convert space-delimited to lines |
|
Split field into array (awk) |
|
Formatted column output |