INC-2026-04-06-001: Investigation

Investigation

Commands Executed

All commands run on vyos-01 via sudo virsh console vyos-01 from kvm-01.

VyOS IoT Configuration Audit

show configuration commands | grep -i iot
Key findings
# IoT Zone Definition
set firewall zone IOT default-action 'drop'
set firewall zone IOT description 'IoT Devices - Limited Access'
set firewall zone IOT member interface 'eth1.40'

# IoT → WAN policy (IOT_WAN)
set firewall ipv4 name IOT_WAN default-action 'drop'
set firewall ipv4 name IOT_WAN rule 10 action 'accept'    # established/related
set firewall ipv4 name IOT_WAN rule 20 action 'accept'    # TCP 80,443 (HTTP/HTTPS)
set firewall ipv4 name IOT_WAN rule 30 action 'accept'    # UDP 123 (NTP)
set firewall ipv4 name IOT_WAN rule 40 action 'accept'    # ICMP
set firewall ipv4 name IOT_WAN rule 50 action 'accept'    # UDP 53 (DNS)
set firewall ipv4 name IOT_WAN rule 60 action 'accept'    # UDP 5246 (CAPWAP control)
set firewall ipv4 name IOT_WAN rule 70 action 'accept'    # UDP 5247 (CAPWAP data)
# DEFAULT: drop — NO rules for ESP, UDP 4501, UDP 500

# NAT — IoT masquerade confirmed
set nat source rule 140 description 'SNAT IOT → WAN'
set nat source rule 140 outbound-interface name 'eth0'
set nat source rule 140 source group network-group 'NET_IOT'
set nat source rule 140 translation address 'masquerade'

# DHCP — IoT DNS points to public resolvers (not internal)
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server '8.8.8.8'
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server '1.1.1.1'

NAT Rules Audit

show configuration commands | grep nat
Result — All VLANs have SNAT masquerade rules
set nat source rule 100 description 'SNAT INFRA → WAN'      # NET_INFRA
set nat source rule 110 description 'SNAT DATA → WAN'       # NET_DATA
set nat source rule 120 description 'SNAT VOICE → WAN'      # NET_VOICE
set nat source rule 130 description 'SNAT GUEST → WAN'      # NET_GUEST
set nat source rule 140 description 'SNAT IOT → WAN'        # NET_IOT ← confirmed
set nat source rule 150 description 'SNAT SECURITY → WAN'   # NET_SECURITY
set nat source rule 160 description 'SNAT SERVICES → WAN'   # NET_SERVICES
set nat source rule 170 description 'k3s pods to internet'   # NET_K3S_PODS

NAT is not the issue. IoT masquerade rule 140 exists and covers the subnet.

Firewall Drop Log Analysis

show log firewall | grep -i iot | tail -30
Result — IOT_MGMT drops (separate issue)
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.100 DST=10.50.1.200 DPT=443 PROTO=TCP SYN
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.104 DST=10.50.1.200 DPT=443 PROTO=TCP SYN
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.120 DST=10.50.1.90 DPT=53 PROTO=UDP
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.120 DST=10.50.1.91 DPT=53 PROTO=TCP

These are IoT→MGMT drops — IoT devices attempting to reach internal MGMT resources (10.50.1.200:443, 10.50.1.90/91:53). Dropped by IOT_MGMT default-action drop. This is correct firewall behavior — IoT should not reach MGMT directly.

10.50.1.200 on TCP 443 is an internal MGMT host, NOT the VPN gateway. The VPN gateway is external (work infrastructure on the internet). 
show log firewall | grep "IOT_WAN" | tail -20
Result
(empty — no logged drops)

IOT_WAN does not have default-log enabled. Drops are occurring but not being logged. This is why the VPN failure appeared silent.

Analysis

How GlobalProtect VPN Works

  1. Tunnel establishment: Client connects to work VPN gateway on TCP 443 (SSL/TLS handshake)

  2. Authentication: User authenticates via SAML/credentials over the TCP 443 session

  3. Data tunnel transition: After auth, GlobalProtect transitions to IPsec for the actual data tunnel:

    • ESP (IP protocol 50): Encapsulated Security Payload — the encrypted data tunnel

    • UDP 4501 (IPsec NAT-T): NAT Traversal — used when client is behind NAT (which IoT devices are, via VyOS masquerade)

    • UDP 500 (IKE): Internet Key Exchange — key negotiation for the IPsec tunnel

  4. Data transport: All user traffic flows through the IPsec tunnel (ESP or UDP 4501)

Why It Fails on Domus-IoT

IOT_WAN rule 20 permits TCP 443 → tunnel handshake succeeds.

After handshake, GlobalProtect switches to IPsec:

  • ESP (protocol 50) — no rule in IOT_WAN → dropped (default action: drop)

  • UDP 4501 — no rule in IOT_WAN → dropped

  • UDP 500 — no rule in IOT_WAN → dropped

The VPN appears connected (handshake completed) but no data flows through the tunnel because the IPsec transport is being silently dropped by the firewall.

Why It Works on Other VLANs

DATA_WAN has default-action: accept — all outbound traffic is permitted, including ESP and IPsec. IoT is the restrictive zone with explicit allow-list and default drop.

DATA_WAN vs IOT_WAN Comparison

Property DATA_WAN IOT_WAN

Default action

accept

drop

TCP 443

Allowed (default accept)

Allowed (rule 20)

ESP (protocol 50)

Allowed (default accept)

Not permitted → DROPPED

UDP 4501

Allowed (default accept)

Not permitted → DROPPED

UDP 500

Allowed (default accept)

Not permitted → DROPPED

VPN behavior

Works

Tunnel up, data fails