INC-2026-04-06-001: Domus-IoT VLAN VPN Connectivity Failure

Incident Summary

Field Value

Detected

2026-04-06 — user reported VPN connectivity degraded on Domus-IoT VLAN

Mitigated

2026-04-07 ~12:05 PST — rule 75 (TCP 2443) applied, tunnel handshake restored

Resolved

2026-04-07 ~12:10 PST — rules 80/85/90 (IPsec ESP, NAT-T, IKE) applied, full VPN data flow restored

Duration

Ongoing

Severity

P3 (Medium) — VPN tunnel establishes but data transport fails. Workaround: disconnect VPN.

Impact

Users on Domus-IoT VLAN (10.50.40.0/24) cannot use Palo Alto GlobalProtect work VPN. Tunnel handshake succeeds on TCP 443 but IPsec data transport is blocked by VyOS firewall.

Root Cause (Suspected)

IOT_WAN firewall policy permits TCP 80/443 (tunnel handshake) but does not permit IPsec protocols (ESP, UDP 4501, UDP 500) required for VPN data transport. Default action is drop. When GlobalProtect transitions from SSL handshake to IPsec data tunnel, VyOS silently drops the traffic.

Environment

Property Value

Affected VLAN

Domus-IoT (VLAN 40, 10.50.40.0/24, interface eth1.40)

VPN Client

Palo Alto GlobalProtect

VPN Gateway

External (work infrastructure, not hosted locally)

Firewall

VyOS 01 (vyos-01, KVM guest on kvm-01)

Firewall Policy

IOT_WAN (IoT zone → WAN zone)

NAT Rule

Rule 140: SNAT IoT → WAN via masquerade (confirmed working)

Timeline

Time Event

2026-04-06

User reports Palo Alto GlobalProtect VPN connects on Domus-IoT VLAN but network access degrades after tunnel establishment.

2026-04-06

Incident raised. 5 hypotheses documented. Investigation deferred to Apr 7.

2026-04-07

VyOS configuration reviewed via virsh console vyos-01 from kvm-01.

2026-04-07

IOT_WAN policy analyzed. TCP 443 permitted (tunnel handshake). ESP, UDP 4501, UDP 500 not permitted (data tunnel). Default action: drop.

2026-04-07

Firewall logs reviewed. IOT_WAN shows zero logged drops (default-log not enabled on this policy). IOT_MGMT logs confirm IoT devices actively communicating.

2026-04-07

Proposed fix: add IPsec rules to IOT_WAN. Proof-of-concept test designed.

2026-04-07 ~11:30

Immediate remediation: Enabled default-log on IOT_WAN to capture previously silent drops. Applied on VyOS via configure → set firewall ipv4 name IOT_WAN default-log → commit → save. No traffic impact — logging only. All future IOT_WAN default-action drops will now appear in show log firewall. This was a visibility gap — drops were occurring but not logged, making the VPN failure appear silent.

Symptoms

  • Palo Alto GlobalProtect VPN connects successfully — tunnel handshake completes on TCP 443

  • After connection, network access through VPN is degraded or broken — work resources unreachable, connections timeout

  • Issue is specific to Domus-IoT VLAN (10.50.40.0/24)

  • Disconnecting VPN restores normal internet connectivity (workaround)

  • Other VLANs (DATA, MGMT) not affected — VPN works from those networks

Investigation

Commands Executed

All commands run on vyos-01 via sudo virsh console vyos-01 from kvm-01.

VyOS IoT Configuration Audit

show configuration commands | grep -i iot
Key findings
# IoT Zone Definition
set firewall zone IOT default-action 'drop'
set firewall zone IOT description 'IoT Devices - Limited Access'
set firewall zone IOT member interface 'eth1.40'

# IoT → WAN policy (IOT_WAN)
set firewall ipv4 name IOT_WAN default-action 'drop'
set firewall ipv4 name IOT_WAN rule 10 action 'accept'    # established/related
set firewall ipv4 name IOT_WAN rule 20 action 'accept'    # TCP 80,443 (HTTP/HTTPS)
set firewall ipv4 name IOT_WAN rule 30 action 'accept'    # UDP 123 (NTP)
set firewall ipv4 name IOT_WAN rule 40 action 'accept'    # ICMP
set firewall ipv4 name IOT_WAN rule 50 action 'accept'    # UDP 53 (DNS)
set firewall ipv4 name IOT_WAN rule 60 action 'accept'    # UDP 5246 (CAPWAP control)
set firewall ipv4 name IOT_WAN rule 70 action 'accept'    # UDP 5247 (CAPWAP data)
# DEFAULT: drop — NO rules for ESP, UDP 4501, UDP 500

# NAT — IoT masquerade confirmed
set nat source rule 140 description 'SNAT IOT → WAN'
set nat source rule 140 outbound-interface name 'eth0'
set nat source rule 140 source group network-group 'NET_IOT'
set nat source rule 140 translation address 'masquerade'

# DHCP — IoT DNS points to public resolvers (not internal)
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server '8.8.8.8'
set service dhcp-server shared-network-name IOT subnet 10.50.40.0/24 option name-server '1.1.1.1'

NAT Rules Audit

show configuration commands | grep nat
Result — All VLANs have SNAT masquerade rules
set nat source rule 100 description 'SNAT INFRA → WAN'      # NET_INFRA
set nat source rule 110 description 'SNAT DATA → WAN'       # NET_DATA
set nat source rule 120 description 'SNAT VOICE → WAN'      # NET_VOICE
set nat source rule 130 description 'SNAT GUEST → WAN'      # NET_GUEST
set nat source rule 140 description 'SNAT IOT → WAN'        # NET_IOT ← confirmed
set nat source rule 150 description 'SNAT SECURITY → WAN'   # NET_SECURITY
set nat source rule 160 description 'SNAT SERVICES → WAN'   # NET_SERVICES
set nat source rule 170 description 'k3s pods to internet'   # NET_K3S_PODS

NAT is not the issue. IoT masquerade rule 140 exists and covers the subnet.

Firewall Drop Log Analysis

show log firewall | grep -i iot | tail -30
Result — IOT_MGMT drops (separate issue)
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.100 DST=10.50.1.200 DPT=443 PROTO=TCP SYN
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.104 DST=10.50.1.200 DPT=443 PROTO=TCP SYN
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.120 DST=10.50.1.90 DPT=53 PROTO=UDP
[ipv4-NAM-IOT_MGMT-default-D] SRC=10.50.40.120 DST=10.50.1.91 DPT=53 PROTO=TCP

These are IoT→MGMT drops — IoT devices attempting to reach internal MGMT resources (10.50.1.200:443, 10.50.1.90/91:53). Dropped by IOT_MGMT default-action drop. This is correct firewall behavior — IoT should not reach MGMT directly.

10.50.1.200 on TCP 443 is an internal MGMT host, NOT the VPN gateway. The VPN gateway is external (work infrastructure on the internet). 
show log firewall | grep "IOT_WAN" | tail -20
Result
(empty — no logged drops)

IOT_WAN does not have default-log enabled. Drops are occurring but not being logged. This is why the VPN failure appeared silent.

Analysis

How GlobalProtect VPN Works

  1. Tunnel establishment: Client connects to work VPN gateway on TCP 443 (SSL/TLS handshake)

  2. Authentication: User authenticates via SAML/credentials over the TCP 443 session

  3. Data tunnel transition: After auth, GlobalProtect transitions to IPsec for the actual data tunnel:

    • ESP (IP protocol 50): Encapsulated Security Payload — the encrypted data tunnel

    • UDP 4501 (IPsec NAT-T): NAT Traversal — used when client is behind NAT (which IoT devices are, via VyOS masquerade)

    • UDP 500 (IKE): Internet Key Exchange — key negotiation for the IPsec tunnel

  4. Data transport: All user traffic flows through the IPsec tunnel (ESP or UDP 4501)

Why It Fails on Domus-IoT

IOT_WAN rule 20 permits TCP 443 → tunnel handshake succeeds.

After handshake, GlobalProtect switches to IPsec:

  • ESP (protocol 50) — no rule in IOT_WAN → dropped (default action: drop)

  • UDP 4501 — no rule in IOT_WAN → dropped

  • UDP 500 — no rule in IOT_WAN → dropped

The VPN appears connected (handshake completed) but no data flows through the tunnel because the IPsec transport is being silently dropped by the firewall.

Why It Works on Other VLANs

DATA_WAN has default-action: accept — all outbound traffic is permitted, including ESP and IPsec. IoT is the restrictive zone with explicit allow-list and default drop.

DATA_WAN vs IOT_WAN Comparison

Property DATA_WAN IOT_WAN

Default action

accept

drop

TCP 443

Allowed (default accept)

Allowed (rule 20)

ESP (protocol 50)

Allowed (default accept)

Not permitted → DROPPED

UDP 4501

Allowed (default accept)

Not permitted → DROPPED

UDP 500

Allowed (default accept)

Not permitted → DROPPED

VPN behavior

Works

Tunnel up, data fails

Proposed Change: CR-2026-04-07-iot-vpn-ipsec

Justification

Users on the Domus-IoT VLAN require the ability to connect to external work VPN (Palo Alto GlobalProtect). The current IOT_WAN policy permits the tunnel handshake (TCP 443) but blocks IPsec data transport protocols. This creates a misleading failure state where the VPN appears connected but no traffic flows.

The proposed rules add the minimum required protocols for IPsec VPN passthrough. These are outbound-only rules (IoT → WAN) and do not permit any inbound access. The security posture of the IoT zone is preserved — devices can only initiate outbound VPN connections, not receive inbound ones.

Proposed VyOS Configuration

configure

# Rule 80: Allow IPsec ESP (IP protocol 50) — encrypted data tunnel
set firewall ipv4 name IOT_WAN rule 80 action accept
set firewall ipv4 name IOT_WAN rule 80 description 'Allow IPsec ESP for VPN'
set firewall ipv4 name IOT_WAN rule 80 protocol esp

# Rule 85: Allow IPsec NAT-T (UDP 4501) — NAT traversal for IPsec
set firewall ipv4 name IOT_WAN rule 85 action accept
set firewall ipv4 name IOT_WAN rule 85 description 'Allow IPsec NAT-T for VPN'
set firewall ipv4 name IOT_WAN rule 85 destination port 4501
set firewall ipv4 name IOT_WAN rule 85 protocol udp

# Rule 90: Allow IKE (UDP 500) — IPsec key exchange
set firewall ipv4 name IOT_WAN rule 90 action accept
set firewall ipv4 name IOT_WAN rule 90 description 'Allow IKE for VPN'
set firewall ipv4 name IOT_WAN rule 90 destination port 500
set firewall ipv4 name IOT_WAN rule 90 protocol udp

# Also enable default-log on IOT_WAN for future visibility
set firewall ipv4 name IOT_WAN default-log

# Review changes before committing
compare

Risk Assessment

Risk Mitigation

Opens ESP/IPsec outbound from IoT

Outbound only — IoT initiates the connection. No inbound rules changed. Return traffic handled by existing established/related rule (rule 10).

Could allow unauthorized VPN use from IoT devices

IoT devices (cameras, smart home) don’t have VPN clients. This enables human users (laptops) temporarily on IoT VLAN. Monitor with default-log.

Blast radius if misconfigured

Changes only affect IOT_WAN (IoT → WAN direction). No other zone policies modified. Rollback: delete rules 80, 85, 90.

Rollback Procedure

configure
delete firewall ipv4 name IOT_WAN rule 80
delete firewall ipv4 name IOT_WAN rule 85
delete firewall ipv4 name IOT_WAN rule 90
compare
commit
save

Proof-of-Concept Test Procedure

Before committing the change, capture evidence that IOT_WAN is dropping IPsec traffic.

Step 1: Enable Drop Logging on IOT_WAN

configure
set firewall ipv4 name IOT_WAN default-log
commit

Step 2: Monitor Firewall Log in Real Time

# Terminal 1 on VyOS — watch for IOT_WAN drops
tail -f /var/log/messages | grep "IOT_WAN"

Step 3: User Connects VPN

Have the user on Domus-IoT connect GlobalProtect VPN. The firewall log should show:

[ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.xxx DST=<vpn-gateway-ip> PROTO=ESP
[ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.xxx DST=<vpn-gateway-ip> PROTO=UDP DPT=4501

This is the proof — ESP or UDP 4501 packets from the IoT client being dropped by IOT_WAN.

Step 4: Apply Fix and Retest

# Apply the proposed rules (from "Proposed VyOS Configuration" above)
# Then have user reconnect VPN
# Verify:
# - VPN connects AND data flows
# - Work resources accessible
# - No new drops in IOT_WAN log

Step 5: Verify and Save

# Confirm no new drops
show log firewall | grep "IOT_WAN" | tail -10

# If working, save
commit
save

# If not working, rollback (see Rollback Procedure)

Verification Checklist

  • Enable default-log on IOT_WAN

  • Capture firewall drops during VPN connection attempt (proof)

  • Apply rules 80, 85, 90

  • User connects VPN — tunnel AND data transport functional

  • User can access work resources through VPN

  • External sites reachable while VPN connected

  • DNS resolution works inside VPN tunnel

  • ping -c 100 through VPN — no packet loss

  • MTU test: ping -M do -s 1400 <work-resource> passes

  • Firewall log shows no new IOT_WAN drops for VPN traffic

  • save configuration to persist across reboot

Impact Assessment

System Status Impact Duration

Domus-IoT VLAN VPN users

Degraded — tunnel up, data blocked

Since at least Mar 8 (earliest log evidence)

Palo Alto GlobalProtect

Functional — not a client-side issue

N/A

VyOS Router

Correct behavior per policy — policy needs update

N/A

Other VLANs

Unaffected

N/A

Business Impact

  • Users on Domus-IoT cannot work remotely via VPN

  • Workaround: connect to DATA VLAN (requires physical move or VLAN reassignment)

  • No data loss

  • Affects personal home network, not production infrastructure

Change Record

Change Record: CR-2026-04-07-iot-wan-vpn-passthrough — Complete

Four rules applied to VyOS firewall ipv4 name IOT_WAN:

  1. Rule 75 (Applied): TCP 2443 — GlobalProtect gateway tunnel

  2. Rule 80 (Applied): IPsec ESP (protocol 50) — encrypted data transport

  3. Rule 85 (Applied): UDP 4501 — IPsec NAT-T data channel

  4. Rule 90 (Applied): UDP 500 — IKE key exchange

  5. default-log enabled — all drops now logged for visibility

All rules verified 2026-04-07 12:10 PST. VPN connectivity restored.

API Queries — domus-api

Query this incident and its linked change record via the documentation REST API.

Start the API

cd ~/atelier/_projects/personal/domus-api && uv run uvicorn domus_api.main:app --host 0.0.0.0 --port 8080

Incident Queries

# Incident title
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.title'

# Incident content (readable plain text)
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' | head -80

# Search for all IOT_WAN references across the documentation system
curl -s 'localhost:8080/search?q=IOT_WAN' | jq -r '.results[] | "\(.title)\t\(.path)"'

# Search for VPN-related content
curl -s 'localhost:8080/search?q=GlobalProtect' | jq -r '.results[] | "\(.title)\t\(.path)"'

Change Record Queries

# CR title
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.title'

# CR content
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' | head -80

# List all change records
curl -s 'localhost:8080/pages?category=case-studies' | jq -r '.pages[] | select(.path | contains("changes")) | "\(.title)\t\(.path)"'

Traceability — Incident to CR

# Find all documents referencing this incident
curl -s 'localhost:8080/search?q=INC-2026-04-06-001' | jq -r '.results[] | {title, path}'

# Find the CR linked to this incident
curl -s 'localhost:8080/search?q=CR-2026-04-07-iot-wan' | jq -r '.results[] | {title, path}'

# Verify bidirectional link — incident references CR and CR references incident
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' | grep -i "CR-2026-04-07"
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' | grep -i "INC-2026-04-06"

Export for Reporting

# Terminal table — all incidents
curl -s 'localhost:8080/pages?category=case-studies' | jq -r '.pages[] | select(.path | contains("incidents")) | [.title, .path] | @tsv' | column -t -s $'\t'

# Export incident as text file for email/ticket
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' > /tmp/INC-2026-04-06-001-report.txt

# Export CR as text file
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' > /tmp/CR-2026-04-07-report.txt

Related

Metadata

Field Value

Incident ID

INC-2026-04-06-001

Author

Evan Rosado

Created

2026-04-06

Last Updated

2026-04-07

Status

Resolved — CR-2026-04-07 applied and verified 2026-04-07 12:10 PST

Post-Incident Review

After fix applied and verified