INC-2026-04-06-001: Domus-IoT VLAN VPN Connectivity Failure

Incident Summary

Field Value

Detected

2026-04-06 — user reported VPN connectivity degraded on Domus-IoT VLAN

Mitigated

2026-04-07 ~12:05 PST — rule 75 (TCP 2443) applied, tunnel handshake restored

Resolved

2026-04-07 ~12:10 PST — rules 80/85/90 (IPsec ESP, NAT-T, IKE) applied, full VPN data flow restored

Duration

Ongoing

Severity

P3 (Medium) — VPN tunnel establishes but data transport fails. Workaround: disconnect VPN.

Impact

Users on Domus-IoT VLAN (10.50.40.0/24) cannot use Palo Alto GlobalProtect work VPN. Tunnel handshake succeeds on TCP 443 but IPsec data transport is blocked by VyOS firewall.

Root Cause (Suspected)

IOT_WAN firewall policy permits TCP 80/443 (tunnel handshake) but does not permit IPsec protocols (ESP, UDP 4501, UDP 500) required for VPN data transport. Default action is drop. When GlobalProtect transitions from SSL handshake to IPsec data tunnel, VyOS silently drops the traffic.

Environment

Property Value

Affected VLAN

Domus-IoT (VLAN 40, 10.50.40.0/24, interface eth1.40)

VPN Client

Palo Alto GlobalProtect

VPN Gateway

External (work infrastructure, not hosted locally)

Firewall

VyOS 01 (vyos-01, KVM guest on kvm-01)

Firewall Policy

IOT_WAN (IoT zone → WAN zone)

NAT Rule

Rule 140: SNAT IoT → WAN via masquerade (confirmed working)

Timeline

Time Event

2026-04-06

User reports Palo Alto GlobalProtect VPN connects on Domus-IoT VLAN but network access degrades after tunnel establishment.

2026-04-06

Incident raised. 5 hypotheses documented. Investigation deferred to Apr 7.

2026-04-07

VyOS configuration reviewed via virsh console vyos-01 from kvm-01.

2026-04-07

IOT_WAN policy analyzed. TCP 443 permitted (tunnel handshake). ESP, UDP 4501, UDP 500 not permitted (data tunnel). Default action: drop.

2026-04-07

Firewall logs reviewed. IOT_WAN shows zero logged drops (default-log not enabled on this policy). IOT_MGMT logs confirm IoT devices actively communicating.

2026-04-07

Proposed fix: add IPsec rules to IOT_WAN. Proof-of-concept test designed.

2026-04-07 ~11:30

Immediate remediation: Enabled default-log on IOT_WAN to capture previously silent drops. Applied on VyOS via configure → set firewall ipv4 name IOT_WAN default-log → commit → save. No traffic impact — logging only. All future IOT_WAN default-action drops will now appear in show log firewall. This was a visibility gap — drops were occurring but not logged, making the VPN failure appear silent.

Symptoms

  • Palo Alto GlobalProtect VPN connects successfully — tunnel handshake completes on TCP 443

  • After connection, network access through VPN is degraded or broken — work resources unreachable, connections timeout

  • Issue is specific to Domus-IoT VLAN (10.50.40.0/24)

  • Disconnecting VPN restores normal internet connectivity (workaround)

  • Other VLANs (DATA, MGMT) not affected — VPN works from those networks

Impact Assessment

System Status Impact Duration

Domus-IoT VLAN VPN users

Degraded — tunnel up, data blocked

Since at least Mar 8 (earliest log evidence)

Palo Alto GlobalProtect

Functional — not a client-side issue

N/A

VyOS Router

Correct behavior per policy — policy needs update

N/A

Other VLANs

Unaffected

N/A

Business Impact

  • Users on Domus-IoT cannot work remotely via VPN

  • Workaround: connect to DATA VLAN (requires physical move or VLAN reassignment)

  • No data loss

  • Affects personal home network, not production infrastructure

Metadata

Field Value

Incident ID

INC-2026-04-06-001

Author

Evan Rosado

Created

2026-04-06

Last Updated

2026-04-07

Status

Resolved — CR-2026-04-07 applied and verified 2026-04-07 12:10 PST

Post-Incident Review

After fix applied and verified