CR: IOT_WAN VPN Passthrough and Drop Logging

1. Change Summary

Field Value

Field	Value
CR ID	CR-2026-04-07-iot-wan-vpn-passthrough
Date	2026-04-07
Priority	P2 - High
Type	Firewall Policy Change
Status	Complete — all 4 rules applied and verified
Requestor	Evan Rosado
Implementor	Evan Rosado
Risk Level	Low (outbound-only rules, no inbound changes, IoT security posture preserved)
Systems Affected	VyOS vyos-01 — `firewall ipv4 name IOT_WAN`
Related Incident	INC-2026-04-06-001: Domus-IoT VPN Connectivity

CR ID

CR-2026-04-07-iot-wan-vpn-passthrough

Date

2026-04-07

Priority

P2 - High

Type

Firewall Policy Change

Status

Complete — all 4 rules applied and verified

Requestor

Evan Rosado

Implementor

Evan Rosado

Risk Level

Low (outbound-only rules, no inbound changes, IoT security posture preserved)

Systems Affected

VyOS vyos-01 — firewall ipv4 name IOT_WAN

Related Incident

INC-2026-04-06-001: Domus-IoT VPN Connectivity

2. Objective

Enable drop logging on IOT_WAN to provide visibility into blocked outbound traffic from the IoT VLAN (10.50.40.0/24)
Add IPsec protocol rules (ESP, UDP 4501, UDP 500) to IOT_WAN to permit Palo Alto GlobalProtect VPN data transport

Users on the Domus-IoT VLAN cannot use Palo Alto GlobalProtect work VPN. The tunnel handshake succeeds on TCP 443 (permitted by IOT_WAN rule 20), but after authentication the VPN client transitions to IPsec for data transport. IPsec uses ESP (IP protocol 50), UDP 4501 (NAT Traversal), and UDP 500 (IKE key exchange) — none of which are permitted by IOT_WAN. The default action is drop.

Additionally, IOT_WAN had no default-log enabled, meaning all drops were silent — no log entries, no visibility. This allowed the issue to persist undetected.

Comparison: DATA_WAN has default-action: accept, so VPN works from the DATA VLAN. Only IoT is affected due to its restrictive allow-list policy.

4. Change 1: Enable Default Drop Logging

4.1. Justification

IOT_WAN default-action is drop but default-log was not enabled. All dropped packets were silent — no log entries in show log firewall. This is a visibility gap that prevented detection of the VPN issue and any other outbound IoT traffic being silently blocked.

4.2. Pre-Verification

# Verify current state — default-log not set
show configuration commands | grep "IOT_WAN default"

Expected output

set firewall ipv4 name IOT_WAN default-action 'drop'
# NO default-log line present

# Confirm no IOT_WAN entries in firewall log
show log firewall | grep "IOT_WAN" | tail -5

Expected output

(empty — no logged drops)

4.3. Change Commands

configure
set firewall ipv4 name IOT_WAN default-log
commit
save

4.4. Post-Verification

# Verify default-log is set
show configuration commands | grep "IOT_WAN default"

Expected output

set firewall ipv4 name IOT_WAN default-action 'drop'
set firewall ipv4 name IOT_WAN default-log

# Verify drops now appear in log (generate traffic from IoT device if needed)
show log firewall | grep "IOT_WAN" | tail -5

4.5. Rollback

configure
delete firewall ipv4 name IOT_WAN default-log
commit
save

4.6. Status

Applied — 2026-04-07 ~11:30 PST

5. Change 2: Add IPsec VPN Passthrough Rules

5.1. Justification

Palo Alto GlobalProtect VPN requires IPsec protocols for data transport after the initial SSL/TLS handshake on TCP 443:

Protocol	Port/ID	Purpose
ESP	IP Protocol 50	Encapsulated Security Payload — the encrypted data tunnel carrying all user traffic
UDP 4501	Destination port	IPsec NAT Traversal — used when the VPN client is behind NAT (which IoT devices are, via VyOS masquerade rule 140)
UDP 500	Destination port	Internet Key Exchange (IKE) — negotiates encryption keys for the IPsec tunnel

Protocol

Port/ID

Purpose

ESP

IP Protocol 50

Encapsulated Security Payload — the encrypted data tunnel carrying all user traffic

UDP 4501

Destination port

IPsec NAT Traversal — used when the VPN client is behind NAT (which IoT devices are, via VyOS masquerade rule 140)

UDP 500

Destination port

Internet Key Exchange (IKE) — negotiates encryption keys for the IPsec tunnel

Without these protocols, the VPN handshake completes (TCP 443 allowed) but the data tunnel fails silently. Users see "VPN Connected" but cannot access any resources.

5.2. Risk Assessment

Risk Mitigation

Risk	Mitigation
Opens ESP/IPsec outbound from IoT	Outbound only — IoT initiates the connection to external VPN gateways. No inbound rules changed. Return traffic handled by existing established/related rule (IOT_WAN rule 10).
Could allow unauthorized VPN use from IoT devices	IoT devices (cameras, smart home sensors) do not have VPN clients. These rules enable human users with laptops temporarily on the IoT VLAN. Monitor via `default-log`.
Blast radius if misconfigured	Changes affect ONLY `IOT_WAN` (IoT → WAN direction). No other zone policies modified. Three discrete rules with clear protocol/port scope.

Opens ESP/IPsec outbound from IoT

Outbound only — IoT initiates the connection to external VPN gateways. No inbound rules changed. Return traffic handled by existing established/related rule (IOT_WAN rule 10).

Could allow unauthorized VPN use from IoT devices

IoT devices (cameras, smart home sensors) do not have VPN clients. These rules enable human users with laptops temporarily on the IoT VLAN. Monitor via default-log.

Blast radius if misconfigured

Changes affect ONLY IOT_WAN (IoT → WAN direction). No other zone policies modified. Three discrete rules with clear protocol/port scope.

5.3. Pre-Verification

# Verify current IOT_WAN rules (should show rules 10-70 only)
show firewall ipv4 name IOT_WAN

# Confirm no ESP/4501/500 rules exist
show configuration commands | grep "IOT_WAN rule 8\|IOT_WAN rule 9"

Expected output

(empty — rules 80/85/90 do not exist yet)

5.4. Proof — Firewall Drop Log Evidence (Captured 2026-04-07 12:00 PST)

After enabling default-log on IOT_WAN (Change 1), the user on Domus-IoT (10.50.40.146) attempted to connect GlobalProtect VPN. The following drops were captured:

show log firewall | grep "IOT_WAN" | tail -20

Captured output

# GlobalProtect VPN gateway — TCP 2443 DROPPED
Apr 07 12:00:53 [ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.146 DST=162.115.35.40 PROTO=TCP SPT=49363 DPT=2443 SYN
Apr 07 12:00:54 [ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.146 DST=162.115.35.40 PROTO=TCP SPT=49363 DPT=2443 SYN
Apr 07 12:00:55 [ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.146 DST=162.115.35.40 PROTO=TCP SPT=49363 DPT=2443 SYN
Apr 07 12:00:56 [ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.146 DST=162.115.35.40 PROTO=TCP SPT=49363 DPT=2443 SYN
# ... repeated SYN retransmits through 12:01:28 — connection never established

# Google push notifications — TCP 5228 DROPPED (separate issue)
Apr 07 11:46:28 [ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.146 DST=74.125.137.188 PROTO=TCP SPT=65533 DPT=5228 SYN

5.5. Analysis of Proof

Drop Destination Port/Proto Impact

Drop	Destination	Port/Proto	Impact
`162.115.35.40:2443`	Palo Alto GlobalProtect VPN gateway	TCP 2443 — alternate GP portal/gateway port	VPN cannot establish tunnel — handshake blocked at SYN
`74.125.137.188:5228`	Google push notification service	TCP 5228 — Android/Chrome push	Push notifications broken on IoT devices

162.115.35.40:2443

Palo Alto GlobalProtect VPN gateway

TCP 2443 — alternate GP portal/gateway port

VPN cannot establish tunnel — handshake blocked at SYN

74.125.137.188:5228

Google push notification service

TCP 5228 — Android/Chrome push

Push notifications broken on IoT devices

Key finding: The VPN failure occurred at two stages:

TCP handshake stage: GlobalProtect uses TCP 2443 for portal/gateway, blocked by IOT_WAN (only 80,443 permitted)
IPsec data stage: After rule 75 (TCP 2443) applied, tunnel established but UDP 4501 (IPsec NAT-T) was blocked, preventing data flow

5.6. Post-Fix Verification (2026-04-07 12:10 PST)

After applying rules 75, 80, 85, 90:

show firewall ipv4 name IOT_WAN

Result — all rules in place

rule 75 { action accept; destination { port 2443 }; protocol tcp }
rule 80 { action accept; description "Allow IPsec ESP for VPN"; protocol esp }
rule 85 { action accept; description "Allow IPsec NAT-T for VPN"; destination { port 4501 }; protocol udp }
rule 90 { action accept; description "Allow IKE for VPN"; destination { port 500 }; protocol udp }

show log firewall | grep "IOT_WAN" | grep "4501\|2443" | tail -10

Result — only PRE-FIX drops, no post-fix drops

# All drops timestamped 12:00-12:08 — BEFORE rules applied
Apr 07 12:00:53 ... DST=162.115.35.40 DPT=2443 SYN  ← pre-fix
Apr 07 12:01:28 ... DST=162.115.35.40 DPT=2443 SYN  ← pre-fix
Apr 07 12:02:13 ... DST=162.115.28.58 DPT=4501       ← pre-fix
Apr 07 12:08:28 ... DST=162.115.28.58 DPT=4501       ← pre-fix
# NO drops after 12:10 — rules are working

show log firewall | grep "IOT_WAN" | tail -10

Result — remaining drops are non-VPN traffic only

# TCP 5228 — Google push notifications (not in scope)
# TCP 5150 — Application service (not in scope)
# TCP 5223 — Apple push notifications (not in scope)
# TCP 53   — DNS over TCP (rule 50 only allows UDP 53, separate issue)
# NO VPN-related drops (2443, 4501, 500, ESP)

User confirmed: GlobalProtect VPN connected and data flowing. Work resources accessible.

Proof status: VERIFIED — Rules 75, 80, 85, 90 resolved the VPN connectivity failure.

The original hypothesis (IPsec ESP/UDP 4501 blocked after tunnel) is still valid for after the tunnel establishes, but the immediate blocker is TCP 2443.

5.7. Proof-of-Concept Test Procedure (Completed)

Step 1: Enabled default-log on IOT_WAN (Change 1, applied 2026-04-07 ~11:30).

Step 2: Monitored firewall log:

tail -f /var/log/messages | grep "IOT_WAN"

Step 3: User on Domus-IoT (10.50.40.146) connected GlobalProtect VPN.

Step 4: Firewall log showed TCP 2443 SYN packets to 162.115.35.40 being dropped by IOT_WAN default action. Repeated SYN retransmits confirm the connection was never established.

Proof status: CONFIRMED — IOT_WAN is blocking the VPN gateway connection on TCP 2443.

[ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.xxx DST=<vpn-gateway-ip> PROTO=ESP
[ipv4-NAM-IOT_WAN-default-D] SRC=10.50.40.xxx DST=<vpn-gateway-ip> PROTO=UDP DPT=4501

This is the proof — ESP or UDP 4501 from the IoT client dropped by IOT_WAN default action.

Step 4: If drops confirmed, proceed to apply the change.

5.8. Change Commands

configure

# Rule 75: Allow GlobalProtect VPN portal (TCP 2443) — CONFIRMED BLOCKED in proof
set firewall ipv4 name IOT_WAN rule 75 action accept
set firewall ipv4 name IOT_WAN rule 75 description 'Allow GlobalProtect VPN portal (TCP 2443)'
set firewall ipv4 name IOT_WAN rule 75 destination port 2443
set firewall ipv4 name IOT_WAN rule 75 protocol tcp

# Rule 80: Allow IPsec ESP (IP protocol 50) — encrypted data tunnel
set firewall ipv4 name IOT_WAN rule 80 action accept
set firewall ipv4 name IOT_WAN rule 80 description 'Allow IPsec ESP for VPN'
set firewall ipv4 name IOT_WAN rule 80 protocol esp

# Rule 85: Allow IPsec NAT-T (UDP 4501) — NAT traversal for IPsec
set firewall ipv4 name IOT_WAN rule 85 action accept
set firewall ipv4 name IOT_WAN rule 85 description 'Allow IPsec NAT-T for VPN'
set firewall ipv4 name IOT_WAN rule 85 destination port 4501
set firewall ipv4 name IOT_WAN rule 85 protocol udp

# Rule 90: Allow IKE (UDP 500) — IPsec key exchange
set firewall ipv4 name IOT_WAN rule 90 action accept
set firewall ipv4 name IOT_WAN rule 90 description 'Allow IKE for VPN'
set firewall ipv4 name IOT_WAN rule 90 destination port 500
set firewall ipv4 name IOT_WAN rule 90 protocol udp

# Review before committing
compare

Review the output of compare. Verify ONLY rules 80, 85, 90 are added. No other changes.

# If compare output is correct:
commit
save

5.9. Post-Verification

# 1. Verify rules are in place
show firewall ipv4 name IOT_WAN

Expected — rules 80, 85, 90 present with zero packet counts

Rule 80: accept ESP
Rule 85: accept UDP DPT=4501
Rule 90: accept UDP DPT=500

# 2. Have user reconnect VPN and test
# On user device:
# - Connect GlobalProtect
# - Access work resources
# - Run: ping -c 10 <work-resource>
# - Run: curl -I https://<work-portal>

# 3. Verify rule hit counters incrementing
show firewall ipv4 name IOT_WAN

Expected — packet counts on rules 80/85/90 incrementing

Rule 80: accept ESP — Packets > 0
Rule 85: accept UDP DPT=4501 — Packets > 0
Rule 90: accept UDP DPT=500 — Packets > 0

# 4. Verify no new default drops for VPN traffic
show log firewall | grep "IOT_WAN" | tail -10

Expected — no ESP/4501/500 drops (only unrelated drops if any)

# 5. Confirm IoT security posture unchanged
# IoT devices should NOT be able to reach internal networks
# Test from IoT device:
# ping 10.50.1.1    ← should still be blocked (IOT_LOCAL only allows DHCP/DNS)
# ping 10.50.10.x   ← should still be blocked (IOT_DATA default drop)

5.10. Rollback

configure
delete firewall ipv4 name IOT_WAN rule 75
delete firewall ipv4 name IOT_WAN rule 80
delete firewall ipv4 name IOT_WAN rule 85
delete firewall ipv4 name IOT_WAN rule 90
compare
commit
save

5.11. Status

Applied — 2026-04-07 ~12:10 PST. Proof verified: no VPN-related drops after commit. User confirmed VPN functional.

6. IOT_WAN Policy Summary (After Both Changes)

Rule	Description	Protocol/Port	Status
10	Established/related	stateful	Existing
20	HTTP/HTTPS	TCP 80,443	Existing
30	NTP	UDP 123	Existing
40	ICMP	icmp	Existing
50	DNS	UDP 53	Existing
60	CAPWAP control	UDP 5246	Existing
70	CAPWAP data	UDP 5247	Existing
75	GlobalProtect VPN portal	TCP 2443	New — PROOF CONFIRMED
80	IPsec ESP	protocol esp	New (pending)
85	IPsec NAT-T	UDP 4501	New (pending)
90	IKE	UDP 500	New (pending)
default	Drop + log	all	Logging added

Rule

Description

Protocol/Port

Status

Established/related

stateful

Existing

HTTP/HTTPS

TCP 80,443

Existing

NTP

UDP 123

Existing

ICMP

icmp

Existing

DNS

UDP 53

Existing

CAPWAP control

UDP 5246

Existing

CAPWAP data

UDP 5247

Existing

GlobalProtect VPN portal

TCP 2443

New — PROOF CONFIRMED

IPsec ESP

protocol esp

New (pending)

IPsec NAT-T

UDP 4501

New (pending)

IKE

UDP 500

New (pending)

default

Drop + log

all

Logging added

7. API Queries — domus-api

Query this incident and its linked change record via the documentation REST API.

7.1. Start the API

cd ~/atelier/_projects/personal/domus-api && uv run uvicorn domus_api.main:app --host 0.0.0.0 --port 8080

7.2. Incident Queries

# Incident title
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.title'

# Incident content (readable plain text)
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' | head -80

# Search for all IOT_WAN references across the documentation system
curl -s 'localhost:8080/search?q=IOT_WAN' | jq -r '.results[] | "\(.title)\t\(.path)"'

# Search for VPN-related content
curl -s 'localhost:8080/search?q=GlobalProtect' | jq -r '.results[] | "\(.title)\t\(.path)"'

7.3. Change Record Queries

# CR title
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.title'

# CR content
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' | head -80

# List all change records
curl -s 'localhost:8080/pages?category=case-studies' | jq -r '.pages[] | select(.path | contains("changes")) | "\(.title)\t\(.path)"'

7.4. Traceability — Incident to CR

# Find all documents referencing this incident
curl -s 'localhost:8080/search?q=INC-2026-04-06-001' | jq -r '.results[] | {title, path}'

# Find the CR linked to this incident
curl -s 'localhost:8080/search?q=CR-2026-04-07-iot-wan' | jq -r '.results[] | {title, path}'

# Verify bidirectional link — incident references CR and CR references incident
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' | grep -i "CR-2026-04-07"
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' | grep -i "INC-2026-04-06"

7.5. Export for Reporting

# Terminal table — all incidents
curl -s 'localhost:8080/pages?category=case-studies' | jq -r '.pages[] | select(.path | contains("incidents")) | [.title, .path] | @tsv' | column -t -s $'\t'

# Export incident as text file for email/ticket
curl -s localhost:8080/pages/case-studies/incidents/INC-2026-04-06-domus-iot-vpn-connectivity | jq -r '.content' > /tmp/INC-2026-04-06-001-report.txt

# Export CR as text file
curl -s localhost:8080/pages/case-studies/changes/CR-2026-04-07-iot-wan-vpn-passthrough | jq -r '.content' > /tmp/CR-2026-04-07-report.txt

8. Related

INC-2026-04-06-001: Domus-IoT VPN Connectivity

9. Metadata

Field	Value
CR ID	CR-2026-04-07-iot-wan-vpn-passthrough
Author	Evan Rosado
Created	2026-04-07
Last Updated	2026-04-07
Status	Complete — All changes applied and verified 2026-04-07 12:10 PST

Field

Value

CR ID

CR-2026-04-07-iot-wan-vpn-passthrough

Author

Evan Rosado

Created

2026-04-07

Last Updated

2026-04-07

Status

Complete — All changes applied and verified 2026-04-07 12:10 PST