INC-2026-04-07-001: Investigation

Investigation

Initial Triage — Network Reachability

# From P16g (10.50.1.x, VLAN 50) to Razer (10.50.1.106, VLAN 10)
ping 10.50.1.106
# Result: 100% packet loss — VyOS drops inter-VLAN traffic VLAN 50→10

VLAN 10 is the restricted data VLAN. VyOS firewall policy denies inbound SSH from VLAN 50 (management) to VLAN 10. This is by design — the restriction is intentional for 802.1X segmentation.

Mitigation — Reverse SSH Tunnel

Since VLAN 10→50 is permitted (Razer can reach P16g), the tunnel is initiated from the allowed direction:

# On modestus-razer (TTY) — establish reverse tunnel
ssh -R 2222:localhost:22 evanusmodestus@modestus-p16g

This binds port 2222 on P16g to port 22 on the Razer. The firewall sees only the outbound SSH from VLAN 10, which is permitted. P16g connects to itself on 2222, traffic forwards through the tunnel to the Razer’s sshd.

modestus-razer:22 ←──tunnel──→ modestus-p16g:2222
      (VLAN 10)                    (VLAN 50)
         └── initiates outbound SSH (allowed) ──→

Failure 1 — Stale Host Key

ssh -p 2222 localhost
# WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
# Offending ECDSA key in known_hosts:45

Previous tunnel session left a [localhost]:2222 entry pointing to a different machine’s host key. StrictHostKeyChecking correctly refused the connection.

Fix:

ssh-keygen -R '[localhost]:2222'

Failure 2 — Root SSH Attempt

sudo ssh -p 2222 localhost
# Permission denied (publickey,gssapi-with-mic)

sudo ssh authenticates as root using root’s SSH keypair. The Razer’s authorized_keys has the user’s public key, not root’s. This was the wrong approach.

Failure 3 — No Keys at Default Paths

ssh -vvv -p 2222 localhost 2>&1 | grep -E 'identity file|agent'
# identity file /home/evanusmodestus/.ssh/id_ed25519 type -1
# agent contains no identities

SSH checks five default key names: id_rsa, id_ecdsa, id_ecdsa_sk, id_ed25519, id_ed25519_sk. None exist. All 16 keys use custom suffixes:

ls ~/.ssh/*.pub
# id_ed25519_bitbucket.pub, id_ed25519_d000.pub, id_ed25519_github.pub,
# id_ed25519_sk_rk_d000.pub, id_ed25519_vault.pub, etc.

The SSH config normally maps these to the correct host via IdentityFile directives in Host blocks. But localhost:2222 matches no configured host — SSH falls back to defaults, finds nothing.

Root Cause

Three independent failures compounded into a single access denial:

  1. Network layer: VyOS firewall denies VLAN 50→VLAN 10 by design. This is correct behavior — the firewall is working as intended for 802.1X segmentation.

  2. Host key layer: Stale known_hosts entry from a previous reverse tunnel session. StrictHostKeyChecking blocked the connection. This is also correct behavior — SSH detected a host key mismatch.

  3. Authentication layer: All SSH keys use custom-suffixed filenames. The SSH config maps them via Host blocks with IdentityFile directives. localhost:2222 matches no block, so SSH tries only the five default key names, finds none, and fails.

Why it happened:

  • Immediate cause: No Host block matches the reverse tunnel endpoint (localhost:2222)

  • Contributing factor: SSH agent was empty after reboot — no keys cached

  • Contributing factor: 16 keys with unique suffixes means zero default-path keys available as fallback

  • Systemic: Reverse SSH tunnels are not part of the documented runbook for cross-VLAN access