INC-2026-04-07-001: SSH Access Failure to modestus-razer Due to VLAN Restriction and Key Configuration

Incident Summary

Field Value

Detected

2026-04-07 06:30 PST (manual discovery during Hyprland troubleshooting)

Mitigated

2026-04-07 06:45 PST (reverse SSH tunnel established)

Resolved

2026-04-07 06:50 PST (authenticated with correct identity file)

Duration

~20 minutes

Severity

P3 (Medium) — Workaround available

Impact

Cannot manage modestus-razer remotely from modestus-p16g. Razer stuck in TTY (Hyprland down), SSH required for remote troubleshooting.

Root Cause

Three compounding failures: (1) VyOS firewall blocks VLAN 50→VLAN 10 SSH by design, (2) SSH keys use custom suffixed paths not matched by default lookup, (3) SSH agent empty after reboot.

Timeline

Time (PST) Event

06:23

Hyprland failure detected on modestus-razer. User drops to TTY.

06:25

Razer can SSH to P16g (VLAN 10→50 allowed). P16g cannot SSH to Razer (VLAN 50→10 denied by VyOS).

06:30

ping 10.50.1.106 — 100% packet loss. Confirms firewall blocking ICMP and SSH cross-VLAN.

06:35

Reverse SSH tunnel established from Razer TTY: ssh -R 2222:localhost:22 evanusmodestus@modestus-p16g

06:38

ssh -p 2222 localhost — host key conflict. Stale known_hosts:45 entry for [localhost]:2222 from previous tunnel session.

06:39

ssh-keygen -R '[localhost]:2222' — removed stale key. Retried. Host key accepted.

06:40

Permission denied (publickey,gssapi-with-mic) — authentication failure.

06:41

sudo ssh -p 2222 localhost — attempted root auth. Wrong approach: root’s keypair, not user’s.

06:43

ssh -vvv debug: identity file .ssh/id_ed25519 type -1 — no key files at default paths. Agent contains no identities.

06:45

ls ~/.ssh/*.pub — 16 keys found, all with custom suffixes (_d000, _github, _sk_rk_d000, etc.). None at default id_ed25519 path.

06:48

ssh -p 2222 -i ~/.ssh/id_ed25519_d000 localhost — passphrase accepted, connection established.

06:50

Authenticated on modestus-razer. Incident resolved.

Symptoms

  • ping 10.50.1.106 — 100% packet loss from P16g to Razer

  • ssh modestus-razer — connection timeout (firewall drop)

  • ssh -p 2222 localhost — REMOTE HOST IDENTIFICATION HAS CHANGED warning

  • After key removal: Permission denied (publickey,gssapi-with-mic)

  • ssh -vvv: agent contains no identities, all default key paths return type -1

Investigation

Initial Triage — Network Reachability

# From P16g (10.50.1.x, VLAN 50) to Razer (10.50.1.106, VLAN 10)
ping 10.50.1.106
# Result: 100% packet loss — VyOS drops inter-VLAN traffic VLAN 50→10

VLAN 10 is the restricted data VLAN. VyOS firewall policy denies inbound SSH from VLAN 50 (management) to VLAN 10. This is by design — the restriction is intentional for 802.1X segmentation.

Mitigation — Reverse SSH Tunnel

Since VLAN 10→50 is permitted (Razer can reach P16g), the tunnel is initiated from the allowed direction:

# On modestus-razer (TTY) — establish reverse tunnel
ssh -R 2222:localhost:22 evanusmodestus@modestus-p16g

This binds port 2222 on P16g to port 22 on the Razer. The firewall sees only the outbound SSH from VLAN 10, which is permitted. P16g connects to itself on 2222, traffic forwards through the tunnel to the Razer’s sshd.

modestus-razer:22 ←──tunnel──→ modestus-p16g:2222
      (VLAN 10)                    (VLAN 50)
         └── initiates outbound SSH (allowed) ──→

Failure 1 — Stale Host Key

ssh -p 2222 localhost
# WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
# Offending ECDSA key in known_hosts:45

Previous tunnel session left a [localhost]:2222 entry pointing to a different machine’s host key. StrictHostKeyChecking correctly refused the connection.

Fix:

ssh-keygen -R '[localhost]:2222'

Failure 2 — Root SSH Attempt

sudo ssh -p 2222 localhost
# Permission denied (publickey,gssapi-with-mic)

sudo ssh authenticates as root using root’s SSH keypair. The Razer’s authorized_keys has the user’s public key, not root’s. This was the wrong approach.

Failure 3 — No Keys at Default Paths

ssh -vvv -p 2222 localhost 2>&1 | grep -E 'identity file|agent'
# identity file /home/evanusmodestus/.ssh/id_ed25519 type -1
# agent contains no identities

SSH checks five default key names: id_rsa, id_ecdsa, id_ecdsa_sk, id_ed25519, id_ed25519_sk. None exist. All 16 keys use custom suffixes:

ls ~/.ssh/*.pub
# id_ed25519_bitbucket.pub, id_ed25519_d000.pub, id_ed25519_github.pub,
# id_ed25519_sk_rk_d000.pub, id_ed25519_vault.pub, etc.

The SSH config normally maps these to the correct host via IdentityFile directives in Host blocks. But localhost:2222 matches no configured host — SSH falls back to defaults, finds nothing.

Resolution

ssh -p 2222 -i ~/.ssh/id_ed25519_d000 localhost
# Enter passphrase: [accepted]
# Connected to modestus-razer

Explicitly specifying the machine key (d000 = primary workstation key) bypasses the Host block matching problem.

Root Cause

Three independent failures compounded into a single access denial:

  1. Network layer: VyOS firewall denies VLAN 50→VLAN 10 by design. This is correct behavior — the firewall is working as intended for 802.1X segmentation.

  2. Host key layer: Stale known_hosts entry from a previous reverse tunnel session. StrictHostKeyChecking blocked the connection. This is also correct behavior — SSH detected a host key mismatch.

  3. Authentication layer: All SSH keys use custom-suffixed filenames. The SSH config maps them via Host blocks with IdentityFile directives. localhost:2222 matches no block, so SSH tries only the five default key names, finds none, and fails.

Why it happened:

  • Immediate cause: No Host block matches the reverse tunnel endpoint (localhost:2222)

  • Contributing factor: SSH agent was empty after reboot — no keys cached

  • Contributing factor: 16 keys with unique suffixes means zero default-path keys available as fallback

  • Systemic: Reverse SSH tunnels are not part of the documented runbook for cross-VLAN access

Resolution

Immediate Fix

# 1. Remove stale host key
ssh-keygen -R '[localhost]:2222'

# 2. Connect with explicit identity
ssh -p 2222 -i ~/.ssh/id_ed25519_d000 localhost

Verification

  • SSH session established to modestus-razer

  • Shell prompt confirmed: evanusmodestus@modestus-razer

  • System info banner: Arch Linux, 24 cores, 62.2 GB RAM, 3 LUKS volumes

Impact Assessment

Systems Affected

System Status Impact Duration

modestus-razer (remote management)

Restored

20 minutes

Hyprland (Razer display server)

Degraded

Ongoing — separate investigation

Business Impact

  • Users affected: 1

  • Data loss: No

  • Workaround: Reverse SSH tunnel from the reachable direction

Prevention

Short-term (This Week)

  • Add SSH config block for reverse tunnel access — Evan

    Host razer-tunnel
    HostName localhost
    Port 2222
    User evanusmodestus
    IdentityFile ~/.ssh/id_ed25519_d000

  • Load keys into agent on login — add ssh-add ~/.ssh/id_ed25519_d000 to shell profile or use AddKeysToAgent yes in SSH config — Evan

  • Document reverse tunnel procedure in infrastructure runbook — Evan

Long-term (This Quarter)

  • Evaluate Tailscale as VLAN-agnostic SSH path — bypasses firewall restrictions entirely — Evan

  • Add VyOS rule for P16g→Razer SSH on management VLAN — assess whether VLAN 50→10 SSH should be permitted for admin hosts — Evan

  • Create ~/.ssh/config block for each machine with explicit IdentityFile for direct and tunnel access — Evan

Lessons Learned

What Went Well

  • Reverse SSH tunnel was identified and deployed within 5 minutes

  • StrictHostKeyChecking correctly prevented connection to mismatched host key — security control worked as designed

  • ssh -vvv debug output immediately revealed the empty agent and missing default keys

What Could Be Improved

  • No documented procedure for cross-VLAN SSH access when direct path is blocked

  • SSH agent not auto-loaded after reboot — 16 keys available but none offered

  • sudo ssh was attempted before diagnosing — wasted a step

Key Takeaways

  1. Reverse SSH tunnels bypass firewall restrictions by initiating from the permitted direction. The tunnel endpoint appears as localhost traffic, invisible to the network firewall.

  2. Custom key filenames require explicit -i or SSH config IdentityFile — SSH only auto-discovers five default names. If all keys are suffixed, default lookup finds nothing.

  3. ssh -vvv is the diagnostic — it shows exactly which keys are offered, which agent is consulted, and why auth fails. Run it before guessing.

  4. sudo ssh uses root’s keys — never the right move for user auth. It authenticates as a different identity entirely.

Reverse SSH Tunnel Reference

How It Works

A reverse tunnel (-R) tells the remote sshd to listen on a port and forward traffic back through the SSH connection to the originating machine.

ssh -R [bind_port]:[target_host]:[target_port] [user]@[remote_host]
Component Meaning

-R 2222:localhost:22

On the remote host, listen on port 2222. Forward connections to localhost:22 on the local machine.

evanusmodestus@modestus-p16g

The remote host where port 2222 will be opened.

Persistent Tunnel with autossh

For long-running tunnel sessions, autossh reconnects automatically on network interruption:

# Install
sudo pacman -S autossh

# Persistent reverse tunnel (reconnects on failure)
autossh -M 0 -f -N -R 2222:localhost:22 evanusmodestus@modestus-p16g \
  -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3"

Security Considerations

  • The tunnel is only as secure as the SSH session carrying it. Use key-based auth, not passwords.

  • Port 2222 is bound to localhost on P16g by default — not exposed to the network.

  • To bind to all interfaces (allowing other machines to use the tunnel): -R 0.0.0.0:2222:localhost:22 — requires GatewayPorts yes in the remote sshd_config. Avoid unless necessary.

Metadata

Field Value

Incident ID

INC-2026-04-07-001

Author

Evan Rosado

Created

2026-04-07

Last Updated

2026-04-07

Status

Final

Post-Incident Review

Complete — documented with prevention items