Runbook: YubiKey SSH Validation

Last Updated

2026-02-19

Owner

evanusmodestus

Frequency

Monthly / After key changes

Purpose

Validate that ALL FOUR YubiKeys can authenticate to all critical infrastructure via SSH. This ensures disaster recovery capability across multiple failure scenarios.

YubiKey Hierarchy

Tier Key Location Use Case

Primary

YubiKey 5C NFC

Keychain

Daily operations

Secondary

YubiKey 5C NFC

Safe/drawer

Hot backup if primary lost

Tertiary

YubiKey 5C Nano (Serial: 35641207)

Plugged into laptop (always in)

Daily driver, convenience auth

Quaternary

YubiKey (touch)

Off-site

Disaster recovery

Key Files

Key Path Type

Primary

~/.ssh/id_ed25519_sk_rk_d000

FIDO2 resident

Secondary

~/.ssh/id_ed25519_sk_rk_d000_secondary

FIDO2 resident

Tertiary (Nano)

~/.ssh/id_ed25519_sk_rk_d000_nano

FIDO2 resident (35641207)

Quaternary (Touch)

~/.ssh/id_ed25519_sk_rk_d000_quaternary

FIDO2 resident

Software fallback

~/.ssh/id_ed25519_d000

Ed25519 (passphrase)

Infrastructure Assets

Linux VMs (Host A - Production)

  • vault-01 (10.50.1.60) - Vault primary, cert management

  • ise-01 (10.50.1.20) - ISE PAN (CLI access)

  • keycloak-01 (10.50.1.80) - SSO server

  • home-dc01 (10.50.1.50) - AD domain controller

  • ipsk-mgr-01 (10.50.1.30) - iPSK manager primary

  • nas-01 (10.50.1.70) - Synology NAS

  • gitea-01 (10.50.1.72) - Git server

  • bind-01 (10.50.1.90) - DNS primary

Linux VMs (Host B - Standby)

  • vault-02 (10.50.1.61) - Vault standby

  • ise-02 (10.50.1.21) - ISE PSN

  • ipsk-mgr-02 (10.50.1.31) - iPSK manager standby

  • nas-02 (10.50.1.71) - NAS replica

  • bind-02 (10.50.1.91) - DNS secondary

Network Devices

  • vyos-01 (10.50.1.2) - VyOS router (primary)

  • vyos-02 (10.50.1.3) - VyOS router (secondary)

  • 9800-wlc-01 (10.50.1.40) - Wireless LAN controller (IOS-XE)

  • Home-3560CX-01 (10.50.1.10) - Access switch (IOS)

  • Home-C9300-01 (10.50.1.11) - Distribution switch (IOS-XE)

Hypervisor & Hardware

  • kvm-01 (10.50.1.110) - KVM hypervisor

  • kvm-01-lan (192.168.1.181) - KVM LAN interface

  • ipmi-01 (10.50.1.200) - IPMI BMC (if SSH enabled)

Procedure

Step 1: Validate Primary YubiKey

Insert PRIMARY YubiKey (keychain) and run validation script:

#!/usr/bin/env bash
# SSH Validation - Primary YubiKey
set -euo pipefail

KEY_LABEL="PRIMARY"
RESULTS=()

validate_host() {
    local host="$1"
    local ip="$2"
    local type="${3:-linux}"

    echo -n "Testing $host ($ip)... "

    if [ "$type" = "cisco" ]; then
        # Cisco devices - just test connection
        if timeout 5 ssh -o BatchMode=yes -o ConnectTimeout=3 "$host" "show version | i uptime" &>/dev/null; then
            echo "[$KEY_LABEL OK]"
            RESULTS+=("PASS: $host")
        else
            echo "[FAILED]"
            RESULTS+=("FAIL: $host")
        fi
    else
        # Linux hosts
        if timeout 5 ssh -o BatchMode=yes -o ConnectTimeout=3 "$host" "hostname" &>/dev/null; then
            echo "[$KEY_LABEL OK]"
            RESULTS+=("PASS: $host")
        else
            echo "[FAILED]"
            RESULTS+=("FAIL: $host")
        fi
    fi
}

echo "=== SSH Validation: $KEY_LABEL YubiKey ==="
echo "Touch YubiKey when prompted"
echo ""

# Linux VMs - Host A
validate_host "vault-01" "10.50.1.60"
validate_host "keycloak-01" "10.50.1.80"
validate_host "ipsk-mgr-01" "10.50.1.30"
validate_host "nas-01" "10.50.1.70"
validate_host "gitea-01" "10.50.1.72"
validate_host "bind-01" "10.50.1.90"

# Linux VMs - Host B
validate_host "vault-02" "10.50.1.61"
validate_host "ipsk-mgr-02" "10.50.1.31"
validate_host "nas-02" "10.50.1.71"
validate_host "bind-02" "10.50.1.91"

# Hypervisor
validate_host "kvm-01" "10.50.1.110"

# Network Devices
validate_host "vyos-01" "10.50.1.2"
validate_host "vyos-02" "10.50.1.3"
validate_host "Home-3560CX-01" "10.50.1.10" "cisco"
validate_host "Home-C9300-01" "10.50.1.11" "cisco"
validate_host "9800-wlc-01" "10.50.1.40" "cisco"

echo ""
echo "=== Summary ==="
printf '%s\n' "${RESULTS[@]}"

Save as ~/bin/ssh-validate.sh and run:

chmod +x ~/bin/ssh-validate.sh
~/bin/ssh-validate.sh

Step 2: Validate Secondary YubiKey

Remove primary, insert SECONDARY YubiKey (from safe/drawer):

# Quick validation - secondary key
for host in vault-01 nas-01 kvm-01 vyos-01; do
    echo -n "$host: "
    ssh -o BatchMode=yes -o ConnectTimeout=3 "$host" "hostname" 2>/dev/null && echo "OK" || echo "FAILED"
done

Step 3: Validate Tertiary YubiKey (Nano)

Insert TERTIARY YubiKey Nano (stays plugged in):

# Nano should already be plugged in
ykman list  # Should show Nano

# Quick validation
for host in vault-01 nas-01 kvm-01 vyos-01; do
    echo -n "$host: "
    ssh -o BatchMode=yes -o ConnectTimeout=3 "$host" "hostname" 2>/dev/null && echo "OK" || echo "FAILED"
done

Step 4: Validate Quaternary YubiKey (Off-site)

When off-site key is available for validation:

# Insert quaternary key
for host in vault-01 nas-01 kvm-01 vyos-01; do
    echo -n "$host: "
    ssh -o BatchMode=yes -o ConnectTimeout=3 "$host" "hostname" 2>/dev/null && echo "OK" || echo "FAILED"
done

Step 5: Validate Software Fallback

Without any YubiKey inserted:

# Test software key explicitly
for host in vault-01 nas-01 kvm-01; do
    echo -n "$host (software): "
    ssh -i ~/.ssh/id_ed25519_d000 -o IdentitiesOnly=yes -o ConnectTimeout=3 "$host" "hostname" 2>/dev/null && echo "OK" || echo "FAILED"
done

SSH Config Template

Ensure ~/.ssh/config has proper IdentityFile order:

# =============================================================================
# INFRASTRUCTURE - Linux VMs
# =============================================================================

Host vault-01
    HostName 10.50.1.60
    User ansible
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_tertiary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_quaternary
    IdentityFile ~/.ssh/id_ed25519_d000

Host nas-01
    HostName 10.50.1.70
    User evanusmodestus
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_tertiary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_quaternary
    IdentityFile ~/.ssh/id_ed25519_d000

Host kvm-01
    HostName 10.50.1.110
    User root
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_tertiary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_quaternary
    IdentityFile ~/.ssh/id_ed25519_d000

# =============================================================================
# NETWORK DEVICES
# =============================================================================

Host vyos-01
    HostName 10.50.1.2
    User vyos
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_tertiary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_quaternary
    IdentityFile ~/.ssh/id_ed25519_d000

Host vyos-02
    HostName 10.50.1.3
    User vyos
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_tertiary
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_quaternary
    IdentityFile ~/.ssh/id_ed25519_d000

Host Home-3560CX-01
    HostName 10.50.1.10
    User admin
    KexAlgorithms +diffie-hellman-group14-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa
    IdentityFile ~/.ssh/id_ed25519_d000

Host Home-C9300-01
    HostName 10.50.1.11
    User admin
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_d000

Host 9800-wlc-01
    HostName 10.50.1.40
    User admin
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_d000
Older Cisco IOS devices may not support Ed25519. Use RSA keys or password auth as fallback.

Network Device Setup

Cisco IOS/IOS-XE SSH Key Import

! Generate RSA keys (if not done)
crypto key generate rsa modulus 4096

! Enable SSH v2
ip ssh version 2

! Create local user with privilege 15
username admin privilege 15 secret <password>

! Enable SSH authentication
line vty 0 15
 login local
 transport input ssh

! Import public key (from workstation)
ip ssh pubkey-chain
 username admin
  key-string
   <paste-public-key-here>
  exit
 exit

Get your public key:

cat ~/.ssh/id_ed25519_d000.pub
# Or for YubiKey:
cat ~/.ssh/id_ed25519_sk_rk_d000.pub

VyOS SSH Key Setup

VyOS uses standard Linux SSH configuration:

# SSH to VyOS
ssh vyos@vyos-01

# Configure SSH key in VyOS config mode
configure
set system login user vyos authentication public-keys primary type ssh-ed25519
set system login user vyos authentication public-keys primary key <your-key-data>
set system login user vyos authentication public-keys secondary type sk-ssh-ed25519@openssh.com
set system login user vyos authentication public-keys secondary key <secondary-key-data>
commit
save
exit

Or copy keys via ssh-copy-id from workstation:

ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000.pub vyos@vyos-01
ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000_secondary.pub vyos@vyos-01

Synology NAS SSH Key Setup

  1. Enable SSH in Control Panel → Terminal & SNMP

  2. SSH to NAS with password first:

    ssh evanusmodestus@nas-01
mkdir -p ~/.ssh
chmod 700 ~/.ssh

  3. From workstation, copy keys:

ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000.pub evanusmodestus@nas-01
ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000_secondary.pub evanusmodestus@nas-01
ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000_tertiary.pub evanusmodestus@nas-01
ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000_quaternary.pub evanusmodestus@nas-01
ssh-copy-id -i ~/.ssh/id_ed25519_d000.pub evanusmodestus@nas-01

Verification Checklist

  • Primary YubiKey works on all Linux VMs

  • Primary YubiKey works on all network devices

  • Secondary YubiKey works on critical hosts (vault, nas, kvm, pfsense)

  • Tertiary YubiKey Nano works on critical hosts

  • Quaternary YubiKey works on critical hosts

  • Software key works as fallback

  • No hosts failed authentication

Troubleshooting

Permission Denied

Symptom: Permission denied (publickey)

Diagnosis:

# Verbose SSH to see which keys are tried
ssh -vvv vault-01 2>&1 | grep -E "(Offering|Trying|Authentication)"

Resolution:

# Export resident keys from YubiKey
ssh-keygen -K

# Copy to remote host
ssh-copy-id -i ~/.ssh/id_ed25519_sk_rk_d000.pub user@host

YubiKey Not Detected

Symptom: No such device or agent refused operation

Diagnosis:

# List connected YubiKeys
ykman list

# Check FIDO2 credentials
ykman fido credentials list

Resolution: Re-insert YubiKey, ensure FIDO2 application is enabled:

ykman config usb --enable FIDO2

Cisco Device Rejects Ed25519

Symptom: no matching key type found on older IOS

Resolution: Use RSA key for legacy devices:

# Generate RSA key if not exists
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_cisco

# Add to SSH config for legacy devices
Host Home-3560CX-01
    HostName 10.50.1.10
    User admin
    IdentityFile ~/.ssh/id_rsa_cisco
    KexAlgorithms +diffie-hellman-group14-sha1
    HostKeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa

Wrong Key Used

Symptom: SSH tries keys in wrong order, prompts for touch when shouldn’t

Resolution: Check ~/.ssh/config IdentityFile order. First listed = first tried.

For hosts where YubiKey is NOT configured, use software key first:

Host legacy-server
    IdentityFile ~/.ssh/id_ed25519_d000
    IdentitiesOnly yes

Export All Public Keys

For adding to new hosts:

echo "=== Software Key (Ed25519) ==="
cat ~/.ssh/id_ed25519_d000.pub

echo ""
echo "=== Primary YubiKey ==="
cat ~/.ssh/id_ed25519_sk_rk_d000.pub

echo ""
echo "=== Secondary YubiKey ==="
cat ~/.ssh/id_ed25519_sk_rk_d000_secondary.pub

echo ""
echo "=== Tertiary YubiKey (Nano) ==="
cat ~/.ssh/id_ed25519_sk_rk_d000_tertiary.pub

echo ""
echo "=== Quaternary YubiKey ==="
cat ~/.ssh/id_ed25519_sk_rk_d000_quaternary.pub

Recovery: Adding Keys to New Host

# One-liner to add all 5 keys to new host
for key in id_ed25519_d000 id_ed25519_sk_rk_d000 id_ed25519_sk_rk_d000_secondary id_ed25519_sk_rk_d000_tertiary id_ed25519_sk_rk_d000_quaternary; do
    [ -f ~/.ssh/${key}.pub ] && ssh-copy-id -i ~/.ssh/${key}.pub user@newhost
done

Notes

  • Tertiary YubiKey Nano should remain plugged in for convenience operations

  • Run full validation monthly and after any YubiKey changes

  • Store quaternary YubiKey off-site (different physical location)

  • Software key is emergency fallback only - prefer hardware keys