SSH Operations

Overview

This runbook documents the complete SSH infrastructure for Domus Digitalis, including:

  • Multi-tier key hierarchy (Vault CA → YubiKey → Software fallback)

  • systemd-managed SSH agent

  • Post-quantum key exchange configuration

  • Per-host key chains with automatic fallback

Key Architecture

SSH Key Hierarchy
Figure 1. SSH Key Hierarchy

Key Inventory

Key File Type TTL Purpose

id_ed25519_vault

Ed25519 + Cert

8h

Infrastructure hosts via Vault SSH CA

id_ed25519_sk_rk_d000_nano

FIDO2 Resident

∞

YubiKey Nano (always present)

id_ed25519_sk_rk_d000

FIDO2 Resident

∞

Primary YubiKey 5C NFC

id_ed25519_sk_rk_d000_secondary

FIDO2 Resident

∞

Backup YubiKey

id_ed25519_d000

Ed25519

∞

Software fallback (passphrase)

id_ed25519_github

Ed25519

∞

GitHub only

id_ed25519_gitlab

Ed25519

∞

GitLab only

id_ed25519_gitea

Ed25519

∞

Gitea (NAS) only

id_rsa_azure

RSA 4096

∞

Azure DevOps (RSA required)

SSH Config Behavior

Your ~/.ssh/config has these key settings that control agent behavior:

Key Config Settings (from ~/.ssh/config)
AddKeysToAgent 1h      # Auto-add keys to agent for 1 hour on first use
IdentitiesOnly yes     # ONLY try keys explicitly listed for each host
ControlMaster auto     # Enable connection multiplexing
ControlPersist 600     # Keep master connection alive 10 minutes

AddKeysToAgent Explained

When you SSH to a host, OpenSSH:

  1. Reads your config to find which IdentityFile to use

  2. Prompts for passphrase (if key is encrypted)

  3. Automatically adds the key to the agent for the duration specified

Table 1. Behavior Matrix
Scenario Passphrase Prompt? Key Lifetime in Agent

First SSH to github.com

Yes (once)

1 hour (from AddKeysToAgent 1h)

Second SSH to github.com (within 1h)

No

Remaining time from first add

Manual ssh-add -t 1d ~/.ssh/id_ed25519_github

Yes (once)

1 day (overrides config)

After 1h expires, SSH again

Yes (once)

1 hour (re-added automatically)

When to Use Manual ssh-add

Use manual ssh-add -t when you want a longer lifetime than the default 1 hour:

# Work session - extend to full day
ssh-add -t 1d ~/.ssh/id_ed25519_github

# Infrastructure work - match Vault cert TTL
ssh-add -t 8h ~/.ssh/id_ed25519_d000

The -t duration overrides the AddKeysToAgent 1h setting for that key.

IdentitiesOnly Explained

With IdentitiesOnly yes, SSH will only try keys explicitly listed in config:

# For github.com, ONLY try id_ed25519_github
Host github.com
    IdentityFile ~/.ssh/id_ed25519_github

# For vault-01, try these in order until one works
Host vault-01
    IdentityFile ~/.ssh/id_ed25519_vault
    CertificateFile ~/.ssh/id_ed25519_vault-cert.pub
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano
    IdentityFile ~/.ssh/id_ed25519_sk_rk_d000
    IdentityFile ~/.ssh/id_ed25519_d000

Without IdentitiesOnly yes, SSH would try ALL keys in the agent, causing:

  • Unnecessary auth attempts

  • Possible lockouts (too many failed attempts)

  • Slower connections

SSH Agent Management

systemd User Service

The SSH agent runs as a systemd user service, ensuring it starts at login and persists across terminals.

Service File Location
~/.config/systemd/user/ssh-agent.service
Service Status
systemctl --user status ssh-agent
Restart Agent
systemctl --user restart ssh-agent
Check Socket
ls -la /run/user/$(id -u)/ssh-agent.socket

Adding Keys to Agent

Add key with 1-day lifetime
ssh-add -t 1d ~/.ssh/id_ed25519_github
Add key with 8-hour lifetime (matches Vault cert TTL)
ssh-add -t 8h ~/.ssh/id_ed25519_d000
Add all Git service keys for extended session
for key in github gitlab codeberg bitbucket gitea; do
  ssh-add -t 1d ~/.ssh/id_ed25519_$key 2>/dev/null && echo "Added: $key"
done
Add Vault key with cert
ssh-add -t 8h ~/.ssh/id_ed25519_vault

The Vault cert (id_ed25519_vault-cert.pub) is automatically associated when the key is added.

Listing Loaded Keys

Show fingerprints
ssh-add -l
Show key comments (more readable)
ssh-add -l | awk '{print $3}'
Count loaded keys
ssh-add -l | wc -l

Removing Keys

Remove specific key
ssh-add -d ~/.ssh/id_ed25519_github
Remove ALL keys (nuclear option)
ssh-add -D

Vault SSH CA Operations

Certificate Renewal

The Vault SSH CA issues 8-hour certificates. Renew with vault-ssh-sign:

vault-ssh-sign
Manual renewal (if script unavailable)
vault write -field=signed_key ssh/sign/domus-client \
  public_key=@$HOME/.ssh/id_ed25519_vault.pub \
  valid_principals="ansible,evanusmodestus,root" \
  >| ~/.ssh/id_ed25519_vault-cert.pub

Verify Certificate

Check validity and extensions
ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub
Extract just validity period
ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub | grep -E "Valid:|Serial:"
Check if expired
ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub | awk '/Valid:/{
  split($5, t, "T");
  print "Expires:", t[1], substr(t[2],1,5)
}'

Reload After Renewal

After renewing the cert, reload it into the agent:

# Remove old
ssh-add -d ~/.ssh/id_ed25519_vault 2>/dev/null

# Add fresh
ssh-add -t 8h ~/.ssh/id_ed25519_vault

YubiKey Operations

Regenerate Resident Keys

If you need to recover keys from YubiKey:

cd ~/.ssh
ssh-keygen -K

This extracts all resident keys from the inserted YubiKey.

Test YubiKey Key

# Requires touch
ssh -i ~/.ssh/id_ed25519_sk_rk_d000 -o IdentitiesOnly=yes user@host

Multiple YubiKeys

The SSH config specifies fallback order:

IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_nano   # Try Nano first
IdentityFile ~/.ssh/id_ed25519_sk_rk_d000        # Then 5C NFC
IdentityFile ~/.ssh/id_ed25519_sk_rk_d000_secondary  # Then backup

The Nano is always plugged in, so it’s tried first. Keep backup YubiKey in safe location.

Git Forge Operations

Per-Service Key Design

Each Git service has a dedicated key to:

  1. Isolate compromise (one key leak doesn’t affect others)

  2. Allow per-service key rotation

  3. Track usage per service

Add Git keys for work session
ssh-add -t 1d ~/.ssh/id_ed25519_github
ssh-add -t 1d ~/.ssh/id_ed25519_gitlab

Test Connection

ssh -T git@github.com
ssh -T git@gitlab.com
ssh -T git@codeberg.org

Azure DevOps (RSA Required)

Azure DevOps doesn’t support Ed25519 as of 2025. Use RSA:

ssh-add -t 1d ~/.ssh/id_rsa_azure
ssh -T git@ssh.dev.azure.com

Connection Multiplexing

How It Works

The config enables ControlMaster with persistent sockets:

ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600

First SSH to a host creates a master connection. Subsequent SSHs reuse it (instant connect, no re-auth).

Check Active Sockets

ls -la ~/.ssh/sockets/
Show socket details with age
find ~/.ssh/sockets/ -type s -printf "%f\t%T+\n" | sort -k2

Force New Connection

If you need a fresh connection (not via master):

ssh -o ControlMaster=no host

Close Master Socket

ssh -O exit host

Troubleshooting

"Permission denied (publickey)"

Diagnostic steps
# 1. Check agent has keys
ssh-add -l

# 2. Test with verbose
ssh -vvv user@host 2>&1 | grep -E "Offering|Trying|Authentication"

# 3. Check key permissions
ls -la ~/.ssh/id_ed25519_*

# 4. Verify correct key for host
ssh -G host | grep -E "identityfile|user|hostname"

"Certificate invalid: not yet valid"

Clock skew between client and server.

Fix on server
sudo systemctl enable --now systemd-timesyncd
timedatectl show | awk -F= '/NTPSynchronized/{print}'

Agent Not Running

# Check status
systemctl --user status ssh-agent

# Restart
systemctl --user restart ssh-agent

# Verify socket
echo $SSH_AUTH_SOCK
ls -la /run/user/$(id -u)/ssh-agent.socket

Wrong Key Used

The IdentitiesOnly yes setting forces SSH to only try keys explicitly listed for that host.

Debug which key SSH tries
ssh -vvv host 2>&1 | grep "Offering public key"

Vault Cert Expired

# Check expiry
ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub | grep Valid

# Renew
vault-ssh-sign

# Reload agent
ssh-add -d ~/.ssh/id_ed25519_vault
ssh-add -t 8h ~/.ssh/id_ed25519_vault

YubiKey Not Detected

# Check YubiKey presence
ykman info

# List SSH-capable keys on YubiKey
ykman fido credentials list

Quick Reference

Table 2. Essential Commands
Action Command

List loaded keys

ssh-add -l

Add key (1 day)

ssh-add -t 1d ~/.ssh/id_ed25519_github

Remove key

ssh-add -d ~/.ssh/id_ed25519_github

Remove ALL keys

ssh-add -D

Renew Vault cert

vault-ssh-sign

Check cert validity

ssh-keygen -Lf ~/.ssh/id_ed25519_vault-cert.pub

Reload Vault key

ssh-add -d ~/.ssh/id_ed25519_vault && ssh-add -t 8h ~/.ssh/id_ed25519_vault

Restart agent

systemctl --user restart ssh-agent

Debug connection

ssh -vvv host 2>&1 | grep Auth

Close multiplex socket

ssh -O exit host

Recover YubiKey keys

cd ~/.ssh && ssh-keygen -K

Daily Workflow

Morning Startup

# 1. Renew Vault cert
vault-ssh-sign

# 2. Add Vault key
ssh-add -t 8h ~/.ssh/id_ed25519_vault

# 3. Add Git keys for push operations
ssh-add -t 1d ~/.ssh/id_ed25519_github
ssh-add -t 1d ~/.ssh/id_ed25519_gitlab

# 4. Verify
ssh-add -l

Before Extended AFK

No action needed - keys auto-expire at set TTL.

Returning to Work

# Check what's still loaded
ssh-add -l

# If empty, re-add
vault-ssh-sign
ssh-add -t 8h ~/.ssh/id_ed25519_vault
ssh-add -t 1d ~/.ssh/id_ed25519_github

Post-Quantum Security

The SSH config enables hybrid post-quantum key exchange:

KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512@openssh.com,...

This provides protection against future quantum computers while maintaining compatibility with current systems.

Post-quantum KEX requires OpenSSH 9.0+ (ideally 9.5+). Older servers will negotiate down to curve25519.

Related Pages