Vault Troubleshooting Runbook

Overview

vault-01.inside.domusdigitalis.dev (10.50.1.60) is the central certificate management server running:

HashiCorp Vault PKI - Internal certificates (EAP-TLS, servers)
Certbot - Let’s Encrypt certificates via Cloudflare DNS-01

Vault is currently a Single Point of Failure. See Vault Enterprise Hardening for HA cluster deployment.

Architecture Reference

See Infrastructure Diagrams:

Certbot Renewal Flow - Full renewal process with error states
PKI Hierarchy - Certificate authority chain
Infrastructure Overview - System topology

Current Certificates

# List all managed certificates
sudo certbot certificates

Table 1. Certificates as of 2026-01-24
Certificate Name	Domains	Days Left	Status
9800-wlc-01.inside.domusdigitalis.dev	9800-wlc-01.inside.domusdigitalis.dev	~90	OK
guest.domusdigitalis.dev	guest.domusdigitalis.dev	~90	BROKEN - Path issue
ipmi-01.inside.domusdigitalis.dev	ipmi-01.inside.domusdigitalis.dev	~90	OK
kvm-01.inside.domusdigitalis.dev	kvm-01.inside.domusdigitalis.dev	~90	BROKEN - DNS NXDOMAIN

Vault Sealed After Restart/Disruption

Vault seals automatically when restarted, when VM is disrupted, or after power loss. This is by design (security). You must unseal it manually.

Symptoms

vault-ssh-sign fails with 503 Vault is sealed
SSH falls back to passphrase or YubiKey (no Vault cert)
Any Vault API call fails

Cluster Details

Property	Value
Storage Type	file
Cluster Name	vault-cluster-904d4b42
Cluster ID	aff339e0-7b48-58ce-cb35-0ae05a230c97
HA Enabled	false (single node)
Threshold	2 of 3 unseal keys required

Property

Value

Storage Type

file

Cluster Name

vault-cluster-904d4b42

Cluster ID

aff339e0-7b48-58ce-cb35-0ae05a230c97

HA Enabled

false (single node)

Threshold

2 of 3 unseal keys required

Check Vault Status

vault status

Sealed output:

Sealed          true
Unseal Progress 0/2

Unseal Vault

Need 2 unseal keys (threshold 2):

vault operator unseal $(gopass show -o v3/domains/d000/vault/unseal-key-1)

vault operator unseal $(gopass show -o v3/domains/d000/vault/unseal-key-2)

vault status

Unsealed output:

Sealed          false

Post-Unseal: Refresh SSH Cert

vault-ssh-sign

vault-ssh-test

Prevention: Auto-Unseal (Future)

Phase 6 of Vault Enterprise Hardening will implement auto-unseal using:

AWS KMS, or
Azure Key Vault, or
Transit secrets engine on another Vault

Until then, manual unseal is required after any Vault restart.

Known Issues and Fixes

CRITICAL: guest.domusdigitalis.dev - Wrong Cloudflare Path

Error Message

Attempting to renew cert (guest.domusdigitalis.dev) from /etc/letsencrypt/renewal/guest.domusdigitalis.dev.conf produced an unexpected error:
dns_cloudflare_credentials: /root/.secrets/cloudflare.ini: No such file or directory

Root Cause

The certificate was configured with credentials path /root/.secrets/cloudflare.ini but credentials are stored at /home/ansible/.secrets/cloudflare.ini.

Fix Steps

# 1. Check current config
sudo cat /etc/letsencrypt/renewal/guest.domusdigitalis.dev.conf | grep cloudflare

# 2. Edit the renewal config
sudo vim /etc/letsencrypt/renewal/guest.domusdigitalis.dev.conf

# 3. Change:
#    dns_cloudflare_credentials = /root/.secrets/cloudflare.ini
# To:
#    dns_cloudflare_credentials = /home/ansible/.secrets/cloudflare.ini

# 4. Test renewal
sudo certbot renew --cert-name guest.domusdigitalis.dev --dry-run

# 5. If successful, force renewal
sudo certbot renew --cert-name guest.domusdigitalis.dev --force-renewal

Prevention

Always use absolute path to ansible user’s secrets:

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /home/ansible/.secrets/cloudflare.ini \
  -d guest.domusdigitalis.dev

CRITICAL: kvm-01.inside.domusdigitalis.dev - DNS NXDOMAIN

Error Message

Encountered NXDOMAIN when looking up TXT record: _acme-challenge.kvm-01.inside.domusdigitalis.dev

Root Cause

DNS TXT record _acme-challenge.kvm-01.inside.domusdigitalis.dev does not exist in Cloudflare zone, OR DNS propagation timing is too short.

Fix Option 1: Increase Propagation Time

# Edit renewal config
sudo vim /etc/letsencrypt/renewal/kvm-01.inside.domusdigitalis.dev.conf

# Add or increase:
dns_cloudflare_propagation_seconds = 60

Fix Option 2: Verify Cloudflare Zone Permissions

# Check API token permissions
# Token needs: Zone:DNS:Edit for domusdigitalis.dev

# List zones accessible by token
curl -X GET "https://api.cloudflare.com/client/v4/zones" \
  -H "Authorization: Bearer $(cat /home/ansible/.secrets/cloudflare.ini | grep api_token | cut -d= -f2 | tr -d ' ')" \
  -H "Content-Type: application/json"

Fix Option 3: Use Wildcard Instead

# Consider using wildcard for *.inside.domusdigitalis.dev
# This reduces individual cert management

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /home/ansible/.secrets/cloudflare.ini \
  -d "*.inside.domusdigitalis.dev" \
  -d "inside.domusdigitalis.dev"

Deploy Hook Failures

Location

/etc/letsencrypt/renewal-hooks/deploy/deploy-certs.sh

Table 2. Common Issues
Issue	Symptom	Fix
ISE API unreachable	`curl: (7) Failed to connect to ise-01.inside.domusdigitalis.dev`	Check ISE is up, verify IP in secrets
age decryption failed	`age: error: no identity matched`	Verify age key in `~/.config/age/`
iPSK SSH failed	`ssh: connect to host ipsk-mgr-01.inside.domusdigitalis.dev port 22: Connection refused`	Check iPSK VM is running, SSH key authorized
Missing environment variable	`ISE_API_TOKEN: unbound variable`	Secrets not loaded, check age file path

Debug Deploy Hook

# Run deploy hook manually with debug
sudo bash -x /etc/letsencrypt/renewal-hooks/deploy/deploy-certs.sh

# Check what certificates triggered the hook
echo $RENEWED_LINEAGE
echo $RENEWED_DOMAINS

Systemd Timer Status

# Check timer status
systemctl status certbot-renew.timer

# Check recent runs
journalctl -u certbot-renew.service -n 50

# Force a renewal check
sudo systemctl start certbot-renew.service

Expected Timer Output

certbot-renew.timer - Run certbot twice daily
   Loaded: loaded (/lib/systemd/system/certbot-renew.timer; enabled)
   Active: active (waiting)
  Trigger: [next trigger time]

Full Renewal Test

# Dry-run all certificates
sudo certbot renew --dry-run

# Expected: All simulated renewals succeeded
# If failures, check each cert individually

Emergency Procedures

Certificate About to Expire

# Force renewal immediately
sudo certbot renew --force-renewal

# If renewal fails, issue new cert
sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /home/ansible/.secrets/cloudflare.ini \
  -d [domain]

Deploy Hook Not Running

# Manually deploy after renewal
export RENEWED_LINEAGE="/etc/letsencrypt/live/[cert-name]"
export RENEWED_DOMAINS="[domain]"
sudo -E /etc/letsencrypt/renewal-hooks/deploy/deploy-certs.sh

ISE Guest Portal Cert Expired

# 1. Renew cert
sudo certbot renew --cert-name guest.domusdigitalis.dev --force-renewal

# 2. Manual deployment to ISE if hook fails
# Use netapi or direct API call
source <(dsec source d000 dev/network)
curl -sk -X POST "https://$<ise-pan-ip>/api/v1/certs/system-certificate/import" \
  -H "Authorization: Basic $<ise-api-token>" \
  -H "Content-Type: application/json" \
  -d @- << EOF
{
  "admin": false,
  "eap": false,
  "portal": true,
  "portalGroupTag": "Default Portal Certificate Group",
  "data": "$(base64 -w0 /etc/letsencrypt/live/guest.domusdigitalis.dev/fullchain.pem)",
  "privateKeyData": "$(base64 -w0 /etc/letsencrypt/live/guest.domusdigitalis.dev/privkey.pem)"
}
EOF

Long-Term Solutions

Priority 1: Fix Current Failures

Fix guest.domusdigitalis.dev cloudflare path
Fix kvm-01.inside.domusdigitalis.dev DNS propagation/NXDOMAIN
Verify all certs renew with --dry-run

Priority 2: Reduce Let’s Encrypt Dependency

Deploy HashiCorp Vault as intermediate CA
Issue internal certs from Vault (ISE admin, EAP-TLS)
Keep Let’s Encrypt only for public-facing (guest portal)

Priority 3: Redundancy

Backup /etc/letsencrypt to nas-01.inside.domusdigitalis.dev
Document manual cert issuance procedures
Consider secondary certbot server