Vault PKI Verification Runbook

1. Overview

This runbook provides comprehensive verification procedures for the DOMUS HashiCorp Vault PKI infrastructure. Use this for:

Daily health checks
Post-reboot verification
Troubleshooting certificate issuance issues
Preparing for Windows EAP-TEAP and BYOD deployments

Vault runs on HTTP (not HTTPS) on certmgr-01

Always set VAULT_ADDR='http://127.0.0.1:8200' before running vault commands on certmgr-01.

If you see this error:

Error checking seal status: Get "https://127.0.0.1:8200/v1/sys/seal-status":
http: server gave HTTP response to HTTPS client

You forgot to set VAULT_ADDR!

2. Quick Start

# SSH to Vault server
ssh certmgr-01

# Set VAULT_ADDR (REQUIRED!)
export VAULT_ADDR='http://127.0.0.1:8200'

# Check Vault status
vault status

3. Architecture Summary

DOMUS-ROOT-CA (Vault pki/, 20 years: 2026-2046)
    ├── Serial: 10:E0:16:3C:7C:35:09:14:E2:9C:5B:78:D7:31:F1:50:1D:B6:55:83
    ├── Algorithm: RSA 4096
    │
    └── DOMUS-ISSUING-CA (Vault pki_int/, 5 years: 2026-2031)
        ├── Serial: 3F:83:98:35:4C:D7:8D:E5:99:3C:B1:94:7D:9C:BB:19:67:3F:10:7D
        ├── Algorithm: RSA 4096
        │
        └── End-Entity Certificates
            ├── Servers (domus-server role, 1 year max)
            ├── Clients (TBD: domus-client role)
            ├── BYOD (TBD: domus-byod role)
            └── Windows (TBD: domus-windows-machine, domus-windows-user)

4. Section 1: Vault Status & Unsealing

4.1. Check Vault Status

# SSH to certmgr-01
ssh certmgr-01

# Set Vault address
export VAULT_ADDR='http://127.0.0.1:8200'

# Check status
vault status

Expected Output (Unsealed)

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.21.2
Storage Type    file
Cluster Name    vault-cluster-904d4b42
Cluster ID      8a9c3c77-5b4f-3e92-0d38-1f5e7a6b2c9d
HA Enabled      false

Expected Output (Sealed - Needs Unsealing)

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          true      ← SEALED!
Total Shares    5
Threshold       3
Unseal Progress 0/3

4.2. Unseal Vault (If Sealed)

Vault seals after reboot or manual seal operations. You need 3 unseal keys (threshold) to unseal.

Unseal keys are stored in:

dsec vault: d000/dev/vault (on local workstation)
Age-encrypted backup: ~/.secrets/certs/vault-unseal-keys.age

4.2.1. Method 1: Manual Unseal (on certmgr-01)

export VAULT_ADDR='http://127.0.0.1:8200'

# Apply first unseal key (will prompt)
vault operator unseal
# Enter key 1: <paste key>

# Apply second unseal key
vault operator unseal
# Enter key 2: <paste key>

# Apply third unseal key (threshold met)
vault operator unseal
# Enter key 3: <paste key>

# Verify unsealed
vault status | grep Sealed
# Expected: Sealed          false

4.2.2. Method 2: Auto-Unseal via netapi (from local workstation)

# On modestus-razer (your workstation)
dsource d000 dev/vault
netapi vault unseal --auto

# Verify
netapi vault status

netapi vault unseal --auto reads VAULT_UNSEAL_KEY_1, VAULT_UNSEAL_KEY_2, VAULT_UNSEAL_KEY_3 from dsec and automatically applies them.

5. Section 2: PKI Secrets Engines

5.1. List Secrets Engines

# On certmgr-01
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'  # Get from dsec or use root token

vault secrets list -detailed

Expected Output

Path          Type         Description
----          ----         -----------
cubbyhole/    cubbyhole    per-token private secret storage
identity/     identity     identity store
pki/          pki          DOMUS-ROOT-CA (20 years)
pki_int/      pki          DOMUS-ISSUING-CA (5 years)
secret/       kv           key/value secret storage
sys/          system       system endpoints used for control, policy and debugging

Required Engines:

✅ pki/ - Root CA
✅ pki_int/ - Issuing/Intermediate CA

If missing, Vault PKI is not configured!

5.2. Verify PKI Engine Configuration

# Check root CA max TTL
vault read sys/mounts/pki/tune
# Expected: max_lease_ttl: 175200h (20 years)

# Check intermediate CA max TTL
vault read sys/mounts/pki_int/tune
# Expected: max_lease_ttl: 43800h (5 years)

6. Section 3: Root CA (pki/)

6.1. Read Root CA Certificate

# On certmgr-01
export VAULT_ADDR='http://127.0.0.1:8200'

# Get root CA certificate
vault read pki/cert/ca -format=json | jq -r '.data.certificate' | openssl x509 -noout -text | grep -E "Subject:|Issuer:|Not Before|Not After|Serial Number"

Expected Output

        Serial Number:
            10:e0:16:3c:7c:35:09:14:e2:9c:5b:78:d7:31:f1:50:1d:b6:55:83
        Issuer: C=US, O=Domus Digitalis, OU=Enterprise PKI, CN=DOMUS-ROOT-CA
        Subject: C=US, O=Domus Digitalis, OU=Enterprise PKI, CN=DOMUS-ROOT-CA
            Not Before: Jan 25 06:57:11 2026 GMT
            Not After : Jan 20 06:57:41 2046 GMT  ← 20 years

Root CA is self-signed: Subject == Issuer

6.2. Check Root CA URLs

vault read pki/config/urls

Expected Output

Key                        Value
---                        -----
crl_distribution_points    [http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki/crl]
issuing_certificates       [http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki/ca]
ocsp_servers               []

7. Section 4: Intermediate CA (pki_int/)

7.1. Read Intermediate CA Certificate

# Get intermediate CA certificate
vault read pki_int/cert/ca -format=json | jq -r '.data.certificate' | openssl x509 -noout -text | grep -E "Subject:|Issuer:|Not Before|Not After|Serial Number"

Expected Output

        Serial Number:
            3f:83:98:35:4c:d7:8d:e5:99:3c:b1:94:7d:9c:bb:19:67:3f:10:7d
        Issuer: C=US, O=Domus Digitalis, OU=Enterprise PKI, CN=DOMUS-ROOT-CA  ← Root CA
        Subject: CN=DOMUS-ISSUING-CA, OU=Enterprise PKI, O=Domus Digitalis
            Not Before: Jan 25 07:05:44 2026 GMT
            Not After : Jan 24 07:06:14 2031 GMT  ← 5 years

Intermediate CA signed by Root:

Issuer: CN=DOMUS-ROOT-CA (root)
Subject: CN=DOMUS-ISSUING-CA (intermediate)

This proves the chain is correct!

7.2. Verify Certificate Chain

# Download certificates
vault read pki/cert/ca -format=json | jq -r '.data.certificate' > /tmp/root.crt
vault read pki_int/cert/ca -format=json | jq -r '.data.certificate' > /tmp/intermediate.crt

# Verify chain
openssl verify -CAfile /tmp/root.crt /tmp/intermediate.crt

Expected Output

/tmp/intermediate.crt: OK

If you get unable to get local issuer certificate, the chain is broken!

7.3. Check Intermediate CA URLs

vault read pki_int/config/urls

Expected Output

Key                        Value
---                        -----
crl_distribution_points    [http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki_int/crl]
issuing_certificates       [http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki_int/ca]
ocsp_servers               []

8. Section 5: PKI Roles

8.1. List Roles

vault list pki_int/roles

Expected Output (Current Deployment)

Keys
----
domus-server

Expected Output (Full Deployment with BYOD/Windows)

Keys
----
domus-server
domus-client
domus-byod
domus-windows-machine
domus-windows-user

8.2. Inspect domus-server Role

vault read pki_int/roles/domus-server

Expected Output

Key                  Value
---                  -----
allow_bare_domains   false
allow_subdomains     true
allowed_domains      [inside.domusdigitalis.dev domusdigitalis.dev]
enforce_hostnames    true
key_bits             2048
key_type             rsa
max_ttl              8760h  ← 1 year max
require_cn           true

8.3. Check for Missing Roles (BYOD/Windows)

# Try to read roles that should exist for full deployment
vault read pki_int/roles/domus-client 2>&1 | grep -q "No value found" && echo "❌ domus-client NOT FOUND"
vault read pki_int/roles/domus-byod 2>&1 | grep -q "No value found" && echo "❌ domus-byod NOT FOUND"
vault read pki_int/roles/domus-windows-machine 2>&1 | grep -q "No value found" && echo "❌ domus-windows-machine NOT FOUND"
vault read pki_int/roles/domus-windows-user 2>&1 | grep -q "No value found" && echo "❌ domus-windows-user NOT FOUND"

If roles are missing, you’ll need to create them before deploying Windows EAP-TEAP or BYOD provisioning. See Section 9.

9. Section 6: Test Certificate Issuance

9.1. Issue Test Certificate (vault CLI)

# On certmgr-01
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'

# Issue test certificate
vault write pki_int/issue/domus-server \
  common_name="test-$(date +%s).inside.domusdigitalis.dev" \
  ttl=24h \
  -format=json > /tmp/test-cert.json

# Verify certificate was issued
cat /tmp/test-cert.json | jq -r '.data.certificate' | openssl x509 -noout -subject -issuer -dates

Expected Output

subject=CN = test-1738047123.inside.domusdigitalis.dev
issuer=CN = DOMUS-ISSUING-CA, OU = Enterprise PKI, O = Domus Digitalis
notBefore=Jan 28 05:32:03 2026 GMT
notAfter=Jan 29 05:32:33 2026 GMT  ← 24 hours

9.2. Verify Full Chain

# Extract certificates
cat /tmp/test-cert.json | jq -r '.data.certificate' > /tmp/test.crt
cat /tmp/test-cert.json | jq -r '.data.issuing_ca' > /tmp/issuing.crt

# Verify against root
openssl verify -CAfile /tmp/root.crt -untrusted /tmp/issuing.crt /tmp/test.crt

Expected Output

/tmp/test.crt: OK

Certificate Chain Verification:

test.crt (end-entity)
  ↓ verified by
issuing.crt (DOMUS-ISSUING-CA)
  ↓ verified by
root.crt (DOMUS-ROOT-CA)

All three steps must succeed!

9.3. Cleanup Test Files

rm -f /tmp/test-cert.json /tmp/test.crt /tmp/root.crt /tmp/intermediate.crt /tmp/issuing.crt

10. Section 7: netapi Vault Commands

10.1. Prerequisites

Run these commands from your local workstation (modestus-razer), not from certmgr-01.

# Load Vault credentials from dsec
dsource d000 dev/vault

# Verify environment variables loaded
env | grep VAULT
# Expected:
#   VAULT_ADDR=http://certmgr-01.inside.domusdigitalis.dev:8200
#   VAULT_TOKEN=<token>
#   VAULT_UNSEAL_KEY_1=<key>
#   VAULT_UNSEAL_KEY_2=<key>
#   VAULT_UNSEAL_KEY_3=<key>

10.2. Check Vault Status via netapi

# From modestus-razer
netapi vault status

Expected Output

         Vault Status: UNSEALED
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Property     ┃ Value                  ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Initialized  │ Yes                    │
│ Sealed       │ No                     │
│ Version      │ 1.21.2                 │
│ Cluster Name │ vault-cluster-904d4b42 │
│ HA Enabled   │ No                     │
│ Storage Type │ file                   │
└──────────────┴────────────────────────┘

10.3. List Secrets Engines via netapi

netapi vault secrets-engines

Expected Output

              Secrets Engines
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Path     ┃ Type   ┃ Description           ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ pki/     │ pki    │ DOMUS-ROOT-CA         │
│ pki_int/ │ pki    │ DOMUS-ISSUING-CA      │
│ secret/  │ kv     │ Key/Value Secrets     │
└──────────┴────────┴───────────────────────┘

10.4. Check PKI Status via netapi

netapi vault pki-status

10.5. Issue Certificate via netapi

# Issue certificate with default TTL (1 year)
netapi vault pki-issue test-netapi.inside.domusdigitalis.dev

# Issue with custom TTL (24 hours)
netapi vault pki-issue test-netapi-24h.inside.domusdigitalis.dev --ttl 24h

# Issue and save to specific directory
netapi vault pki-issue web-server.inside.domusdigitalis.dev --ttl 8760h -o /tmp/certs

Expected Output

✓ Certificate issued successfully
  Common Name: test-netapi.inside.domusdigitalis.dev
  Serial: 3a:1f:5c:89:...
  Not After: 2027-01-28 05:45:00 UTC

Files created:
  test-netapi.inside.domusdigitalis.dev.crt
  test-netapi.inside.domusdigitalis.dev.key
  test-netapi.inside.domusdigitalis.dev.chain.crt

11. Section 8: Certificate Revocation List (CRL)

11.1. Check CRL (Root CA)

# Download and view root CRL
curl -s http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki/crl | openssl crl -inform DER -text -noout | head -20

11.2. Check CRL (Intermediate CA)

# Download and view intermediate CRL
curl -s http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki_int/crl | openssl crl -inform DER -text -noout | head -20

Expected Output

Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=DOMUS-ISSUING-CA, OU=Enterprise PKI, O=Domus Digitalis
        Last Update: Jan 28 05:50:00 2026 GMT
        Next Update: Jan 31 05:50:00 2026 GMT  ← Updated every 72 hours
No Revoked Certificates.

12. Section 9: Create Missing Roles (for BYOD/Windows)

If roles for BYOD or Windows are missing, create them:

12.1. Create domus-client Role (Linux Workstations)

vault write pki_int/roles/domus-client \
  allowed_domains="inside.domusdigitalis.dev" \
  allow_subdomains=true \
  allow_bare_domains=false \
  max_ttl=8760h \
  key_type=rsa \
  key_bits=2048 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ClientAuth" \
  require_cn=true \
  enforce_hostnames=true

12.2. Create domus-byod Role (Mobile Devices)

vault write pki_int/roles/domus-byod \
  allowed_domains="byod.inside.domusdigitalis.dev" \
  allow_subdomains=true \
  allow_bare_domains=false \
  max_ttl=2160h \
  key_type=rsa \
  key_bits=2048 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ClientAuth" \
  require_cn=true \
  enforce_hostnames=true \
  no_store=false

BYOD TTL: 2160h = 90 days (shorter than workstations for security)

12.3. Create domus-windows-machine Role

vault write pki_int/roles/domus-windows-machine \
  allowed_domains="inside.domusdigitalis.dev" \
  allow_subdomains=true \
  allow_bare_domains=false \
  max_ttl=8760h \
  key_type=rsa \
  key_bits=2048 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ClientAuth" \
  require_cn=true \
  enforce_hostnames=true

12.4. Create domus-windows-user Role

vault write pki_int/roles/domus-windows-user \
  allowed_domains="inside.domusdigitalis.dev" \
  allow_subdomains=true \
  allow_bare_domains=false \
  max_ttl=8760h \
  key_type=rsa \
  key_bits=2048 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ClientAuth,EmailProtection" \
  require_cn=true \
  enforce_hostnames=true

13. Section 10: Troubleshooting

13.1. Vault is Sealed After Reboot

Symptoms:

vault status | grep Sealed
Sealed          true

Solution:

# Method 1: From local workstation
dsource d000 dev/vault
netapi vault unseal --auto

# Method 2: On certmgr-01
vault operator unseal  # Apply 3 keys

13.2. Certificate Issuance Fails

Symptoms:

Error writing data to pki_int/issue/domus-server: Error making API request.

Diagnosis:

# Check if role exists
vault list pki_int/roles | grep domus-server

# Check role configuration
vault read pki_int/roles/domus-server

# Check if CN matches allowed domains
# CN must end with: .inside.domusdigitalis.dev or .domusdigitalis.dev

13.3. Chain Verification Fails

Symptoms:

openssl verify -CAfile root.crt -untrusted intermediate.crt test.crt
unable to get local issuer certificate

Diagnosis:

# Check intermediate was signed by root
openssl x509 -in intermediate.crt -noout -issuer
# Should show: issuer=CN=DOMUS-ROOT-CA

# Check root is self-signed
openssl x509 -in root.crt -noout -subject -issuer
# Subject and Issuer should match

13.4. netapi Commands Fail

Symptoms:

netapi vault status
Error: VAULT_ADDR not set

Solution:

# Load Vault credentials
dsource d000 dev/vault

# Verify variables loaded
env | grep VAULT_

# If VAULT_TOKEN is missing, regenerate token on certmgr-01

14. Section 11: Daily Checklist

Run this daily to verify Vault PKI health:

# 1. Check status (from workstation)
dsource d000 dev/vault
netapi vault status
# Expected: UNSEALED

# 2. Check secrets engines
netapi vault secrets-engines
# Expected: pki/ and pki_int/ present

# 3. Test certificate issuance
netapi vault pki-issue daily-test-$(date +%Y%m%d).inside.domusdigitalis.dev --ttl 1h
# Expected: Success

# 4. Check CRL is accessible
curl -I http://certmgr-01.inside.domusdigitalis.dev:8200/v1/pki_int/crl
# Expected: HTTP/1.1 200 OK

# 5. Verify backup is recent (on certmgr-01)
ssh certmgr-01 "ls -lh /opt/vault/backups/ | head -5"
# Expected: Recent backup within 24 hours

15. Related Documentation

16. Appendix: Quick Command Reference

Task Command

Task	Command
Check Vault status	`netapi vault status`
Unseal Vault	`netapi vault unseal --auto`
List PKI roles	`vault list pki_int/roles`
Issue certificate	`netapi vault pki-issue <hostname>`
Verify chain	`openssl verify -CAfile root.crt -untrusted int.crt cert.crt`
Check CRL	`curl certmgr-01:8200/v1/pki_int/crl`
View role config	`vault read pki_int/roles/<role-name>`

Check Vault status

netapi vault status

Unseal Vault

netapi vault unseal --auto

List PKI roles

vault list pki_int/roles

Issue certificate

netapi vault pki-issue <hostname>

Verify chain

openssl verify -CAfile root.crt -untrusted int.crt cert.crt

Check CRL

curl certmgr-01:8200/v1/pki_int/crl

View role config

vault read pki_int/roles/<role-name>