Vault Certificate Deployment Runbook

Step-by-step procedures for issuing and deploying certificates from HashiCorp Vault PKI to infrastructure services.

1. Overview

This runbook covers deploying Vault-issued certificates to:

ISE (Admin, EAP, Portal, pxGrid)
Keycloak (IdP)
Other internal services

Prerequisites:

Vault unsealed on certmgr-01.inside.domusdigitalis.dev
Vault token with pki_int/issue/domus-server permissions
SSH access to target servers
DOMUS-ROOT-CA in client trust stores

2. Certificate Issuance from Vault

2.1. Standard Server Certificate

# SSH to certmgr-01
ssh certmgr-01.inside.domusdigitalis.dev

# Set Vault environment
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'  # From dsec: dsource d000 dev/vault

# Issue certificate
vault write -format=json pki_int/issue/domus-server \
  common_name="<hostname>.inside.domusdigitalis.dev" \
  alt_names="<hostname>.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/<service>.json

# Extract components
jq -r '.data.certificate' /tmp/<service>.json > /tmp/<service>.crt
jq -r '.data.private_key' /tmp/<service>.json > /tmp/<service>.key
jq -r '.data.ca_chain[]' /tmp/<service>.json > /tmp/<service>-chain.crt

# Create fullchain (cert + CA chain) - needed for most services
cat /tmp/<service>.crt /tmp/<service>-chain.crt > /tmp/<service>-fullchain.crt

# Verify
openssl x509 -in /tmp/<service>.crt -noout -subject -dates

3. Keycloak Deployment

Keycloak runs in Docker on keycloak-01, with certs mounted at /opt/keycloak/certs/.

3.1. Issue Certificate

ssh certmgr-01.inside.domusdigitalis.dev
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'

vault write -format=json pki_int/issue/domus-server \
  common_name="keycloak-01.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/keycloak.json

jq -r '.data.certificate' /tmp/keycloak.json > /tmp/keycloak.crt
jq -r '.data.private_key' /tmp/keycloak.json > /tmp/keycloak.key
jq -r '.data.ca_chain[]' /tmp/keycloak.json > /tmp/keycloak-chain.crt

# Keycloak needs fullchain for browsers to verify
cat /tmp/keycloak.crt /tmp/keycloak-chain.crt > /tmp/keycloak-fullchain.crt

3.2. Deploy to Keycloak

# From your workstation - transfer files
scp certmgr-01.inside.domusdigitalis.dev:/tmp/keycloak-fullchain.crt /tmp/
scp certmgr-01.inside.domusdigitalis.dev:/tmp/keycloak.key /tmp/
scp /tmp/keycloak-fullchain.crt keycloak-01:/tmp/
scp /tmp/keycloak.key keycloak-01:/tmp/

# On keycloak-01
ssh keycloak-01
sudo cp /tmp/keycloak-fullchain.crt /opt/keycloak/certs/cert.pem
sudo cp /tmp/keycloak.key /opt/keycloak/certs/key.pem
sudo chown evanusmodestus:evanusmodestus /opt/keycloak/certs/cert.pem /opt/keycloak/certs/key.pem
sudo chmod 444 /opt/keycloak/certs/cert.pem
sudo chmod 400 /opt/keycloak/certs/key.pem
docker restart keycloak

3.3. Verify Deployment

# Check certificate chain is served
echo | openssl s_client -connect keycloak-01:8443 -showcerts 2>/dev/null | grep -c "BEGIN CERTIFICATE"
# Expected: 2 or 3 (cert + chain)

# Check subject and issuer
echo | openssl s_client -connect keycloak-01:8443 2>/dev/null | openssl x509 -noout -subject -issuer
# Expected:
#   subject=CN=keycloak-01.inside.domusdigitalis.dev
#   issuer=CN=DOMUS-ISSUING-CA

4. ISE Certificate Deployment

ISE certificates can be deployed via GUI (reliable) or OpenAPI (may have issues).

4.1. Issue Certificate for ISE

ssh certmgr-01.inside.domusdigitalis.dev
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<token>'

# Issue EAP/Admin certificate
vault write -format=json pki_int/issue/domus-server \
  common_name="ise-02.inside.domusdigitalis.dev" \
  alt_names="ise-02.inside.domusdigitalis.dev,ise-02.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/ise-eap.json

jq -r '.data.certificate' /tmp/ise-eap.json > /tmp/ise-eap.crt
jq -r '.data.private_key' /tmp/ise-eap.json > /tmp/ise-eap.key
jq -r '.data.ca_chain[]' /tmp/ise-eap.json > /tmp/ise-eap-chain.crt

4.2. Deploy via GUI (Recommended)

Transfer cert/key to your workstation
Login to ISE Admin: ise-02.inside.domusdigitalis.dev:8443
Navigate: Administration > System > Certificates > System Certificates
Click Import
Configure:
- Friendly Name: ISE-VAULT-EAP-ADMIN
- Certificate File: Upload ise-eap.crt
- Private Key File: Upload ise-eap.key
- Password: (leave empty)
Select usages: [x] Admin, [x] EAP Authentication
Click Submit

ISE will restart services when Admin usage is changed. This takes 2-5 minutes.

4.3. Deploy via netapi (OpenAPI)

# Load credentials
dsource d000 dev/network

# Import certificate
netapi ise cert import /tmp/ise-eap.crt /tmp/ise-eap.key \
  --name "ISE-VAULT-EAP-ADMIN" \
  --hostname ise-02 \
  --admin \
  --eap

Known Issue: OpenAPI may return 200 OK but not actually import the certificate. If the cert doesn’t appear in ISE, use the GUI method instead.

4.4. Verify ISE Certificate

# Check certificate via openssl
echo | openssl s_client -connect ise-02.inside.domusdigitalis.dev:8443 2>/dev/null | openssl x509 -noout -subject -issuer

# Check via netapi
netapi ise cert list-system | grep -i vault

5. Client Trust Store Setup

For clients to trust Vault-issued certificates, DOMUS-ROOT-CA must be in their trust store.

5.1. Download CA from Vault

# Via SSH and Vault CLI (recommended - returns PEM format)
ssh certmgr-01.inside.domusdigitalis.dev "export VAULT_ADDR='http://127.0.0.1:8200' && vault read -field=certificate pki/cert/ca" > /tmp/DOMUS-ROOT-CA.crt

# Verify
openssl x509 -in /tmp/DOMUS-ROOT-CA.crt -noout -subject
# Expected: subject=C=US, O=Domus Digitalis, OU=Enterprise PKI, CN=DOMUS-ROOT-CA

The HTTP endpoint certmgr-01:8200/v1/pki/ca returns DER (binary) format. Use the Vault CLI with -field=certificate for PEM format.

5.2. Linux Trust Store Installation

Distribution Commands

Table 1. Trust Store Paths by Distribution
Distribution	Commands
Arch Linux	`sudo cp /tmp/DOMUS-ROOT-CA.crt /etc/ca-certificates/trust-source/anchors/ sudo update-ca-trust trust list \| grep -i domus`
RHEL/Fedora/CentOS	`sudo cp /tmp/DOMUS-ROOT-CA.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust trust list \| grep -i domus`
Debian/Ubuntu	`sudo cp /tmp/DOMUS-ROOT-CA.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates trust list \| grep -i domus`

Arch Linux

sudo cp /tmp/DOMUS-ROOT-CA.crt /etc/ca-certificates/trust-source/anchors/
sudo update-ca-trust
trust list | grep -i domus

RHEL/Fedora/CentOS

sudo cp /tmp/DOMUS-ROOT-CA.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
trust list | grep -i domus

Debian/Ubuntu

sudo cp /tmp/DOMUS-ROOT-CA.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
trust list | grep -i domus

5.3. Firefox Browser Trust

Firefox uses its own certificate store by default. To use system certificates:

Open Firefox
Navigate to about:config
Search for security.enterprise_roots.enabled
Set to true
Restart Firefox

Alternatively, import CA manually: Settings > Privacy & Security > Certificates > View Certificates > Authorities > Import

5.4. Android Device Trust

Transfer DOMUS-ROOT-CA.crt to device
Settings > Security > Encryption & credentials > Install a certificate
Select CA certificate
Confirm installation
Verify: Settings > Trusted credentials > User tab

6. 802.1X Client Certificate Deployment

Issue and deploy EAP-TLS client certificates for Linux workstations authenticating to ISE.

6.1. Prerequisites

Vault unsealed on certmgr-01.inside.domusdigitalis.dev
DOMUS-ROOT-CA already installed on target client (see Linux Trust Store Installation)
SSH access to target workstation

6.2. Step 1: Unseal Vault (if sealed)

ssh {certmgr-hostname}

# Check status
vault status

# If sealed, unseal with 3 keys
vault operator unseal  # Enter key 1
vault operator unseal  # Enter key 2
vault operator unseal  # Enter key 3

# Authenticate
vault login  # Enter root token

6.3. Step 2: List Available Roles

vault list pki_int/roles
# Available roles:
# - domus-byod      (BYOD devices)
# - domus-client    (Linux/managed clients)
# - domus-server    (Server certificates)
# - domus-windows-machine
# - domus-windows-user

6.4. Step 3: Issue Client Certificate

HOSTNAME="<target-hostname>"  # e.g., modestus-p50

vault write -format=json pki_int/issue/domus-client \
  common_name="$<hostname>.inside.domusdigitalis.dev" \
  ttl="8760h" > /tmp/$<hostname>.json

# Extract certificate and private key
jq -r '.data.certificate' /tmp/$<hostname>.json > /tmp/$<hostname>-eaptls.pem
jq -r '.data.private_key' /tmp/$<hostname>.json > /tmp/$<hostname>-eaptls.key

# Verify certificate
openssl x509 -in /tmp/$<hostname>-eaptls.pem -noout -subject -issuer
# Expected:
#   subject=CN=<hostname>.inside.domusdigitalis.dev
#   issuer=CN=DOMUS-ISSUING-CA

6.5. Step 4: Deploy to Target Workstation

HOSTNAME="<target-hostname>"
TARGET_IP="<target-ip>"

# From your workstation - transfer files
scp certmgr-01.inside.domusdigitalis.dev:/tmp/$<hostname>-eaptls.pem /tmp/
scp certmgr-01.inside.domusdigitalis.dev:/tmp/$<hostname>-eaptls.key /tmp/
scp /tmp/$<hostname>-eaptls.* $<username>@$<target-ip>:/tmp/

# On target workstation
ssh $<username>@$<target-ip>
sudo cp /tmp/$<hostname>-eaptls.pem /etc/ssl/certs/
sudo cp /tmp/$<hostname>-eaptls.key /etc/ssl/private/
sudo chmod 644 /etc/ssl/certs/$<hostname>-eaptls.pem
sudo chmod 600 /etc/ssl/private/$<hostname>-eaptls.key
sudo chown root:root /etc/ssl/private/$<hostname>-eaptls.key

6.6. Step 5: Update NetworkManager Connection

HOSTNAME=$(cat /etc/hostname)

# Update WiFi 802.1X connection
sudo nmcli connection modify Domus-Secure \
  802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
  802-1x.client-cert /etc/ssl/certs/$<hostname>-eaptls.pem \
  802-1x.private-key /etc/ssl/private/$<hostname>-eaptls.key

# Update Wired 802.1X connection (if applicable)
sudo nmcli connection modify Wired-802.1X \
  802-1x.ca-cert /etc/ssl/certs/DOMUS-ROOT-CA.pem \
  802-1x.client-cert /etc/ssl/certs/$<hostname>-eaptls.pem \
  802-1x.private-key /etc/ssl/private/$<hostname>-eaptls.key

# Connect
nmcli connection up Domus-Secure

6.7. Step 6: Verify Authentication

# Check connection status
nmcli device status

# Watch authentication logs
journalctl -u NetworkManager -u wpa_supplicant --since "5 minutes ago" | grep -E "EAP|TLS|SUCCESS"

# Verify on switch
# show access-session interface <port> details

Expected Success Log

CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
CTRL-EVENT-CONNECTED - Connection to <BSSID> completed
Activation: successful, device activated.

6.8. Cleanup

# On certmgr-01 - remove temporary files
rm /tmp/$<hostname>.json /tmp/$<hostname>-eaptls.*

# On your workstation
rm /tmp/$<hostname>-eaptls.*

7. Verification Commands

7.1. Check Session via netapi

# Table format (default)
netapi ise mnt session <MAC>

# JSON format with policy details
netapi ise mnt -f json session <MAC>

# Key fields in JSON output:
# - selected_azn_profiles: Authorization profile applied
# - ISEPolicySetName: Policy set matched
# - AuthorizationPolicyMatchedRule: Authorization rule matched
# - IdentityPolicyMatchedRule: Identity rule matched

7.2. Check Authentication Status

netapi ise mnt auth-status <MAC>

8. Services Deployed (Current State)

Table 2. Certificate Deployment Status
Service	Certificate	Issuer	Deployed
ISE ise-02 (Admin/EAP)	ISE-VAULT-EAP-ADMIN	DOMUS-ISSUING-CA	2026-01-31
Keycloak keycloak-01	keycloak-01.inside.domusdigitalis.dev	DOMUS-ISSUING-CA	2026-01-31
ISE Guest Portal	LetsEncrypt-Portal	Let’s Encrypt E8	Auto-renewed

9. Switch Port Management (netapi)

During 802.1X troubleshooting, you may need to temporarily bypass authentication on a switch port. This section documents safe procedures using netapi.

9.1. Temporary Access (Remove 802.1X)

Only use for troubleshooting. Restore template immediately after issue resolution.

View Current Interface Configuration

netapi ios exec "show running-config interface GigabitEthernet1/0/X"
netapi ios exec "show derived-config interface GigabitEthernet1/0/X"

Remove IBNS 2.0 Template (Emergency Access)

# Remove authentication template
netapi ios config "interface GigabitEthernet1/0/X" "no source template DefaultWiredDot1xClosedAuth" --save

# Verify port is now open
netapi ios exec "show access-session interface GigabitEthernet1/0/X"

9.2. Restore 802.1X Template

After troubleshooting, immediately restore the IBNS 2.0 template.

Standard User Port

netapi ios config \
  "interface GigabitEthernet1/0/X" \
  "description [DOT1X] User Access Port" \
  "ip arp inspection trust" \
  "source template DefaultWiredDot1xClosedAuth" \
  "spanning-tree portfast edge" \
  --save

Verify Template Applied

# Check running config
netapi ios exec "show running-config interface GigabitEthernet1/0/X"

# Check derived config (shows template expansion)
netapi ios exec "show derived-config interface GigabitEthernet1/0/X"

# Check authentication session
netapi ios exec "show access-session interface GigabitEthernet1/0/X details"

9.3. Verify Client Authentication

Check Active Session

netapi ios exec "show access-session interface GigabitEthernet1/0/X details"

Expected Output (Successful EAP-TLS)

            Interface:  GigabitEthernet1/0/X
          MAC Address:  98bb.1e1f.a713
         IPv4 Address:  10.50.10.130
            User-Name:  hostname.inside.domusdigitalis.dev
               Status:  Authorized
               Domain:  DATA

Server Policies:
           Vlan Group:  Vlan: 10
              ACS ACL:  xACSACLx-IP-LINUX_EAPTLS_PERMIT_ALL-69680320

Method status list:
      Method            State
      dot1x              Authc Success
      mab                Stopped

Expected Output (MAB Fallback)

            Interface:  GigabitEthernet1/0/X
          MAC Address:  c85b.76c6.5962
         IPv4 Address:  10.50.40.100
            User-Name:  C8-5B-76-C6-59-62
               Status:  Authorized
               Domain:  DATA

Method status list:
      Method            State
      dot1x              Stopped
      mab                Authc Success