Enterprise PKI Strategy

Overview

This document defines the enterprise Public Key Infrastructure (PKI) strategy, implementing a two-tier architecture with HashiCorp Vault as the primary internal CA.

Core Principle: Internal must stay internal. Public-facing services use Let’s Encrypt; all internal services use Vault PKI.

AD CS Deprecation: HOME-ROOT-CA on home-dc01.inside.domusdigitalis.dev is deprecated and will be decommissioned by 2026-07. All new certificates must be issued from Vault. The DC retains AD services (authentication, DNS, LDAP) but no CA role.

Architecture

Tier Authority Domain Use Case

Tier 1: Public

Let’s Encrypt

*.domusdigitalis.dev

Guest portals, public APIs, external-facing services

Tier 2: Internal

HashiCorp Vault (DOMUS-ROOT-CA)

*.inside.domusdigitalis.dev

ISE, switches, APs, NAS, workstations, BYOD, pxGrid, automation

Certificate Hierarchy

DOMUS-ROOT-CA (Root CA - RSA 4096, 20 years)
└── DOMUS-ISSUING-CA (Issuing CA - RSA 4096, 5 years)
    ├── Server Certificates (ISE, Keycloak, NAS)
    ├── Workstation Certificates (EAP-TLS 802.1X)
    ├── BYOD Certificates (90-day mobile devices)
    ├── pxGrid Certificates (netapi-pxgrid)
    └── Automation Certificates (24-72h short-lived)

Why This Architecture?

  1. Security: Internal hostnames never exposed to external CA validation

  2. Cost: No Microsoft licensing fees for CA services

  3. Skill Investment: HashiCorp Vault is industry-standard for secrets management

  4. Automation: Vault enables dynamic short-lived certificates via API

  5. Simplicity: Single CA hierarchy instead of parallel AD CS and Vault

Tier 1: Public Services (Let’s Encrypt)

Managed Certificates

Certificate: public-portals
  Domains:
    - guest.domusdigitalis.dev
    - sponsor.domusdigitalis.dev
  Location: /etc/letsencrypt/live/public-portals/
  Renewal: Automatic (certbot timer)

Infrastructure

  • Cert Manager: certmgr-01.inside.domusdigitalis.dev

  • DNS Provider: Cloudflare (DNS-01 challenge)

  • Renewal Timer: systemd certbot.timer (twice daily)

Renewal Process

# Check certificate status
ssh certmgr-01.inside.domusdigitalis.dev "sudo certbot certificates"

# Manual dry-run test
ssh certmgr-01.inside.domusdigitalis.dev "sudo certbot renew --dry-run"

# View timer status
ssh certmgr-01.inside.domusdigitalis.dev "systemctl status certbot.timer"

Deploy Hook

After renewal, certificates are automatically deployed to:

  • ISE (REST API import)

  • iPSK Manager (SCP + Apache reload)

Deploy hook location: /etc/letsencrypt/renewal-hooks/deploy/deploy-certs.sh

ISE Portal Certificate Import

Bug CSCvq85152: The ISE API rejects certificates with apostrophes in the subject (e.g., "O=Let’s Encrypt"). This blocks API import of the E8 intermediate. Server certificates (CN=guest.domusdigitalis.dev) work fine via API.

One-Time Setup: E8 Intermediate (GUI Required)

The Let’s Encrypt E8 intermediate must be imported via ISE GUI due to the apostrophe bug:

# Download E8 intermediate
curl -o /tmp/e8.pem https://letsencrypt.org/certs/2024/e8.pem

# Verify certificate
openssl x509 -in /tmp/e8.pem -noout -subject -dates
# Subject: CN=E8, O=Let's Encrypt, C=US
# Valid until: 2027

GUI Steps:

  1. Administration → System → Certificates → Trusted Certificates

  2. Import → Browse → Select e8.pem

  3. Friendly Name: letsencrypt-intermediate-e8

  4. Trust for: ✓ Trust for client authentication

  5. Submit

Server Certificate Renewal (Automated)

The 90-day server certificate renewal is fully automated via the deploy hook:

# Manual test via netapi CLI
dsource d000 dev/network
netapi ise cert import-from-certmgr \
  --cert-dir public-portals \
  --domain guest.domusdigitalis.dev \
  --name guest-portal-letsencrypt \
  --portal

Verification Commands

# List ISE certificates
dsource d000 dev/network
netapi ise cert list

# Check specific cert usage
netapi ise cert list | grep -i portal

# View deploy hook logs
ssh certmgr-01.inside.domusdigitalis.dev "sudo tail -50 /var/log/certbot-deploy.log"

Certificate Chain

Server Cert: CN=guest.domusdigitalis.dev
     ↓ Signed by
Intermediate: CN=E8, O=Let's Encrypt (in ISE trust store)
     ↓ Signed by
Root: CN=ISRG Root X1 (in ISE trust store)

Tier 2: Internal Services (Vault PKI)

Server Details

Server:     certmgr-01.inside.domusdigitalis.dev:8200
Root CA:    DOMUS-ROOT-CA (RSA 4096, expires 2046)
Issuing CA: DOMUS-ISSUING-CA (RSA 4096, expires 2031)
Storage:    File (/opt/vault/data)
Status:     Active, Unsealed
Backup:     Daily to NAS (vault-backup.timer)

PKI Roles

Role Use Case TTL

domus-server

Server certificates (ISE, Keycloak, NAS)

1 year

domus-workstation

Linux workstation EAP-TLS 802.1X

1 year

domus-byod

BYOD mobile device certificates

90 days

domus-pxgrid

pxGrid client authentication

1 year

domus-automation

Ephemeral automation certificates

24-72h

Issuing Certificates

Server Certificate

# Load vault credentials
dsource d000 dev/vault

# Issue server certificate
netapi vault pki-issue nas-01.inside.domusdigitalis.dev \
  --role domus-server \
  --ttl 8760h \
  -o /tmp/nas-cert

Workstation Certificate (EAP-TLS)

# Issue workstation certificate
netapi vault pki-issue workstation.inside.domusdigitalis.dev \
  --role domus-workstation \
  --ttl 8760h \
  -o /etc/pki/tls

BYOD Certificate

# Issue BYOD certificate with PKCS#12 bundle
netapi vault pki-issue byod-device.inside.domusdigitalis.dev \
  --role domus-byod \
  --ttl 2160h \
  --pkcs12 \
  -o /home/ansible/byod-certs/device.p12

ISE Trust Store Setup

ISE must trust both the root and issuing CA for certificate validation to work.

 
# Export CA certificates from Vault
vault read -field=certificate pki/cert/ca > DOMUS-ROOT-CA.pem
vault read -field=certificate pki_int/cert/ca > DOMUS-ISSUING-CA.pem

# Import to ISE via GUI:
# Administration → System → Certificates → Trusted Certificates
# 1. Import DOMUS-ROOT-CA.pem - Trust for client auth + EAP
# 2. Import DOMUS-ISSUING-CA.pem - Trust for client auth + EAP

netapi vault Commands

Command Description

netapi vault status

Show Vault status (sealed/unsealed)

netapi vault health

Health check endpoint

netapi vault unseal --auto

Auto-unseal using dsec keys

netapi vault seal --force

Emergency seal operation

netapi vault secrets-engines

List enabled secrets engines

netapi vault backup --upload-nas

Backup to NAS

netapi vault pki-status

PKI secrets engine status

netapi vault pki-issue <cn>

Issue certificate from Vault PKI

Backup Strategy

  • Timer: vault-backup.timer runs daily at 02:00

  • Destination: nas-01.inside.domusdigitalis.dev:/volume1/backups/vault/

  • Retention: 30 days

  • Format: Compressed tar of /opt/vault/data

Deprecated: AD CS (Remove by 2026-07)

AD CS on home-dc01.inside.domusdigitalis.dev is deprecated. Do not issue new certificates from this CA.

Current Status

CA Name:    HOME-ROOT-CA
CA Type:    Enterprise Root CA
Server:     home-dc01.inside.domusdigitalis.dev
Status:     DEPRECATED - Remove CA role by 2026-07

Migration Checklist

  • Export all active certificates from AD CS

  • Re-issue required certs from Vault PKI

  • Update ISE trust store (remove HOME-ROOT-CA, add Vault CAs)

  • Remove CA role from home-dc01.inside.domusdigitalis.dev

  • Retain AD services only: Authentication, DNS, LDAP

DC Retained Services

After CA removal, home-dc01.inside.domusdigitalis.dev continues to provide:

  • Active Directory authentication (Kerberos, NTLM)

  • DNS for inside.domusdigitalis.dev zone

  • LDAP directory services

  • Group Policy (for Windows clients if any)

DNS Architecture

inside.domusdigitalis.dev is delegated to internal DNS. Cloudflare is authoritative for the parent zone but delegates inside.* to home-dc01.inside.domusdigitalis.dev.

 
domusdigitalis.dev (Cloudflare - Public)
├── guest.domusdigitalis.dev → 216.150.1.1 (Public IP)
├── sponsor.domusdigitalis.dev → 216.150.1.1 (Public IP)
└── inside.domusdigitalis.dev NS → home-dc01.inside.domusdigitalis.dev (Delegated)
    ├── ise-01.inside.domusdigitalis.dev → 10.50.1.20
    ├── ise-02.inside.domusdigitalis.dev → 10.50.1.21
    ├── nas-01.inside.domusdigitalis.dev → 10.50.1.70
    └── ... (all internal hosts)

This is why Let’s Encrypt DNS-01 challenges fail for *.inside.domusdigitalis.dev - the challenge TXT records can’t be validated externally.

Troubleshooting

Let’s Encrypt Renewal Fails

# Check logs
ssh certmgr-01.inside.domusdigitalis.dev "sudo tail -50 /var/log/letsencrypt/letsencrypt.log"

# Verify Cloudflare credentials
ssh certmgr-01.inside.domusdigitalis.dev "sudo cat /home/ansible/.secrets/cloudflare.ini"

# Test DNS propagation
dig _acme-challenge.guest.domusdigitalis.dev TXT @8.8.8.8

Vault Sealed After Reboot

# Load credentials and unseal
dsource d000 dev/vault
netapi vault unseal --auto

BYOD Certificate Not Working

  1. Verify ISE trusts both Vault CAs (root + issuing)

  2. Check Certificate Authentication Profile in ISE

  3. Verify PKCS#12 bundle was imported correctly on device

  4. Check ISE Live Logs for certificate validation errors

# View ISE trusted certificates
dsource d000 dev/network
netapi ise cert list --trusted

# Check certificate chain
openssl verify -CAfile DOMUS-ROOT-CA.pem -untrusted DOMUS-ISSUING-CA.pem client.pem

Quick Reference

Certificate Locations

Type Path

Let’s Encrypt Public

/etc/letsencrypt/live/public-portals/

Vault PKI Output

Specified via -o flag

Vault CA Certificates

/opt/vault/data/ or exported via vault read

BYOD Bundles

/home/ansible/byod-certs/

Key Commands

# Check all Let's Encrypt certs
ssh certmgr-01.inside.domusdigitalis.dev "sudo certbot certificates"

# Check Vault status
dsource d000 dev/vault && netapi vault status

# Issue internal cert via Vault
netapi vault pki-issue webserver.inside.domusdigitalis.dev

# Export CA certificates
vault read -field=certificate pki/cert/ca > root-ca.pem
vault read -field=certificate pki_int/cert/ca > issuing-ca.pem

# Issue BYOD certificate
netapi vault pki-issue byod.inside.domusdigitalis.dev --role domus-byod --pkcs12 -o device.p12

Related Documentation