Enterprise PKI Setup for pxGrid

Overview

This guide covers issuing pxGrid client certificates from Active Directory Certificate Services (AD CS) - the enterprise approach that integrates with your existing PKI infrastructure.

Why AD CS over ISE Internal CA?

  • Centralized certificate lifecycle management

  • Existing trust relationships (domain-joined devices already trust AD CS)

  • Audit trail in AD CS certificate database

  • Consistent PKI across all enterprise systems

pxGrid 2.0 Authentication Requirements

pxGrid uses mutual TLS (mTLS) with an additional requirement: even with certificate authentication, you must send a Basic Auth header with username only (no password). The username identifies the pxGrid client account.

netapi handles this automatically - just ensure ISE_PXGRID_CLIENT_NAME matches your certificate CN.

Prerequisites

  • AD CS deployed with Enterprise CA

  • pxGrid certificate template published (see AD CS Template Configuration)

  • SSH or RDP access to Domain Controller

  • OpenSSL installed on Linux workstation

Step-by-Step Setup

Step 1: Generate Private Key and CSR

# Create secrets directory if needed
mkdir -p ~/.secrets/certs/d000/ise

# Generate 2048-bit RSA key and CSR
openssl req -new -newkey rsa:2048 -nodes \
  -keyout ~/.secrets/certs/d000/ise/pxgrid-client.key \
  -out /tmp/pxgrid-client.csr \
  -subj "/CN=netapi-pxgrid/O=DomusDigitalis/OU=NetworkAutomation/C=US"

The private key must be unencrypted (-nodes flag). pxGrid client libraries don’t support encrypted keys.

Step 2: Submit CSR to AD CS

Option A: SSH to Domain Controller (Recommended)

# Copy CSR to DC (use SCP or copy contents manually)
scp /tmp/pxgrid-client.csr admin@home-dc01.inside.domusdigitalis.dev:C:/temp/

Then on the Windows DC via SSH or RDP:

# List available certificate templates (find the pxGrid template name)
certutil -CATemplates | findstr -i pxgrid

# Submit CSR with explicit CA config and template
# Format: -config "FQDN\CA-Name"
certreq -submit `
  -config "HOME-DC01.inside.domusdigitalis.dev\HOME-ROOT-CA" `
  -attrib "CertificateTemplate:ISE-pxGrid" `
  C:\temp\pxgrid-client.csr `
  C:\temp\pxgrid-client.crt

The -config flag is required when running non-interactively (SSH). Without it, certreq waits for GUI input and hangs.

Template names are case-sensitive. Use certutil -CATemplates to find the exact name.

Option B: PowerShell One-Liner

# Submit and retrieve in one command
$csr = Get-Content C:\temp\pxgrid-client.csr -Raw
$result = certreq -submit -config "HOME-DC01.inside.domusdigitalis.dev\HOME-ROOT-CA" `
  -attrib "CertificateTemplate:ISE-pxGrid" C:\temp\pxgrid-client.csr C:\temp\pxgrid-client.crt
Write-Host $result

Option C: Web Enrollment

If AD CS web enrollment is enabled:

  1. Navigate to: home-dc01.inside.domusdigitalis.dev/certsrv

  2. Click Request a certificate

  3. Click Submit an advanced certificate request

  4. Paste CSR contents, select pxGrid template

  5. Submit and download Base64 certificate

Option D: Certipy (Linux CLI)

# Install certipy
pip install certipy-ad

# Request certificate using CSR
certipy req \
  -u 'automation@inside.domusdigitalis.dev' \
  -p 'password' \
  -ca 'HOME-ROOT-CA' \
  -target home-dc01.inside.domusdigitalis.dev \
  -template 'ISE-pxGrid' \
  -csr /tmp/pxgrid-client.csr \
  -out ~/.secrets/certs/d000/ise/pxgrid-client

Step 3: Retrieve Signed Certificate

# Copy signed cert back from DC
scp admin@home-dc01:/tmp/pxgrid-client.cer ~/.secrets/certs/d000/ise/

# Verify certificate
openssl x509 -in ~/.secrets/certs/d000/ise/pxgrid-client.cer -noout -text | head -20

Step 4: Extract ROOT-CA from Domain Controller

ISE must trust the CA that signed your pxGrid client certificate. Export the ROOT-CA from your DC:

Via PowerShell on DC (Recommended)

# Export root CA certificate
$cert = Get-ChildItem -Path Cert:\LocalMachine\Root | Where-Object {$_.Subject -like "*HOME-ROOT-CA*"}
[System.IO.File]::WriteAllBytes("C:\temp\HOME-ROOT-CA.cer", $cert.Export("Cert"))

# Convert to PEM/Base64 format
certutil -encode C:\temp\HOME-ROOT-CA.cer C:\temp\HOME-ROOT-CA.pem

# Display for copying
type C:\temp\HOME-ROOT-CA.pem

Copy the PEM output to your local machine:

mkdir -p ~/.secrets/certs/d000/ca

cat > ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt << 'EOF'
-----BEGIN CERTIFICATE-----
<paste certificate content here>
-----END CERTIFICATE-----
EOF

# Verify it's the root CA (subject should equal issuer)
openssl x509 -in ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt -noout -subject -issuer

Step 5: Import ROOT-CA to ISE

ISE must trust your enterprise CA to authenticate pxGrid clients.

# Load ISE credentials
dsource d000 dev/network

# Import CA with pxGrid trust enabled
netapi ise cert add-trusted ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt \
  --name HOME-ROOT-CA \
  --pxgrid \
  --desc "AD CS Root CA for pxGrid client authentication"

Or via ISE GUI:

  1. Navigate to: Administration > System > Certificates > Trusted Certificates

  2. Click Import

  3. Upload the ROOT-CA.crt file

  4. Enable: Trust for authentication within ISE (Infrastructure)

  5. Optionally enable: Trust for client authentication (Endpoints)

Step 6: Verify Certificate Chain

# Verify client cert is signed by your CA
openssl verify \
  -CAfile ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt \
  ~/.secrets/certs/d000/ise/pxgrid-client.crt

# Expected output:
# pxgrid-client.crt: OK

# Verify key matches certificate
CERT_MOD=$(openssl x509 -in ~/.secrets/certs/d000/ise/pxgrid-client.crt -noout -modulus | md5sum)
KEY_MOD=$(openssl rsa -in ~/.secrets/certs/d000/ise/pxgrid-client.key -noout -modulus | md5sum)

if [ "$CERT_MOD" = "$KEY_MOD" ]; then
  echo "Key and certificate match"
else
  echo "ERROR: Key and certificate DO NOT match"
fi

Step 7: Configure Environment

Add to your dsec secrets file (d000 dev/network):

# pxGrid Configuration (FQDN required - IP addresses NOT supported)
ISE_PXGRID_FQDN=ise-02.inside.domusdigitalis.dev
ISE_PXGRID_CERT=~/.secrets/certs/d000/ise/pxgrid-client.crt
ISE_PXGRID_KEY=~/.secrets/certs/d000/ise/pxgrid-client.key
ISE_PXGRID_CA=~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt
ISE_PXGRID_CLIENT_NAME=netapi-pxgrid

ISE_PXGRID_CLIENT_NAME must match the CN in your certificate (netapi-pxgrid). This is used in the Basic Auth header that pxGrid requires.

Step 8: Activate pxGrid Account

# Load secrets
dsource d000 dev/network

# First activation - will be PENDING
netapi ise pxgrid activate

Expected output:

pxGrid account pending approval: PENDING
Approve client in ISE: Administration > pxGrid Services > Clients

Step 9: Approve in ISE

In ISE GUI:

  1. Navigate to: Administration > pxGrid Services > Clients

  2. Find client: netapi-pxgrid

  3. Click Approve

The client only appears after the first activate call. If you don’t see it:

  • Verify pxGrid is enabled: Administration > pxGrid Services > Settings

  • Check that the ROOT-CA is trusted with "Infrastructure" trust

  • Review ISE logs for certificate validation errors

Step 10: Verify Connection

# Re-activate (should show ENABLED now)
netapi ise pxgrid activate

# Test full connection
netapi ise pxgrid test

# List available services
netapi ise pxgrid services

Expected output:

pxGrid connection successful
Host: ppan.inside.domusdigitalis.dev:8910
Client: netapi-pxgrid

AD CS Template Configuration

If the pxGrid certificate template doesn’t exist, create it:

Via Certificate Templates MMC

  1. Open certtmpl.msc on CA server

  2. Right-click Certificate Templates > Duplicate Template

  3. Base on: Computer or Web Server

  4. General tab:

    • Template display name: pxGrid

    • Validity period: 1-2 years

  5. Subject Name tab:

    • Select Supply in the request

  6. Extensions tab:

    • Application Policies: Client Authentication, Server Authentication

  7. Security tab:

    • Grant Enroll to automation accounts

  8. Right-click new template > All Tasks > Publish

Via PowerShell

# Create pxGrid template from Web Server template
$ConfigContext = ([ADSI]"LDAP://RootDSE").configurationNamingContext
$TemplatePath = "CN=Certificate Templates,CN=Public Key Services,CN=Services,$ConfigContext"

# Clone Web Server template
$WebServer = [ADSI]"LDAP://CN=WebServer,$TemplatePath"
$pxGrid = $WebServer.PSBase.Copy("CN=pxGrid,$TemplatePath")
$pxGrid.displayName = "pxGrid"
$pxGrid.SetInfo()

# Publish on CA
certutil -SetCATemplates +pxGrid

Certificate Storage Best Practices

Recommended Directory Structure

~/.secrets/certs/
└── d000/                          # Domain/environment
    └── ise/
        ├── ROOT-CA.crt            # Enterprise root CA (trust anchor)
        ├── pxgrid-client.cer      # pxGrid client certificate
        ├── pxgrid-client.key      # pxGrid private key (unencrypted)
        ├── dataconnect.crt        # DataConnect server cert
        └── admin.crt              # ISE admin interface cert

Age Encryption (Optional)

For additional security, encrypt private keys with Age:

# Encrypt private key
age -r age1... -o pxgrid-client.key.age pxgrid-client.key
rm pxgrid-client.key

# netapi auto-decrypts .age files when ISE_PXGRID_KEY points to them
export ISE_PXGRID_KEY=~/.secrets/certs/d000/ise/pxgrid-client.key.age

Renewal Workflow

Certificates should be renewed before expiration:

# Check expiration
openssl x509 -in ~/.secrets/certs/d000/ise/pxgrid-client.cer -noout -enddate

# Generate new CSR (reuse existing key or generate new)
openssl req -new \
  -key ~/.secrets/certs/d000/ise/pxgrid-client.key \
  -out /tmp/pxgrid-client-renewal.csr \
  -subj "/CN=netapi-pxgrid/O=DomusDigitalis/C=US"

# Submit to AD CS (same process as initial request)
# ...

# Replace certificate
mv ~/.secrets/certs/d000/ise/pxgrid-client.cer{,.old}
mv /tmp/new-pxgrid-client.cer ~/.secrets/certs/d000/ise/pxgrid-client.cer

# Test connection
netapi ise pxgrid test

Troubleshooting

401 Unauthorized Despite Valid Certificate

Symptom: TLS handshake succeeds but API returns 401.

# TLS works (Verify return code: 0)
openssl s_client -connect ise.example.com:8910 -cert client.crt -key client.key

# But activate fails with 401
netapi ise pxgrid activate
# Error: [401] Authentication failed - check certificates

Cause: pxGrid requires Basic Auth header with username even for certificate auth. The username identifies the pxGrid client account.

Solution: Ensure ISE_PXGRID_CLIENT_NAME matches your certificate CN. netapi v1.x+ handles this automatically.

Debug:

# Test mTLS connection directly
echo | openssl s_client -connect ise.example.com:8910 \
  -cert ~/.secrets/certs/d000/ise/pxgrid-client.crt \
  -key ~/.secrets/certs/d000/ise/pxgrid-client.key \
  -CAfile ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt \
  2>&1 | grep "Verify return code"

# Should show: Verify return code: 0 (ok)

Client Not Appearing in ISE

Symptom: After activate, client doesn’t appear in ISE Clients list.

Causes:

  1. ROOT-CA not trusted: Import CA with "Infrastructure" trust

  2. pxGrid disabled: Check Administration > pxGrid Services > Settings

  3. Certificate CN mismatch: CN must match ISE_PXGRID_CLIENT_NAME

Solution:

# Verify CA is in ISE trusted certs with correct trust
netapi ise cert list-trusted | grep -i home

# Import if missing
netapi ise cert add-trusted ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt \
  --name HOME-ROOT-CA --pxgrid

Certificate Not Trusted by ISE

Ensure ISE trusts your AD CS root CA:

# Method 1: via netapi (recommended)
netapi ise cert add-trusted ~/.secrets/certs/d000/ca/HOME-ROOT-CA.crt \
  --name HOME-ROOT-CA --pxgrid

# Method 2: via ISE GUI
# Administration > System > Certificates > Trusted Certificates > Import
# Enable: "Trust for authentication within ISE" (Infrastructure)

certreq Hangs on SSH

Symptom: certreq command hangs when run via SSH.

Cause: Missing -config flag causes certreq to wait for GUI input.

Solution: Always specify the full CA config:

certreq -submit -config "DC-FQDN\CA-Name" -attrib "CertificateTemplate:Name" ...

Template Not Available

# List published templates
certutil -CATemplates

# Find template by name
certutil -CATemplates | findstr -i pxgrid

# Verify template permissions
Get-ACL "AD:CN=pxGrid,CN=Certificate Templates,CN=Public Key Services,CN=Services,CN=Configuration,DC=inside,DC=domusdigitalis,DC=dev"

Request Denied

Check CA event logs:

Get-WinEvent -LogName "Microsoft-Windows-CertificationAuthority/Operational" -MaxEvents 20

FQDN vs IP Address

Symptom: Connection fails with IP address.

Cause: pxGrid requires FQDN - IP addresses are not supported.

Solution: Use FQDN in ISE_PXGRID_FQDN:

# Wrong
ISE_PXGRID_FQDN=10.50.1.21

# Correct
ISE_PXGRID_FQDN=ise-02.inside.domusdigitalis.dev

References