Issues Encountered

Issues Encountered

ISE Eval License Expiration — Admin Lockout

Date: 2026-06-07
Severity: P2 — Admin UI inaccessible, auth engine operational
Impact: Cannot view RADIUS live logs, manage policies, or troubleshoot via GUI

The eval license expired on ise-02, displaying:

Out of Compliance: Your Evaluation license has expired.
Your administrative access to Cisco ISE is restricted until you
purchase Essential, Advantage, or Premier licenses.

Key insight: The auth engine continues processing RADIUS requests. The lockout is admin-only — existing 802.1X clients continue authenticating. This means the rotation is not time-critical for auth continuity, only for admin visibility.

Printer Discovery Blocked by VLAN Segmentation

While attempting to configure CUPS printing, discovered that the home printer cannot be located because:

  1. Printer is on the IOT VLAN (10.50.40.0/24) via WiFi (WLC/iPSK)

  2. Printer may not be connected — iPSK credentials need to be configured on the device

  3. ISE admin lockout prevents viewing auth failure logs for the printer’s connection attempt

  4. MGMT→IOT firewall is default-action accept in VyOS, but print ports (631, 9100, 515) show filtered from workstation — investigation needed

Runbook References pfSense (Decommissioned) — RESOLVED

The ISE 3.5 deployment runbook in domus-infra-ops referenced pfSense for gateway, NTP, NADs, CPU map. All references updated to 10.50.1.1 on 2026-06-07. pfSense NAD removed. Bridge corrected to br-mgmt.

kvm-01 NAS Mounts Missing

Date: 2026-06-07
Impact: ISO copy failed — /mnt/nas/isos/ empty on kvm-01

kvm-02 has NFS auto-mounts for isos, vms, backups from nas-01 (10.50.1.70). kvm-01 has the mount directories but no NFS mounts configured. Required manual mount:

sudo mount -t nfs4 10.50.1.70:/volume1/isos /mnt/nas/isos

TODO: Add NFS mounts to kvm-01 /etc/fstab for persistence.

kvm-01 Root Filesystem Full (13G)

Date: 2026-06-07
Impact: ISO copy to /var/lib/libvirt/images/ failed — no space left on device

kvm-01 root partition is only 13G. The ISE ISO is 14G. Copy to /mnt/onboard-ssd/vms/ instead (722G available). Also required chown qemu:qemu on the ISO for QEMU permissions.

Boot Loop After ISE Installation

Date: 2026-06-07
Impact: ISE booted back to 5-option install menu after completing installation

virt-install with --boot cdrom,hd keeps CD-ROM as first boot device. After installation completes and the VM reboots, it boots from ISO again instead of disk.

Fix — must run immediately after install finishes:

sudo virsh destroy ise-01
sudo virsh change-media ise-01 sda --eject --config
sudo virt-xml ise-01 --edit --boot hd,cdrom
sudo virsh start ise-01

Runbook updated with post-install step.

VyOS NTP Not Syncing — No Upstream Source

Date: 2026-06-07
Impact: ISE setup wizard NTP sync failed repeatedly with 10.50.1.1

VyOS was configured as NTP listener on 10.50.1.1 but had no upstream NTP source. Stratum 0, not synchronised. Fixed by adding upstream on vyos-01:

configure
set service ntp server pool.ntp.org
commit
save

ISE install used pool.ntp.org directly to unblock. Must change back to 10.50.1.1 after install. vyos-02 also needs the upstream NTP source added.

ISE Deprecation Notice: ERS Port 9060 and SMBv1

Date: 2026-06-07
Severity: P2 — not breaking yet, will break on future ISE upgrade
Source: ISE 3.5.0.527 post-install banner

Two deprecations announced:

  1. ERS APIs moving from port 9060 to 443. Future ISE releases will drop 9060 entirely. netapi and any ERS client scripts must migrate to port 443 while both ports still work. This is the safer time to make the change — test on 443 now, confirm parity, then the upgrade won’t break automation.

  2. SMBv1 removed, SMBv3 introduced. Only relevant if ISE communicates with SMB shares (AD profiling, WMI). Current environment uses NFS for backups, so likely not impacted. Verify with show running-config | include smb on ISE CLI.

Action items:

  • Audit netapi for port 9060 references — migrate to 443

  • Audit domus-infra-ops runbooks and codex for port 9060 references

  • Test ERS API calls on port 443 after restore

  • Inform CHLA team — same deprecation affects production ISE

  • Verify SMB usage: show running-config | include smb

Vault Cluster Version Skew and HA State

Date: 2026-06-07
Severity: P3 — functional but needs attention

Health check revealed version skew and HA state issues:

vault-01: Sealed=true,  Version=1.21.2, HA Enabled=true
vault-02: Sealed=false, Version=1.21.4, HA Mode=standby
vault-03: Sealed=true,  Version=1.21.4, HA Enabled=true

Issues:

  1. vault-01 is on 1.21.2 while vault-02 and vault-03 are on 1.21.4 — version skew across the cluster

  2. vault-02 is the only unsealed node in standby mode — no active leader since vault-01 (usual leader) is sealed

  3. vault-03 is sealed — only vault-02 is operational

Health check command:

for v in vault-01 vault-02 vault-03; do
  printf "\n=== %s ===\n" "$v"
  ssh "$v" \
    "VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=1 vault status" \
    2>/dev/null |
    awk '
      /Sealed/      {print}
      /Version/     {print}
      /HA Enabled/  {print}
      /HA Mode/     {print}
      /Cluster Name/{print}
      /Cluster ID/  {print}
    '
done

Action items:

  • Upgrade vault-01 from 1.21.2 to 1.21.4 to match cluster

  • Establish vault unseal/seal procedure for routine operations

  • Document which nodes should be unsealed during normal operations

Expired Client Certificate — 9C:83:06:CE:89:46 (Samsung)

Date: 2026-06-07
Severity: P2 — device failing EAP-TLS authentication
ISE Failure Code: 12516

DataConnect query revealed:

12516 EAP-TLS failed SSL/TLS handshake because of an expired certificate in the client certificates chain

Device with MAC 9C:83:06:CE:89:46 (Samsung OUI) is attempting dot1x/EAP-TLS against the Domus_8021X policy set and failing twice. The client certificate has expired and needs reissuance from Vault PKI.

Action:

# Issue new client cert
dsource d000 dev/vault
vault write pki_int/issue/domus-client \
  common_name="<device-hostname>.inside.domusdigitalis.dev" \
  ttl="8760h" \
  -format=json > /dev/shm/<device>-cert.json

# Extract
jq -r '.data.certificate' /dev/shm/<device>-cert.json > /dev/shm/<device>.crt
jq -r '.data.private_key' /dev/shm/<device>-cert.json > /dev/shm/<device>.key
jq -r '.data.ca_chain[]' /dev/shm/<device>-cert.json > /dev/shm/<device>-chain.pem

# Install on device and reconnect to WiFi

TODO: Identify which device this MAC belongs to (Remarkable? ZFold?) and reissue cert.

USB Storage Module Missing from Running Kernel

Date: 2026-06-07
Impact: Cannot mount USB flash drive for sneakernet printing

Kernel 7.0.9-arch1-1 does not include usb_storage module. Installed kernel 7.0.11-arch1-1 has it but insmod cross-version fails. Requires reboot to 7.0.11.