Daily Worklog: 2026-02-10 (Tuesday)

Overview

Date: 2026-02-10 (Tuesday)

Location: Remote

Focus: ISE 3.4 Migration Prep, WLC iPSK Configuration Fix, KVM CPU Pinning

Sessions

Session 1: ISE Backup & KVM CPU Pinning

Goal: Prepare for ISE 3.4 deployment by backing up current ISE and pinning critical VMs to prevent resource contention.

ISE Backup

dsource d000 dev/storage
netapi ise backup --repo nas-01 --name "pre-upgrade-3.5" --wait
Output
Repository: nas-01 (NFS)
Server: 10.50.1.70:/volume1/ise_backups
Name: pre-upgrade-3.5

Backup completed: pre-upgrade-3.5

KVM CPU Pinning

Problem: ISE installation can spike CPU and starve other VMs (pfSense, WLC), causing network outages.

Solution: Pin critical VMs to dedicated CPUs.

VM CPUs Command

pfSense-FW01

0-3

sudo virsh vcpupin pfSense-FW01 0 0 && sudo virsh vcpupin pfSense-FW01 1 1 && sudo virsh vcpupin pfSense-FW01 2 2 && sudo virsh vcpupin pfSense-FW01 3 3

home-dc01

4-5

sudo virsh vcpupin home-dc01 0 4 && sudo virsh vcpupin home-dc01 1 5

ise-02

6-9

sudo virsh vcpupin ise-02 0 6 && sudo virsh vcpupin ise-02 1 7 && sudo virsh vcpupin ise-02 2 8 && sudo virsh vcpupin ise-02 3 9

9800-CL-WLC

10-13

sudo virsh vcpupin 9800-CL-WLC 0 10 && sudo virsh vcpupin 9800-CL-WLC 1 11 && sudo virsh vcpupin 9800-CL-WLC 2 12 && sudo virsh vcpupin 9800-CL-WLC 3 13

Persistence: Edit VM XML with sudo virsh edit <vm-name> and add after <vcpu>:

<cputune>
  <vcpupin vcpu='0' cpuset='X'/>
  <vcpupin vcpu='1' cpuset='Y'/>
  <!-- etc -->
</cputune>

Verification:

sudo virsh vcpuinfo home-dc01
Output
VCPU:           0
CPU:            4
State:          running
CPU Affinity:   ----y-----------

VCPU:           1
CPU:            5
State:          running
CPU Affinity:   -----y----------

Memory Note

virsh "Used memory" = allocated, not consumed. This is normal:

# Check actual QEMU process memory
ps aux --sort=-%mem | grep qemu | awk '{print $6/1024/1024 " GB", $NF}'

Host with 125GB RAM, 53GB allocated to VMs, 75GB available - no memory issues.

Session 2: Domus-IoT iPSK WLAN Fix

Problem: Device BC:D0:74:0C:05:7E failing to connect to Domus-IoT SSID.

Investigation:

netapi ise dc query "SELECT CALLING_STATION_ID, POLICY_SET_NAME, PASSED, FAILURE_REASON
  FROM RADIUS_AUTHENTICATIONS
  WHERE CALLING_STATION_ID LIKE '%BC:D0:74:0C:05:7E%'
  ORDER BY TIMESTAMP_TIMEZONE DESC FETCH FIRST 3 ROWS ONLY"

ISE was passing auth (Domus-IoT iPSK policy), but client not connecting.

Root Cause: WLAN configured with 802.1x authentication instead of PSK for iPSK.

netapi wlc run "show wlan name Domus-IoT" | grep -A5 "Auth Key Management"
Output (WRONG)
Auth Key Management
    802.1x                             : Enabled
    PSK                                : Disabled

iPSK requires PSK mode - WLC sends MAC to ISE via MAC filtering, ISE returns per-device PSK via RADIUS attribute.

Fix Attempt 1: netapi wlc create-wlan (Failed)

# Delete and recreate
netapi wlc delete-wlan-ssh HomeRF --id 1 --save
netapi wlc delete-wlan-ssh Domus-IoT --id 5 --save

netapi wlc create-wlan Domus-IoT --id 5 \
    --security wpa2-psk \
    --psk "FallbackPSK2026" \
    --mac-filtering ISE-AuthZ \
    --enabled --save

Result: Command succeeded but WLAN still had 802.1x enabled. Bug in create-wlan or default behavior.

Fix Attempt 2: RESTCONF API (Not Supported)

curl -sk -u "${WLC_USER}:${WLC_PASS}" \
  "https://${WLC_IP}/restconf/data/Cisco-IOS-XE-wireless-wlan-cfg:wlan-cfg-data" \
  -H "Accept: application/yang-data+json" | jq .

Result: Security AKM settings not exposed via RESTCONF YANG model. Only basic config visible:

{
  "profile-name": "Domus-IoT",
  "wlan-id": 5,
  "psk": "FallbackPSK2026",
  "mac-filtering-list": "ISE-AuthZ"
}

Limitation: Cisco WLC 9800 RESTCONF doesn’t expose all WLAN security settings.

Fix Attempt 3: Direct SSH (Success)

ssh admin@${WLC_IP}
WLC-LAB# conf t
WLC-LAB(config)# wlan Domus-IoT 5
WLC-LAB(config-wlan)# shutdown
WLC-LAB(config-wlan)# no security wpa akm dot1x
WLC-LAB(config-wlan)# security wpa akm psk
WLC-LAB(config-wlan)# no shutdown
WLC-LAB(config-wlan)# end
WLC-LAB# write memory

WLAN must be shut down before changing security settings. Error without shutdown:

% WLAN needs to be disabled before performing this operation.

Verification:

WLC-LAB# show wlan name Domus-IoT | include Auth Key|802.1x|PSK
802.1x authentication list name                : Disabled
             Auth Key Management
                 802.1x                             : Disabled
                 PSK                                : Enabled

Add WLAN to Policy Tag

WLC-LAB# conf t
WLC-LAB(config)# wireless tag policy default-policy-tag
WLC-LAB(config-policy-tag)# wlan Domus-IoT policy POLICY-DOMUS_IoT
WLC-LAB(config-policy-tag)# end
WLC-LAB# write memory

Client Connected

netapi wlc get-client BC:D0:74:0C:05:7E
Output
bc:d0:74:0c:05:7e
──────────────────────────────────────────────── Run ────────
          Connection
  AP                  AP4800
  WLAN ID             5
  Radio               dot11ac
  Client Type         normal
  Slot                1

             Authentication
  Username            BC-D0-74-0C-05-7E
  Method              MAB-ID
  AAA Override        Yes

                  WLAN Policy
  Switching Mode          Central
  Central Auth            client-is-non-hreap
  Central DHCP            Yes

ISE auth confirmed:

netapi ise dc query "SELECT CALLING_STATION_ID, POLICY_SET_NAME, PASSED
  FROM RADIUS_AUTHENTICATIONS
  WHERE CALLING_STATION_ID LIKE '%BC:D0:74:0C:05:7E%'
  ORDER BY TIMESTAMP_TIMEZONE DESC FETCH FIRST 1 ROWS ONLY"
Output
CALLING_STATION_ID   POLICY_SET_NAME   PASSED
BC:D0:74:0C:05:7E    Domus-IoT iPSK    Pass

Key Learnings

iPSK WLAN Requirements

For iPSK (Identity PSK) to work on Cisco 9800 WLC:

  1. Security Mode: WPA2-PSK (not 802.1x)

  2. MAC Filtering: Enabled, pointing to ISE AAA server

  3. Fallback PSK: Required (used if ISE unreachable)

  4. ISE Policy: Returns device-specific PSK via RADIUS attribute

Correct Configuration
Auth Key Management
    802.1x                             : Disabled
    PSK                                : Enabled
Mac Filter Authorization list name     : ISE-AuthZ

WLC RESTCONF Limitations

Not all WLC configurations are exposed via RESTCONF:

Setting RESTCONF Available

WLAN name, SSID, ID

Yes

PSK value

Yes

MAC filtering list

Yes

Security AKM mode (802.1x vs PSK)

No

Authentication list assignment

No

Workaround: Use SSH/CLI for security AKM changes.

netapi wlc run Timeout Issues

netapi wlc run has prompt detection issues with some WLCs:

ReadTimeout:
Pattern not detected: 'WLC\\-LAB\\#' in output.

Workaround: Use direct SSH for complex commands.

KVM CPU Pinning

  • Pin critical VMs before ISE install to prevent resource starvation

  • Use sudo virsh edit for persistent pinning

  • Use sudo virsh vcpupin for live (non-persistent) pinning

  • virsh "Used memory" = allocated, not consumed

Session 3: Pre-Migration Backup

Second backup before ISE 3.4 deployment:

dsource d000 dev/storage
netapi ise backup --repo nas-01 --name "pre-ise34-migration" --wait

Requires dev/storage for ISE_BACKUP_KEY, then dev/network for ISE API credentials.

Session 4: ISE Policy Set Cleanup

Removed stale policy set:

netapi ise api-call openapi DELETE "/api/v1/policy/network-access/policy-set/9ace369d-7dea-47a0-8a55-40fe2e2892f5"
Final Policy Sets
Domus-Wired MAB      enabled
Domus-Wired 802.1X   enabled
Domus-Secure 802.1X  enabled
Domus-IoT iPSK       enabled
Domus-Guest          disabled
Default              enabled

Session 5: ISE 3.4 Download

Cisco Software Center Options:

File Disk Size Use Case

Cisco-vISE-300-3.4.0.608b.ova

300GB

VMware ESXi ONLY

Cisco-vISE-600-3.4.0.608b.ova

600GB

VMware ESXi ONLY

ise-3.4.0.608b.SPA.x86_64.iso

N/A

KVM/QEMU (required)

Session 6: OVA vs ISO - Lesson Learned

OVA files are VMware ESXi format ONLY. They will NOT work on KVM.

Attempted OVA → qcow2 conversion resulted in dracut emergency shell - VM could not find root filesystem because VMware drivers (VMXNET3, PVSCSI) don’t work with KVM’s VirtIO drivers.

Per Cisco ISE 3.5 Installation Guide:

For non-VMware hypervisors (KVM, Nutanix AHV, OpenShift), the standard ISO installation method is required.

Correct approach for KVM:

# Transfer ISO to /tmp first (user-writable)
rsync -avP ~/Downloads/ise-3.4.0.608b.SPA.x86_64.iso kvm-01:/tmp/

# Move to libvirt images (requires sudo)
ssh kvm-01 "sudo mv /tmp/ise-3.4.0.608b.SPA.x86_64.iso /var/lib/libvirt/images/"

# Create VM with ISO install (not --import)
sudo virt-install \
  --name ise-01 \
  --memory 16384 \
  --vcpus 4 \
  --cpu host-passthrough \
  --machine q35 \
  --cdrom /var/lib/libvirt/images/ise-3.4.0.608b.SPA.x86_64.iso \
  --disk path=/mnt/onboard-ssd/vms/ise-01.qcow2,size=600,format=qcow2,bus=virtio \
  --os-variant rhel8.0 \
  --network bridge=virbr0,model=virtio \
  --graphics vnc,listen=0.0.0.0 \
  --video virtio \
  --noautoconsole

Session 7: Red Hat Exam Study

Researched Red Hat subscription options for RHCSA/RHCE exam prep.

Recommendation: Use free Red Hat Developer Subscription instead of paid self-support:

  • URL: developers.redhat.com/register

  • Includes: Full RHEL access (same OS as the exam)

  • Entitlement: 16 systems

  • Cost: Free

  • Perfect for: Lab study, exam prep

The paid "Self-Support" subscription adds nothing useful for exam prep - no support included anyway, and the free developer subscription provides the same RHEL access.

Downloaded:

~/Downloads/rhel-10.1-x86_64-dvd.iso

Next: Create RHEL VM on KVM for exam lab practice.

Session 8: KVM Host Disk Cleanup

Root partition was 92% full - cleaned up:

# Remove orphaned packages
sudo pacman -Rns $(pacman -Qdtq)  # Removed go (191MB)

# Clear pacman cache
sudo pacman -Sc  # Removed ~2GB

# Vacuum journal logs
sudo journalctl --vacuum-time=7d

Result: 92% → 84% (2.2GB free)

Recommendation: Move pacman cache to SSD permanently:

sudo mkdir -p /mnt/onboard-ssd/pacman-cache
sudo ln -sf /mnt/onboard-ssd/pacman-cache /var/cache/pacman/pkg

Session 9: Supermicro Upgrade Research

Current system: Supermicro X11SDV with Xeon D-2146NT (8c/16t, 125GB RAM)

Xeon D Upgrade Path (same compact form factor):

Model CPU Cores Max RAM Est. Price

X11SDV (current)

Xeon D-2146NT

8c/16t

512GB

-

X12SDV-8C

Xeon D-2733NT

8c/16t

512GB

$1500-2000

X12SDV-16C

Xeon D-2775TE

16c/32t

512GB

$2500-3000

X13SDV

Xeon D-2800 series

Up to 24c/48t

1TB

$2000-4000

Alternative Compact Options:

Model CPU Max RAM Notes

Minisforum MS-01

i9-13900H

96GB

Very small, 2x 10GbE, popular for home enterprise

Intel NUC 13 Pro

i7-1370P

64GB

Tiny, low power

HPE MicroServer Gen10+

Xeon E-2300

128GB

Quiet, 4x 3.5" bays

Verdict: Current system is solid. If more CPU needed, X12SDV-16C doubles cores while keeping same compact form factor.

Next Steps

  • Issue DOMUS PKI certificate for ise-01 (Phase 4)

  • Join ISE to AD (Phase 5)

  • Configure SAML SSO with Keycloak (Phase 5a)

Future Project: Home Private Cloud

Goal: Build self-hosted cloud infrastructure at home.

Platform Options:

Platform Complexity Notes

Proxmox VE

Easy

KVM + LXC, great WebUI, can cluster later

XCP-ng

Easy

Xen-based, free XenServer alternative

OpenStack (MicroStack)

Medium

Full cloud APIs, AWS-like, steep learning curve

TrueNAS Scale

Easy

Storage-first with VMs + K8s

oVirt

Medium

Red Hat ecosystem, good for RHCSA/RHCE study

Harvester

Medium

K8s-native VM management (Rancher)

Recommendation: Start with Proxmox VE - can migrate current KVM VMs easily, adds WebUI, snapshots, clustering. Later add second node for HA.

Hardware scaling options: - Add second Supermicro X12SDV for cluster - Or Minisforum MS-01 as compact second node - Shared storage via NFS (nas-01) or Ceph

Upcoming Projects

KVM Host OS Migration (Priority: Medium)

Current: Arch Linux (rolling) on Supermicro KVM host

Problem: Rolling distro on hypervisor = risk. One bad kernel update takes down all VMs.

Options:

  1. Rocky Linux 9 - RHEL clone, stable, aligns with RHCSA/RHCE study

  2. Proxmox VE - Purpose-built hypervisor, WebUI, clustering

  3. Stay on Arch - Works but maintenance burden

Available Hardware:

  • Production: Supermicro X11SDV (Xeon D-2146NT, 125GB RAM) - current Arch KVM host

  • Spare: Supermicro X11SDV (same model) - available for testing

Migration approach (zero-risk):

  1. Install Rocky 9 or Proxmox on spare X11SDV

  2. Test drive the new hypervisor without disrupting production

  3. Migrate VMs one-by-one from Arch host to new hypervisor

  4. Keep Arch host running until everything validated

  5. Repurpose original X11SDV as second cluster node (Proxmox HA) or storage

Benefits of spare server approach:

  • Zero production risk - Arch stays running until validated

  • Can test both Rocky 9 AND Proxmox if desired

  • End state: Two-node cluster instead of single hypervisor

Timeline: After ISE 3.4 migration complete.

Tomorrow

  • Complete ISE 3.4 post-install configuration

  • Verify 802.1X authentication

  • Issue DOMUS PKI certificate for ise-01