Cisco 9800-CL WLC HA SSO Deployment

Deploy redundant Cisco Catalyst 9800-CL virtual wireless controllers with SSO (Stateful Switchover) for zero-downtime failover. This maintains your CCNP ENWLSI skills while providing production-grade wireless infrastructure.

1. Executive Summary

Item Value

Active WLC

9800-WLC-01 (10.50.1.40) on kvm-01

Standby WLC

9800-WLC-02 (10.50.1.41) on kvm-02

HA Mode

SSO (Stateful Switchover) - 1+1 Redundancy

Failover Time

< 1 second (sub-second with SSO)

CCNP Relevance

ENWLSI 300-430 - WLC HA, RF Management, Security

2. Architecture

2.1. SSO HA Topology

                    ┌─────────────────────────────────────┐
                    │         Redundancy Port (RP)        │
                    │         VLAN 100 - Port 9800        │
                    └─────────────────────────────────────┘
                              ▲                 ▲
                              │                 │
              ┌───────────────┴───┐       ┌─────┴───────────────┐
              │   9800-WLC-01     │       │   9800-WLC-02       │
              │   10.50.1.40      │       │   10.50.1.41        │
              │   kvm-01          │       │   kvm-02            │
              │                   │       │                     │
              │   ACTIVE          │◄─────►│   HOT STANDBY       │
              │   (owns SSID/AP)  │  SSO  │   (ready takeover)  │
              │                   │  sync │                     │
              │   4 vCPU          │       │   4 vCPU            │
              │   16GB RAM        │       │   16GB RAM          │
              └───────────────────┘       └───────────────────────┘
                       │                           │
                       ▼                           ▼
              ┌─────────────────────────────────────────────────┐
              │              Management VLAN 100                │
              │              10.50.1.0/24                       │
              └─────────────────────────────────────────────────┘
                                    │
                                    ▼
              ┌─────────────────────────────────────────────────┐
              │                   APs                           │
              │   Catalyst 9120AX    │    Cisco OEAP            │
              │   CAPWAP to Active WLC (auto-failover)          │
              └─────────────────────────────────────────────────┘

2.2. What SSO Provides

Feature Without SSO With SSO

Failover Time

2-5 minutes (AP re-join)

< 1 second (stateful)

Client Impact

Full re-authentication

Seamless (session preserved)

AP Impact

Full CAPWAP re-establishment

Instant switchover

Config Sync

Manual (or Prime)

Automatic real-time

License

Separate per WLC

Shared (Network Advantage)

3. Prerequisites

3.1. VM Resources

9800-CL requires 3 NICs for HA SSO. Single-NIC VMs cannot support HA - port 9800 (Redundancy Port) will never listen.

Per Cisco 9800-CL Deployment Guide:

  • GigabitEthernet1 = Service port (out-of-band management)

  • GigabitEthernet2 = Wireless Management Interface (WMI) - trunk for VLANs

  • GigabitEthernet3 = HA Redundancy Port (RP) - peer-to-peer communication
Resource WLC-01 WLC-02 Notes

vCPU

4

4

Minimum for production

RAM

16GB

16GB

Minimum for 1000 APs

Disk

16GB

16GB

IOS-XE + config

vNIC1 (Gi1)

br-mgmt (or VLAN 100 bridge)

br-mgmt

Service port - SSH/HTTPS management

vNIC2 (Gi2)

br-mgmt (or VLAN 100 bridge)

br-mgmt

WMI trunk - CAPWAP, AP management

vNIC3 (Gi3)

br-mgmt (or VLAN 100 bridge)

br-mgmt

HA Redundancy Port - port 9800 (MUST be same L2 segment as WLC-02/Gi3)

Do NOT use virbr0 for Gi3. It’s an isolated NAT bridge with no L2 connectivity to other hosts.

Both WLCs' Gi3 interfaces MUST be on the same broadcast domain for link-local HA communication.

3.2. HA Interface Options

There are three approaches for HA interface configuration, depending on your VM NIC count:

Approach Interface IPs Use When

3-NIC (Dedicated HA)

GigabitEthernet3

169.254.1.1/2 (link-local)

Fresh 3-NIC VM deployment

2-NIC (Gi2 HA)

GigabitEthernet2

169.254.1.1/2 (link-local)

VMs with only Gi1/Gi2 available

1-NIC (Shared Mgmt)

GigabitEthernet1

10.50.1.40/41 (management)

Single-NIC VMs, simplest setup

3.2.1. Option 1: 3-NIC with Gi3 (Recommended for New Deployments)

WLC Gi3 IP Notes

WLC-01

169.254.1.1/24

HA local IP

WLC-02

169.254.1.2/24

HA remote IP

3.2.2. Option 2: 2-NIC with Gi2 (Validated 2026-03-08)

If your VMs only have Gi1 and Gi2 (no Gi3), use Gi2 for HA:

WLC Gi2 IP Notes

WLC-01

169.254.1.1/24

HA local IP

WLC-02

169.254.1.2/24

HA remote IP

Commands:

On WLC-01:

chassis redundancy ha-interface GigabitEthernet 2 local-ip 169.254.1.1 /24 remote-ip 169.254.1.2
write memory

On WLC-02:

chassis redundancy ha-interface GigabitEthernet 2 local-ip 169.254.1.2 /24 remote-ip 169.254.1.1
write memory

3.2.3. Option 3: 1-NIC with Gi1 (Management IPs)

Uses existing management network for HA traffic:

WLC Gi1 IP Notes

WLC-01

10.50.1.40/24

Same as management

WLC-02

10.50.1.41/24

Same as management

Link-local 169.254.x.x is recommended by Cisco. Both HA interfaces must be on the same L2 segment (same VLAN across kvm-01 and kvm-02).

3.3. Network Requirements

Requirement Value Purpose

RP Port

TCP/UDP 9800

Redundancy heartbeat + sync

Management

VLAN 100

AP CAPWAP, admin access

Latency

< 80ms RTT

Between Active/Standby

MTU

1500 (or jumbo)

RP communication

3.4. Licensing

SSO requires Network Advantage tier on BOTH controllers.

# Verify license on each WLC
show license summary

If no licenses show, enable Network Advantage:

configure terminal
license air level air-network-advantage
end
write memory
reload

After reload, verify:

show license summary
Expected output
License Usage:
  License                 Entitlement Tag               Count  Status
  -----------------------------------------------------------------------------
  air-network-advantage   (AIR-DNA-A)                   1      IN USE

Home Enterprise Licensing:

  • 9800-CL runs in evaluation mode (90 days) with full features

  • After 90 days: "Out of compliance" status but features keep working

  • Smart Licensing Using Policy (SLUP) tracks usage but doesn’t hard-disable features

  • For production/commercial use, proper licensing is required

  • For home enterprise: Run it - SSO and all features continue to work

4. Phase 0: Configure kvm-02 br-mgmt VLAN Trunking

kvm-02’s br-mgmt bridge must have VLAN filtering enabled for the WLC to trunk tagged traffic (VLANs 10, 20, 30, 40, 100, 110, 120). Without this, only untagged VLAN 1 traffic passes.

4.1. Linux Bridge = Virtual Switch (Cisco Mental Model)

Understanding Linux bridges from a Cisco switch perspective:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     CISCO SWITCH vs LINUX BRIDGE                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  CISCO SWITCH                          LINUX BRIDGE (br-mgmt)               │
│  ═══════════                          ══════════════════════                │
│                                                                             │
│  ┌─────────────┐                       ┌─────────────┐                      │
│  │  Gi1/0/1    │ ◄─── trunk ───►       │   eno8      │  Physical uplink    │
│  │  (uplink)   │                       │  (uplink)   │  to physical switch │
│  └─────────────┘                       └─────────────┘                      │
│        │                                     │                              │
│        ▼                                     ▼                              │
│  ┌─────────────────────────────┐       ┌─────────────────────────────┐      │
│  │      VLAN DATABASE          │       │      BRIDGE VLAN TABLE      │      │
│  │  vlan 1,10,20,30,40,100,    │       │  vid 1,10,20,30,40,100,     │      │
│  │       110,120               │       │       110,120               │      │
│  └─────────────────────────────┘       └─────────────────────────────┘      │
│        │                                     │                              │
│        ▼                                     ▼                              │
│  ┌─────────────┐                       ┌─────────────┐                      │
│  │  Gi1/0/2    │ ◄─── access ───►      │   vnet0     │  VM interface       │
│  │  (host)     │                       │  (WLC-02)   │  (virtual NIC)      │
│  └─────────────┘                       └─────────────┘                      │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  CISCO COMMANDS                        LINUX COMMANDS                       │
│  ══════════════                        ══════════════                       │
│                                                                             │
│  vlan 10,20,30,40,100,110,120          bridge vlan add vid 10 dev br-mgmt   │
│                                                                             │
│  interface Gi1/0/1                     # For physical uplink:               │
│    switchport mode trunk               bridge vlan add vid 10 dev eno8      │
│    switchport trunk allowed vlan                                            │
│      1,10,20,30,40,100,110,120                                              │
│                                                                             │
│  interface Gi1/0/2                     # For VM interface:                  │
│    switchport mode trunk               bridge vlan add vid 10 dev vnet0     │
│    switchport trunk allowed vlan                                            │
│      1,10,20,30,40,100,110,120                                              │
│                                                                             │
│  vtp mode transparent                  vlan_filtering = 0 (OFF)             │
│  (VLANs pass but not filtered)         (all traffic passes unfiltered)      │
│                                                                             │
│  vtp mode server                       vlan_filtering = 1 (ON)              │
│  (VLANs enforced)                      (only configured VLANs pass)         │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│  DANGER ZONE (what locked you out)                                          │
│  ═════════════════════════════════                                          │
│                                                                             │
│  CISCO:                                LINUX:                               │
│  switchport trunk allowed vlan 10      ip link set br-mgmt type bridge      │
│  (REMOVES all other VLANs!)               vlan_filtering 1                  │
│                                        (without adding VLANs first!)        │
│                                                                             │
│  FIX: switchport trunk allowed         FIX: Add VLANs BEFORE enabling       │
│       vlan add 1,20,30...                   vlan_filtering                  │
└─────────────────────────────────────────────────────────────────────────────┘

4.2. VLAN Configuration Flow

                    CORRECT ORDER (what we're doing)
                    ═════════════════════════════════

   Step 1: Add VLANs to bridge          Step 2: Add VLANs to uplink
   ┌─────────────────────────┐          ┌─────────────────────────┐
   │  bridge vlan add vid X  │          │  bridge vlan add vid X  │
   │    dev br-mgmt self     │    ──►   │    dev eno8             │
   │                         │          │                         │
   │  "self" = the bridge    │          │  (no self) = the port   │
   │  itself, like VLAN DB   │          │  like trunk allowed     │
   └─────────────────────────┘          └─────────────────────────┘
              │                                    │
              ▼                                    ▼
   Step 3: Verify before enabling        Step 4: Enable filtering
   ┌─────────────────────────┐          ┌─────────────────────────┐
   │  bridge vlan show       │          │  ip link set br-mgmt    │
   │                         │    ──►   │    type bridge          │
   │  Confirm ALL VLANs      │          │    vlan_filtering 1     │
   │  appear on br-mgmt      │          │                         │
   │  AND eno8               │          │  NOW SAFE because VLANs │
   └─────────────────────────┘          │  are already configured │
                                        └─────────────────────────┘

4.3. 0.1 Verify Current State

ssh kvm-02.inside.domusdigitalis.dev
cat /sys/class/net/br-mgmt/bridge/vlan_filtering
Expected (BEFORE fix)
0
bridge vlan show dev br-mgmt
Expected (BEFORE fix) - empty or only VLAN 1
port              vlan-id
br-mgmt           1 PVID Egress Untagged

4.4. 0.2 Find Physical Interface

bridge link show master br-mgmt | awk '/state forwarding/ && !/vnet/ {print $2}' | tr -d ':'
# Set variable for remaining commands
PHYS_IF="eno8"  # Replace with actual output from above
echo "Physical interface: $PHYS_IF"
Verify
echo "PHYS_IF is set to: $PHYS_IF"

4.5. 0.3 Add VLANs to Bridge (BEFORE enabling filtering)

ADD VLANs FIRST, then enable filtering. Enabling filtering without VLANs configured will drop ALL traffic and lock you out.

This is like running switchport trunk allowed vlan 10 on Cisco - it REMOVES all other VLANs. You must configure the allowed VLANs BEFORE the switch enforces them.

 
# Add VLAN 100 as PVID (native) - matches WLC trunk native VLAN 100
sudo bridge vlan add vid 100 dev br-mgmt self pvid untagged
# Add tagged VLANs for WLC and infrastructure trunking
# VLAN names MUST match switch/WLC for ISE policy consistency
# Client VLANs
sudo bridge vlan add vid 10 dev br-mgmt self   # DATA_VLAN
sudo bridge vlan add vid 20 dev br-mgmt self   # VOICE_VLAN
sudo bridge vlan add vid 30 dev br-mgmt self   # GUEST_VLAN
sudo bridge vlan add vid 40 dev br-mgmt self   # IOT_VLAN
# Infrastructure VLANs
sudo bridge vlan add vid 110 dev br-mgmt self  # SECURITY_VLAN (ISE, Vault)
sudo bridge vlan add vid 120 dev br-mgmt self  # SERVICES_VLAN (k3s)
# Special VLANs
sudo bridge vlan add vid 666 dev br-mgmt self  # NATIVE_VLAN_UNUSED
sudo bridge vlan add vid 999 dev br-mgmt self  # CRITICAL_AUTH_VLAN
Verify Step 0.3
bridge vlan show dev br-mgmt
Expected (bridge has VLANs, uplink not yet)
port              vlan-id
br-mgmt           10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999

4.6. 0.4 Add VLANs to Physical Interface (Uplink)

# VLAN 100 as PVID (native) - matches WLC trunk native VLAN 100
sudo bridge vlan add vid 100 dev $PHYS_IF pvid untagged
# Client VLANs (names match switch for ISE policy)
sudo bridge vlan add vid 10 dev $PHYS_IF   # DATA_VLAN
sudo bridge vlan add vid 20 dev $PHYS_IF   # VOICE_VLAN
sudo bridge vlan add vid 30 dev $PHYS_IF   # GUEST_VLAN
sudo bridge vlan add vid 40 dev $PHYS_IF   # IOT_VLAN
# Infrastructure VLANs
sudo bridge vlan add vid 110 dev $PHYS_IF  # SECURITY_VLAN
sudo bridge vlan add vid 120 dev $PHYS_IF  # SERVICES_VLAN
# Special VLANs
sudo bridge vlan add vid 666 dev $PHYS_IF  # NATIVE_VLAN_UNUSED
sudo bridge vlan add vid 999 dev $PHYS_IF  # CRITICAL_AUTH_VLAN
Verify Step 0.4
bridge vlan show dev $PHYS_IF
Expected (uplink now has VLANs)
port              vlan-id
eno8              10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999

4.7. 0.5 Verify ALL VLANs Configured (Pre-Flight Check)

bridge vlan show
Expected Output (VLANs on BOTH br-mgmt AND eno8, filtering still OFF)
port              vlan-id
eno8              10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999
br-mgmt           10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999

DO NOT proceed to 0.6 unless BOTH br-mgmt AND eno8 show all VLANs. Missing VLANs = lockout when filtering is enabled.

4.8. 0.6 Enable VLAN Filtering (NOW safe)

sudo ip link set br-mgmt type bridge vlan_filtering 1
Verify Step 0.6 - Filtering Enabled
cat /sys/class/net/br-mgmt/bridge/vlan_filtering
Expected
1
Verify Step 0.6 - Connectivity Still Works
# Use VyOS gateway (VRRP VIP 10.50.1.1 pending Phase 15 of VyOS deployment)
ping -c2 10.50.1.3
Expected (if you see replies, VLANs are working)
64 bytes from 10.50.1.3: icmp_seq=1 ttl=64 time=0.3 ms
64 bytes from 10.50.1.3: icmp_seq=2 ttl=64 time=0.2 ms

4.9. 0.7 Make Persistent (NetworkManager)

The bridge vlan commands are runtime only. NetworkManager persists them across reboots.

sudo nmcli connection modify br-mgmt bridge.vlan-filtering yes
sudo nmcli connection modify br-mgmt bridge.vlans "100 pvid untagged, 10, 20, 30, 40, 110, 120, 666, 999"
Find the port connection name (NOT the bridge itself)
nmcli connection show | grep -i eno8
Expected (note the connection name - likely "br-mgmt-port")
br-mgmt-port  ebd955da-b7e8-495e-b0f1-3c7f852a75d8  ethernet  eno8
# Add VLANs to the port connection (use actual name from above)
sudo nmcli connection modify "br-mgmt-port" bridge-port.vlans "100 pvid untagged, 10, 20, 30, 40, 110, 120, 666, 999"
Verify Step 0.7 - Bridge Settings Saved
nmcli connection show br-mgmt | grep -i vlan
Expected
bridge.vlan-filtering:                  yes
bridge.vlans:                           100 pvid untagged, 10, 20, 30, 40, 110, 120, 666, 999
Verify Step 0.7 - Port Settings Saved
nmcli connection show br-mgmt-port | grep -i vlan
Expected
bridge-port.vlans:                      100 pvid untagged, 10, 20, 30, 40, 110, 120, 666, 999

4.10. 0.8 Final Verification

cat /sys/class/net/br-mgmt/bridge/vlan_filtering
Expected
1
bridge vlan show | awk '/br-mgmt|eno8/ {found=1} found && /^[a-z]/ && !/br-mgmt|eno8/ {found=0} found'
Expected (both have all VLANs, PVID on 100)
eno8              10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999
br-mgmt           10
                  20
                  30
                  40
                  100 PVID Egress Untagged
                  110
                  120
                  666
                  999

4.11. Phase 0 Complete

kvm-02’s br-mgmt bridge is now a VLAN-aware trunk, equivalent to:

interface GigabitEthernet1/0/1
  description kvm-02-uplink
  switchport mode trunk
  switchport trunk native vlan 100
  switchport trunk allowed vlan 10,20,30,40,100,110,120,666,999

WLC-02 VMs connected to br-mgmt can now trunk all configured VLANs.

VLAN Names for ISE Policy Consistency:

| VLAN ID | Name | Purpose | |---------|------|---------| | 10 | DATA_VLAN | Corporate data | | 20 | VOICE_VLAN | VoIP phones | | 30 | GUEST_VLAN | Guest wireless | | 40 | IOT_VLAN | IoT devices | | 100 | MGMT_VLAN | Management/Infrastructure | | 110 | SECURITY_VLAN | ISE, Vault | | 120 | SERVICES_VLAN | k3s services | | 666 | NATIVE_VLAN_UNUSED | Unused native (security) | | 999 | CRITICAL_AUTH_VLAN | 802.1X auth failure |

These names MUST match on switch, WLC, and ISE authorization profiles.

5. Upgrade Existing Single-NIC WLCs to 3-NIC

If you have existing single-NIC WLC VMs, add the required NICs for HA SSO.

5.1. Prerequisites

  • WLC VM is shut down

  • Existing qcow2 disk preserves all config (hostname, IPs, VLANs)

  • Bridge must support VLAN trunking

5.2. Add NICs to WLC-02 (kvm-02)

# SSH to kvm-02
ssh kvm-02
# Verify VM is running
sudo virsh list --all | grep -i wlc
# Shutdown WLC-02
sudo virsh shutdown 9800-WLC-02
# Wait for shutdown (check status)
sudo virsh list --all | grep -i wlc
# Add vNIC2 (Gi2 - WMI trunk)
sudo virsh attach-interface 9800-WLC-02 --type bridge --source br-mgmt --model virtio --persistent
# Add vNIC3 (Gi3 - HA Redundancy Port)
sudo virsh attach-interface 9800-WLC-02 --type bridge --source br-mgmt --model virtio --persistent
# Verify 3 NICs attached
sudo virsh domiflist 9800-WLC-02
Expected output
Interface   Type     Source    Model    MAC
---------------------------------------------------------
vnetX       bridge   br-mgmt   virtio   52:54:00:xx:xx:xx
vnetY       bridge   br-mgmt   virtio   52:54:00:yy:yy:yy
vnetZ       bridge   br-mgmt   virtio   52:54:00:zz:zz:zz
# Start WLC-02
sudo virsh start 9800-WLC-02

5.3. Add NICs to WLC-01 (kvm-01)

Gi3 (HA port) MUST be on the same L2 segment as WLC-02/Gi3.

If WLC-02/Gi3 is on br-mgmt (VLAN 100), then WLC-01/Gi3 must ALSO be on a bridge connected to VLAN 100.

virbr0 is libvirt’s NAT bridge (192.168.122.x) - it has NO L2 connectivity to br-mgmt. Using virbr0 for Gi3 will cause HA to fail with "Communications = Down".

First, check what bridges exist on kvm-01:

ip link show type bridge

Use whichever bridge provides L2 connectivity to VLAN 100 (typically br-mgmt or similar).

# SSH to kvm-01
ssh kvm-01
# Check existing bridge for WLC-01 Gi1 (the working management interface)
sudo virsh domiflist 9800-WLC-01

Use the SAME bridge for Gi2 and Gi3 to ensure L2 connectivity.

# Shutdown WLC-01
sudo virsh shutdown 9800-WLC-01
# Add vNIC2 (Gi2 - WMI trunk)
# Replace BRIDGE with actual bridge name (e.g., br-mgmt, NOT virbr0)
sudo virsh attach-interface 9800-WLC-01 --type bridge --source BRIDGE --model virtio --persistent
# Add vNIC3 (Gi3 - HA Redundancy Port)
# CRITICAL: Must be same L2 segment as WLC-02/Gi3
sudo virsh attach-interface 9800-WLC-01 --type bridge --source BRIDGE --model virtio --persistent
# Verify 3 NICs attached
sudo virsh domiflist 9800-WLC-01
# Start WLC-01
sudo virsh start 9800-WLC-01

5.4. Configure IOS-XE Interfaces

After adding NICs, configure the new interfaces on each WLC.

5.4.1. On WLC-02

! Verify all 3 interfaces exist
show ip interface brief | include Gig
Expected
GigabitEthernet1    10.50.1.41      YES NVRAM  up         up
GigabitEthernet2    unassigned      YES unset  down       down
GigabitEthernet3    unassigned      YES unset  down       down

Do NOT use ip address on Gi3 - it will fail with % Invalid input detected.

On 9800 WLCs, the HA interface is configured via chassis redundancy ha-interface command only.

 
configure terminal
!
interface GigabitEthernet3
 description HA-REDUNDANCY-PORT
 no shutdown
!
end
write memory

5.4.2. On WLC-01

configure terminal
!
interface GigabitEthernet3
 description HA-REDUNDANCY-PORT
 no shutdown
!
end
write memory

5.5. Configure HA on Gi3

After Gi3 is up on both WLCs (no shutdown applied):

5.5.1. On WLC-01

chassis redundancy ha-interface GigabitEthernet 3 local-ip 169.254.1.1 /24 remote-ip 169.254.1.2
write memory

5.5.2. On WLC-02

chassis redundancy ha-interface GigabitEthernet 3 local-ip 169.254.1.2 /24 remote-ip 169.254.1.1
write memory

5.6. Verify HA Communication

! From WLC-01
ping 169.254.1.2

! From WLC-02
ping 169.254.1.1

! Test HA port
telnet 169.254.1.2 9800

If telnet connects (or shows banner), HA port is listening. Reload both WLCs simultaneously:

reload

6. Phase 1: Deploy 9800-WLC-02 VM

All commands use FULL PATHS. Do NOT use cd - sudo cd doesn’t work (cd is a shell builtin). Stay in your home directory and reference files with absolute paths.

6.1. 1.1 Download C9800-CL OVA (from workstation)

# From Cisco software download (CCO account required)
# Product: Catalyst 9800-CL Cloud Wireless Controller
# Version: 17.15.x (match WLC-01 or upgrade both)
# File: C9800-CL-universalk9.17.15.04d.ova

6.2. 1.2 Copy OVA to kvm-02 NAS Storage

# SCP to home first (avoid permission issues)
scp ~/Downloads/C9800-CL-universalk9.17.15.04d.ova kvm-02:~/
# Then move to NAS (requires sudo for NAS write)
ssh kvm-02.inside.domusdigitalis.dev "sudo mv ~/C9800-CL-universalk9.17.15.04d.ova /mnt/nas/isos/"
Verify OVA in place
ssh kvm-02.inside.domusdigitalis.dev "ls -lh /mnt/nas/isos/*.ova"

6.3. 1.3 SSH to kvm-02

ssh kvm-02.inside.domusdigitalis.dev

6.4. 1.4 Extract OVA

sudo tar -xvf /mnt/nas/isos/C9800-CL-universalk9.17.15.04d.ova -C /mnt/nas/isos/
Expected Output
C9800-CL-universalk9_vga.17.15.04d.ovf
C9800-CL-universalk9_vga.17.15.04d.mf
vwlc_harddisk.vmdk
README-OVF.txt
C9800-CL-universalk9_vga.17.15.04d.iso
Verify VMDK extracted
ls -lh /mnt/nas/isos/vwlc_harddisk.vmdk

6.5. 1.5 Create Empty Disk for Installation

The VMDK in the OVA is a STUB file (~2.6MB), NOT a real disk image. The actual 9800-CL image is on the ISO (~1.6GB). You must:

  1. Create an EMPTY qcow2 disk

  2. Boot from the ISO (CDROM)

  3. The ISO installs IOS-XE to the empty disk

This is like how you install Windows - boot from ISO, it installs to the blank hard drive.

 
# Verify the VMDK is just a stub (should be ~2.6MB, not 8GB)
ls -lh /mnt/nas/isos/vwlc_harddisk.vmdk
Expected (stub file - DO NOT USE for VM)
-rw-r--r-- 1 root root 2.6M Mar  4 22:15 /mnt/nas/isos/vwlc_harddisk.vmdk
# Create EMPTY 16GB disk for IOS-XE installation
sudo qemu-img create -f qcow2 /mnt/nas/vms/9800-WLC-02.qcow2 16G
Expected
Formatting '/mnt/nas/vms/9800-WLC-02.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=17179869184 lazy_refcounts=off refcount_bits=16
Verify empty disk created
ls -lh /mnt/nas/vms/9800-WLC-02.qcow2
Expected (small sparse file - will grow during installation)
-rw-r--r-- 1 root root 193K Mar  4 22:30 /mnt/nas/vms/9800-WLC-02.qcow2

6.6. 1.6 Fix NFS Permissions for libvirt (REQUIRED)

libvirt runs as uid:107 (qemu user), not root. NFS mounts typically don’t allow the qemu user to traverse directories. Without this fix, virt-install fails with:

ERROR    Cannot access storage file '/mnt/nas/...' (as uid:107, gid:107): Permission denied

This is like a user needing x (execute/search) permission on every directory in a path to reach a file.

 
# Grant others read+execute on ALL NFS paths used by VMs
# setfacl doesn't work on NFS ("Operation not supported"), use chmod instead
sudo chmod o+rx /mnt/nas
sudo chmod o+rx /mnt/nas/vms
sudo chmod o+rx /mnt/nas/isos
# Grant qemu user ownership of the disk file
sudo chown qemu:qemu /mnt/nas/vms/9800-WLC-02.qcow2
Why this is needed
Path traversal requires execute (x) permission on EVERY directory:

  /mnt              ← qemu needs x here
    └── nas         ← qemu needs x here
        ├── vms     ← qemu needs x here (for disk images)
        │   └── 9800-WLC-02.qcow2  ← qemu needs r here
        └── isos    ← qemu needs x here (for ISO/CDROM)
            └── C9800-CL-...iso    ← qemu needs r here

NFS default: root creates dirs → mode 755 → others have rx ✓
NFS gotcha:  sometimes restricted → mode 750 → others blocked ✗

setfacl fails on NFS because NFS doesn't support POSIX ACLs.
chmod o+rx is the portable fix.

Run this ONCE on kvm-02 after NAS mount. It persists across reboots (stored on NAS, not local).

6.7. 1.7 Create VM (Boot from ISO)

sudo virt-install \
  --name 9800-WLC-02 \
  --memory 16384 \
  --vcpus 4 \
  --disk path=/mnt/nas/vms/9800-WLC-02.qcow2,format=qcow2 \
  --cdrom /mnt/nas/isos/C9800-CL-universalk9_vga.17.15.04d.iso \
  --network bridge=br-mgmt,model=virtio \
  --os-variant generic \
  --graphics vnc,listen=0.0.0.0 \
  --boot cdrom,hd \
  --noautoconsole

Key options explained:

  • --cdrom - Attach the ISO as a CDROM device (IOS-XE installer)

  • --boot cdrom,hd - Boot from CDROM first, then hard disk (like BIOS boot order)

  • --disk - The empty 16GB qcow2 where IOS-XE will be installed

  • NO --import - We’re NOT importing an existing disk, we’re installing fresh
Expected output
Starting install...
Domain is still running. Installation may be in progress.
Verify VM created and running
sudo virsh list --all | grep -i wlc
Expected
 -    9800-WLC-02    running

6.8. 1.7.1 Convert to Serial Console (Preferred)

Remove VNC graphics and use serial console for installation:

sudo virsh dumpxml 9800-WLC-02 > /tmp/wlc02.xml
# Remove VNC graphics section with sed
sed -i '/<graphics type=.vnc/,/<\/graphics>/d' /tmp/wlc02.xml
# Verify graphics removed
grep -c graphics /tmp/wlc02.xml
Expected
0
# Redefine and restart VM
sudo virsh define /tmp/wlc02.xml
sudo virsh destroy 9800-WLC-02
sudo virsh start 9800-WLC-02
# Connect via serial console
sudo virsh console 9800-WLC-02

Press Enter to wake console. You should see the GRUB boot menu.

Exit console
Ctrl + ]

6.9. 1.7.2 VNC Console (Alternative)

If serial console doesn’t work, use VNC:

sudo virsh vncdisplay 9800-WLC-02
Expected (port 5900 + display number)
:1

This means VNC port 5901. Connect with: vncviewer kvm-02:5901

6.10. 1.7.3 IOS-XE Installation from ISO

At the GRUB boot menu, select vWLC - GOLDEN IMAGE and press Enter.

GRUB Boot Menu
 +----------------------------------------------------------------------------+
 | vWLC - GOLDEN IMAGE                                                        |
 |                                                                            |
 +----------------------------------------------------------------------------+

      Use the UP and DOWN arrow keys to select which entry is highlighted.

The installer will:

  1. Boot the IOS-XE kernel

  2. Format the virtual disk

  3. Install IOS-XE to disk (~5-10 minutes)

  4. Reboot automatically

Installation Progress
Installing IOS-XE...
Formatting disk...
Copying files...
Installation complete. Rebooting...

After reboot, you’ll see the IOS-XE initial configuration dialog:

         --- System Configuration Dialog ---

Would you like to enter the initial configuration dialog? [yes/no]:

Type no and press Enter - we’ll configure manually.

6.11. 1.7.4 Configure vnet Interface PVID (CRITICAL)

libvirt creates vnet10 with PVID 1 by default. The WLC sends management traffic untagged (native VLAN 100), but without PVID 100 on vnet10, the bridge tags it as VLAN 1 and drops it.

This is the #1 cause of "WLC boots but no connectivity" issues.

 
# Find the vnet interface for WLC-02
sudo virsh domiflist 9800-WLC-02 | awk '/br-mgmt/ {print $1}'
Expected (something like vnet10)
vnet10
# Check current VLAN config (likely shows PVID 1)
bridge vlan show dev vnet10
Expected (BEFORE fix) - PVID on wrong VLAN
port              vlan-id
vnet10            1 PVID Egress Untagged
                  100
# Set PVID to 100 (matches WLC native VLAN)
sudo bridge vlan del vid 1 dev vnet10 pvid untagged
sudo bridge vlan add vid 100 dev vnet10 pvid untagged
# Verify PVID is now 100
bridge vlan show dev vnet10
Expected (AFTER fix) - PVID on VLAN 100
port              vlan-id
vnet10            100 PVID Egress Untagged
# Test connectivity
ping -c2 10.50.1.41

Why PVID matters:

  • WLC sends management traffic untagged on native VLAN 100

  • Without PVID 100, bridge tags it as VLAN 1 → frames go to wrong VLAN → no response

  • With PVID 100, bridge tags untagged frames as VLAN 100 → matches br-mgmt PVID → works

This is equivalent to setting switchport access vlan 100 on a Cisco switch port, except the WLC is in trunk mode with native VLAN 100.

6.12. 1.8 Remove CDROM After Installation

After IOS-XE installs and the WLC boots normally, remove the CDROM so it boots from disk:

# Check current CDROM attachment
sudo virsh domblklist 9800-WLC-02 | grep -i cdrom
# Eject the ISO
sudo virsh change-media 9800-WLC-02 sda --eject
# Update boot order to disk only (persistent)
sudo virsh dumpxml 9800-WLC-02 > /tmp/wlc02.xml

# Edit XML: Change boot order from "cdrom,hd" to just "hd"
# Or use virt-manager GUI

# Apply changes
sudo virsh define /tmp/wlc02.xml

You can also use virt-manager GUI to eject the CDROM and change boot order. This is often easier than XML editing.

6.13. 1.9 Initial Configuration (Console)

Connect via VNC console (serial doesn’t work with VGA ISO) and configure:

Gateway: Use VyOS (10.50.1.2 or 10.50.1.3) until VRRP VIP (10.50.1.1) is configured in Phase 15 of VyOS deployment. For WLC-02 on kvm-02, use 10.50.1.3.

 
! Basic setup
hostname 9800-WLC-02
!
! Create ALL VLANs with consistent names (matches switch/ISE)
vlan 10
 name DATA_VLAN
vlan 20
 name VOICE_VLAN
vlan 30
 name GUEST_VLAN
vlan 40
 name IOT_VLAN
vlan 100
 name MGMT_VLAN
vlan 110
 name SECURITY_VLAN
vlan 120
 name SERVICES_VLAN
vlan 666
 name NATIVE_VLAN_UNUSED
vlan 999
 name CRITICAL_AUTH_VLAN
!
! Management SVI on VLAN 100
interface Vlan100
 ip address 10.50.1.41 255.255.255.0
 no shutdown
!
! Trunk interface - native VLAN 100, all VLANs allowed
interface GigabitEthernet1
 switchport mode trunk
 switchport trunk native vlan 100
 switchport trunk allowed vlan 10,20,30,40,100,110,120,666,999
 no shutdown
!
! Gateway - VyOS (VRRP VIP pending)
ip route 0.0.0.0 0.0.0.0 10.50.1.3
ip name-server 10.50.1.90
ip name-server 10.50.1.91
ip domain name inside.domusdigitalis.dev
!
! Enable SSH
crypto key generate rsa modulus 4096
ip ssh version 2
!
line vty 0 15
 login local
 transport input ssh
!
! Admin user (get password from gopass)
username admin privilege 15 secret <password>
!
! Save
write memory

Why trunk with native VLAN 100?

  • WLC sends management traffic untagged (native VLAN)

  • Linux bridge vnet has PVID 100 (tags untagged frames as VLAN 100)

  • This matches the switch config where MGMT_VLAN is the infrastructure VLAN

  • All wireless client VLANs configured upfront for SSO sync

7. Phase 2: Configure SSO on Active (WLC-01)

7.1. 2.1 Enable Redundancy

! On 9800-WLC-01 (Active)
configure terminal
!
redundancy
 mode sso
 exit
!
! Set chassis priority (higher = preferred active)
chassis 1 priority 200
!
! Configure HA interface with local and remote IPs
! WARNING: Do not use management IP - use dedicated HA subnet or same subnet with different IPs
chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.40 /24 remote-ip 10.50.1.41
!
end
write memory

17.15.x Syntax Changes:

  • Priority: chassis 1 priority <value> (not chassis redundancy chassis-priority)

  • HA Interface: chassis redundancy ha-interface GigabitEthernet <num> local-ip <IP> /<mask> remote-ip <peer-IP>

  • No separate wireless management interface command needed

  • No credentials in the command - authentication handled differently

7.2. 2.2 Verify RP Configuration

show redundancy

Expected output (before standby joins):

Redundant System Information :
       Available system uptime = 45 days, 3 hours, 22 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Simplex
              Configured Redundancy Mode = sso
              Operating Redundancy Mode = Non-redundant

8. Phase 3: Configure SSO on Standby (WLC-02)

8.1. 3.1 Enable Redundancy

! On 9800-WLC-02 (Standby)
configure terminal
!
redundancy
 mode sso
 exit
!
! Set chassis priority (lower = standby)
chassis 1 priority 100
!
! Configure HA interface with local and remote IPs
chassis redundancy ha-interface GigabitEthernet 1 local-ip 10.50.1.41 /24 remote-ip 10.50.1.40
!
end
write memory

8.2. 3.2 Verify Redundancy Pair

# On WLC-01 (Active)
show redundancy

Expected output (after sync):

Redundant System Information :
       Available system uptime = 45 days, 3 hours, 30 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Duplex
              Configured Redundancy Mode = sso
              Operating Redundancy Mode = sso
                     Maintenance Mode = Disabled
                        RMI-state = ACTIVE
                  Auto-sync = Enabled

             Current Processor Information :
----------------------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 45 days, 3 hours, 30 minutes

             Peer Processor Information :
----------------------------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT
       Uptime in current state = 0 days, 0 hours, 8 minutes

8.3. 3.3 Verify Sync Status

show redundancy states

Expected:

       Unit = Primary
        Redundancy Mode (Operational) = sso
        Redundancy Mode (Configured)  = sso
        Redundancy State              = ACTIVE
        Manual Swact                  = enabled
        Communications                = Up

        Other Side (Standby) Not Ready or Not Present
        ^^^^^ This becomes "STANDBY HOT" when synced

9. Phase 4: Validation

9.1. 4.1 Show Commands

# Redundancy summary
show redundancy

# Detailed HA status
show chassis redundancy ha-intf summary

# Config sync status
show redundancy config-sync failures

# AP status (should show connected to Active)
show ap summary

9.2. 4.2 Failover Test (Controlled)

This will cause a brief wireless outage. Test during maintenance window.

 
# On Active WLC (WLC-01)
redundancy force-switchover

Verify:

# On new Active (WLC-02)
show redundancy

Expected: WLC-02 now shows ACTIVE, WLC-01 shows STANDBY HOT

9.3. 4.3 Verify AP Failover

# On new Active
show ap summary | include Up

All APs should remain connected (no re-join needed with SSO).

10. Phase 5: ISE Integration

Both WLCs must be registered in ISE as NADs.

10.1. 5.1 Add WLC-02 to ISE

# Using netapi
netapi ise create-nad \
  --name "9800-WLC-02" \
  --ip "10.50.1.41" \
  --type "Wireless" \
  --secret "<radius-secret>" \
  --coa

10.2. 5.2 Verify Both WLCs in ISE

netapi ise get-nads | jq '.[] | select(.name | contains("WLC"))'

11. Troubleshooting

11.1. RP Communication Issues

# Check RP connectivity
ping 10.50.1.41 source 10.50.1.40

# Check RP port
show chassis redundancy ha-intf summary

# Debug (use sparingly)
debug redundancy ios

11.2. Sync Failures

# Check sync errors
show redundancy config-sync failures

# Force re-sync
redundancy config-sync bulk

11.3. Standby Not Coming Up

Common causes: 1. Version mismatch - Both WLCs must run IDENTICAL IOS-XE version 2. License mismatch - Both need Network Advantage 3. MTU issues - Ensure jumbo frames or 1500 MTU on RP path 4. Firewall blocking - Port 9800 TCP/UDP must be open

# Check version
show version | include Version

# Check license
show license summary

12. CCNP Wireless Study Notes

12.1. Exam Topics Covered (ENWLSI 300-430)

Topic Relevance

2.1 HA Design

SSO vs N+1, RPO/RTO, AP groups

2.2 Mobility

Same-WLC roaming (intra) vs inter-WLC

3.1 RF

RRM, DCA, TPC - managed by Active WLC

4.1 Security

802.1X auth happens on Active only

5.1 Troubleshooting

Show commands, debug redundancy

12.2. Key Concepts

SSO (Stateful Switchover)

Both WLCs maintain synchronized state. Failover is sub-second because standby already has all client/AP state.

N+1 Redundancy

Alternative to SSO. Multiple WLCs in a mobility group, APs failover to any available WLC. Longer failover (2-5 min).

Redundancy Port (RP)

Dedicated or shared interface for heartbeat and state sync between Active/Standby.

Chassis Priority

Higher value = preferred Active. WLC-01 (200) preferred over WLC-02 (100).

Preemption

If enabled, higher-priority WLC takes over when it recovers. Default: disabled.

13. Appendix A: Session Variables

# WLC-01 (Active)
WLC01_HOSTNAME="9800-WLC-01"
WLC01_IP="10.50.1.40"
WLC01_PRIORITY=200

# WLC-02 (Standby)
WLC02_HOSTNAME="9800-WLC-02"
WLC02_IP="10.50.1.41"
WLC02_PRIORITY=100

# RP Configuration
RP_PORT=9800
RP_VLAN=100

14. Appendix B: VLAN Reference (ISE Policy Consistency)

VLAN names MUST match across all devices for ISE authorization profiles to work correctly.

ISE uses VLAN names in authorization profiles (e.g., airespace-interface-name=DATA_VLAN). If the WLC has VLAN0010 but ISE expects DATA_VLAN, the profile won’t match.
VLAN ID Name ISE Policy Use Purpose

10

DATA_VLAN

Corporate users (EAP-TLS)

Authenticated employee traffic

20

VOICE_VLAN

VoIP phones

802.1p QoS marking

30

GUEST_VLAN

CWA/WebAuth

Captive portal redirect

40

IOT_VLAN

MAB/iPSK

IoT devices, printers

100

MGMT_VLAN

Infrastructure

WLC management, AP CAPWAP

110

SECURITY_VLAN

ISE, Vault (mTLS)

Security infrastructure

120

SERVICES_VLAN

k3s workloads

Container services

666

NATIVE_VLAN_UNUSED

Security (unused)

Prevent VLAN hopping

999

CRITICAL_AUTH_VLAN

Auth failure

802.1X critical VLAN

14.1. WLC VLAN Configuration (Add After HA Setup)

After SSO is established, add wireless client VLANs to both WLCs:

! Create VLANs with CONSISTENT names
vlan 10
 name DATA_VLAN
vlan 20
 name VOICE_VLAN
vlan 30
 name GUEST_VLAN
vlan 40
 name IOT_VLAN
vlan 110
 name SECURITY_VLAN
vlan 120
 name SERVICES_VLAN
vlan 666
 name NATIVE_VLAN_UNUSED
vlan 999
 name CRITICAL_AUTH_VLAN
!
! Add to trunk
interface GigabitEthernet1
 switchport trunk allowed vlan add 10,20,30,40,110,120,666,999
!
! Create wireless interfaces (dynamic interfaces)
wireless interface DATA_VLAN vlan 10
wireless interface VOICE_VLAN vlan 20
wireless interface GUEST_VLAN vlan 30
wireless interface IOT_VLAN vlan 40

14.2. ISE Authorization Profile Example

# ISE Authorization Profile: Corporate_Full_Access
VLAN:                   DATA_VLAN
Airespace-Interface-Name: DATA_VLAN
dACL:                   PERMIT_ALL

15. Appendix C: Quick Reference

Command Purpose

show redundancy

Overall HA status

show redundancy states

Detailed state machine

show chassis redundancy ha-intf summary

RP interface status

show redundancy config-sync failures

Sync error log

redundancy force-switchover

Manual failover (Active→Standby)

redundancy config-sync bulk

Force full config sync

show ap summary

Verify APs connected post-failover

16. Appendix D: Troubleshooting

16.1. WLC-01 Gi1 Not Detected (kvm-01/virbr0)

Symptom: WLC-01 shows only Gi2, not Gi1. Or neither Gi1 nor Gi2 after NIC removal.

Root Cause: IOS-XE maps PCI slots to GigabitEthernet interfaces. Two NICs on virbr0 (untagged bridge) causes detection issues.

Fix: Recreate VM with single NIC using existing qcow2:

# 1. Shutdown and destroy old VM definition
sudo virsh shutdown 9800-WLC-01
sudo virsh destroy 9800-WLC-01 2>/dev/null
sudo virsh undefine 9800-WLC-01
# 2. Recreate with single NIC (preserves existing qcow2 config)
sudo virt-install \
  --name 9800-WLC-01 \
  --memory 16384 \
  --vcpus 4 \
  --import \
  --disk path=/mnt/onboard-ssd/vms/C9800-CL-universalk9.17.15.03.qcow2,format=qcow2 \
  --network bridge=virbr0,model=virtio \
  --os-variant generic \
  --graphics vnc \
  --noautoconsole
# 3. Verify single NIC attached
sudo virsh domiflist 9800-WLC-01
Expected Output
Interface  Type     Source   Model   MAC
-------------------------------------------------------
vnet0      bridge   virbr0   virtio  52:54:00:xx:xx:xx
# 4. Start and verify Gi1 in WLC
sudo virsh start 9800-WLC-01
# Wait 2-3 minutes for IOS-XE boot
ssh admin@10.50.1.40 "show ip int brief | include Gi"
Expected Output
GigabitEthernet1       10.50.1.40      YES NVRAM  up                    up

Key Learning (2026-03-05):

  • kvm-01/virbr0 = untagged bridge, no VLAN filtering → single NIC required

  • kvm-02/br-mgmt = also requires single NIC (despite VLAN-aware bridge)

  • --import preserves existing qcow2 config (IP, VLANs, hostname)

  • Renaming: virsh domrename <old> <new> (VM must be shutdown)

16.2. WLC-02 Gi1 Not Detected (kvm-02/br-mgmt)

Symptom: WLC-02 shows only Gi2 or no interfaces. Cannot SSH to 10.50.1.41 but ping works from kvm-02.

Root Cause: Same as WLC-01 - IOS-XE PCI slot mapping issue affects both bridges.

Fix: Recreate VM with single NIC using existing qcow2:

# 1. Destroy old VM definition
sudo virsh destroy 9800-WLC-02
sudo virsh undefine 9800-WLC-02
# 2. Recreate with single NIC (preserves existing qcow2 config)
sudo virt-install \
  --name 9800-WLC-02 \
  --memory 16384 \
  --vcpus 4 \
  --import \
  --disk path=/mnt/nas/vms/9800-WLC-02.qcow2,format=qcow2 \
  --network bridge=br-mgmt,model=virtio \
  --os-variant generic \
  --graphics vnc \
  --noautoconsole
# 3. Verify single NIC attached
sudo virsh domiflist 9800-WLC-02
Expected Output
Interface  Type     Source    Model   MAC
-------------------------------------------------------
vnet0      bridge   br-mgmt   virtio  52:54:00:xx:xx:xx
# 4. Verify Gi1 in WLC (after 2-3 min boot)
ssh admin@10.50.1.41 "show ip int brief | include Gi"
Expected Output
GigabitEthernet1       10.50.1.41      YES NVRAM  up                    up

Key Learning (2026-03-06): br-mgmt (VLAN-aware) also requires single NIC for 9800-CL.

16.3. Corrupted Disk Recovery (No Bootable Media)

Symptom: Console shows no bootable media or similar BIOS error. VM won’t boot.

Diagnosis:

# Check if disk is locked (VM may still be "running" in bad state)
sudo qemu-img check /mnt/nas/vms/9800-WLC-02.qcow2
Corrupted disk output
qemu-img: Could not open '/mnt/nas/vms/9800-WLC-02.qcow2': Failed to get "write" lock
Is another process using the image [/mnt/nas/vms/9800-WLC-02.qcow2]?
# VM stuck in running state
sudo virsh list --all | grep -i wlc
Expected (VM running but broken)
 -    9800-WLC-02    running

Root Cause: NAS I/O failure, storage corruption, or ISO installation failure corrupted the qcow2 filesystem.

Recovery Procedure:

# Step 1: Force destroy the stuck VM
sudo virsh destroy 9800-WLC-02
# Step 2: Undefine the VM (removes libvirt config, keeps disk)
sudo virsh undefine 9800-WLC-02
# Step 3: Delete corrupted disk
sudo rm /mnt/nas/vms/9800-WLC-02.qcow2
# Step 4: Create fresh empty disk (16G minimum)
sudo qemu-img create -f qcow2 /mnt/nas/vms/9800-WLC-02.qcow2 16G
# Step 5: Fix permissions for libvirt
sudo chown qemu:qemu /mnt/nas/vms/9800-WLC-02.qcow2
# Step 6: Redeploy from ISO (boots from CDROM, installs to empty disk)
sudo virt-install \
  --name 9800-WLC-02 \
  --memory 16384 \
  --vcpus 4 \
  --disk path=/mnt/nas/vms/9800-WLC-02.qcow2,format=qcow2 \
  --cdrom /mnt/nas/isos/C9800-CL-universalk9_vga.17.15.04d.iso \
  --network bridge=br-mgmt,model=virtio \
  --os-variant generic \
  --graphics vnc,listen=0.0.0.0 \
  --boot cdrom,hd \
  --noautoconsole
# Step 7: Connect via Cockpit VNC (PREFERRED)
# Open browser: https://kvm-02:9090 → Virtual Machines → 9800-WLC-02 → Console
Alternative: CLI VNC
sudo virsh vncdisplay 9800-WLC-02
# Then: vncviewer kvm-02:590X (where X is display number)

At GRUB menu, select vWLC - GOLDEN IMAGE. Wait for installation (~5-10 min).

After reboot, complete initial configuration per Section 1.9.

# Step 8: Configure vnet PVID (CRITICAL - WLC won't ping without this)
VNET=$(sudo virsh domiflist 9800-WLC-02 | awk '/br-mgmt/ {print $1}')
sudo bridge vlan del vid 1 dev $VNET pvid untagged
sudo bridge vlan add vid 100 dev $VNET pvid untagged
# Step 9: Verify connectivity
ping -c2 10.50.1.41

Key Learning (2026-03-05):

  • NAS I/O issues during bridge config changes can corrupt VM disks

  • qcow2 "write lock" error + "no bootable media" = disk is toast, redeploy

  • ISO installation requires empty disk (VMDK in OVA is a stub)

  • PVID 100 on vnet is required for management traffic

17. Troubleshooting HA SSO

17.1. HA Communications Down, Reason: Failure

Symptom:

WLC-01#show redundancy
...
Communications = Down      Reason: Failure
Peer (slot: 0) information is not available because it is in 'DISABLED' state

And show ip interface brief shows Gi3 as "unassigned":

GigabitEthernet3       unassigned      YES unset  up                    up

Root Cause: Gi3 on WLC-01 and Gi3 on WLC-02 are on different L2 segments.

Link-local addresses (169.254.x.x) are non-routable. They require direct Layer 2 adjacency - both WLCs' Gi3 interfaces MUST be on the same broadcast domain.

17.1.1. Verify Bridge Assignments

On kvm-01:

sudo virsh domiflist 9800-WLC-01

On kvm-02:

sudo virsh domiflist 9800-WLC-02

PROBLEM: If Gi3 bridges don’t match (e.g., virbr0 vs br-mgmt), HA cannot form.

Bridge Network L2 Connectivity

virbr0

192.168.122.0/24 (NAT, isolated)

Only within same host

br-mgmt

VLAN 100 (10.50.1.0/24)

Spans physical network

17.1.2. Fix: Move Gi3 to Common Bridge

Both WLCs' Gi3 must be on a bridge that provides L2 connectivity. Since br-mgmt spans the physical network via VLAN 100, use it for both.

Step 1: Check what bridge kvm-01 has for VLAN 100

# On kvm-01
bridge link show | awk '{print $2}'
ip link show type bridge

If kvm-01 has br-mgmt or similar, use that. If not, you may need to create one or use the physical interface with VLAN tagging.

Step 2: Shut down WLC-01

ssh kvm-01
sudo virsh shutdown 9800-WLC-01

Step 3: Identify and detach the wrong Gi3 vnet

# List current interfaces
sudo virsh domiflist 9800-WLC-01

Note the MAC address of the third interface (Gi3).

# Detach the wrong interface (replace MAC with actual)
sudo virsh detach-interface 9800-WLC-01 bridge --mac 52:54:00:xx:xx:xx --persistent

Step 4: Attach Gi3 to correct bridge

# Attach to br-mgmt (or whatever bridge provides L2 to VLAN 100)
sudo virsh attach-interface 9800-WLC-01 --type bridge --source br-mgmt --model virtio --persistent

Step 5: Start WLC-01 and verify

sudo virsh start 9800-WLC-01
sudo virsh domiflist 9800-WLC-01

All 3 NICs should now show the same bridge (or bridges with L2 connectivity).

Step 6: Set PVID on new vnet

# Get the new vnet name for Gi3
VNET=$(sudo virsh domiflist 9800-WLC-01 | awk 'NR==4 {print $1}')
echo "Gi3 vnet: $VNET"
# Set PVID 100 (CRITICAL for untagged traffic)
sudo bridge vlan del vid 1 dev $VNET pvid untagged
sudo bridge vlan add vid 100 dev $VNET pvid untagged
# Verify
bridge vlan show dev $VNET

17.1.3. Verify L2 Connectivity

After fixing bridge assignments, verify L2 connectivity from within the WLCs:

! On WLC-01
ping 169.254.1.2
! On WLC-02
ping 169.254.1.1

If ping succeeds, HA should form after reload.

17.2. Gi3 Shows "unassigned" After chassis redundancy Command

Symptom:

WLC-01#show ip interface brief
GigabitEthernet3       unassigned      YES unset  up                    up

Even after running:

chassis redundancy ha-interface GigabitEthernet 3 local-ip 169.254.1.1 /24 remote-ip 169.254.1.2

Root Cause: The chassis redundancy ha-interface command doesn’t show the IP in show ip interface brief. This is normal behavior.

Verify HA interface status instead:

show chassis redundancy ha-intf summary

The HA interface is managed by the redundancy subsystem, not the standard IP stack.

17.3. ip address Command Fails on Gi3

Symptom:

WLC-01(config-if)#ip address 169.254.1.1 255.255.255.0
% Invalid input detected at '^' marker.

Root Cause: On 9800 WLCs, Gi3 (HA Redundancy Port) cannot be configured with ip address in interface config mode.

Fix: Use the chassis redundancy ha-interface command in global config:

chassis redundancy ha-interface GigabitEthernet 3 local-ip 169.254.1.1 /24 remote-ip 169.254.1.2
write memory

17.4. HA Still Won’t Form After All Fixes

If L2 connectivity is confirmed (pings work) but HA still shows DISABLED:

  1. Reload both WLCs simultaneously - Per Cisco docs, both controllers must be reloaded at the same time for HA to form:

    ! On both WLCs at the same time
reload

  2. Check version mismatch - Both WLCs must run the exact same IOS-XE version:

    show version | include IOSXE

    If versions differ (e.g., 17.15.3 vs 17.15.4d), HA will not form. Upgrade the older WLC.

  3. Check chassis priority - In 17.15.x, priority values are 1-2 only (not 200/100):

    show chassis redundancy

17.5. Key Learning (2026-03-06)

  • L2 adjacency is mandatory for Gi3 (HA port) - link-local IPs don’t route

  • virbr0 (libvirt NAT) is isolated per host - NEVER use for HA

  • ip address command doesn’t work on Gi3 - use chassis redundancy ha-interface

  • show ip interface brief shows Gi3 as "unassigned" - this is normal for HA ports

  • Both WLCs must have identical IOS-XE versions for SSO

  • Reload both WLCs simultaneously after HA config for pairing to occur

18. Migrate WLC VM Between KVM Hosts

When WLCs are on different KVM hosts with incompatible bridges, temporarily migrate to the same host to establish HA, then migrate back.

18.1. Use Case

  • WLC-01 on kvm-01 with virbr0 (no L2 to VLAN 100)

  • WLC-02 on kvm-02 with br-mgmt (VLAN 100)

  • Solution: Move WLC-01 to kvm-02, configure HA, then move back later

18.2. Step 1: Export VM Definition (Source Host)

ssh kvm-01
sudo virsh dumpxml 9800-WLC-01 > /tmp/9800-WLC-01.xml

18.3. Step 2: Check Disk Location

sudo virsh domblklist 9800-WLC-01
Example output
Target   Source
---------------------------------------------------------
sda      /mnt/onboard-ssd/libvirt/images/9800-WLC-01.qcow2
sdb      /var/lib/libvirt/images/C9800-CL-universalk9.17.15.03.iso
If disk is on shared NFS (nas-01), skip the copy step - just update the path in XML.

18.4. Step 3: Shut Down VM

sudo virsh shutdown 9800-WLC-01
# Wait for clean shutdown
watch -n2 'sudo virsh list --all | grep -i wlc'

18.5. Step 4: Copy Disk to Destination Host

# Copy qcow2 (skip if on shared NFS)
scp /mnt/onboard-ssd/libvirt/images/9800-WLC-01.qcow2 kvm-02:/mnt/onboard-ssd/libvirt/images/
# Copy XML definition
scp /tmp/9800-WLC-01.xml kvm-02:/tmp/

18.6. Step 5: Edit XML for Destination Host

ssh kvm-02
# Edit the XML
sudo vim /tmp/9800-WLC-01.xml

Changes required:

  1. Bridge names - Change all virbr0 to br-mgmt:

    <!-- BEFORE -->
<source bridge='virbr0'/>

<!-- AFTER -->
<source bridge='br-mgmt'/>

  2. Disk path - Update to kvm-02 location:

    <!-- BEFORE -->
<source file='/mnt/onboard-ssd/libvirt/images/9800-WLC-01.qcow2'/>

<!-- AFTER (if path differs) -->
<source file='/mnt/onboard-ssd/libvirt/images/9800-WLC-01.qcow2'/>

  3. Remove UUID (optional) - Let libvirt generate new one:

    <!-- Remove this line -->
<uuid>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</uuid>

Quick sed replacement:

# Replace all virbr0 with br-mgmt
sed -i 's/virbr0/br-mgmt/g' /tmp/9800-WLC-01.xml
# Verify changes
grep -E "bridge=|source file=" /tmp/9800-WLC-01.xml

18.7. Step 6: Define and Start on Destination Host

sudo virsh define /tmp/9800-WLC-01.xml
sudo virsh start 9800-WLC-01

18.8. Step 7: Set PVID on All vNets

# Set PVID 100 on all WLC-01 interfaces
for VNET in $(sudo virsh domiflist 9800-WLC-01 | awk 'NR>2 {print $1}'); do
  echo "Setting PVID 100 on $VNET"
  sudo bridge vlan del vid 1 dev $VNET pvid untagged 2>/dev/null
  sudo bridge vlan add vid 100 dev $VNET pvid untagged
done
# Verify
for VNET in $(sudo virsh domiflist 9800-WLC-01 | awk 'NR>2 {print $1}'); do
  echo "=== $VNET ==="
  bridge vlan show dev $VNET
done

18.9. Step 8: Verify Both WLCs Running

sudo virsh list | grep -i wlc
Expected
 Id   Name           State
--------------------------------
 XX   9800-WLC-01    running
 YY   9800-WLC-02    running

18.10. Step 9: Undefine on Source Host

After confirming WLC-01 works on kvm-02:

ssh kvm-01
# Remove the old definition (disk already copied)
sudo virsh undefine 9800-WLC-01

18.11. Step 10: Test HA Connectivity

Now both WLCs are on kvm-02 with br-mgmt. Test L2 connectivity:

On WLC-01:

ping 169.254.1.2

On WLC-02:

ping 169.254.1.1

If pings succeed, reload both WLCs simultaneously for HA to form.

18.12. Migrate Back to Original Host (Later)

After HA is working and you want to spread VMs across hosts:

  1. Ensure kvm-01 has a bridge with L2 connectivity to VLAN 100

  2. Create br-mgmt on kvm-01 if it doesn’t exist

  3. Follow the same migration procedure in reverse

  4. Both WLCs must be on bridges with L2 adjacency for Gi3

19. Appendix E: Cross-KVM Platform Differences

kvm-01 (Arch Linux) and kvm-02 (Rocky Linux) use different QEMU/libvirt configurations.

VM XML definitions exported from one host will NOT work on the other without modification.

19.1. Platform Comparison Table

Item kvm-01 (Arch Linux) kvm-02 (Rocky Linux 9)

QEMU Emulator Path

/usr/bin/qemu-system-x86_64

/usr/libexec/qemu-kvm

Machine Type

pc-i440fx-10.1

pc-i440fx-rhel7.6.0

virsh console

Works (sometimes)

Doesn’t work with WLC VMs

Preferred Console

Cockpit VNC + SSH

Cockpit VNC + SSH

IP Address

10.50.1.99

10.50.1.111

19.2. XML Modifications Required (kvm-01 → kvm-02)

When moving a VM definition from kvm-01 to kvm-02, apply these sed transformations:

# 1. Fix emulator path
sed -i 's|/usr/bin/qemu-system-x86_64|/usr/libexec/qemu-kvm|' /tmp/vm.xml
# 2. Fix machine type
sed -i "s/machine='pc-i440fx-10.1'/machine='pc-i440fx-rhel7.6.0'/" /tmp/vm.xml
# 3. Update bridge names (if needed)
sed -i "s|source bridge='virbr0'|source bridge='br-mgmt'|g" /tmp/vm.xml
# 4. Update disk path (if on local storage)
sed -i "s|/mnt/onboard-ssd/vms/|/mnt/nas/vms/|g" /tmp/vm.xml

19.3. XML Modifications Required (kvm-02 → kvm-01)

When moving a VM definition from kvm-02 back to kvm-01:

# 1. Fix emulator path
sed -i 's|/usr/libexec/qemu-kvm|/usr/bin/qemu-system-x86_64|' /tmp/vm.xml
# 2. Fix machine type (check available types on kvm-01)
# On kvm-01: qemu-system-x86_64 -machine help | grep pc-i440fx
sed -i "s/machine='pc-i440fx-rhel7.6.0'/machine='pc-i440fx-10.1'/" /tmp/vm.xml

19.4. Finding Available Machine Types

Each QEMU installation supports different machine types. Check before migrating:

# On kvm-01 (Arch)
qemu-system-x86_64 -machine help | grep pc-i440fx | head -5
# On kvm-02 (Rocky)
/usr/libexec/qemu-kvm -machine help | grep pc-i440fx | head -5

19.5. Console Access for WLC VMs

virsh console does NOT work with 9800-CL WLC VMs. The VGA ISO boots with graphical console only.

Use Cockpit VNC instead:

  1. Open browser: kvm-02:9090

  2. Login with admin credentials

  3. Navigate to: Virtual Machines → 9800-WLC-XX → Console

  4. After initial config, SSH to WLC directly

19.6. Key Learnings (2026-03-06)

  • XML is NOT portable between distros - Different QEMU packaging = different paths

  • Machine types vary by QEMU version - RHEL/Rocky uses rhel7.6.0 suffix

  • Always verify emulator path - virsh define fails silently with wrong path, then virsh start errors

  • NAS storage helps - Disk on /mnt/nas/vms/ accessible from both hosts

  • Document platform differences - Prevents future confusion during migrations

20. Appendix F: 3 NICs Attached But Only Gi1/Gi2 Detected

20.1. Symptom

virsh domiflist shows 3 NICs attached, but IOS-XE only detects Gi1 and Gi2:

sudo virsh domiflist 9800-WLC-01
libvirt shows 3 NICs
Interface  Type     Source    Model   MAC
-------------------------------------------------------
vnet0      bridge   br-mgmt   virtio  52:54:00:aa:aa:aa
vnet1      bridge   br-mgmt   virtio  52:54:00:bb:bb:bb
vnet2      bridge   br-mgmt   virtio  52:54:00:cc:cc:cc

But inside the WLC:

9800-WLC-01#show ip interface brief | include Gig
GigabitEthernet1       10.50.1.40      YES NVRAM  up         up
GigabitEthernet2       unassigned      YES unset  down       down

No Gi3 appears despite 3 vNICs being attached.

20.2. Root Cause: PCI Slot Ordering Corruption

IOS-XE maps PCI slot numbers to GigabitEthernet interfaces:

  • PCI slot 0 → GigabitEthernet1

  • PCI slot 1 → GigabitEthernet2

  • PCI slot 2 → GigabitEthernet3

When you use virsh attach-interface and virsh detach-interface multiple times (during VM migration, troubleshooting, etc.), libvirt assigns non-sequential PCI slots. IOS-XE can’t handle gaps or out-of-order slot assignments.

Example of corrupted PCI slots:

<!-- What we want (sequential) -->
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

<!-- What we get after attach/detach cycles (gaps) -->
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

20.3. Fix: Recreate VM with virt-install

The only reliable fix is to recreate the VM definition with all 3 --network options specified upfront. This ensures clean, sequential PCI slot assignments.

The qcow2 disk is preserved - all config (hostname, IP, VLANs) remains intact. Only the VM definition (XML) is recreated.

20.3.1. Step 1: Shutdown and Undefine

sudo virsh shutdown 9800-WLC-01
# Wait for clean shutdown
watch -n2 'sudo virsh list --all | grep -i wlc'
# Undefine removes XML, keeps disk
sudo virsh undefine 9800-WLC-01

20.3.2. Step 2: Recreate with 3 NICs

sudo virt-install \
  --name 9800-WLC-01 \
  --memory 16384 \
  --vcpus 4 \
  --import \
  --disk path=/mnt/nas/vms/9800-WLC-01.qcow2,format=qcow2 \
  --network bridge=br-mgmt,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --os-variant generic \
  --noautoconsole

Key options:

  • --import - Use existing qcow2 (preserves all config)

  • --network x3 - Creates 3 NICs with proper PCI slot ordering

  • No --graphics - Default to none (use SSH after boot)

20.3.3. Step 3: Verify 3 NICs

sudo virsh domiflist 9800-WLC-01
Expected (3 NICs with sequential vnet numbers)
Interface  Type     Source    Model   MAC
-------------------------------------------------------
vnet0      bridge   br-mgmt   virtio  52:54:00:xx:xx:xx
vnet1      bridge   br-mgmt   virtio  52:54:00:yy:yy:yy
vnet2      bridge   br-mgmt   virtio  52:54:00:zz:zz:zz

20.3.4. Step 4: Set PVID on All vNets

for VNET in $(sudo virsh domiflist 9800-WLC-01 | awk 'NR>2 {print $1}'); do
  echo "Setting PVID 100 on $VNET"
  sudo bridge vlan del vid 1 dev $VNET pvid untagged 2>/dev/null
  sudo bridge vlan add vid 100 dev $VNET pvid untagged
done

20.3.5. Step 5: Verify Inside WLC

Wait 2-3 minutes for IOS-XE boot, then verify all 3 interfaces:

ssh admin@10.50.1.40 "show ip interface brief | include Gig"
Expected (all 3 GigabitEthernet interfaces)
GigabitEthernet1       10.50.1.40      YES NVRAM  up         up
GigabitEthernet2       unassigned      YES unset  up         up
GigabitEthernet3       unassigned      YES unset  up         up

20.4. Prevention: Always Use virt-install for Multi-NIC VMs

NEVER use virsh attach-interface for 9800-CL WLCs.

Always recreate the VM definition with virt-install --network repeated for each NIC.
Correct approach (all NICs at creation time)
sudo virt-install \
  --name 9800-WLC-XX \
  --network bridge=br-mgmt,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  --network bridge=br-mgmt,model=virtio \
  ...
Incorrect approach (causes PCI slot corruption)
# DON'T DO THIS
sudo virsh attach-interface 9800-WLC-XX --type bridge --source br-mgmt --persistent
sudo virsh attach-interface 9800-WLC-XX --type bridge --source br-mgmt --persistent

20.5. Key Learnings (2026-03-06)

  • PCI slot ordering = interface mapping - Gi1/Gi2/Gi3 map to slots 3/4/5 (or similar sequential)

  • attach/detach corrupts slot ordering - libvirt fills gaps, causing non-sequential slots

  • virt-install creates clean slots - All --network options processed sequentially

  • qcow2 preserves config - Undefine + reimport keeps all IOS-XE settings

  • PVID still required - Set PVID 100 on all vnet interfaces after recreation