Phase 6: SAML, Cutover & Validation

Phase 6: SAML, Cutover & Validation

Reconfigure Keycloak SAML (Phase 5a)

ISE generates a new Entity ID with each deployment. The Keycloak client must be updated.

  1. In ISE: Administration → Identity Management → External Identity Sources → SAML Id Providers

  2. Export Service Provider Info — note the Entity ID

  3. In Keycloak: update the ISE client’s Client ID to match the new Entity ID

  4. Update Valid Redirect URIs: ise-01.inside.domusdigitalis.dev:8443/*

  5. Test: log in via SAML at ise-01.inside.domusdigitalis.dev/admin

Fallback local login: ise-01.inside.domusdigitalis.dev/admin/LoginAction.do?local=true

Fix VyOS Firewall for NTP (if not already done)

VyOS MGMT_LOCAL drops NTP (UDP 123) by default. Add a rule:

# On vyos-01 (and vyos-02)
configure
set firewall ipv4 name MGMT_LOCAL rule 45 action accept
set firewall ipv4 name MGMT_LOCAL rule 45 description 'NTP'
set firewall ipv4 name MGMT_LOCAL rule 45 destination port 123
set firewall ipv4 name MGMT_LOCAL rule 45 protocol udp
commit
save
Verify from workstation (heredoc — avoids quoting issues)
python3 <<'EOF'
import socket, struct, time
msg = b'\x1b' + 47 * b'\0'
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(3)
s.sendto(msg, ('10.50.1.1', 123))
data, _ = s.recvfrom(1024)
t = struct.unpack('!12I', data[:48])[10] - 2208988800
print("VyOS NTP response:", time.ctime(t))
s.close()
EOF
Check VyOS firewall counters (proof of fix)
# From VyOS operational mode ($ prompt)
show firewall ipv4 name MGMT_LOCAL

# If in configure mode (# prompt), prefix with 'run':
run show firewall ipv4 name MGMT_LOCAL

# Rule 45 should show incrementing packet count
# NOTE: 'show firewall' without 'run' in configure mode shows config, NOT counters

Change ISE NTP to VyOS (Remove pool.ntp.org)

configure terminal
ntp server 10.50.1.1
no ntp server pool.ntp.org
exit
Verify ISE NTP sync
show ntp
Expected — VyOS VIP as selected source (^*), Stratum 3, Leap: Normal
^* vyos-vip.inside.domusdig>     2   6    17     0   -380ns[  -13us] +/- 6574us

Switch NAD Cutover (3560CX)

Connect via console:

sudo picocom -b 9600 /dev/ttyUSB0
Verify current RADIUS config
show run aaa
show radius server-group all
show radius server-group ISE-RADIUS
Swap ISE-RADIUS group: ise-01 primary, ise-02 fallback
configure terminal
aaa group server radius ISE-RADIUS
  no server name ISE-02
  server name ISE-01
  server name ISE-02
  exit
end
Verify and save
show radius server-group ISE-RADIUS
write memory
Expected — ise-01 (10.50.1.20) listed first
Server group ISE-RADIUS
    Server(10.50.1.20:1812,1813)
    Server(10.50.1.21:1812,1813)

Both CoA (Change of Authorization) clients are already configured in ISE from the restore:

aaa server radius dynamic-author
 client 10.50.1.20 server-key 7 ...
 client 10.50.1.21 server-key 7 ...

No change needed — ISE can push CoA from either node.

WLC NAD Cutover (9800-CL)

Connect via SSH (WLC uses password auth, not vault cert):

ssh -o PubkeyAuthentication=no admin@9800-wlc-01.inside.domusdigitalis.dev
# Or via sudo if SSH config conflicts:
sudo ssh admin@9800-wlc-01.inside.domusdigitalis.dev
Verify current RADIUS config
show run | section radius
show aaa servers
Swap ISE-GROUP: ise-01 primary, ise-02 fallback
configure terminal
aaa group server radius ISE-GROUP
  no server name ISE-2
  no server name ISE-1
  server name ISE-1
  server name ISE-2
  exit
end
Verify and save
show aaa servers
write memory

The 9800 WLC may display %Failed to update configuration in the database — this is a known cosmetic bug. The config applies correctly despite the error. Verify with show aaa servers — ISE-1 should appear first with State: current UP.
Test WiFi re-authentication from workstation
sudo nmcli connection down "Domus-WiFi-Mgmt-VLAN100" && \
sudo nmcli connection up "Domus-WiFi-Mgmt-VLAN100"
Verify connection dashboard
nmcli -t -f NAME,TYPE,DEVICE con show | awk -F: '{t=$2;
gsub("802-11-wireless","wifi",t); gsub("802-3-ethernet","eth",t);
printf "%s %-30s %-8s %s\n",($3?"active":"offline"),$1,t,($3?$3:"offline")}'

Rejoin Active Directory

The restore left AD. Rejoin from ISE GUI:

  1. Administration → Identity Management → External Identity Sources → Active Directory

  2. Select DOMUS_AD → Join

  3. Credentials: gopass show -o v2/DOMUS/ad/svc-ise-adjoin (user: svc-ise-adjoin)

  4. OU: OU=ISE,OU=Tier 1 - Servers,DC=inside,DC=domusdigitalis,DC=dev

Update dsec Environment

dsec edit d000 dev/network
# Change ISE_PAN_FQDN to ise-01.inside.domusdigitalis.dev
# Change ISE_PAN_IP to 10.50.1.20
dsource d000 dev/network

Test Authentication (One Client First)

sudo nmcli connection up "Wired-802.1X-Vault"
journalctl -u wpa_supplicant@eth0 -f

Shut Down Old Node

ssh kvm-02 "sudo virsh shutdown ise-02"

API Validation (All 5 Surfaces)

# ERS
netapi ise get-cert-profiles

# OpenAPI
netapi ise api-call openapi GET '/api/v1/deployment/node' \
  | jq -r '.response[] | "\(.hostname) (\(.fqdn))"'

# MnT
netapi ise mnt count

# DataConnect
netapi ise dc test

# pxGrid
netapi ise pxgrid test

Post-Cutover Monitoring

# Recent authentications
netapi ise dc query "SELECT USERNAME, ISE_NODE, POLICY_SET_NAME, PASSED
FROM RADIUS_AUTHENTICATIONS
WHERE TIMESTAMP_TIMEZONE > SYSDATE - INTERVAL '1' HOUR
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 20 ROWS ONLY"

# Failures
netapi ise dc query "SELECT USERNAME, FAILURE_REASON, TIMESTAMP_TIMEZONE
FROM RADIUS_AUTHENTICATIONS
WHERE PASSED = 0 AND TIMESTAMP_TIMEZONE > SYSDATE - 1
ORDER BY TIMESTAMP_TIMEZONE DESC
FETCH FIRST 20 ROWS ONLY"

Keep ise-02 for Rollback (1-2 weeks)

Do NOT decommission immediately. If issues arise:

ssh kvm-01 "sudo virsh shutdown ise-01"
ssh kvm-02 "sudo virsh start ise-02"
# Revert dsec environment back to ise-02