Runbook: iPSK Manager Failover

Last Updated

2026-01-25

Owner

evanusmodestus

Frequency

As Needed (on primary failure)

Purpose

Failover iPSK Manager from primary (ipsk-mgr-01) to secondary (ipsk-mgr-02) when primary server is unavailable.

Prerequisites

  • Secondary server with MySQL slave configured and synced

  • ISE admin credentials for ODBC configuration

  • SSH access to secondary server

Procedure

Step 1: Verify Primary is Down

ping {ipsk-mgr}
ssh {ipsk-mgr} "systemctl status apache2 mysql"

Expected output (primary down):

ping: ipsk-mgr-01: No route to host
# OR
ssh: connect to host ipsk-mgr-01 port 22: Connection refused

Step 2: Check Replication Status on Secondary

ssh ipsk-mgr-02 "mysql -e 'SHOW SLAVE STATUS\G'" | grep -E "Slave_IO|Slave_SQL|Seconds_Behind"

Expected output:

Slave_IO_Running: Yes (or Connecting if primary down)
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0 (or NULL if primary down)

Step 3: Promote Secondary to Master

ssh ipsk-mgr-02 "mysql -e 'STOP SLAVE; RESET SLAVE ALL; SET GLOBAL read_only = 0;'"

Verify:

ssh ipsk-mgr-02 "mysql -e 'SELECT @@read_only;'"

Expected output:

@@read_only
0

Step 4: Verify Apache Running on Secondary

ssh ipsk-mgr-02 "systemctl status apache2"

If not running:

ssh ipsk-mgr-02 "sudo systemctl start apache2"

Step 5: Update ISE ODBC Configuration

  1. Navigate to: Administration > Identity Management > External Identity Sources > ODBC

  2. Click iPSKManager (or your ODBC source name)

  3. Change Hostname/IP from primary IP to secondary IP

  4. Click Test Connection - must show success

  5. Click Save

Step 6: Verify Authentication

Test a device connection:

# From netapi (home lab)
dsource d000 dev/network
netapi ise mnt sessions --ssid IoT_Net

Or check ISE Live Logs:

  • Navigate to: Operations > RADIUS > Live Logs

  • Filter by SSID or MAC address

  • Verify successful authentication with iPSKManager identity store

Verification

# Verify secondary MySQL is master (read_only=0)
ssh ipsk-mgr-02 "mysql -e 'SELECT @@read_only, @@server_id;'"

# Verify Apache responding
curl -sk https://ipsk-mgr-02.{internal-domain}/ | head -5

# Verify ISE ODBC
# Test Connection in ISE GUI should succeed

# Verify device authentication
# Test device WiFi reconnection

Troubleshooting

Issue 1: ISE ODBC Test Connection Fails

Symptom: "Connection failed" after updating IP

Cause: Firewall, MySQL not listening, wrong credentials

Resolution:

# Check MySQL listening
ssh ipsk-mgr-02 "ss -tlnp | grep 3306"

# Check firewall
ssh ipsk-mgr-02 "sudo ufw status"

# Test MySQL connection from ISE subnet
mysql -h ipsk-mgr-02 -u iseodbc -p ipsk -e "SELECT 1;"

Issue 2: Device Authentication Fails After Failover

Symptom: ISE returns "Authentication failed" or "User not found"

Cause: ODBC not updated, stored procedure missing, replication gap

Resolution:

# Verify ODBC pointing to secondary
# Check ISE GUI: Administration > ODBC > View Hostname

# Verify stored procedures exist
ssh ipsk-mgr-02 "mysql ipsk -e 'SHOW PROCEDURE STATUS WHERE Db = \"ipsk\";'"

# Verify endpoint exists
ssh ipsk-mgr-02 "mysql ipsk -e 'SELECT macAddress, vlan FROM endpoints LIMIT 5;'"

Issue 3: read_only Still Enabled

Symptom: iPSK Manager UI shows errors on save

Cause: Forgot to disable read_only after promotion

Resolution:

ssh ipsk-mgr-02 "mysql -e 'SET GLOBAL read_only = 0;'"

Rollback

If secondary is unstable after promotion:

# 1. If primary is restored, revert ISE ODBC to primary IP
# (via ISE GUI as in Step 5)

# 2. Reconfigure secondary as slave
ssh ipsk-mgr-02 "mysql -e '
  SET GLOBAL read_only = 1;
  CHANGE MASTER TO
    MASTER_HOST=\"ipsk-mgr-01\",
    MASTER_USER=\"repl_user\",
    MASTER_PASSWORD=\"<password>\",
    MASTER_AUTO_POSITION=1;
  START SLAVE;
'"

Notes

  • RTO target: < 15 minutes

  • ISE caches ODBC connections; may need to wait 1-2 minutes after update

  • Document actual failover time for compliance reporting