KVM High Availability with Shared Storage

Overview

This runbook documents KVM high availability configuration using Synology NAS as shared storage, enabling live VM migration between kvm-01 and kvm-02.

KVM HA Architecture

Architecture

Hypervisor Cluster

Host Hardware Resources Role

kvm-01

Supermicro A

16 cores, 64GB RAM

Primary hypervisor

kvm-02

Supermicro B

16 cores, 64GB RAM

Secondary hypervisor

Storage Tiers

Tier Path Use Case

Local SSD

/mnt/onboard-ssd/libvirt/images/

Performance-critical VMs (ISE, WLC, VyOS, DC)

NAS NFS

/mnt/nas/libvirt/

HA-capable VMs (k3s, Vault, Keycloak, BIND)

VM Classification

HA VMs (NAS Storage) - Can live migrate between hypervisors:

  • k3s-master-* (1-3)

  • vault-* (1-3)

  • keycloak-01

  • bind-01

Performance VMs (Local SSD) - Pinned to single hypervisor:

  • ise-01/02 (Cisco ISE)

  • 9800-WLC-01/02 (Cisco WLC)

  • vyos-01/02 (VyOS HA)

  • home-dc01 (Windows DC)

Phase 1: NAS Configuration

1.1 Create NFS Export on Synology

# SSH to NAS
ssh nas-01

# Create directory structure
sudo mkdir -p /volume1/libvirt/images
sudo chown -R 1000:1000 /volume1/libvirt

Via DSM Web UI:

  1. Control Panel → Shared Folder → Create

  2. Name: libvirt

  3. Location: Volume 1

  4. Enable NFS permissions:

    • Hostname: 10.50.1.98 (kvm-01), 10.50.1.99 (kvm-02)

    • Privilege: Read/Write

    • Squash: Map root to admin

    • Security: sys

    • Enable async

1.2 Verify NFS Export

# From NAS
showmount -e localhost
Expected output
Export list for localhost:
/volume1/libvirt 10.50.1.98,10.50.1.99

Phase 2: Hypervisor NFS Mount

Run on both kvm-01 and kvm-02:

2.1 Install NFS Client

sudo pacman -S nfs-utils
sudo systemctl enable --now nfs-client.target

2.2 Create Mount Point

sudo mkdir -p /mnt/nas/libvirt

2.3 Add fstab Entry

echo '10.50.1.70:/volume1/libvirt /mnt/nas/libvirt nfs4 defaults,_netdev,soft,timeo=150,retrans=3 0 0' | sudo tee -a /etc/fstab

2.4 Mount and Verify

sudo mount -a
df -h /mnt/nas/libvirt

2.5 Test Write Access

# On kvm-01
echo "test from kvm-01" | sudo tee /mnt/nas/libvirt/test-kvm-01.txt

# On kvm-02
cat /mnt/nas/libvirt/test-kvm-01.txt
echo "test from kvm-02" | sudo tee /mnt/nas/libvirt/test-kvm-02.txt

# Cleanup
sudo rm /mnt/nas/libvirt/test-*.txt

Phase 3: Libvirt Storage Pool

Run on both hypervisors:

3.1 Define NAS Storage Pool

sudo virsh pool-define-as nas-libvirt dir --target /mnt/nas/libvirt
sudo virsh pool-autostart nas-libvirt
sudo virsh pool-start nas-libvirt

3.2 Verify Pool

sudo virsh pool-list --all
sudo virsh pool-info nas-libvirt

Phase 4: VM Migration

4.1 Prerequisites

Before migration:

# Verify VM uses NAS storage
sudo virsh domblklist <vm-name> | awk 'NR>2 {print $2}'

# Verify both hypervisors can access the disk
ls -la /mnt/nas/libvirt/<vm-disk>.qcow2

4.2 Live Migration Command

# From source hypervisor (e.g., kvm-01)
sudo virsh migrate --live --persistent --undefine-source \
  <vm-name> \
  qemu+ssh://root@kvm-02/system

4.3 Migration with Verbose Output

sudo virsh migrate --live --persistent --undefine-source --verbose \
  <vm-name> \
  qemu+ssh://root@kvm-02/system

4.4 Verify Migration

# On destination (kvm-02)
sudo virsh list --all | grep <vm-name>

# Verify VM is running
sudo virsh domstate <vm-name>

Phase 5: Moving Existing VMs to NAS

For VMs currently on local SSD that should be HA-capable:

5.1 Shutdown VM

sudo virsh shutdown <vm-name>
# Wait for clean shutdown
sudo virsh domstate <vm-name>

5.2 Copy Disk to NAS

sudo rsync -avP /mnt/onboard-ssd/libvirt/images/<vm-name>.qcow2 /mnt/nas/libvirt/

5.3 Update VM Configuration

sudo virsh edit <vm-name>

Change disk source path:

<source file='/mnt/nas/libvirt/<vm-name>.qcow2'/>

5.4 Start and Verify

sudo virsh start <vm-name>
sudo virsh domblklist <vm-name>

5.5 Remove Old Disk (After Verification)

# Only after confirming VM works correctly
sudo rm /mnt/onboard-ssd/libvirt/images/<vm-name>.qcow2

Troubleshooting

NFS Mount Fails

# Check NFS server is reachable
showmount -e 10.50.1.70

# Check firewall on NAS
# DSM → Control Panel → Security → Firewall

# Manual mount test
sudo mount -t nfs4 10.50.1.70:/volume1/libvirt /mnt/nas/libvirt -v

Migration Fails: "Cannot access storage"

Both hypervisors must have identical paths to the storage:

# Verify on both hosts
ls -la /mnt/nas/libvirt/<vm-disk>.qcow2

Migration Fails: SSH Key

# Ensure root SSH between hypervisors works
sudo ssh root@kvm-02 hostname
sudo ssh root@kvm-01 hostname

Migration Hangs

# Check migration status
sudo virsh domjobinfo <vm-name>

# Cancel stuck migration
sudo virsh domjobabort <vm-name>

Quick Reference

Task Command

List NAS pool VMs

sudo virsh vol-list nas-libvirt

Check VM disk location

sudo virsh domblklist <vm> | awk 'NR>2'

Live migrate

sudo virsh migrate --live --persistent --undefine-source <vm> qemu+ssh://root@kvm-02/system

Migration status

sudo virsh domjobinfo <vm>

Pool status

sudo virsh pool-info nas-libvirt