Keycloak Rebuild Runbook

Complete rebuild of keycloak-01 from scratch. Use when VM is corrupted or Keycloak needs fresh deployment.

Executive Summary

Item Value

VM

keycloak-01

Hypervisor

kvm-01

IP

10.50.1.80

OS

Fedora Cloud 43

Service

Keycloak (Docker Compose)

Purpose

SAML/OIDC IdP for ISE admin, dashboards

Prerequisites

  • SSH access to kvm-01

  • Vault SSH cert valid (vault-ssh-sign)

  • dsec credentials loaded (dsource d000 dev/identity)

  • Fedora Cloud image available on kvm-01

Phase 0: Discovery

Determine current state before proceeding.

0.1 Check VM State

ssh kvm-01 "sudo virsh list --all | grep -i keycloak"
Table 1. Possible outputs:
Output Action

keycloak-01 running

VM running - check container: 0.2 Check Container (if VM running)

keycloak-01 shut off

VM stopped - try start: 0.3 Start VM (if stopped)

keycloak-01 paused

VM paused - resume: sudo virsh resume keycloak-01

(no output)

VM missing - proceed to 1.1 Export Realm (if possible)

0.2 Check Container (if VM running)

ssh keycloak-01 "sudo docker ps -a"

If container unhealthy, try restart:

ssh keycloak-01 "cd /opt/keycloak && sudo docker compose restart"

0.3 Start VM (if stopped)

ssh kvm-01 "sudo virsh start keycloak-01"

Wait 30 seconds, then test SSH:

ssh keycloak-01 "hostnamectl"

If SSH works, skip to 4.1 Install Docker.

Phase 1: Destroy Corrupted VM

This destroys all data on the VM. Ensure you have realm export if needed.

1.1 Export Realm (if possible)

If Keycloak is still accessible:

# Export realm config before destruction
curl -ks -X POST "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms/domusdigitalis/partial-export?exportClients=true&exportGroupsAndRoles=true" \
  -H "Authorization: Bearer $(cat /tmp/kc-token)" \
  -o /tmp/domusdigitalis-realm-export.json

1.2 Destroy VM

ssh kvm-01 << 'EOF'
# Stop VM
sudo virsh destroy keycloak-01 2>/dev/null

# Remove VM and storage
sudo virsh undefine keycloak-01 --remove-all-storage

# Verify
sudo virsh list --all | grep keycloak
# Expected: (no output)
EOF

1.3 Clean Up Cloud-Init ISO

ssh kvm-01 "sudo rm -f /mnt/onboard-ssd/isos/keycloak-01-cloud-init.iso"

Phase 2: Create VM

2.1 Verify Fedora Cloud Image

ssh kvm-01 "ls -lh /mnt/onboard-ssd/vms/Fedora-Cloud-Base-Generic-43*.qcow2"

If missing, download:

ssh kvm-01 << 'EOF'
cd /mnt/onboard-ssd/vms
sudo curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/43/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-43-1.6.x86_64.qcow2
EOF

2.2 Create Cloud-Init ISO

ssh kvm-01 << 'EOF'
cd /mnt/onboard-ssd/isos

cat > meta-data <<METADATA
instance-id: keycloak-01
local-hostname: keycloak-01
METADATA

cat > user-data <<'USERDATA'
#cloud-config
hostname: keycloak-01
fqdn: keycloak-01.inside.domusdigitalis.dev
users:
  - name: evanusmodestus
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: wheel
    shell: /bin/bash
    lock_passwd: false
    plain_text_passwd: changeme
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFKcJlxTF8z3O3Y3F6lMdZMXfA8R0hbLs2KfQJYuJ+Yk evanusmodestus@modestus-razer
USERDATA

genisoimage -output keycloak-01-cloud-init.iso -volid cidata -joliet -rock user-data meta-data
rm user-data meta-data
ls -lh keycloak-01-cloud-init.iso
EOF

2.3 Create VM Disk

ssh kvm-01 << 'EOF'
cd /mnt/onboard-ssd/vms

# Copy base image
sudo cp Fedora-Cloud-Base-Generic-43-1.6.x86_64.qcow2 keycloak-01.qcow2

# Resize to 20GB
sudo qemu-img resize keycloak-01.qcow2 20G

# Verify
ls -lh keycloak-01.qcow2
EOF

2.4 Create VM

ssh kvm-01 << 'EOF'
sudo virt-install \
  --name keycloak-01 \
  --vcpus 2 \
  --memory 4096 \
  --disk /mnt/onboard-ssd/vms/keycloak-01.qcow2,bus=virtio \
  --disk /mnt/onboard-ssd/isos/keycloak-01-cloud-init.iso,device=cdrom \
  --os-variant fedora-unknown \
  --network bridge=virbr0,model=virtio \
  --graphics none \
  --import \
  --noautoconsole

# Verify running
sudo virsh list | grep keycloak
EOF

Phase 3: Configure VM

3.1 Connect to Console

ssh kvm-01 "sudo virsh console keycloak-01"

Login: evanusmodestus / changeme

Immediately change password:

passwd

3.2 Configure Static IP

sudo tee /etc/NetworkManager/system-connections/enp1s0.nmconnection <<'EOF'
[connection]
id=enp1s0
type=ethernet
interface-name=enp1s0
autoconnect=true

[ipv4]
method=manual
addresses=10.50.1.80/24
gateway=10.50.1.1
dns=10.50.1.90

[ipv6]
method=disabled
EOF

sudo chmod 600 /etc/NetworkManager/system-connections/enp1s0.nmconnection
sudo nmcli con reload
sudo nmcli con up enp1s0

3.3 Verify Network

# Check IP
ip -4 addr show enp1s0 | awk '/inet/{print $2}'
# Expected: 10.50.1.80/24

# Test gateway
ping -c2 10.50.1.1

# Test DNS
ping -c2 bind-01.inside.domusdigitalis.dev

Exit console: Ctrl+]

3.4 Eject Cloud-Init CD

ssh kvm-01 "sudo virsh change-media keycloak-01 sda --eject"

3.5 Test SSH

ssh keycloak-01 "hostnamectl"

Phase 4: Install Keycloak

4.1 Install Docker

ssh keycloak-01 << 'EOF'
# Install Docker
sudo dnf install -y docker docker-compose

# Enable and start
sudo systemctl enable --now docker

# Verify
sudo docker --version
sudo docker compose version
EOF

4.2 Create Directory Structure

ssh keycloak-01 << 'EOF'
sudo mkdir -p /opt/keycloak/certs
sudo chown -R $(id -u):$(id -g) /opt/keycloak
EOF

4.3 Create Docker Compose File

ssh keycloak-01 << 'ENDSSH'
cat > /opt/keycloak/docker-compose.yml <<'EOF'
services:
  postgres:
    image: postgres:16
    container_name: keycloak-db
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD: ${KC_DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  keycloak:
    image: quay.io/keycloak/keycloak:latest
    container_name: keycloak
    command: start
    environment:
      KC_DB: postgres
      KC_DB_URL: jdbc:postgresql://postgres:5432/keycloak
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD: ${KC_DB_PASSWORD}
      KC_HOSTNAME: keycloak-01.inside.domusdigitalis.dev
      KC_HTTPS_CERTIFICATE_FILE: /opt/keycloak/conf/cert.pem
      KC_HTTPS_CERTIFICATE_KEY_FILE: /opt/keycloak/conf/key.pem
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: ${KC_ADMIN_PASSWORD}
    ports:
      - "8443:8443"
    volumes:
      - ./certs:/opt/keycloak/conf:ro
    depends_on:
      - postgres
    restart: unless-stopped

volumes:
  postgres_data:
EOF
ENDSSH

4.4 Get TLS Certificate from Vault PKI

Issue certificate from Vault (not AD CS):

# On workstation - issue cert
vault write -format=json pki_int/issue/domus-server \
  common_name="keycloak-01.inside.domusdigitalis.dev" \
  alt_names="keycloak-01" \
  ttl="8760h" > /tmp/keycloak-cert.json

# Extract cert and key
jq -r '.data.certificate' /tmp/keycloak-cert.json > /tmp/cert.pem
jq -r '.data.private_key' /tmp/keycloak-cert.json > /tmp/key.pem
jq -r '.data.ca_chain[]' /tmp/keycloak-cert.json >> /tmp/cert.pem

# Verify
openssl x509 -in /tmp/cert.pem -noout -subject -dates

# Transfer to keycloak-01
scp /tmp/cert.pem /tmp/key.pem keycloak-01:/tmp/

4.5 Install Certificate

ssh keycloak-01 << 'EOF'
# Move certs to keycloak directory
sudo mv /tmp/cert.pem /tmp/key.pem /opt/keycloak/certs/

# Set ownership for Keycloak container (UID 1000)
sudo chown 1000:1000 /opt/keycloak/certs/*.pem

# Set permissions
sudo chmod 400 /opt/keycloak/certs/key.pem
sudo chmod 444 /opt/keycloak/certs/cert.pem

# SELinux context (Fedora)
sudo chcon -Rt svirt_sandbox_file_t /opt/keycloak/certs/

# Verify
ls -laZ /opt/keycloak/certs/
EOF

4.6 Start Keycloak

# Load secrets on workstation
eval "$(dsource d000 dev/identity)"

# Start containers with secrets
ssh keycloak-01 "cd /opt/keycloak && sudo env KC_DB_PASSWORD='$KC_DB_PASSWORD' KC_ADMIN_PASSWORD='$KC_ADMIN_PASSWORD' docker compose up -d"

4.7 Verify Deployment

# Check containers
ssh keycloak-01 "sudo docker ps"

# Check logs (wait for startup)
ssh keycloak-01 "sudo docker logs keycloak 2>&1 | tail -30"

# Test HTTPS
curl -sk https://keycloak-01.inside.domusdigitalis.dev:8443/ -w "%{http_code}\n" -o /dev/null
# Expected: 302

# Verify certificate
echo | openssl s_client -connect keycloak-01.inside.domusdigitalis.dev:8443 2>/dev/null | openssl x509 -noout -subject -issuer

Phase 5: Configure Realm

5.1 Create Realm

Access Keycloak admin console:

https://keycloak-01.inside.domusdigitalis.dev:8443/admin/

Login: admin / (from dsec KC_ADMIN_PASSWORD)

  1. Click Create realm

  2. Name: domusdigitalis

  3. Click Create

5.2 Create ISE SAML Client

  1. Clients → Create client

  2. Client type: SAML

  3. Client ID: (get from ISE SP metadata - format CiscoISE/{UUID})

  4. Configure:

5.3 Create Groups

Create groups that map to ISE admin roles:

Keycloak Group ISE Admin Group

ise-super-admin

Super Admin

ise-read-only

Read Only Admin

ise-helpdesk

Help Desk Admin

5.4 Add User to Group

netapi keycloak add-user-to-group domusdigitalis evanusmodestus ise-super-admin

5.5 Export SAML Metadata

netapi keycloak get-saml-metadata domusdigitalis -o /tmp/keycloak-saml-metadata.xml

Import this XML into ISE: Administration → Identity Management → External Identity Sources → SAML Id Providers

Phase 6: Validation

6.1 Test SAML Login

  1. Open ISE admin portal: ise-02.inside.domusdigitalis.dev/admin/

  2. Click Log in with SAML

  3. Authenticate with Keycloak credentials

  4. Verify ISE admin access

6.2 Verify Group Mapping

After SAML login, check ISE shows correct admin group.

Rollback

If rebuild fails, restore from previous state:

Option A: Restore VM from Snapshot

ssh kvm-01 "sudo virsh snapshot-revert keycloak-01 --snapshotname pre-rebuild"

Option B: Restore Realm from Export

If you exported realm in Phase 1:

# Import realm
curl -ks -X POST "https://keycloak-01.inside.domusdigitalis.dev:8443/admin/realms" \
  -H "Authorization: Bearer $(cat /tmp/kc-token)" \
  -H "Content-Type: application/json" \
  -d @/tmp/domusdigitalis-realm-export.json

Related