Runbook Template

Overview

Purpose: What this runbook accomplishes.

When to use: Circumstances requiring this procedure.

Time estimate: X minutes (uninterrupted)

Risk level: Low / Medium / High

Pre-Flight Checklist

Complete ALL checks before proceeding.

  • Verify you have root/sudo access

  • Backup critical data: rsync -av /path/to/data /backup/location/

  • Confirm maintenance window (if applicable)

  • Notify stakeholders (if applicable)

  • Check current state:

# Capture current state for rollback reference
systemctl status service-name
cat /etc/config/file > /tmp/config-backup-$(date +%F)

Variables

Set these before running commands:

# REQUIRED - Set these values
export TARGET_HOST="hostname"
export SERVICE_NAME="service"
export BACKUP_PATH="/backup/$(date +%F)"

# Verify
echo "Target: $TARGET_HOST, Service: $SERVICE_NAME, Backup: $BACKUP_PATH"

Execution

Step 1: [First Action]

Purpose: Why this step is necessary.

# Command with explanation
sudo systemctl stop $SERVICE_NAME

Expected output:

[expected output here]

If this fails: Jump to Rollback Procedure.

Step 2: [Second Action]

# Next command

Checkpoint: Verify step completed:

# Verification command

Step 3: [Third Action]

  • Arch

  • RHEL/Fedora

  • Debian/Ubuntu

sudo pacman -S package
sudo dnf install package
sudo apt install package

Step 4: Restart Services

sudo systemctl daemon-reload
sudo systemctl start $SERVICE_NAME
sudo systemctl enable $SERVICE_NAME

Verification

Do not skip verification.

Functional Tests

# Test 1: Service running
systemctl is-active $SERVICE_NAME

# Test 2: Ports listening
ss -tlnp | grep service

# Test 3: Functionality test
curl -s http://localhost:port/health

Expected Results

Test Expected Actual Pass/Fail

Service status

active (running)

_

[ ]

Port listening

LISTEN on :port

_

[ ]

Health check

HTTP 200

_

[ ]

Post-Execution

Cleanup

# Remove temporary files
rm -f /tmp/config-backup-*

# Unset variables
unset TARGET_HOST SERVICE_NAME BACKUP_PATH

Documentation

  • Update CMDB/asset inventory

  • Record changes in change management system

  • Notify stakeholders of completion

Rollback Procedure

Use if any step fails or verification fails.

Quick Rollback

# Stop the broken service
sudo systemctl stop $SERVICE_NAME

# Restore configuration
sudo cp /tmp/config-backup-* /etc/config/file

# Restart with old config
sudo systemctl start $SERVICE_NAME

Full Rollback

If quick rollback fails:

# Nuclear option - reinstall
sudo systemctl stop $SERVICE_NAME
sudo dnf remove package  # or pacman -Rns / apt purge
sudo dnf install package
sudo cp /tmp/config-backup-* /etc/config/file
sudo systemctl start $SERVICE_NAME

Troubleshooting

Common Issues

Issue Cause Resolution

Service won’t start

Config syntax error

journalctl -u $SERVICE_NAME -n 50

Permission denied

SELinux/AppArmor

ausearch -m avc -ts recent (RHEL)

Port already in use

Zombie process

fuser -k port/tcp

Related