Incident Response

Incident response lifecycle — detection, triage, containment, and recovery procedures.

Incident Response

NIST IR Lifecycle (SP 800-61)

Phase 1: PREPARATION
  - IR plan documented and tested
  - Tools ready (forensic workstation, jump bag, contact list)
  - Logging configured and validated
  - Team roles assigned

Phase 2: DETECTION & ANALYSIS
  - Alert triage — true positive or false positive?
  - Scope assessment — how many systems affected?
  - Evidence collection — volatile first
  - IOC identification — IPs, hashes, domains, user accounts

Phase 3: CONTAINMENT, ERADICATION & RECOVERY
  - Short-term containment: isolate the host, block the IP
  - Evidence preservation: image before cleanup
  - Eradication: remove malware, close access vector
  - Recovery: restore from known-good, validate, monitor

Phase 4: POST-INCIDENT ACTIVITY
  - Lessons learned meeting (within 5 business days)
  - Root cause analysis (RCA)
  - Update detection rules
  - Update IR playbook

Volatile Data Collection — First 5 Minutes

Collect in order of volatility. Run on the live system before any containment action. Save output to external media or network share.

Network connections — who is talking to this host right now

ss -tulnp | tee /media/usb/netstat_$(hostname)_$(date +%s).txt

Running processes with full command lines

ps auxf | tee /media/usb/ps_$(hostname)_$(date +%s).txt

Established connections with process mapping — find the C2 channel

ss -tnp state established | tee /media/usb/established_$(hostname)_$(date +%s).txt

Logged-in users and their activity

w | tee /media/usb/who_$(hostname)_$(date +%s).txt

Open files — what processes have which files open

lsof -nP | tee /media/usb/lsof_$(hostname)_$(date +%s).txt

Loaded kernel modules — check for rootkits

lsmod | tee /media/usb/lsmod_$(hostname)_$(date +%s).txt

ARP cache and routing table — network context

ip neigh | tee /media/usb/arp_$(hostname)_$(date +%s).txt
ip route | tee /media/usb/routes_$(hostname)_$(date +%s).txt

DNS cache — recent lookups (systemd-resolved)

resolvectl statistics | tee /media/usb/dns_stats_$(hostname)_$(date +%s).txt

Containment Actions

Network isolation — block all traffic except management

# Aggressive: drop all traffic, allow only SSH from jump host
sudo iptables -I INPUT -s 10.50.1.5 -p tcp --dport 22 -j ACCEPT
sudo iptables -I INPUT -j DROP
sudo iptables -I OUTPUT -d 10.50.1.5 -p tcp --sport 22 -j ACCEPT
sudo iptables -I OUTPUT -j DROP

Block a specific IOC IP at the firewall

# Linux host firewall
sudo iptables -I OUTPUT -d 203.0.113.50 -j DROP
sudo iptables -I INPUT -s 203.0.113.50 -j DROP

# Verify
sudo iptables -L -n | awk '/203\.0\.113\.50/ {print}'

Disable a compromised user account

# Lock the account — prevents password login
sudo passwd -l compromised_user

# Expire the account — prevents all login methods
sudo usermod -e 1 compromised_user

# Kill all sessions for that user
sudo pkill -u compromised_user

# Verify
sudo passwd -S compromised_user

Disable SSH authorized keys for a compromised account

sudo mv /home/compromised_user/.ssh/authorized_keys \
  /evidence/case001/compromised_authorized_keys
sudo chmod 000 /home/compromised_user/.ssh/

Evidence Collection — Disk Preservation

Create a forensic disk image before eradication

# Always image BEFORE cleanup — you can't un-delete evidence
sudo dc3dd if=/dev/sda of=/evidence/case001/$(hostname)_disk.img \
  hash=sha256 log=/evidence/case001/$(hostname)_imaging.log

Collect logs to evidence directory

CASE="/evidence/case001"
HOST=$(hostname)
mkdir -p "$CASE/$HOST"

# Authentication logs
cp -a /var/log/auth.log* "$CASE/$HOST/"
cp -a /var/log/secure* "$CASE/$HOST/" 2>/dev/null

# System logs
cp -a /var/log/syslog* "$CASE/$HOST/"
cp -a /var/log/messages* "$CASE/$HOST/" 2>/dev/null

# Application-specific
cp -a /var/log/apache2/ "$CASE/$HOST/apache2/" 2>/dev/null
cp -a /var/log/nginx/ "$CASE/$HOST/nginx/" 2>/dev/null

# Login records
last -aiF > "$CASE/$HOST/last.txt"
lastb -aiF > "$CASE/$HOST/lastb.txt" 2>/dev/null
lastlog > "$CASE/$HOST/lastlog.txt"

# Hash all collected evidence
find "$CASE/$HOST" -type f -exec sha256sum {} \; > "$CASE/$HOST/evidence_hashes.sha256"

IOC Checking

Check IP reputation with curl — abuse databases

# AbuseIPDB
curl -s -H "Key: $ABUSEIPDB_KEY" \
  "https://api.abuseipdb.com/api/v2/check?ipAddress=203.0.113.50&maxAgeInDays=90" \
  | jq '{ip: .data.ipAddress, abuse_score: .data.abuseConfidenceScore, country: .data.countryCode, total_reports: .data.totalReports}'

Check file hash against VirusTotal

HASH=$(sha256sum suspicious_file | awk '{print $1}')
curl -s -H "x-apikey: $VT_API_KEY" \
  "https://www.virustotal.com/api/v3/files/$HASH" \
  | jq '{malicious: .data.attributes.last_analysis_stats.malicious, undetected: .data.attributes.last_analysis_stats.undetected, name: .data.attributes.meaningful_name}'

Search for IOCs across log files — IP, domain, or hash

# Search all logs for a suspicious IP
grep -r "203.0.113.50" /var/log/ 2>/dev/null | head -20

# Search for a malicious domain
grep -rP "evil-domain\.com" /var/log/ 2>/dev/null

# Find files matching a known-bad hash
find / -type f -exec sha256sum {} \; 2>/dev/null | grep "abc123deadbeef"

Check for unauthorized SSH keys across all users

for keyfile in /home/*/.ssh/authorized_keys /root/.ssh/authorized_keys; do
  if [ -f "$keyfile" ]; then
    echo "=== $keyfile ==="
    awk '{print NR": "$0}' "$keyfile"
    echo "Modified: $(stat -c '%y' "$keyfile")"
    echo
  fi
done

Eradication

Remove malware — after imaging and evidence collection

# Identify the malicious file
file /tmp/.hidden_binary
sha256sum /tmp/.hidden_binary | tee /evidence/case001/malware_hash.txt

# Remove it
sudo rm /tmp/.hidden_binary

# Check for persistence
sudo grep -r ".hidden_binary" /etc/systemd/ /etc/cron* /etc/init.d/ /home/*/

Check for unauthorized cron jobs

# System cron
cat /etc/crontab
ls -la /etc/cron.d/

# Per-user cron
for user in $(awk -F: '$3>=1000 {print $1}' /etc/passwd) root; do
  echo "=== $user ==="
  sudo crontab -u "$user" -l 2>/dev/null
done

Check for unauthorized systemd services — find recently added

# Services created after OS installation
find /etc/systemd/system/ /usr/lib/systemd/system/ \
  -name "*.service" -newer /etc/os-release -type f \
  -exec ls -la {} \;

Check for LD_PRELOAD hijacking — rootkit technique

# This file should not exist or should be empty
cat /etc/ld.so.preload 2>/dev/null
echo "Exit code: $?"

# Check environment for LD_PRELOAD
env | grep LD_PRELOAD

Recovery

Restore from known-good backup — verify integrity first

# Verify backup integrity
sha256sum /backups/web-server-01_2026-04-08.tar.gz

# Restore
sudo tar xzf /backups/web-server-01_2026-04-08.tar.gz -C /

# Reset all passwords for affected accounts
sudo passwd affected_user

Post-recovery monitoring — watch for re-compromise

# Monitor authentication in real time
tail -f /var/log/auth.log | awk '/Failed|Accepted|session opened/ {print strftime("%H:%M:%S"), $0}'

# Watch network connections for C2 callback
watch -n 5 'ss -tnp state established | awk "!/127.0.0.1/ && !/::1/ {print}"'

Communication Templates

Initial notification to management

Subject: Security Incident — [Severity] — [Brief Description]

At [TIME UTC], the SOC detected [brief description of the incident].

Current Status: [Detection / Containment / Eradication / Recovery]
Affected Systems: [List]
Affected Users: [Count or list]
Business Impact: [Description]

Actions Taken:
1. [Action 1]
2. [Action 2]

Next Steps:
1. [Planned action]

Next update: [Time]

Incident Commander: [Name]

Escalation criteria

P1 (Critical): Active breach, data exfiltration, ransomware
   → Notify CISO within 15 minutes
   → All-hands IR team activation

P2 (High): Compromised account, malware detected, unauthorized access
   → Notify Security Manager within 1 hour
   → Dedicated IR team assigned

P3 (Medium): Policy violation, suspicious activity, failed attack
   → Notify SOC Lead within 4 hours
   → SOC analyst handles

P4 (Low): Informational, vulnerability discovered, false positive investigation
   → Document and track
   → Handle during business hours