Threat Hunting

Overview

Threat hunting is proactive searching for indicators of compromise (IOCs) that evade traditional detection. These commands help identify unauthorized access, persistence mechanisms, and lateral movement.

Run these on systems you own or have authorization to test. Some commands require root privileges.

Quick Triage

Run this first when investigating a potentially compromised system:

# One-liner triage
echo "=== QUICK TRIAGE ===" && \
w && echo "---" && \
last -10 && echo "---" && \
ss -tlnp && echo "---" && \
ps aux --sort=-%cpu | head -10

Persistence Hunting

Attackers establish persistence to survive reboots. Check these locations:

Cron Jobs

# User crontabs
crontab -l 2>/dev/null
for user in $(cut -f1 -d: /etc/passwd); do
  echo "=== $user ===" && crontab -u $user -l 2>/dev/null
done

# System cron directories
ls -la /etc/cron.*
cat /etc/crontab

# Anacron
cat /etc/anacrontab 2>/dev/null

systemd Services

# Enabled services (what starts at boot)
systemctl list-unit-files --state=enabled

# Running services
systemctl list-units --type=service --state=running

# Recently created service files (suspicious)
find /etc/systemd /usr/lib/systemd -name "*.service" -mtime -7 -ls 2>/dev/null

# User-level services (often overlooked)
ls -la ~/.config/systemd/user/*.service 2>/dev/null

Shell Startup Files

# Check for modifications
ls -la ~/.bashrc ~/.bash_profile ~/.profile ~/.zshrc 2>/dev/null

# Look for suspicious additions
grep -E "curl|wget|nc|bash -i|/dev/tcp" ~/.bashrc ~/.zshrc ~/.profile 2>/dev/null

# System-wide
ls -la /etc/profile.d/
cat /etc/profile

SSH Authorized Keys

# SSH Authorized Keys Audit - shows permissions, types, fingerprints
find /home -name "authorized_keys" -exec sh -c '
  echo "--- {} ---"
  echo "Permissions: $(stat -c "%a %U:%G" "{}")"
  echo "Keys:"
  while read -r line; do
    [[ -z "$line" || "$line" =~ ^# ]] && continue
    type=$(echo "$line" | awk "{print \$1}")
    fp=$(echo "$line" | ssh-keygen -lf - 2>/dev/null | awk "{print \$2}")
    comment=$(echo "$line" | awk "{print \$NF}")
    options=$(echo "$line" | grep -oE "^[^s].*=" | head -1)
    printf "  %-12s %-47s %s %s\n" "$type" "$fp" "$comment" "${options:+[OPTIONS: $options]}"
  done < "{}"
' \; 2>/dev/null

# Root authorized_keys (should be empty or non-existent)
cat /root/.ssh/authorized_keys 2>/dev/null

File System Analysis

Recently Modified Files

# Modified in last 24 hours (exclude /proc, /sys)
find / -type f -mtime -1 -ls 2>/dev/null | grep -v -E "^/(proc|sys|run)" | head -50

# Specifically in sensitive directories
find /etc /usr/bin /usr/sbin -type f -mtime -7 -ls 2>/dev/null

Hidden Files

# Hidden files in /tmp (classic attacker staging)
ls -la /tmp/.*
find /tmp -name ".*" -type f -ls 2>/dev/null

# Hidden directories anywhere
find / -type d -name ".*" 2>/dev/null | grep -v -E "^\.(git|cache|config)"

# Files with spaces or weird names (evasion)
find / -name "* *" -o -name "*[[:cntrl:]]*" 2>/dev/null

SUID/SGID Binaries

# All SUID binaries (potential privesc)
find / -type f -perm -4000 -ls 2>/dev/null

# SUID binaries not owned by root (very suspicious)
find / -type f -perm -4000 ! -user root -ls 2>/dev/null

# Recently modified SUID (backdoor indicator)
find / -type f -perm -4000 -mtime -7 -ls 2>/dev/null

World-Writable Files

# World-writable files (excluding /tmp, /var/tmp)
find / -type f -perm -002 ! -path "/tmp/*" ! -path "/var/tmp/*" -ls 2>/dev/null

# World-writable directories without sticky bit
find / -type d -perm -002 ! -perm -1000 -ls 2>/dev/null

Network Analysis

Listening Ports

# All listening ports with process
ss -tlnp
ss -ulnp

# Compare against expected (document your baseline!)
ss -tlnp | awk '{print $4}' | sort -u

Established Connections

# Current connections with process info
ss -tnp state established

# Foreign connections (not localhost)
ss -tnp state established | grep -v "127.0.0.1"

# Connections to unusual ports (C2 often uses 443, 8080, 4444)
ss -tnp state established | grep -E ":(4444|5555|6666|1337|31337)"

DNS Analysis

# Check resolv.conf (DNS hijacking)
cat /etc/resolv.conf

# Recent DNS queries (if systemd-resolved)
resolvectl statistics
journalctl -u systemd-resolved --since "1 hour ago" | grep -i query

Network Scanning

Fast host discovery for identifying unknown assets or lateral movement.

Parallel Port Scanning

Parallel netcat SSH discovery (50 concurrent)

# Massively parallel SSH port scan using xargs
# Scans 254 hosts in ~5 seconds
echo 192.168.1.{1..254} | tr ' ' '\n' | \
  xargs -P 50 -I {} sh -c 'timeout 1 nc -zv {} 22 2>&1 | grep -q succeeded && echo "{}: SSH open"'

Parallel multi-port discovery

# Scan multiple ports across subnet
for port in 22 80 443 3389 5900; do
  echo "=== Port $port ==="
  echo 192.168.1.{1..254} | tr ' ' '\n' | \
    xargs -P 50 -I {} sh -c "timeout 1 nc -zv {} $port 2>&1 | grep -q succeeded && echo '{}'"
done

nmap Quick Scans

nmap quick scans

# Fast SSH discovery (single subnet)
nmap -p 22 192.168.1.0/24 --open -T4

# Multiple common ports
nmap -p 22,80,443,3389 192.168.1.0/24 --open -T4

# Top 100 ports
nmap -F 192.168.1.0/24 --open -T4

nmap stealth scans

# SYN scan (requires root, stealthier)
sudo nmap -sS -p 22 192.168.1.0/24 --open -T4

# No ping, just port check
nmap -Pn -p 22 192.168.1.0/24 --open

# Randomize scan order
nmap -p 22 192.168.1.0/24 --open --randomize-hosts

Host Discovery

Find live hosts (ICMP)

# Parallel ping sweep
echo 192.168.1.{1..254} | tr ' ' '\n' | \
  xargs -P 50 -I {} sh -c 'ping -c 1 -W 1 {} >/dev/null 2>&1 && echo "{}: alive"'

# nmap ping sweep
nmap -sn 192.168.1.0/24

ARP scanning (local segment only)

# arp-scan for local network discovery
sudo arp-scan -l

# Specific interface
sudo arp-scan -I eth0 192.168.1.0/24

Tool Comparison

Table 1. Tool comparison
Method	Pros	Cons
xargs + nc	No deps, fast, precise control	Limited features, requires parsing
nmap	Feature-rich, service detection	Slower, may require root
masscan	Extremely fast for large ranges	Requires root, noisy

Process Analysis

Process Tree

# Full process tree
ps auxf

# Processes without controlling terminal (background/daemon)
ps aux | awk '$7 == "?"'

# Processes running as root
ps aux | awk '$1 == "root"'

Suspicious Processes

# Processes with deleted binaries (in-memory only)
ls -l /proc/*/exe 2>/dev/null | grep deleted

# Processes running from /tmp or /dev/shm
ls -l /proc/*/exe 2>/dev/null | grep -E "(tmp|shm)"

# High CPU processes
ps aux --sort=-%cpu | head -10

# High memory processes
ps aux --sort=-%mem | head -10

Open Files

# Files opened by a process
lsof -p <PID>

# Network connections by process
lsof -i -P -n

# Who has /etc/passwd open (credential access)
lsof /etc/passwd /etc/shadow 2>/dev/null

User Analysis

Login Activity

# Currently logged in
w
who

# Recent logins
last -20

# Failed login attempts
lastb -20 2>/dev/null || journalctl -u sshd | grep -i failed

Account Anomalies

# Users with UID 0 (should only be root)
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Recently modified passwd/shadow
ls -la /etc/passwd /etc/shadow /etc/group

# Empty password fields
awk -F: '$2 == ""' /etc/shadow 2>/dev/null

Log Analysis

Authentication Logs

# SSH auth attempts
journalctl -u sshd --since "24 hours ago" | grep -E "(Accepted|Failed)"

# sudo usage
journalctl | grep sudo | tail -50

# su usage
grep "su:" /var/log/auth.log 2>/dev/null || journalctl | grep "su:"

Command History

# All users' history files
find /home -name ".*history" -ls 2>/dev/null
cat /root/.bash_history 2>/dev/null | tail -50

# Commands run as root
grep -h "sudo\|su -" /home/*/.bash_history 2>/dev/null

Rootkit Detection

Quick Checks

# Compare ls output (rootkits hide files)
ls /tmp | wc -l
find /tmp -maxdepth 1 | wc -l  # Should match

# Check for hidden kernel modules
lsmod | wc -l
cat /proc/modules | wc -l  # Should match

# Verify system binaries (RHEL/Fedora)
rpm -Va 2>/dev/null | grep -E "^..5"

# Verify system binaries (Debian/Ubuntu)
debsums -c 2>/dev/null

Kernel Module Analysis

# Kernel modules loaded
lsmod

# Hidden modules (compare output)
cat /proc/modules

# Recently loaded modules
dmesg | grep -i module | tail -20

Incident Response One-Liners

Capture System State

# Capture system state for analysis
tar czf /tmp/ir-$(hostname)-$(date +%Y%m%d).tar.gz \
  /etc/passwd /etc/shadow /etc/group \
  /etc/crontab /etc/cron.* \
  /var/log/auth.log* /var/log/secure* \
  /root/.bash_history \
  2>/dev/null

Freeze Suspicious Process

# Kill suspicious process and preserve
kill -STOP <PID>  # Freeze, don't kill
cp -r /proc/<PID>/ /tmp/frozen_proc_<PID>/

Block Attacker IP

# Block IP immediately (iptables)
iptables -I INPUT -s <ATTACKER_IP> -j DROP

# Block IP (nftables)
nft add rule inet filter input ip saddr <ATTACKER_IP> drop

# Block IP (firewalld)
firewall-cmd --add-rich-rule='rule family="ipv4" source address="<ATTACKER_IP>" drop'

Baseline Documentation

Document these on clean systems for comparison:

# Network baseline
ss -tlnp > /root/baseline_ports.txt

# Process baseline
ps aux > /root/baseline_procs.txt

# SUID baseline
find / -type f -perm -4000 -ls > /root/baseline_suid.txt 2>/dev/null

# Cron baseline
cat /etc/crontab /etc/cron.d/* > /root/baseline_cron.txt 2>/dev/null