SED Mastery Reference

sed uses regex patterns. Foundation learning:

Overview

sed (Stream Editor) is the Swiss Army knife of text transformation. Master these patterns for log analysis, config management, and data manipulation.

Core Syntax

sed [options] 'command' file
sed [options] -e 'cmd1' -e 'cmd2' file
sed [options] -f script.sed file
Option Description

-i

Edit in place (DANGER - no backup)

-i.bak

Edit in place with backup

-n

Suppress auto-print (use with p)

-E / -r

Extended regex (ERE)

-e

Multiple commands

-f

Script file

Substitution: The s Command

Basic Substitution

# Replace first occurrence per line
sed 's/old/new/' file

# Replace ALL occurrences (global)
sed 's/old/new/g' file

# Replace Nth occurrence
sed 's/old/new/2' file

# Case-insensitive
sed 's/old/new/gi' file

# Print only changed lines
sed -n 's/old/new/p' file

Delimiter Flexibility

# Any delimiter works - great for paths!
sed 's|/var/log|/tmp/log|g' file
sed 's#http://old#https://new#g' file
sed 's@pattern@replacement@g' file

Capture Groups & Backreferences

# Swap order: "first,last" → "last,first"
echo "John,Doe" | sed -E 's/([^,]+),([^,]+)/\2,\1/'
# Output: Doe,John

# Surround matches
echo "192.168.1.1" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/[\1]/'
# Output: [192.168.1.1]

# Duplicate with modification
echo "ERROR: disk full" | sed -E 's/(ERROR): (.*)/\1 → \2 ← \1/'
# Output: ERROR → disk full ← ERROR

Address Ranges

# Line 5 only
sed '5s/old/new/' file

# Lines 5-10
sed '5,10s/old/new/' file

# Line 5 to end
sed '5,$s/old/new/' file

# Pattern match
sed '/ERROR/s/old/new/' file

# Between patterns (inclusive)
sed '/BEGIN/,/END/s/old/new/' file

# First match only
sed '0,/pattern/s/pattern/replacement/' file

# NOT matching pattern
sed '/DEBUG/!s/old/new/' file

SIEM & Log Analysis Patterns

Extract IP Addresses

# Extract source IPs from firewall logs
sed -nE 's/.*SRC=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).*/\1/p' /var/log/firewall.log

# Extract both source and destination
sed -nE 's/.*SRC=([0-9.]+).*DST=([0-9.]+).*/SRC:\1 DST:\2/p' firewall.log

Parse Windows Event Logs

# Extract username from 4625 failed login
sed -nE 's/.*Account Name:[[:space:]]+([^[:space:]]+).*/\1/p' security.log

# Extract event ID and message
sed -nE 's/.*EventID=([0-9]+).*Message="([^"]+)".*/[\1] \2/p' events.log

QRadar AQL Results Processing

# Convert QRadar JSON to CSV-like
sed -E 's/\{"sourceip":"([^"]+)","destinationip":"([^"]+)","count":([0-9]+)\}/\1,\2,\3/g'

# Clean up AQL output
sed -E 's/[[:space:]]+/ /g; s/^ //; s/ $//'

ISE Log Processing

# Extract MAC addresses from ISE logs
sed -nE 's/.*CallingStationId=([0-9A-Fa-f:]{17}).*/\1/p' ise.log

# Extract authentication results
sed -nE 's/.*AuthenticationResult=([A-Za-z]+).*/\1/p' ise.log | sort | uniq -c

Timestamp Transformation

# ISO to readable: 2026-02-12T14:30:00Z → Feb 12, 2026 14:30:00
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})T([0-9:]+)Z/\2\/\3\/\1 \4/'

# Epoch to placeholder (real conversion needs date)
sed -E 's/timestamp":([0-9]{10})/timestamp":"EPOCH_\1"/g'

Multi-Line Operations

Join Lines

# Join continuation lines (ending with \)
sed ':a; /\\$/{N; s/\\\n//; ba}'

# Join every 2 lines
sed 'N;s/\n/ /'

# Join all lines into one
sed ':a;N;$!ba;s/\n/ /g'

Delete Multi-Line Blocks

# Delete from BEGIN to END (inclusive)
sed '/BEGIN/,/END/d' file

# Delete everything EXCEPT between BEGIN and END
sed '/BEGIN/,/END/!d' file

Insert Before/After

# Insert line BEFORE pattern
sed '/pattern/i\New line before' file

# Insert line AFTER pattern
sed '/pattern/a\New line after' file

# Insert at line number
sed '5i\Inserted at line 5' file

Config File Manipulation

Update Key=Value

# Change specific setting
sed -i 's/^ServerName=.*/ServerName=newserver/' config.ini

# Uncomment a line
sed -i 's/^#\(ServerName=.*\)/\1/' config.ini

# Comment a line
sed -i 's/^\(ServerName=.*\)/#\1/' config.ini

# Add setting if not exists (complex)
grep -q '^ServerName=' config.ini || echo 'ServerName=default' >> config.ini

YAML/JSON Manipulation

# Update YAML value (simple cases)
sed -i 's/^\([[:space:]]*port:\).*/\1 8443/' config.yaml

# JSON value update (simple)
sed -i 's/"port":[[:space:]]*[0-9]*/"port": 8443/' config.json

Batch File Operations

Recursive In-Place Editing

# Replace in all .adoc files
find . -name "*.adoc" -exec sed -i 's/old/new/g' {} +

# With backup
find . -name "*.conf" -exec sed -i.bak 's/10.50.1.1/10.50.1.100/g' {} +

Pipeline Integration

# Chain with other tools
cat logs/*.log | sed -nE 's/.*ERROR[[:space:]]+(.+)/\1/p' | sort | uniq -c | sort -rn | head -20

# Process and save
netapi ise mnt sessions | sed '1d' | sed 's/  */ /g' > cleaned_sessions.txt

Advanced Patterns

Hold Space Operations

# Reverse lines (tac equivalent)
sed -n '1!G;h;$p' file

# Print line numbers with content
sed = file | sed 'N;s/\n/: /'

Conditional Processing

# If line contains ERROR, uppercase it
sed '/ERROR/s/.*/\U&/' file

# If line contains DEBUG, delete it
sed '/DEBUG/d' file

# Replace only if previous line matches
sed -n '/^Host:/{n;s/Port:.*/Port: 443/p}' config

Transform Commands

# Uppercase
sed 's/.*/\U&/' file

# Lowercase
sed 's/.*/\L&/' file

# Capitalize first letter
sed 's/\b\(.\)/\U\1/g' file

# Character transliteration (like tr)
sed 'y/abc/ABC/' file

Security Operations

Credential Scrubbing

# Mask passwords in logs
sed -E 's/(password[=:][[:space:]]*)([^[:space:]]+)/\1********/gi' log.txt

# Mask API keys
sed -E 's/(api[_-]?key[=:][[:space:]]*)([A-Za-z0-9]{20,})/\1[REDACTED]/gi'

# Mask IPs for sharing
sed -E 's/([0-9]+\.[0-9]+)\.[0-9]+\.[0-9]+/\1.xxx.xxx/g'

Certificate Processing

# Extract certificate subject
openssl x509 -in cert.pem -noout -subject | sed 's/subject=//'

# Clean up openssl output
openssl x509 -in cert.pem -text | sed -n '/Subject:/p; /Issuer:/p; /Not After/p'

One-Liner Power Collection

# Remove blank lines
sed '/^$/d' file

# Remove leading/trailing whitespace
sed 's/^[[:space:]]*//; s/[[:space:]]*$//' file

# Remove comments (# style)
sed 's/#.*//' file

# Remove HTML tags
sed 's/<[^>]*>//g' file.html

# Double-space a file
sed G file

# Number non-blank lines
sed '/./=' file | sed 'N; s/\n/ /'

# Print first 10 lines (like head)
sed 10q file

# Print last 10 lines (like tail) - complex
sed -e :a -e '$q;N;11,$D;ba' file

# Remove duplicate consecutive lines (like uniq)
sed '$!N; /^\(.*\)\n\1$/!P; D' file

# Print lines between two patterns
sed -n '/START/,/END/p' file

# Delete lines matching pattern
sed '/pattern/d' file

# Print only lines matching pattern
sed -n '/pattern/p' file

# Reverse each line
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' file

SIEM Quick Reference Card

┌─────────────────────────────────────────────────────────────────────────────┐
│                         SED SIEM QUICK REFERENCE                            │
├─────────────────────────────────────────────────────────────────────────────┤
│ EXTRACT PATTERNS                                                            │
│   IP addresses:     sed -nE 's/.*SRC=([0-9.]+).*/\1/p'                     │
│   MAC addresses:    sed -nE 's/.*MAC=([0-9A-Fa-f:]{17}).*/\1/p'            │
│   Usernames:        sed -nE 's/.*user=([^[:space:]]+).*/\1/p'              │
│   Event IDs:        sed -nE 's/.*EventID=([0-9]+).*/\1/p'                  │
│   Error messages:   sed -nE 's/.*ERROR[[:space:]]+(.+)/\1/p'               │
├─────────────────────────────────────────────────────────────────────────────┤
│ CLEAN OUTPUT                                                                │
│   Remove header:    sed '1d'                                                │
│   Trim whitespace:  sed 's/^[[:space:]]*//; s/[[:space:]]*$//'             │
│   Normalize spaces: sed 's/[[:space:]]\+/ /g'                              │
│   Remove blanks:    sed '/^$/d'                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ TRANSFORM DATA                                                              │
│   CSV to TSV:       sed 's/,/\t/g'                                          │
│   Add prefix:       sed 's/^/PREFIX: /'                                     │
│   Add suffix:       sed 's/$/ SUFFIX/'                                      │
│   Quote fields:     sed 's/.*/"&"/'                                         │
├─────────────────────────────────────────────────────────────────────────────┤
│ FILTER LINES                                                                │
│   Keep matches:     sed -n '/pattern/p'                                     │
│   Delete matches:   sed '/pattern/d'                                        │
│   Keep range:       sed -n '5,10p'                                          │
│   Delete range:     sed '5,10d'                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│ IN-PLACE EDITING                                                            │
│   With backup:      sed -i.bak 's/old/new/g' file                          │
│   No backup:        sed -i 's/old/new/g' file  # DANGEROUS                 │
│   Multiple files:   find . -name "*.log" -exec sed -i 's/old/new/g' {} +   │
└─────────────────────────────────────────────────────────────────────────────┘

Real-World SIEM Workflows

Daily Log Summary

#!/bin/bash
# Extract and summarize errors from today's logs

TODAY=$(date +%Y-%m-%d)

echo "=== Error Summary for $TODAY ==="
cat /var/log/app/*.log | \
    sed -n "/$TODAY/p" | \
    sed -nE 's/.*ERROR[[:space:]]+\[([^\]]+)\][[:space:]]+(.*)/[\1] \2/p' | \
    sort | uniq -c | sort -rn | head -20

ISE Session Cleanup

# Clean up netapi ISE output for CSV export
netapi ise mnt sessions | \
    sed '1d' | \                    # Remove header
    sed 's/[[:space:]]\+/,/g' | \   # Spaces to commas
    sed 's/^,//' | \                # Remove leading comma
    sed 's/,$//' > sessions.csv     # Remove trailing comma

QRadar Log Normalization

# Normalize varied timestamp formats to ISO
sed -E '
    s/([0-9]{2})\/([0-9]{2})\/([0-9]{4})/\3-\1-\2/g
    s/([A-Za-z]{3}) ([0-9]{2}), ([0-9]{4})/\3-\1-\2/g
' raw_logs.txt > normalized.txt

Related