Regex Session 05: sed Mastery

sed is the stream editor - it transforms text line by line using regex patterns. Master sed and you can reshape any text file.

sed Basics

Command Syntax Purpose

Substitute

s/pattern/replacement/flags

Replace text

Delete

/pattern/d

Remove lines

Print

/pattern/p

Print matching lines

Insert

i\text

Insert before line

Append

a\text

Append after line

Test File Setup

cat << 'EOF' > /tmp/sed-practice.txt
# Server Configuration
server_name=web-prod-01
server_ip=192.168.1.100
server_port=8080
admin_email=admin@example.com

# Database Configuration
db_host=db-prod-01.internal
db_port=5432
db_user=app_user
db_pass=oldpassword123

# Feature Flags
feature_dark_mode=false
feature_api_v2=true
feature_beta=false

# Log entries
2026-03-15T10:30:45 INFO Server started
2026-03-15T10:31:02 WARN Disk space low
2026-03-15T10:32:00 ERROR Connection failed
2026-03-15T10:33:00 DEBUG Query completed
EOF

Lesson 1: Basic Substitution

Syntax: sed 's/pattern/replacement/' file

# Replace first occurrence per line
sed 's/prod/staging/' /tmp/sed-practice.txt

# Replace ALL occurrences per line (global flag)
sed 's/prod/staging/g' /tmp/sed-practice.txt

# Case-insensitive replacement
sed 's/error/CRITICAL/i' /tmp/sed-practice.txt

In-Place Editing

# Edit file directly (creates backup with .bak)
sed -i.bak 's/old/new/g' file.txt

# Edit without backup (DANGEROUS - no undo)
sed -i 's/old/new/g' file.txt

Lesson 2: ERE Mode (-E)

Use -E for extended regex - same as grep -E.

# Without -E: need to escape +, ?, |, (), {}
sed 's/[0-9]\+/NUMBER/' file.txt

# With -E: clean syntax
sed -E 's/[0-9]+/NUMBER/' file.txt

# Match alternatives with |
sed -E 's/(prod|staging)/ENV/' /tmp/sed-practice.txt

Lesson 3: Capture Groups

Use \1, \2, etc. to reference captured groups.

# Swap key=value to value=key
sed -E 's/^([a-z_]+)=(.+)$/\2=\1/' /tmp/sed-practice.txt

# Extract and reformat dates
echo "2026-03-15" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\2\/\3\/\1/'
# Output: 03/15/2026

# Wrap log levels in brackets
sed -E 's/(INFO|WARN|ERROR|DEBUG)/[\1]/g' /tmp/sed-practice.txt

Exercise: Add prefix to all values

# Change value=X to value=PREFIX_X
sed -E 's/^([a-z_]+)=(.+)$/\1=PREFIX_\2/' /tmp/sed-practice.txt

Lesson 4: Address Ranges

Operate on specific lines or line ranges.

# Only line 5
sed '5s/prod/dev/' /tmp/sed-practice.txt

# Lines 5-10
sed '5,10s/prod/dev/' /tmp/sed-practice.txt

# From line 5 to end
sed '5,$s/prod/dev/' /tmp/sed-practice.txt

# Lines matching pattern
sed '/Database/,/Log/s/prod/dev/' /tmp/sed-practice.txt

Lesson 5: Delete Lines

# Delete lines containing pattern
sed '/^#/d' /tmp/sed-practice.txt  # Remove comments

# Delete blank lines
sed '/^$/d' /tmp/sed-practice.txt

# Delete lines matching multiple patterns
sed -E '/(DEBUG|INFO)/d' /tmp/sed-practice.txt

# Delete line range
sed '5,10d' /tmp/sed-practice.txt

Lesson 6: Print Specific Lines

Useful for extracting sections.

# Print only matching lines (with -n to suppress default output)
sed -n '/ERROR/p' /tmp/sed-practice.txt

# Print line range
sed -n '5,10p' /tmp/sed-practice.txt

# Print from pattern to pattern
sed -n '/Database/,/Feature/p' /tmp/sed-practice.txt

Lesson 7: Multiple Commands

# Chain with semicolon
sed 's/prod/staging/g; s/8080/9090/g' /tmp/sed-practice.txt

# Chain with -e
sed -e 's/prod/staging/g' -e 's/8080/9090/g' /tmp/sed-practice.txt

# From a script file
cat << 'EOF' > /tmp/transform.sed
s/prod/staging/g
s/8080/9090/g
/^#/d
EOF
sed -f /tmp/transform.sed /tmp/sed-practice.txt

Lesson 8: Insert and Append

# Insert line BEFORE match
sed '/Database/i\# --- DATABASE SECTION ---' /tmp/sed-practice.txt

# Append line AFTER match
sed '/server_port/a\server_ssl=true' /tmp/sed-practice.txt

# Insert at specific line number
sed '1i\# Configuration File' /tmp/sed-practice.txt

Practical Applications

Config File Modifications

# Change database password
sed -i 's/db_pass=.*/db_pass=newSecurePass456/' /tmp/sed-practice.txt

# Enable feature flags
sed -i 's/feature_.*=false/&_DISABLED/' /tmp/sed-practice.txt

# Comment out a section
sed -i '/feature_beta/s/^/# DISABLED: /' /tmp/sed-practice.txt

Log Transformation

# Convert timestamps to epoch-friendly format
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})T/\1\2\3 /' /tmp/sed-practice.txt

# Anonymize IP addresses
sed -E 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/X.X.X.X/g' /tmp/sed-practice.txt

# Extract just error messages
sed -n 's/.*ERROR \(.*\)/\1/p' /tmp/sed-practice.txt

Code Refactoring

# Rename function calls
sed -E 's/oldFunction\(/newFunction(/g' code.py

# Update import statements
sed -i 's/from oldmodule import/from newmodule import/' *.py

# Convert tabs to spaces
sed 's/\t/    /g' file.txt

Security Sanitization

# Mask passwords in logs
sed -E 's/(password|passwd|pwd)=[^ ]+/\1=***MASKED***/gi' /tmp/sed-practice.txt

# Remove API keys
sed -E 's/api_key=[A-Za-z0-9_-]+/api_key=REDACTED/g' /tmp/sed-practice.txt

# Mask email addresses
sed -E 's/([A-Za-z0-9._%+-]+)@([A-Za-z0-9.-]+)/***@\2/g' /tmp/sed-practice.txt

Common Patterns Reference

Task Command

Remove trailing whitespace

sed 's/*$//'

Remove leading whitespace

sed 's/^*//'

Remove blank lines

sed '/^$/d'

Remove comments

sed '/^#/d'

Double-space file

sed 'G'

Number lines

sed = file | sed 'N;s/\n/\t/'

Reverse lines

sed '1!G;h;$!d'

Get first line

sed 'q' or sed -n '1p'

Get last line

sed -n '$p'

Exercises to Complete

  1. [ ] Replace all port numbers with "PORT"

  2. [ ] Extract only the values (right side of =)

  3. [ ] Convert all log levels to lowercase

  4. [ ] Remove all comment lines and blank lines

  5. [ ] Add "[MASKED]" after any password values

Self-Check

Solutions 
# 1. Replace port numbers
sed -E 's/[0-9]{2,5}/PORT/g' /tmp/sed-practice.txt

# 2. Extract only values
sed -E 's/^[a-z_]+=(.*)$/\1/' /tmp/sed-practice.txt | grep -v '^#'

# 3. Lowercase log levels
sed -E 's/(INFO|WARN|ERROR|DEBUG)/\L\1/g' /tmp/sed-practice.txt

# 4. Remove comments and blank lines
sed '/^#/d; /^$/d' /tmp/sed-practice.txt

# 5. Mask passwords
sed -E 's/(pass=.*)$/\1 [MASKED]/' /tmp/sed-practice.txt

Next Session

Session 06: awk Regex Power - Pattern-action processing with field extraction.