AWK Range Patterns + SED Multi-Substitution

Advanced pattern for extracting file sections and replacing placeholders with real values.

The Pattern

awk '/START_PATTERN/,/END_PATTERN/' file | grep 'filter' | sed -e 's/old/new/g' -e 's/old2/new2/g'

Real-World Example

Extract firewall group commands from AsciiDoc and resolve Antora attributes to IPs:

awk '/^set firewall group/,/^----$/' vyos-firewall.adoc | \
  grep '^set firewall' | \
  sed -e 's/{ise-01-ip}/10.50.1.20/g' \
      -e 's/{vault-01-ip}/10.50.1.60/g' \
      -e 's/{homedc-ip}/10.50.1.50/g'

Component Breakdown

1. AWK Range Pattern

awk '/^set firewall group/,/^----$/' file
Component Meaning

/pattern1/,/pattern2/

Range operator - print all lines FROM pattern1 TO pattern2 (inclusive)

^set firewall group

Start: lines beginning with "set firewall group"

^----$

End: lines that are exactly "----" (AsciiDoc code block delimiter)

How it works:

  1. AWK scans line by line

  2. When it sees ^set firewall group, it turns ON printing

  3. It prints every line until it sees ^----$

  4. Then it turns OFF printing

  5. If another ^set firewall group appears, cycle repeats

Alternatives:

# Print lines 10-50
awk 'NR>=10 && NR<=50' file

# Print from pattern to EOF
awk '/START/,0' file

# Print from line 1 to pattern
awk '1; /END/{exit}' file

2. GREP Filter

grep '^set firewall'
Component Meaning

^set firewall

Lines starting with "set firewall"

Purpose

Remove comments and blank lines from AWK output

The AWK range includes everything between markers - comments, blank lines, etc. GREP filters to only the actual commands.

Why not do this in AWK?

You could:

awk '/^set firewall group/,/^----$/ { if (/^set firewall/) print }' file

But piping to grep is more readable and follows Unix philosophy (each tool does one thing).

3. SED Multi-Substitution

sed -e 's/{old}/new/g' -e 's/{old2}/new2/g'
Component Meaning

-e

Execute this expression (allows multiple)

s/old/new/g

Substitute "old" with "new", globally (all occurrences)

{…​}

Curly braces are literal (Antora attribute syntax)

Key insight: Each -e is applied in sequence to the same line. Order matters if substitutions overlap.

Alternative syntax:

# Semicolon-separated (same result)
sed 's/{old}/new/g; s/{old2}/new2/g'

# From a file of substitutions
sed -f substitutions.sed file

Building the Substitution List

Extract Attributes from antora.yml

# List all IP attributes
grep -E '^\s{4}[a-z].*-ip:' docs/asciidoc/antora.yml | awk '{print $1, $2}'

Generate sed Commands Automatically

# Convert antora.yml attributes to sed expressions
grep -E '^\s{4}[a-z].*-ip:' docs/asciidoc/antora.yml | \
  awk '{gsub(/:/, ""); printf "-e '\''s/{%s}/%s/g'\'' \\\n", $1, $2}'

Output:

-e 's/{ise-01-ip}/10.50.1.20/g' \
-e 's/{vault-01-ip}/10.50.1.60/g' \
...

Complete Pipeline (VyOS Firewall Groups)

awk '/^set firewall group/,/^----$/' \
  docs/asciidoc/modules/ROOT/partials/vyos-firewall.adoc | \
  grep '^set firewall' | \
  sed -e 's/{wazuh-manager-vip}/10.50.1.134/g' \
      -e 's/{bind-ip}/10.50.1.90/g' \
      -e 's/{bind-02-ip}/10.50.1.91/g' \
      -e 's/{traefik-vip}/10.50.1.130/g' \
      -e 's/{wazuh-indexer-vip}/10.50.1.131/g' \
      -e 's/{wazuh-dashboard-vip}/10.50.1.132/g' \
      -e 's/{wazuh-workers-vip}/10.50.1.133/g' \
      -e 's/{k3s-master-01-ip}/10.50.1.120/g' \
      -e 's/{k3s-master-02-ip}/10.50.1.121/g' \
      -e 's/{k3s-master-03-ip}/10.50.1.122/g' \
      -e 's/{k3s-worker-01-ip}/10.50.1.123/g' \
      -e 's/{k3s-worker-02-ip}/10.50.1.124/g' \
      -e 's/{k3s-worker-03-ip}/10.50.1.125/g' \
      -e 's/{ws-razer-ip}/10.50.10.106/g' \
      -e 's/{ws-aw-ip}/10.50.10.107/g' \
      -e 's/{ws-p50-ip}/10.50.10.108/g' \
      -e 's/{ise-01-ip}/10.50.1.20/g' \
      -e 's/{ise-02-ip}/10.50.1.21/g' \
      -e 's/{vault-01-ip}/10.50.1.60/g' \
      -e 's/{vault-02-ip}/10.50.1.61/g' \
      -e 's/{vault-03-ip}/10.50.1.62/g' \
      -e 's/{ipsk-mgr-ip}/10.50.1.30/g' \
      -e 's/{ipsk-mgr-02-ip}/10.50.1.31/g' \
      -e 's/{homedc-ip}/10.50.1.50/g' \
      -e 's/{homedc-02-ip}/10.50.1.51/g' \
      -e 's/{keycloak-ip}/10.50.1.80/g' \
      -e 's/{keycloak-02-ip}/10.50.1.81/g' \
      -e 's/{ipa-ip}/10.50.1.100/g' \
      -e 's/{ipa-02-ip}/10.50.1.101/g' \
      -e 's/{wlc-ip}/10.50.1.40/g' \
      -e 's/{wlc-02-ip}/10.50.1.41/g' \
      -e 's/{c9300-ip}/10.50.1.11/g' \
      -e 's/{c3560-ip}/10.50.1.10/g' \
      -e 's/{vyos-01-ip}/10.50.1.3/g' \
      -e 's/{vyos-02-ip}/10.50.1.2/g' \
      -e 's/{pfsense-ip}/10.50.1.1/g' \
      -e 's/{nas-ip}/10.50.1.70/g' \
      -e 's/{nas-02-ip}/10.50.1.71/g' \
      -e 's/{gitea-ip}/10.50.1.72/g' \
      -e 's/{minio-ip}/10.50.1.73/g' \
      -e 's/{kvm-01-ip}/10.50.1.110/g' \
      -e 's/{kvm-02-ip}/10.50.1.111/g' \
      -e 's/{ipmi-01-ip}/10.50.1.200/g' \
      -e 's/{ipmi-02-ip}/10.50.1.201/g' \
      -e 's/{zabbix-ip}/10.50.1.135/g' \
      -e 's/{port-dns}/53/g'

Mental Model

┌─────────────────────────────────────────────────────────────────┐
│  SOURCE FILE (vyos-firewall.adoc)                               │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ [source,bash]                                             │  │
│  │ ----                                                      │  │
│  │ # RFC1918 Private Networks           ◄── AWK starts here │  │
│  │ set firewall group network-group...                       │  │
│  │ set firewall group address-group...  ◄── GREP keeps these │  │
│  │ # Comment line                       ◄── GREP removes     │  │
│  │ set firewall group port-group...                          │  │
│  │ ----                                 ◄── AWK stops here   │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  AWK OUTPUT (range extracted)                                   │
│  set firewall group network-group RFC1918 network 10.0.0.0/8    │
│  set firewall group address-group ISE_NODES address {ise-01-ip} │
│  # Comment line                                                  │
│  set firewall group port-group RADIUS port 1812                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  GREP OUTPUT (comments removed)                                 │
│  set firewall group network-group RFC1918 network 10.0.0.0/8    │
│  set firewall group address-group ISE_NODES address {ise-01-ip} │
│  set firewall group port-group RADIUS port 1812                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  SED OUTPUT (attributes resolved)                               │
│  set firewall group network-group RFC1918 network 10.0.0.0/8    │
│  set firewall group address-group ISE_NODES address 10.50.1.20  │
│  set firewall group port-group RADIUS port 1812                 │
└─────────────────────────────────────────────────────────────────┘

Related Patterns

Extract Multiple Sections

# AWK range resets after each match - extracts ALL matching sections
awk '/^== Phase/,/^== /' runbook.adoc

Exclusive Range (don’t include markers)

# Skip the start/end lines themselves
awk '/^----$/,/^----$/ { if (!/^----$/) print }' file

Named Captures with sed

# Capture groups for reordering
echo "10.50.1.20" | sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/'
# Output: 20.1.50.10

Quick Reference

Pattern Use Case

awk '/A/,/B/'

Extract lines between A and B (inclusive)

awk 'NR>=10 && NR⇐20'

Extract lines 10-20

grep '^prefix'

Filter to lines starting with prefix

sed -e 's/a/b/g' -e 's/c/d/g'

Multiple substitutions in sequence

sed 's/a/b/g; s/c/d/g'

Same as above, semicolon syntax