Advanced Shell Patterns

Advanced shell patterns for power users.

Process Substitution <() and >()

Process substitution treats command output as a file. The shell creates a named pipe (/dev/fd/NN) that commands can read from.

Basic Pattern: Remote File Access

# Read remote file with line numbers
awk 'NR>=74 && NR<=90 {print NR": "$0}' <(ssh bind-01 "sudo cat /var/named/inside.domusdigitalis.dev.zone")

# Same pattern, different filter
grep -n "CNAME" <(ssh bind-01 "sudo cat /var/named/inside.domusdigitalis.dev.zone")

Diff Remote Files

# Compare zone files between two DNS servers
diff <(ssh bind-01 "sudo cat /var/named/zone.db") \
     <(ssh bind-02 "sudo cat /var/named/zone.db")

# Side-by-side comparison
diff -y <(ssh kvm-01 "cat /etc/hosts") \
        <(ssh kvm-02 "cat /etc/hosts")

# Colorized diff
diff --color=always <(ssh host1 "cat /etc/resolv.conf") \
                    <(ssh host2 "cat /etc/resolv.conf")

Compare Local vs Remote

# Check if local config matches remote
diff /etc/chrony.conf <(ssh kvm-02 "cat /etc/chrony.conf")

# Compare sorted outputs
diff <(sort /etc/hosts) <(ssh remote "sort /etc/hosts")

Multi-Source Aggregation

# Combine outputs from multiple hosts
cat <(ssh kvm-01 "hostname; uptime") \
    <(ssh kvm-02 "hostname; uptime") \
    <(ssh kvm-03 "hostname; uptime")

# With headers
{
  echo "=== kvm-01 ===" && ssh kvm-01 "df -h /"
  echo "=== kvm-02 ===" && ssh kvm-02 "df -h /"
}

# Parallel execution with paste (side by side)
paste <(ssh kvm-01 "vmstat 1 5") <(ssh kvm-02 "vmstat 1 5")

Filter Remote Logs

# Extract specific time range from remote syslog
awk '/Mar  1 19:/ && /sshd/' <(ssh kvm-02 "sudo journalctl --no-pager")

# Filter remote auth logs for failures
grep "Failed password" <(ssh kvm-02 "sudo cat /var/log/secure")

# Real-time remote log filtering (blocks)
grep --line-buffered "error" <(ssh kvm-02 "sudo tail -f /var/log/messages")

LVM/Disk Comparison

# Compare LVM layouts
diff <(ssh kvm-01 "sudo lvs --noheadings") \
     <(ssh kvm-02 "sudo lvs --noheadings")

# Compare partition tables
diff <(ssh kvm-01 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT") \
     <(ssh kvm-02 "lsblk -o NAME,SIZE,TYPE,MOUNTPOINT")

Join/Paste Remote Data

# Join user lists from two systems (find common users)
comm -12 <(ssh host1 "cut -d: -f1 /etc/passwd | sort") \
         <(ssh host2 "cut -d: -f1 /etc/passwd | sort")

# Find users on host1 but NOT on host2
comm -23 <(ssh host1 "cut -d: -f1 /etc/passwd | sort") \
         <(ssh host2 "cut -d: -f1 /etc/passwd | sort")

Output Process Substitution >()

Write to multiple destinations:

# Tee to file AND process
echo "test" | tee >(gzip > test.gz) >(sha256sum > test.sha256)

# Log to file while also sending to remote
command 2>&1 | tee >(ssh loghost "cat >> /var/log/remote.log")

IPMI/Network Comparison

# Compare IPMI settings across hosts
diff <(ssh kvm-01 "sudo ipmitool lan print 1") \
     <(ssh kvm-02 "sudo ipmitool lan print 1")

# Compare firewall rules
diff <(ssh kvm-01 "sudo firewall-cmd --list-all") \
     <(ssh kvm-02 "sudo firewall-cmd --list-all")

AWK with Line Ranges

# Print lines 50-60 with line numbers
awk 'NR>=50 && NR<=60 {print NR": "$0}' <(ssh host "cat /etc/ssh/sshd_config")

# Print first match and 5 lines after
awk '/^PermitRootLogin/,NR==FNR+5' <(ssh host "cat /etc/ssh/sshd_config")

# Extract section between markers
awk '/BEGIN_SECTION/,/END_SECTION/' <(ssh host "cat config.conf")

Verification Patterns

# Verify DNS zone serial across servers
echo "bind-01: $(ssh bind-01 "sudo grep -oP 'Serial.*\K\d+' /var/named/zone.db")"
echo "bind-02: $(ssh bind-02 "sudo grep -oP 'Serial.*\K\d+' /var/named/zone.db")"

# One-liner comparison
diff <(ssh bind-01 "sudo grep Serial /var/named/zone.db") \
     <(ssh bind-02 "sudo grep Serial /var/named/zone.db") && echo "MATCH" || echo "DRIFT!"

Command Substitution $()

Capture output into variables or inline:

# Capture for variable
UUID=$(sudo blkid -s UUID -o value /dev/nvme0n1p1)
echo "UUID=$UUID /var/lib/libvirt/images xfs defaults 0 0"

# Inline substitution
echo "Uptime: $(ssh kvm-02 uptime)"

# Nested substitution
echo "Kernel: $(ssh kvm-02 "uname -r") on $(ssh kvm-02 hostname)"

Here Strings <<<

# Feed string to command
grep "pattern" <<< "$variable"

# Process JSON inline
jq '.name' <<< '{"name": "test"}'

The Magic: Why This Works

<(ssh host "cat file")
        ↓
    /dev/fd/63
        ↓
   Named pipe (FIFO)
        ↓
   awk reads it like a file!

  • No temp file on disk

  • Streams directly through kernel buffer (~64KB)

  • Writer (ssh) and reader (awk) synchronize automatically

  • Clean, composable, Unix-philosophy

Performance Notes

  • Process substitution uses kernel pipes, not disk

  • Multiple <() in one command run in parallel

  • Large outputs may block if reader is slow (pipe buffer fills)

  • For huge files, consider scp then local processing

See Also

See Also