Process Substitution

Process substitution lets you treat command output as a file. Master <() and >() for comparing outputs, multi-input processing, and advanced pipelines.

Basic Concept

# <(command) creates a pseudo-file containing command output
# >(command) creates a pseudo-file that feeds into command

Input Process Substitution <()

Compare Command Outputs

# Compare two directories
diff <(ls dir1) <(ls dir2)

# Compare sorted versions
diff <(sort file1) <(sort file2)

# Compare configs from two servers
diff <(ssh server1 cat /etc/hosts) <(ssh server2 cat /etc/hosts)

Multiple Inputs to One Command

# Paste outputs side by side
paste <(cut -d: -f1 /etc/passwd) <(cut -d: -f3 /etc/passwd)

# Join two command outputs
join <(sort file1) <(sort file2)

Commands That Need Files

# wc with multiple "files"
wc -l <(grep ERROR log1) <(grep ERROR log2)

# Source from command output
source <(kubectl completion bash)

# Feed to program expecting filename
program_that_needs_file <(generate_data)

Output Process Substitution >()

Tee to Multiple Destinations

# Process and log simultaneously
command | tee >(grep ERROR > errors.log) >(grep WARN > warnings.log)

# Multiple processing paths
cat data.txt | tee >(awk '{sum+=$1} END{print sum}' > sum.txt) \
                   >(wc -l > count.txt)

Pipeline with Side Effects

# Log while processing
cat file | tee >(logger -t myapp) | process_data

# Archive while transmitting
tar cf - /data | tee >(gzip > backup.tar.gz) | ssh remote "cat > /tmp/data.tar"

Subshells vs Process Substitution

Subshells $()

Captures output as a string:

# String result
files=$(ls)
echo "$files"

# In command
grep "pattern" $(find . -name "*.txt")

Process Substitution <()

Creates a file descriptor:

# File-like input
diff <(sort file1) <(sort file2)

# Can't capture to variable directly
# This creates a filename like /dev/fd/63
echo <(ls)      # Prints: /dev/fd/63

Command Grouping

Braces { }

Execute in current shell, share variables:

# Commands in current shell
{ echo "start"; date; echo "end"; } > output.txt

# Variable persists
{ x=42; echo $x; }
echo $x    # 42 (still available)

Parentheses ( )

Execute in subshell, isolated:

# Commands in subshell
( cd /tmp; pwd; ls )
pwd    # Still in original directory

# Variable isolated
( x=42; echo $x )
echo $x    # Empty (not set in parent)

FIFOs (Named Pipes)

Create and Use

# Create named pipe
mkfifo mypipe

# Write to pipe (blocks until reader)
echo "data" > mypipe &

# Read from pipe
cat mypipe

# Clean up
rm mypipe

Producer-Consumer Pattern

mkfifo pipe1

# Producer (background)
generate_data > pipe1 &

# Consumer
process_data < pipe1

rm pipe1

Infrastructure Patterns

Compare Remote Configs

# Diff configs across servers
diff <(ssh prod1 cat /etc/nginx/nginx.conf) \
     <(ssh prod2 cat /etc/nginx/nginx.conf)

# Compare kubectl outputs
diff <(kubectl get pods -n prod) <(kubectl get pods -n staging)

Compare Before/After

# Capture before
before=$(ss -tn)

# Make changes...

# Compare
diff <(echo "$before") <(ss -tn)

Multi-Stream Processing

# Process log with multiple analyses
tail -f /var/log/app.log | tee \
    >(grep ERROR | logger -t errors) \
    >(awk '/slow/ {print strftime(), $0}' > slow_queries.log) \
    >(grep -c ERROR | while read n; do [[ $n -gt 100 ]] && alert; done)

Parallel Processing

# Process file multiple ways simultaneously
cat large_file | tee \
    >(gzip > file.gz) \
    >(sha256sum > file.sha256) \
    >(wc -l > file.lines) > /dev/null

Join Data Sources

# Join user data from two sources
join -t, \
    <(sort -t, -k1 users.csv) \
    <(sort -t, -k1 permissions.csv)

# Merge API responses
paste -d, \
    <(curl -s api/users | jq -r '.[].name') \
    <(curl -s api/roles | jq -r '.[].role')

Dynamic Source Loading

# Source completion from command
source <(kubectl completion bash)
source <(helm completion bash)

# Source secrets (be careful!)
source <(vault kv get -format=json secret/app | jq -r '.data | to_entries | .[] | "export \(.key)=\(.value)"')

Process ISE Data

# Compare active sessions between nodes
diff <(netapi ise mnt active-sessions --node ise-01 -f json | jq -r '.[].mac' | sort) \
     <(netapi ise mnt active-sessions --node ise-02 -f json | jq -r '.[].mac' | sort)

Git Comparisons

# Compare branches
diff <(git show main:file.txt) <(git show feature:file.txt)

# List unique commits
comm -23 <(git log --oneline main | sort) <(git log --oneline feature | sort)

While Loop with Process Substitution

Problem: Variables Lost in Pipe

# WRONG: count stays 0 (subshell)
count=0
cat file | while read line; do
    ((count++))
done
echo $count    # 0

Solution: Process Substitution

# RIGHT: count persists
count=0
while read line; do
    ((count++))
done < <(cat file)
echo $count    # Correct value

Alternative: Here String

count=0
while read line; do
    ((count++))
done <<< "$(cat file)"
echo $count

Heredocs

Basic Here Document

cat << 'EOF'
This is literal text.
Variables like $HOME are not expanded.
EOF

With Expansion

cat << EOF
Current user: $USER
Home directory: $HOME
EOF

As Command Input

mysql -u user -p << 'SQL'
SELECT * FROM users;
SQL

kubectl apply -f - << 'YAML'
apiVersion: v1
kind: ConfigMap
metadata:
  name: test
YAML

Here String

# Single line input
grep "pattern" <<< "search this text"

# From variable
grep "error" <<< "$log_content"

Coprocesses

# Start coprocess
coproc myproc { command; }

# Write to coprocess
echo "input" >&${myproc[1]}

# Read from coprocess
read output <&${myproc[0]}

# Close
exec {myproc[1]}>&-

Error Handling

Capture stderr

# stderr to variable
errors=$(command 2>&1 >/dev/null)

# Both stdout and stderr
diff <(cmd1 2>&1) <(cmd2 2>&1)

Separate stdout/stderr

command > >(tee stdout.log) 2> >(tee stderr.log >&2)

Quick Reference

# Process substitution
<(command)              # Output as file input
>(command)              # Input as file output

# Grouping
{ commands; }           # Current shell (braces + space + semicolon)
( commands )            # Subshell

# Here documents
<< 'EOF' ... EOF        # Literal (no expansion)
<< EOF ... EOF          # With expansion
<<- EOF ... EOF         # Strip leading tabs
<<< "string"            # Here string

# Common patterns
diff <(cmd1) <(cmd2)    # Compare outputs
while read x; do ...; done < <(cmd)  # Preserve variables
cmd | tee >(log) >(process)          # Split output
source <(cmd)           # Dynamic sourcing

Key Takeaways

  1. <(cmd) - Treat output as file input

  2. >(cmd) - Treat file as command input

  3. diff <() <() - Compare command outputs

  4. < <(cmd) - Preserve while-loop variables

  5. { } vs ( ) - Current shell vs subshell

  6. Heredocs - Multi-line input

  7. FIFOs - Persistent named pipes

Next Module

Signals & Jobs - Process control and traps.