Phase 2: Process State & System Calls

Target: Weeks 4-7 (May 19 - Jun 8, 2026)

Your colleague flagged this. He’s right. This is where the competition separates engineers who USE Linux from engineers who UNDERSTAND Linux. System calls are the kernel API — every command you run translates to syscalls.

Man Pages to Master

Reference What It Teaches

Reference	What It Teaches
`man 2 intro`	Overview of ALL system calls — the kernel interface
`man 2 open`	Opening files — flags (O_RDONLY, O_CREAT, O_TRUNC), file descriptors returned
`man 2 read` / `man 2 write`	Reading/writing bytes — fd, buffer, count
`man 2 close`	Closing file descriptors — when the kernel reclaims resources
`man 2 fork`	Process creation — parent gets child PID, child gets 0
`man 2 execve`	Replace process image — this is how programs actually run
`man 2 wait`	Parent waits for child — zombie prevention
`man 2 pipe`	Creating pipes — fd[0] read end, fd[1] write end
`man 2 dup2`	Redirect file descriptors — how shell redirection works under the hood
`man 2 socket` / `man 2 bind` / `man 2 listen` / `man 2 accept`	Network socket lifecycle
`man 2 kill`	Sending signals — not just termination
`man 7 signal`	All signals: SIGTERM, SIGKILL, SIGSTOP, SIGCONT, SIGHUP, SIGUSR1/2
`man strace`	Trace system calls of a running process — `-e trace=`, `-p`, `-f`
`man lsof`	List open files — `-p PID`, `-i :port`, `+D dir`, `+L1` deleted
`man ps`	Process listing — `aux` vs `-ef`, custom format with `-o`
`man pstree`	Process hierarchy — `-sp PID` shows ancestors
`man proc`	`/proc/<pid>/` virtual files — status, fd, maps, cmdline, environ

man 2 intro

Overview of ALL system calls — the kernel interface

man 2 open

Opening files — flags (O_RDONLY, O_CREAT, O_TRUNC), file descriptors returned

man 2 read / man 2 write

Reading/writing bytes — fd, buffer, count

man 2 close

Closing file descriptors — when the kernel reclaims resources

man 2 fork

Process creation — parent gets child PID, child gets 0

man 2 execve

Replace process image — this is how programs actually run

man 2 wait

Parent waits for child — zombie prevention

man 2 pipe

Creating pipes — fd[0] read end, fd[1] write end

man 2 dup2

Redirect file descriptors — how shell redirection works under the hood

man 2 socket / man 2 bind / man 2 listen / man 2 accept

Network socket lifecycle

man 2 kill

Sending signals — not just termination

man 7 signal

All signals: SIGTERM, SIGKILL, SIGSTOP, SIGCONT, SIGHUP, SIGUSR1/2

man strace

Trace system calls of a running process — -e trace=, -p, -f

man lsof

List open files — -p PID, -i :port, +D dir, +L1 deleted

man ps

Process listing — aux vs -ef, custom format with -o

man pstree

Process hierarchy — -sp PID shows ancestors

man proc

/proc/<pid>/ virtual files — status, fd, maps, cmdline, environ

Concepts to Internalize

How a Command Actually Runs

You type: cat /etc/hostname

What the kernel does:
1. fork()     — shell creates a child process (copy of itself)
2. execve()   — child replaces itself with /usr/bin/cat
3. open()     — cat opens /etc/hostname (returns fd 3)
4. read()     — cat reads bytes from fd 3 into a buffer
5. write()    — cat writes buffer to fd 1 (stdout)
6. close()    — cat closes fd 3
7. exit()     — cat exits with status 0
8. wait()     — shell reaps child, gets exit status

Every command follows this pattern. strace shows you each step.

File Descriptors

Every process starts with 3 file descriptors:
  fd 0 = stdin   (keyboard or pipe input)
  fd 1 = stdout  (terminal or pipe output)
  fd 2 = stderr  (terminal, error messages)

New files open at the next available fd (usually 3, 4, 5...)

Shell redirection is fd manipulation:
  cmd > file        →  open(file, O_WRONLY|O_CREAT|O_TRUNC) → dup2(fd, 1)
  cmd 2>&1          →  dup2(1, 2) — stderr points to same place as stdout
  cmd < file        →  open(file, O_RDONLY) → dup2(fd, 0)
  cmd | cmd2        →  pipe() creates fd pair, fork(), dup2() connects them

Verify with:
  ls -la /proc/$$/fd       — your shell's open fds
  ls -la /proc/<pid>/fd    — any process's open fds

Signals

Signals are asynchronous notifications to a process.

Critical signals to know:
  SIGTERM (15) — polite termination request (kill default)
  SIGKILL (9)  — immediate death, cannot be caught or ignored
  SIGSTOP (19) — pause process, cannot be caught
  SIGCONT (18) — resume stopped process
  SIGHUP  (1)  — terminal hangup, often used to reload config
  SIGINT  (2)  — Ctrl+C
  SIGQUIT (3)  — Ctrl+\ (core dump)
  SIGUSR1 (10) — user-defined
  SIGUSR2 (12) — user-defined
  SIGCHLD (17) — child process state changed

Trap in bash:
  trap 'echo caught' SIGTERM
  trap cleanup EXIT          — runs on ANY exit (normal or error)

Send:
  kill -SIGTERM <pid>        — or just: kill <pid>
  kill -9 <pid>              — SIGKILL (last resort)
  kill -0 <pid>              — test if process exists (no signal sent)

Process States

STAT column in ps aux:

R  = Running or runnable
S  = Sleeping (interruptible — waiting for I/O)
D  = Uninterruptible sleep (usually disk I/O — cannot be killed)
T  = Stopped (SIGSTOP or Ctrl+Z)
Z  = Zombie (exited but parent hasn't called wait())
I  = Idle (kernel thread)

Modifiers:
<  = High priority
N  = Low priority (nice)
s  = Session leader
l  = Multi-threaded
+  = Foreground process group

Competition scenario: "Why can't I kill this process?"
  D state → waiting for I/O → SIGKILL won't work until I/O completes
  Z state → already dead → parent needs to wait() or parent must die

Drills

Drill 2.1 — strace a simple command

# Trace cat reading a file. Identify the syscall sequence:
strace -e trace=openat,read,write,close cat /etc/hostname 2>&1

# Expected sequence: openat() → read() → write() → close()
# man strace → -e trace=SET

Drill 2.2 — strace a failing command

# Why does this fail? Use strace to find the actual error:
strace -e trace=openat cat /nonexistent/file 2>&1

# Look for: openat() returning -1 ENOENT
# man 2 open → ERRORS section

Drill 2.3 — file descriptor investigation

# Start a background process, inspect its fds:
sleep 300 &
PID=$!

ls -la /proc/$PID/fd/
cat /proc/$PID/status | head -10
cat /proc/$PID/cmdline | tr '\0' ' ' && echo

kill $PID

Drill 2.4 — zombie process creation and cleanup

# Create a zombie:
bash -c 'sleep 1 & exec sleep 300' &
PARENT=$!
sleep 2

# Find the zombie:
ps aux | grep Z

# The sleep 1 child exited but parent (sleep 300) hasn't wait()ed.
# Kill the parent to reparent zombie to init (PID 1), which reaps it:
kill $PARENT
sleep 1
ps aux | grep Z    # zombie should be gone

Drill 2.5 — signal handling in bash

# Write a script that catches SIGTERM and cleans up:
cat << 'SCRIPT' > /tmp/drill-signal.sh
#!/usr/bin/env bash
set -euo pipefail

TMPFILE=$(mktemp)
trap 'echo "Caught signal, cleaning up $TMPFILE"; rm -f "$TMPFILE"; exit 0' SIGTERM SIGINT
trap 'rm -f "$TMPFILE"' EXIT

echo "PID: $$, tmpfile: $TMPFILE"
echo "Send: kill -SIGTERM $$"

while true; do
  date >> "$TMPFILE"
  sleep 1
done
SCRIPT

chmod +x /tmp/drill-signal.sh
/tmp/drill-signal.sh &
# Then: kill -SIGTERM <pid>

Drill 2.6 — pipe internals

# Trace what happens in a pipe:
strace -f -e trace=pipe,fork,dup2,read,write sh -c 'echo hello | cat' 2>&1 | head -30

# -f follows forks (pipe creates child processes)
# You'll see: pipe() → fork() → dup2() → write() → read()
# man 2 pipe → DESCRIPTION

Drill 2.7 — lsof practical

# What files does sshd have open?
lsof -c sshd | head -20

# What process is listening on port 22?
lsof -i :22

# What processes have files open in /etc/?
lsof +D /etc/ 2>/dev/null | head -20

# What deleted files are still held open?
lsof +L1

# man lsof → OUTPUT section (understand every column)