Phase 1: Filesystem & Inode Mastery

Target: Weeks 2-4 (Apr 28 - May 18, 2026)

Man Pages to Master

Reference What It Teaches

Reference	What It Teaches
`man 7 inode`	Inode structure, timestamps (atime/mtime/ctime), permission bits, link count
`man 2 stat`	The `struct stat` — every field the kernel exposes about a file
`man 1 stat`	`stat` command — format strings (`-c`), all format sequences
`man find`	TESTS section: `-inum`, `-links`, `-type`, `-size`, `-mtime`, `-newer`, `-exec`
`man ln`	Hard links vs symbolic, behavior on deletion
`man df`	Filesystem-level space reporting — `-i` for inode usage
`man du`	Directory-level space reporting — `-s`, `-h`, `--apparent-size`
`man lsof`	Open files, `+L1` for deleted-but-open, `-p` for PID, `-i` for network
`man proc`	The `/proc` filesystem — `/proc/self`, `/proc/<pid>/fd`, `/proc/mounts`
`man findmnt`	Mount tree — filesystem types, mount options, source devices
`man lsblk`	Block device tree — partitions, sizes, mount points, filesystem types
`man fsck`	Filesystem check — when and why, inode recovery
`man debugfs`	ext4 debugging — inode inspection, undelete, superblock recovery

man 7 inode

Inode structure, timestamps (atime/mtime/ctime), permission bits, link count

man 2 stat

The struct stat — every field the kernel exposes about a file

man 1 stat

stat command — format strings (-c), all format sequences

man find

TESTS section: -inum, -links, -type, -size, -mtime, -newer, -exec

man ln

Hard links vs symbolic, behavior on deletion

man df

Filesystem-level space reporting — -i for inode usage

man du

Directory-level space reporting — -s, -h, --apparent-size

man lsof

Open files, +L1 for deleted-but-open, -p for PID, -i for network

man proc

The /proc filesystem — /proc/self, /proc/<pid>/fd, /proc/mounts

man findmnt

Mount tree — filesystem types, mount options, source devices

man lsblk

Block device tree — partitions, sizes, mount points, filesystem types

man fsck

Filesystem check — when and why, inode recovery

man debugfs

ext4 debugging — inode inspection, undelete, superblock recovery

Concepts to Internalize

The Inode Model

Directory entry (name) → inode number → inode structure → data blocks

Inode stores:
  - File type (regular, directory, symlink, device, pipe, socket)
  - Permissions (rwx) + special bits (SUID, SGID, sticky)
  - Owner UID, Group GID
  - Size in bytes
  - Timestamps: atime (access), mtime (modify), ctime (change metadata)
  - Link count (how many directory entries point here)
  - Block pointers (direct, indirect, double-indirect, triple-indirect)

Inode does NOT store: the filename. Names live in directory entries.

Hard Link vs Symbolic Link

Hard link:
  - Another directory entry pointing to the SAME inode
  - Same inode number, link count increments
  - Deleting original: hard link still works (data exists until link count = 0)
  - Cannot cross filesystem boundaries
  - Cannot link to directories (prevent cycles)

Symbolic link:
  - A new file whose DATA is a path string
  - Different inode number, own permissions
  - Deleting original: symlink breaks (dangling)
  - Can cross filesystems
  - Can link to directories

stat Format Strings

every format you’ll need — man stat → FORMAT SEQUENCES

stat -c '%n'   file    # filename
stat -c '%i'   file    # inode number
stat -c '%h'   file    # hard link count
stat -c '%s'   file    # size in bytes
stat -c '%b'   file    # blocks allocated (512-byte units)
stat -c '%a'   file    # permissions (octal)
stat -c '%A'   file    # permissions (human: -rwxr-xr-x)
stat -c '%U'   file    # owner name
stat -c '%G'   file    # group name
stat -c '%x'   file    # atime (last access)
stat -c '%y'   file    # mtime (last modify)
stat -c '%z'   file    # ctime (last status change)
stat -c '%F'   file    # file type (regular file, directory, symbolic link)
stat -c '%t %T' file   # major/minor device number (for device files)

# Combined — the competition dump:
stat -c 'inode=%i links=%h perm=%a owner=%U:%G size=%s type=%F' file

The df vs du Mismatch (Competition Favorite)

Scenario: df shows 90% full. du shows only 60% used. Where's the missing space?

Answer: Deleted files still held open by a process.

Diagnosis:
  lsof +L1           # files with link count < 1 (deleted but open)

The file's directory entry is removed (du can't see it).
The inode still exists because a process has an open file descriptor.
df counts at the filesystem level (sees the blocks as used).
du counts by walking directory entries (can't see the deleted name).

Fix: identify the process, close or restart it. Space reclaimed.

/proc Filesystem

/proc is not on disk. It's a virtual filesystem — the kernel exposes runtime state as files.

/proc/self/          → your current process
/proc/<pid>/         → any process by PID
/proc/<pid>/fd/      → open file descriptors (symlinks to actual files)
/proc/<pid>/status   → name, state, memory, UIDs, threads
/proc/<pid>/cmdline  → command that started this process (null-separated)
/proc/<pid>/environ  → environment variables (null-separated)
/proc/<pid>/maps     → memory map (shared libs, heap, stack)
/proc/<pid>/cwd      → symlink to current working directory
/proc/<pid>/exe      → symlink to the executable binary
/proc/<pid>/root     → symlink to the filesystem root (may differ in chroot)

/proc/mounts         → current mounts (equivalent to findmnt)
/proc/filesystems    → supported filesystem types
/proc/cpuinfo        → CPU details
/proc/meminfo        → memory stats
/proc/loadavg        → load averages
/proc/uptime         → system uptime in seconds
/proc/sys/           → tunable kernel parameters (sysctl)

Drills

Drill 1.1 — inode inspection

# Create a file. Inspect its inode completely.
echo "arena drill" > /tmp/drill-1-1
stat /tmp/drill-1-1

# Extract just the inode number:
stat -c '%i' /tmp/drill-1-1

# Find all files on this filesystem with 2+ hard links:
find / -xdev -type f -links +1 2>/dev/null | head -10
# man find → -links, -xdev (don't cross filesystem boundaries)

Drill 1.2 — link count lifecycle

# Predict the link count at each step. Then verify with stat -c '%h'.
echo "test" > /tmp/drill-1-2          # link count = ?
ln /tmp/drill-1-2 /tmp/drill-1-2-hard # link count = ?
ln /tmp/drill-1-2 /tmp/drill-1-2-h2   # link count = ?
rm /tmp/drill-1-2                      # link count on remaining files = ?
rm /tmp/drill-1-2-hard                 # link count on last file = ?
cat /tmp/drill-1-2-h2                  # works? data intact?
rm /tmp/drill-1-2-h2                   # inode freed (link count = 0, no open FDs)

Drill 1.3 — deleted file recovery via /proc

# Open a file, delete it, recover from /proc
echo "critical data" > /tmp/drill-1-3
tail -f /tmp/drill-1-3 &
PID=$!
rm /tmp/drill-1-3

# File is deleted but PID holds it open
ls -la /proc/$PID/fd/ | grep deleted

# Recover:
cp /proc/$PID/fd/3 /tmp/drill-1-3-recovered
cat /tmp/drill-1-3-recovered

kill $PID

Drill 1.4 — df vs du discrepancy

# Simulate: create a large file, open it, delete it
dd if=/dev/zero of=/tmp/drill-1-4 bs=1M count=100
tail -f /tmp/drill-1-4 &
PID=$!
rm /tmp/drill-1-4

# du won't see it:
du -sh /tmp/ | grep drill    # nothing

# lsof will:
lsof +L1 | grep drill-1-4   # shows the deleted-but-open file

kill $PID
# Space reclaimed after process exits

Drill 1.5 — filesystem inode exhaustion

# Check inode usage:
df -i

# A filesystem can be out of inodes but have free space:
# Many tiny files = many inodes, little data
# man df → -i (show inode information instead of block usage)