Phase 1: Syscall Tracing — From C to Kernel

The Bridge: Your C → Syscalls → Kernel

Three lines of C generate three syscalls. The kernel executes 40+ internal functions to service them. This exercise connects your C education to kernel internals, AppArmor enforcement, and infrastructure (ISE/DNS/TLS).

The Program

// /tmp/trace-me.c
#include <stdio.h>
int main(void) {
    FILE *f = fopen("/tmp/hello.txt", "w");
    fprintf(f, "ftrace sees this\n");
    fclose(f);
    return 0;
}

gcc -o /tmp/trace-me /tmp/trace-me.c

Layer 1: strace — What Syscalls Does Your C Make?

strace intercepts the boundary between userspace (your C) and kernel.

strace -e openat,write,close /tmp/trace-me

Output (2026-04-27)

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3    (1)
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3  (2)
close(3)                                = 0
openat(AT_FDCWD, "/tmp/hello.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3  (3)
write(3, "ftrace sees this\n", 17)      = 17                     (4)
close(3)                                = 0                       (5)

1	Dynamic linker loading shared library cache
2	Loading libc — every C program needs this
3	Your `fopen("w")` → `openat` with `O_WRONLY\|O_CREAT\|O_TRUNC` (write, create if missing, truncate to zero)
4	Your `fprintf` → `write` syscall, 17 bytes
5	Your `fclose` → `close` syscall

Key insight: fopen is a libc wrapper. The kernel never sees fopen — it sees openat. Your C and the kernel speak different languages; libc translates.

Layer 2: ftrace — What Does the Kernel Do Internally?

strace shows the syscall boundary. ftrace shows what happens inside the kernel when it services your syscall. This is Steven Rostedt’s infrastructure — built into every Linux kernel.

Find ftrace

mount | grep tracefs
# → tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)

# How many kernel functions can you trace?
sudo cat /sys/kernel/tracing/available_filter_functions | wc -l
# → 91,083

# Available tracers
sudo cat /sys/kernel/tracing/available_tracers
# → timerlat osnoise hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop

# How many syscalls can you hook?
ls /sys/kernel/tracing/events/syscalls/ | wc -l
# → 734

Old path: /sys/kernel/debug/tracing/ (via debugfs). New path: /sys/kernel/tracing/ (tracefs). Rostedt split tracefs out as its own filesystem — smaller attack surface than mounting all of debugfs.

Trace do_sys_openat2

# Set up the tracer
sudo bash -c '
  echo function_graph > /sys/kernel/tracing/current_tracer
  echo do_sys_openat2 > /sys/kernel/tracing/set_graph_function
  echo > /sys/kernel/tracing/trace
  echo 1 > /sys/kernel/tracing/tracing_on
'

# Run your program
/tmp/trace-me

# Capture and stop
sudo bash -c '
  echo 0 > /sys/kernel/tracing/tracing_on
  cat /sys/kernel/tracing/trace
' | head -80

Output (2026-04-27) — kernel’s C responding to your fopen

# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
  5)               |            file_ra_state_init() {
  5)   0.318 us    |              inode_to_bdi();
  5)   1.484 us    |            }
  5) * 15872.94 us |          } /* do_dentry_open */
  5) * 15873.30 us |        } /* vfs_open */
  5)               |        security_file_post_open() {
  5)   0.137 us    |          bpf_lsm_file_post_open();
  5)   0.451 us    |        }
  5)               |        security_file_truncate() {
  5)   0.225 us    |          hook_file_truncate();
  5)               |          apparmor_file_truncate() {        (1)
  5)   0.207 us    |            make_vfsuid();
  5)   0.534 us    |            common_perm();                  (2)
  5)   1.308 us    |          }
  5)   0.127 us    |          bpf_lsm_file_truncate();
  5)   2.287 us    |        }

1	AppArmor checking your file operation — this is the MAC enforcement from Phase 12
2	`common_perm()` — the function that returns DENIED when a browser tries to read `~/.secrets/`

Clean Up

sudo bash -c '
  echo nop > /sys/kernel/tracing/current_tracer
  echo > /sys/kernel/tracing/set_graph_function
'

Save a Trace

sudo cat /sys/kernel/tracing/trace > /tmp/ftrace-openat-$(date +%F).txt
wc -l /tmp/ftrace-openat-$(date +%F).txt

Layer 3: strace on Real Infrastructure

Your ISE API shell functions are just curl — and curl is C making syscalls.

dsource d000 dev/network/ise

strace -e openat,socket,connect -f curl -sS --cacert "${ISE_CA_CERT}" \
  -u "${ISE_API_USER}:${ISE_API_PASS}" \
  -H "Accept: application/json" \
  "https://${ISE_PAN_FQDN}:${ISE_ERS_PORT}/ers/config/endpoint?size=1" 2>&1 \
  | grep -E 'openat|connect|socket'

Output (2026-04-27) — your entire infrastructure visible in syscalls

openat("/usr/lib/libcurl.so.4")                                       (1)
openat("/usr/lib/libssl.so.3")                                        (2)
openat("/usr/lib/libcrypto.so.3")
openat("/usr/lib/libkrb5.so.3")                                      (3)
openat("/etc/nsswitch.conf")
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK)
connect(5, {sin_port=htons(53), sin_addr=inet_addr("10.50.1.90")})   (4)
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK)
connect(5, {sin_port=htons(9060), sin_addr=inet_addr("10.50.1.21")}) (5)
openat("/home/evanusmodestus/.secrets/certs/d000/ise/ROOT-CA.crt")    (6)

1	curl loads 26 shared libraries before doing anything
2	OpenSSL for TLS
3	Kerberos libraries (loaded even if not used — dynamic linker resolves all deps)
4	DNS query to your BIND server (10.50.1.90:53) resolving `ise-02.inside.domusdigitalis.dev`
5	TCP connection to ISE (10.50.1.21:9060) — ERS API port
6	Loading your CA cert from age-managed secrets to validate ISE’s TLS certificate

The Full Stack

Your Code          Shell function: ers "/endpoint?size=1"
    ↓
Binary             curl (compiled C)
    ↓
libc               fopen → openat, connect, write, read
    ↓
Syscall boundary   ← strace sees this
    ↓
Kernel VFS         do_sys_openat2 → vfs_open → do_dentry_open
    ↓
Security hooks     apparmor_file_truncate → common_perm    ← ftrace sees this
    ↓
Filesystem         write to inode on disk (or tmpfs in memory)
    ↓
Network stack      socket → connect → TCP handshake → TLS → HTTP
    ↓
Wire               Packets to ISE at 10.50.1.21:9060

Everything is syscalls. C is just the language they’re written in.

What to Explore Next

Trace write syscall through ftrace (see the VFS write path)
Write C that reads from /sys/kernel/tracing/trace — your C reading the kernel’s C
Use perf to profile your C program — see which functions take the most time
Trace AppArmor denials in real-time: sudo cat /sys/kernel/tracing/trace_pipe | grep apparmor
Write a C program that makes a raw openat syscall without libc (syscall(SYS_openat, …))

Key People

Steven Rostedt	Author of `ftrace`, the kernel’s built-in tracing framework. Maintains the ring buffer that all kernel tracing flows through. 20+ years as kernel developer.
Linus Torvalds	Created Linux, maintains the kernel. Rostedt collaborates with him — different domains, same codebase.