Trace System Calls with bpftrace

By Kokil Thapa | Last reviewed: September 2026

A slow Laravel queue worker, a stuck PHP-FPM request, or a mystery spike in disk I/O often starts with one question: which system calls is this process actually making? You can trace system calls with bpftrace on modern Linux without attaching a heavy debugger or restarting the service. On production Ubuntu servers I maintain for booking and CRM applications, bpftrace has replaced many ad-hoc strace sessions because it adds near-zero overhead and exits cleanly. This guide walks through install steps, copy-paste one-liners, filtering patterns, and the trade-offs against strace and perf.

What is bpftrace and why trace system calls at all?

System calls are the boundary between user space and the kernel. Every file open, socket connect, and memory allocation crosses that line. When application logs stay silent, syscall traces reveal the truth: repeated stat() on missing cache files, blocking read() on a pipe, or thousands of failed connect() attempts to a dead upstream.

bpftrace is a high-level front end for eBPF on Linux. You write short scripts; the tool compiles them to BPF bytecode and attaches probes in the kernel. Unlike classic tracers that use ptrace, eBPF runs in-kernel with bounded cost. That matters on shared VPS or EC2 hosts where a mis-tuned tracer can starve PHP-FPM workers.

If you already think in terms of observability versus monitoring, syscall tracing sits beside metrics and structured logs. It answers “what did the kernel see?” when your app never logged the failure. For broader context on how traces fit a stack, see the comparison of metrics, logs, and traces.

When syscall tracing beats application logs

Logs tell you what developers instrumented. Syscalls tell you what actually happened. Common cases I see on Linux system administration engagements include permission errors masked as generic 500 responses, DNS stalls visible only as long connect() latencies, and runaway cron jobs hammering unlink() on temp directories.

How do you install bpftrace on Ubuntu Linux?

Ubuntu 22.04 and 24.04 ship bpftrace in the main repositories. You need a kernel with eBPF and tracepoint support. Most stock Ubuntu cloud images qualify. Root or CAP_BPF plus CAP_PERFMON is required to load programs.

1. Update package lists and install bpftrace with kernel headers for BTF:

sudo apt update
sudo apt install -y bpftrace linux-headers-$(uname -r)

Verify the install and kernel BTF availability:

bpftrace --version
ls /sys/kernel/btf/vmlinux

If /sys/kernel/btf/vmlinux exists, you get richer introspection through CO-RE. Without BTF, many scripts still work via tracepoint probes. Check your kernel config when probes fail to attach:

grep CONFIG_BPF /boot/config-$(uname -r)
grep CONFIG_FTRACE /boot/config-$(uname -r)

On hardened hosts, AppArmor or LSM policies may block BPF loading. For managed servers under support and maintenance contracts, document which environments allow tracing before an incident.

Quick sanity check script

Run a five-second syscall count to confirm everything works:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter { @[probe] = count(); }
    interval:s:5 { exit(); }'

You should see histogram lines per syscall name. No output usually means permissions, not a broken install.

How do you trace system calls with bpftrace one-liners?

The fastest way to trace system calls with bpftrace is a single -e argument. Syscall tracepoints follow the pattern tracepoint:syscalls:sys_enter_NAME and sys_exit_NAME. Argument names differ per syscall; the bpftrace docs and bpftrace -l help you discover them.

Trace every openat call with filename

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat
{
  printf("%s %d %s\n", comm, pid, str(args->filename));
}'

This prints process name, PID, and the path passed to openat. On a busy web server the stream is noisy. Add a filter next.

Filter by process name

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat
/ comm == "php-fpm8.3" /
{
  printf("PID %d opened %s\n", pid, str(args->filename));
}'

Replace the comm string with your binary: nginx, mysqld, redis-server, or a custom queue worker. The comm field is truncated to 16 characters on most kernels.

Filter by PID

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read
/pid == 4821/
{
  @bytes = hist(args->count);
}'

Find the PID with pgrep first. PID filters are precise when several workers share the same comm.

Measure syscall latency on exit

Pair enter and exit probes with a timestamp map:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read  /@start[tid]/
{
  @us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

The histogram buckets show microsecond latency for read(). Spikes often correlate with slow disks or blocked pipes. This pattern generalises to write, sendto, and fsync.

Save and reuse a script file

Move repeated probes into /usr/local/bin/traces/openat.bt:

#!/usr/bin/env bpftrace

tracepoint:syscalls:sys_enter_openat
/ comm == $1 /
{
  printf("%s %d %s\n", comm, pid, str(args->filename));
}

Run with sudo bpftrace openat.bt php-fpm8.3. Parameterised scripts keep one-liners out of shell history.

Which bpftrace patterns help filter and aggregate syscalls?

Raw syscall streams overwhelm terminals within seconds. Production tracing needs aggregation: counts, histograms, and top-N summaries. bpftrace map types handle this in-kernel so only summaries reach user space.

Top syscalls by count for one process

sudo bpftrace -e 'tracepoint:syscalls:sys_enter
/ comm == "nginx" /
{
  @[probe] = count();
}
interval:s:10 { print(@); clear(@); }

Every ten seconds you get a fresh ranking. Watch for unexpected sys_enter_futex or sys_enter_epoll_wait dominance when you expected disk I/O.

Failed syscalls only

sudo bpftrace -e 'tracepoint:syscalls:sys_exit_openat
/ args->ret < 0 /
{
  printf("FAIL pid=%d ret=%d comm=%s\n", pid, args->ret, comm);
}'

Negative return values are errno codes. A burst of -2 (ENOENT) on config paths often explains startup failures better than PHP stack traces alone.

Network connect tracing

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect
{
  $sa = args->uservaddr;
  printf("connect pid=%d comm=%s\n", pid, comm);
}'

For full IP parsing, use built-in helpers or kprobes on tcp_connect. Start simple; add detail only when the probe cost stays acceptable.

Discover available syscall probes

bpftrace -l 'tracepoint:syscalls:*' | head -20
man 2 syscalls

List probes before writing filters. The official bpftrace reference guide documents map types, builtins, and probe syntax.

When you need to validate regex filters for log pipelines alongside syscall work, a regex tester saves round trips. For JSON log payloads from tracing sidecars, a JSON formatter keeps field inspection quick.

How do you trace syscalls for a specific process safely in production?

Production tracing demands a short blast radius. Run as root on the host, not inside an unprivileged container missing BPF caps. Cap probe duration, scope by PID or cgroup, and prefer aggregated maps over per-event printf.

• Set a hard timeout: wrap scripts with timeout 30s bpftrace ....
• Trace one PID, not the whole comm, when dozens of workers share a name.
• Avoid tracing high-frequency syscalls (read, write, futex) without filters on busy hosts.
• Run during a maintenance window for first-time scripts on payment or booking paths.
• Record output to a file; do not pipe through tools that block on backpressure.

On a legal-tech portal deployment, I traced a document upload stall by filtering write and fsync on a single PHP-FPM PID. The culprit was a synchronous NFS mount option, not the application code. That class of issue aligns with testing and optimization work where the bottleneck lives outside the repo.

Attach to a running PID with duration cap

TARGET=$(pgrep -n php-fpm8.3)
timeout 20s sudo bpftrace -e "
tracepoint:syscalls:sys_enter_openat /pid == $TARGET/ {
  @files[str(args->filename)] = count();
}
interval:s:5 { print(@files); clear(@files); }
"

Adjust the interval and timeout to match traffic. Twenty seconds is often enough for a single failing request path.

Integrate with on-call workflows

Store vetted one-liners in your incident runbook next to disk and memory checks. When paging fires during a deploy, syscall traces confirm whether the new release changed file access patterns. Pair this habit with an on-call and incident response runbook so traces are step three, not a panic grep.

How does bpftrace compare to strace and perf for syscall tracing?

All three tools answer overlapping questions with different cost models. Pick based on environment, skill depth, and whether the process can tolerate ptrace overhead.

strace -c gives a summary similar to bpftrace maps but still uses ptrace. On a PHP-FPM pool serving concurrent requests, ptrace attachment can slow every worker. bpftrace avoids that stop-start cycle.

perf trace sits between strace and bpftrace for syscall tracing. It is excellent when you already collect perf data for flame graphs. bpftrace wins when you want a ten-line script with custom maps and printf control. The Linux syscalls(2) man page lists errno semantics shared by all three tools.

For capacity reviews, syscall rates inform disk and network headroom. Combine trace findings with capacity planning for growing systems so a fix today does not become next quarter's outage.

Multi-cloud and container notes

bpftrace runs on the host or in privileged containers with BPF capabilities. Standard Kubernetes pods without caps cannot load probes. For multi-host fleets, centralise findings through your existing log pipeline rather than leaving SSH sessions open. The pattern mirrors multi-cloud observability where host-level tools feed higher-level dashboards.

On sister sites sharing Deployer releases, I keep identical bpftrace packages on each Ubuntu node. Version drift breaks script portability. Treat tracing tools like any other ops dependency pinned in your bootstrap docs alongside notes on network baseline configuration.

Key Takeaways

• Install bpftrace on Ubuntu with apt install bpftrace and confirm BTF at /sys/kernel/btf/vmlinux.
• Trace system calls with bpftrace via tracepoint:syscalls:sys_enter_* and sys_exit_* probes, not ptrace.
• Always filter by PID or comm and cap runtime with timeout on production hosts.
• Use maps (count(), hist()) instead of raw printf on high-frequency syscalls.
• Prefer bpftrace over strace for live servers; keep strace for local single-process debugging.
• Store proven one-liners in your incident runbook next to disk, memory, and queue checks.

People Also Ask

Do you need root to run bpftrace?

Loading BPF programs requires root or the CAP_BPF and CAP_PERFMON capabilities on modern kernels. Read-only listing with bpftrace -l may work unprivileged. On production servers, use sudo with command logging rather than sharing root passwords.

Can bpftrace trace all syscalls at once?

Yes, attach to tracepoint:syscalls:sys_enter without a suffix. Output volume explodes on busy hosts. Start with one syscall family like openat or connect, then widen only if needed.

Does bpftrace work in Docker containers?

Only privileged containers with BPF capabilities and access to host tracepoints. Default bridge containers cannot load probes. Trace from the host PID namespace or use nsenter against the container PID on the node.

Is bpftrace available on Amazon Linux and RHEL?

Yes, through distribution packages or the bpftrace GitHub project build instructions. Ubuntu 22.04 and 24.04 remain the simplest path for teams already on that stack.

Next steps for your Linux fleet

You now have install commands, filtered one-liners, aggregation patterns, and production guardrails to trace system calls with bpftrace on live Ubuntu servers. Start on a staging node mirroring production PHP-FPM or nginx versions. Copy three scripts into your runbook: open path audit, connect latency histogram, and failed syscall counter. Run each for twenty seconds during normal traffic before the next incident forces your hand.

If your team lacks time to build observability into deploy pipelines and incident docs, enterprise application development and Linux administration support can wire tracing playbooks into the same GitLab CI workflows that ship your code. Review shipped work on the portfolio or read more on the blog, including fleet automation with AWS Systems Manager.

Need hands-on help tracing a production bottleneck on your stack? Contact us with your distro, kernel version, and the process name — we will suggest a scoped bpftrace probe set before anyone attaches strace to a hot pool.