OSORIO.SYS

Configuring eBPF for production profiling

eBPF profilers are cheap enough to leave on. They are also easy to misconfigure so that you either miss the interesting stacks or you melt a core on symbolization.

This is the setup I keep on production Linux boxes.

What to collect

  • On-CPU: perf_event samples at 49 Hz, not 99 Hz. The extra resolution is rarely worth the noise.
  • Off-CPU: finish_task_switch with a minimum blocked time, otherwise you profile every futex wait.
  • Kernel and user stacks. User-only profiles lie about syscall time.

What not to do

Do not symbolize on the box. Dump stack hashes and resolve them on a worker with the matching debuginfod cache. Production should ship maps, not spend CPU walking DWARF.

Pin programs with bpffs so a daemon restart does not drop the profile window you actually needed.

Verification

After attach, check:

sudo bpftool prog show
sudo cat /sys/kernel/debug/tracing/trace_pipe | head

If lost events climb, lower the sample rate before you add more probes. A complete 49 Hz profile beats a 99 Hz profile with holes.