Measure your test suite

py-spy vs tachyon: benchmarking sampling profilers on pytest suites

We profiled pytest suites with py-spy and Python 3.15's tachyon at different settings. Non-blocking sampling silently drops up to 90% of samples, and the ones it keeps are biased.

flakytest profiles pytest suites with a sampling profiler: py-spy on Python 3.11 to 3.14, and tachyon, the sampling profiler that ships with Python 3.15 as profiling.sampling, on 3.15. We wanted numbers for what each one costs, so we benchmarked both at the settings people actually use.

The overhead turned out to be the less interesting part. The interesting part is how many samples each profiler loses, and that the ones it loses aren't random:

• py-spy in non-blocking mode, the mode most people use because it doesn't pause your program, dropped 42% to 94% of its samples in our microbenchmarks and 62% on a Django test suite. It records the failures as an error count at the very end of its output, which is easy to miss.
• tachyon in non-blocking mode dropped 10% to 54%, depending on the workload.
• The dropped samples are biased. Code that calls lots of functions (most real code) is under-counted, and tight loops look more expensive than they are. On a workload that spent exactly half its time in each kind of code, non-blocking py-spy reported a 20/80 split and non-blocking tachyon 37/63.
• Blocking mode fixes this for both, but py-spy's blocking mode is so expensive per sample that above about 50 samples per second it can slow a Django suite down 24 times, or effectively stop it. tachyon's blocking mode captured 99% of samples at 1000 Hz for 6% to 15% overhead.

The rest of this page shows how we measured it and why it happens: tachyon reads a stack with 5 to 16 memory reads where py-spy needs 110 to 1,100, thanks to support that Python 3.15 added to the interpreter.

How we measured

Everything ran on one cloud VM: 4 vCPUs (Intel Xeon), 16 GB of RAM, Linux. py-spy 0.4.2 on CPython 3.14.6, tachyon on CPython 3.15.0b4. py-spy 0.4.2 refuses to attach to Python 3.15 ("Unsupported version of Python"), so the two can't be compared on the same interpreter. Instead every number is relative to an unprofiled run on the same Python version. Runs were shuffled so that drift on the machine spreads across all configurations, and we report the median of 3 (microbenchmarks) or 5 (suites) repetitions.

Two kinds of benchmarks:

1. Microbenchmark. A single Python process in a tight loop that recurses to a fixed depth (10, 50 or 150 frames), calls a small function and returns, for 6 seconds. The sampler attaches from outside, like py-spy and tachyon normally do. We measure how much the program slows down (iterations per second against an unprofiled run), the CPU the sampler uses, and how many samples it actually records versus how many it failed to read.

2. Whole suites through flakytest. Three pytest suites run with pytest --flakytest-profile, exactly as a user would:

   • cpu: 300 tests of about 30 ms of pure-Python work each. The stack is 47 frames deep in the middle of a test (pytest, pluggy and the test itself).
   • deep: the same work 150 frames further down: 187 frames.
   • django: 600 Django TestCase tests against SQLite: ORM queries, templates, the test client. Median stack depth 44 frames, p90 65.

   We also ran the cpu suite with pytest -n 4. Each test records how long it spent in two functions with perf_counter, so we know the true split and can check how close the profile gets.

Both profilers were run the way flakytest runs them: py-spy as py-spy record --nonblocking --threads --idle --format raw (flakytest's default is 500 Hz), and tachyon through profiling.sampling.SampleProfiler at 1000 Hz (flakytest's default), in its default non-blocking mode. "Blocking" variants drop --nonblocking for py-spy, and pass blocking=True to tachyon.

The harness, the test suites and the raw results are in the flakytest-python repository, with instructions to reproduce them.

Result 1: non-blocking sampling loses most samples

The microbenchmark at the default rates, with failed reads as a share of all attempts:

At depth 50 and 150, non-blocking py-spy also exited without writing any samples in 16 of 24 runs. When we reproduced it by hand it failed while attaching ("Failed to find python version from target process", "Failed to find a python interpreter in the .data section"). That's partly an artifact of attaching right as the target starts; through flakytest, which attaches to an already running pytest, it never failed to attach. The failed reads, though, are just as visible in the real suites:

(Slightly over 100% because the rate is relative to the measured test time, which excludes a bit of startup.)

Django is the interesting row: its stack isn't deep, but it changes constantly. A Django test calls thousands of small functions (ORM, template rendering, middleware), so the stack rarely looks the same for long. That turns out to be what matters most.

Result 2: the samples that are lost aren't random

A missing sample is harmless if every part of your code is equally likely to lose one: you get the same profile, just with more noise. So we checked. The target alternates between two functions for 50 ms each, for 6 seconds:

• flat() runs a loop in one frame: the stack doesn't change while it runs.
• churn() calls a 30-deep recursive function over and over: the stack changes every few microseconds.

The true split is 50/50. Here's what each profiler reported:

Non-blocking py-spy made flat() look four times as expensive as churn(), when they cost the same. Code that makes lots of short calls (most application code) is under-represented, and code that loops in one place is over-represented. You'd optimise the wrong thing.

Fewer samples also mean noisier per-test results. In the deep suite, where non-blocking py-spy recorded 20% of its samples, its estimate of how each test split its time between two functions was off by 24 percentage points on average. tachyon's was off by 3.5 points non-blocking and 2.4 points blocking.

Result 3: what each setting costs

Overhead of a whole flakytest-profiled run against an unprofiled run on the same Python, and the CPU the profiler process itself used:

The "without a profiler" column is flakytest's own bookkeeping (per-test timings, fixture timings, query counting): about 10% on the Django suite, where it counts every SQL query, and nothing measurable on pure-Python tests.

Three things stand out:

py-spy burns a CPU core. At 500 Hz on the cpu suite, py-spy used 8.1 seconds of CPU during a 9.6-second run, and at 1000 Hz it saturated a core and still only managed 53% of the requested rate. tachyon used under a second for twice the rate. On a laptop with idle cores you barely notice. Under pytest-xdist you do: -n 4 on a 4-CPU machine starts four workers and four py-spy processes, each wanting a full core, and the run took 13.5 seconds instead of 3.2.

py-spy's blocking mode has a cliff. At 50 Hz it costs 37% on the Django suite. At 100 Hz the same suite took 178 seconds instead of 7.3 (24 times slower), and at 500 Hz it took 99 minutes. A cpu-suite run at 500 Hz was still going after 20 minutes, when we stopped it. More on why below.

tachyon is cheap at any rate we tried. Even at 5000 Hz non-blocking it added 6% on the cpu suite and used 2.5 seconds of CPU. Blocking costs more than non-blocking (on Django, +36% against +21%), because the program is paused while each stack is read. But it's the only setting in this table that records every sample without a bias.

Why: one read per frame versus one read per stack

Both profilers work the same way from the outside: a separate process reads the target's memory (process_vm_readv on Linux) and reconstructs the Python stack from the interpreter's data structures. The difference is how much they read per sample. We counted the system calls with strace at 500 Hz (py-spy in blocking mode, so that every sample completes):

py-spy reads every frame separately

py-spy's stack walker follows the frame chain one frame at a time. For each frame it copies the frame struct, then the code object it points to, then the filename string, the function name string, and the line number table, each as a separate read, every sample. That's about 7 reads per frame, and it doesn't reuse anything from the previous sample.

In non-blocking mode the program keeps running while those hundreds of reads happen. If a function returns halfway through, the frame py-spy is about to read is gone (or reused by another call), the next read returns garbage, and the whole sample is discarded. The more reads a sample takes (deeper stacks), and the faster the stack changes (code that makes many calls), the more likely that is. That's exactly the pattern in the tables above, and it's where the bias comes from: the code most likely to lose samples is code that is busy calling functions.

In blocking mode py-spy pauses the program first. Per sample, its remoteprocess dependency lists /proc/<pid>/task and, for every thread, attaches with PTRACE_SEIZE, stops it with PTRACE_INTERRUPT, waits for the stop, and after the read detaches again. The program is paused for the whole read: about 0.7 ms per sample at depth 50 in our microbenchmark, and more on real pytest stacks. Once that pause gets close to the sampling interval (2 ms at 500 Hz, 10 ms at 100 Hz, with the line tables of big pytest and Django functions to copy), py-spy falls behind. It then takes samples back to back to catch up, and the program spends nearly all its time paused. That's the cliff in Result 3: the program still runs, but in the gaps between samples.

Python 3.15 helps tachyon read the stack

tachyon lives in CPython itself (Modules/_remote_debugging), so it doesn't have to reverse-engineer the interpreter. Three things make its reads few and cheap:

1. The interpreter publishes its own layout. Since Python 3.13, CPython embeds a _Py_DebugOffsets table in a named section of the binary, with the offset of every field a profiler needs. tachyon reads it once at attach. py-spy instead ships its own struct definitions for every Python version and locates the interpreter from its symbols or by scanning its memory, which is why each new Python release needs a new py-spy release, and why 0.4.2 can't attach to 3.15.
2. Frames are copied in bulk. Since Python 3.11, a thread's frames live in contiguous "data stack chunks". On a cache miss tachyon copies whole chunks in one read each and parses the frames from that snapshot, instead of chasing pointers frame by frame. Code objects and their names are cached across samples (99% hit rate on the Django suite).
3. The interpreter tells tachyon what changed. Python 3.15 added two fields to every thread state, last_profiled_frame and last_profiled_frame_seq. After each sample tachyon writes the address of the frame it just saw into the target. When the interpreter pops that frame, it moves the marker to the parent and bumps the sequence number. On the next sample tachyon walks from the current frame up only until it meets the marker: everything above it is unchanged since the last sample and comes from tachyon's cache.

Fewer reads mean a much shorter window for the stack to change under the reader, so non-blocking tachyon loses far fewer samples than non-blocking py-spy. It still loses some, because it still reads a moving target. In blocking mode tachyon pauses threads with the same ptrace calls as py-spy, but the pause is short: about 0.08 ms per sample at depth 50 in our microbenchmark, against py-spy's 0.7 ms.

What to use

• On Python 3.15+, use tachyon. If you call profiling.sampling yourself, pass blocking=True when you care about where time goes (the bias above), and keep the default non-blocking mode when you only need a rough picture with the lowest overhead.
• With py-spy on 3.14 and older, non-blocking mode is the only one that's cheap at useful rates. Check the Errors: count py-spy prints at the end: if it's a large share of Samples:, treat the profile with suspicion, especially for call-heavy code. Use blocking mode (without --nonblocking) at low rates only (around 50 Hz for a typical pytest suite) and over longer runs, to collect enough samples. Avoid running one py-spy per xdist worker on a machine without spare cores.
• Compare like with like. A profile from a non-blocking profiler that dropped 60% of its samples isn't comparable to one that dropped 10%, even of the same code.

What this means for flakytest

flakytest runs py-spy non-blocking at 500 Hz on Python 3.11 to 3.14, and tachyon non-blocking at 1000 Hz on 3.15+. Its per-test durations, fixture times and query counts are measured exactly and don't depend on the profiler. The breakdown inside each test does, and these results apply to it.

Caveats

• One machine, one configuration of each profiler. A virtual machine with 4 vCPUs; results on bare metal or with more cores will differ, especially for the xdist case.
• Python 3.15.0b4 is a beta, and tachyon is new. Its numbers may change before and after the 3.15.0 release, and py-spy will likely gain 3.15 support.
• py-spy and tachyon ran on different Python versions (3.14 and 3.15). We compared each with an unprofiled run on its own version to account for that, but the interpreters themselves differ.
• The microbenchmark is deliberately hostile: a stack that changes every few microseconds. Real code is usually somewhere between it and the flat loop. The suite results are the better guide to what you'll see.

Further reading

• Python 3.15: profiling.sampling
• py-spy, and its --nonblocking option in the README
• The benchmark harness and raw results
• How to profile a pytest suite

Related guides

• How to profile a pytest suite: Profile slow pytest tests with cProfile, pytest-profiling, py-spy, pyinstrument, Python 3.15's tachyon or flakytest: the commands, what each one shows, and which to use when.
• Find your slowest pytest tests: List the slowest pytest tests with --durations and --durations-min, read setup vs call vs teardown times, then fix, split or mark slow tests so they stop holding up the suite.
• How to speed up a slow pytest suite: A checklist for slow Python test suites, from the fixes that usually save the most time (password hashing, migrations, test data) to parallelism and CI. Each item links to a detailed guide.

Updated October 19, 2026 · Suggest a change