Documentation

How flakytest works

Everything about the flakytest pytest plugin and CLI, and the optional cloud at flakytest.dev. In a hurry? The get started guide takes two minutes.

Overview

flakytest runs your pytest suite under a sampling profiler and explains, for every test, class and file, where the time goes: setup, the test itself and teardown, and inside those the database, password hashing, network calls, sleeps, fixtures, factories and imports, down to the line of your code that causes it. Then it ranks what to fix first, with the exact setting or line to change.

• Local and free. The plugin needs no account. Every run writes an HTML report and a profile.json to .flakytest/runs/.
• Made for agents too. flakytest report prints the same as Markdown, and flakytest mcp serves your runs to coding agents.
• flakytest.dev is optional. With a project token, runs from every developer machine and CI job are collected, so you can compare environments, follow trends, and track flaky tests.

Install

$ pip install flakytest              # or: uv add --dev flakytest
$ pip install 'flakytest[py-spy]'    # Python 3.11–3.14: adds the py-spy profiler

Install it into the environment your tests run in. It supports Python 3.11+ and pytest 6+. On Python 3.15+ flakytest uses tachyon, the sampling profiler in the standard library (profiling.sampling). On older versions it uses py-spy. Without either, you still get exact durations, fixture times and query counts, just no breakdown inside tests.

Run flakytest doctor to see which profiler will be used and whether it's allowed to attach (see troubleshooting).

Profile a run

$ flakytest run                   # pytest with profiling, then opens the report
$ flakytest run -- tests/api -x   # everything after -- goes to pytest
$ pytest --flakytest-profile      # the same from pytest (or FLAKYTEST_PROFILE=1 pytest)

At the end of the run pytest prints a summary:

============================== flakytest profile ===============================
812 tests in 1.4 min (test time 5.0 min) · local:anze@mbp · Python 3.15.0 · tachyon @ 1000 Hz
Time goes to: password hashing 41% · database 30% · test code 9% · sleep 6%
Slowest tests:
    3.21s  tests/test_orders.py::TestCheckout::test_checkout  (database 70%, setUp (call) 2.1s, 412 queries)
Fix next:
  1. [high] Use a fast password hasher in tests (~2.0 min per run)
  2. [high] Create shared test data in setUpTestData instead of setUp (~41s per run)
Report: .flakytest/runs/20261016-210311-8c3f2a/index.html

Every run writes .flakytest/runs/<run id>/index.html (the report) and profile.json (everything in it, as data). .flakytest/ adds its own .gitignore, so runs never end up in git. Runs are identified by their directory name; latest and previous work wherever a run is expected.

The report

index.html is a single file that works offline. On the left is every test, grouped by file and class, with a bar showing how its time splits into categories. Filter by name, show failed, the slowest 10% or under-sampled tests, and sort by time or name. Click a test, class or file to see:

• Where the time goes: setup, call and teardown, and the time by category.
• Biggest pieces of work: the top-level things each phase did: fixtures, setUp, the test body, Django's transaction setup.
• Hot lines in your code: the deepest line of your project in each sample, with its source and category. "Which line of mine is slow."
• Call tree and hot functions, with pytest and pluggy internals hidden.
• Fixtures: exact setup time of every fixture the test ran, and its scope. Class, module and session fixtures are charged to the test that triggered them, and marked as shared.
• Database: query count and time for Django and SQLAlchemy, and repeated queries (N+1).
• Fix next: the suggestions that apply here.

The session view adds a timeline (startup, collection, tests, teardown), the slowest imports during startup and collection, and fixture totals. Copy for agent copies what you're looking at as Markdown. flakytest serve serves all runs on localhost if you'd rather not open files.

CLI reference

The CLI looks for .flakytest/ in the current directory and its parents, so it works from anywhere in your project.

Options and environment

How profiling works

Attributing samples to tests

pytest runs every test phase from CallInfo.from_call. flakytest wraps it so each phase runs inside a tiny generated function named after the test and phase, like __flakytest_call_1f3a9c0d2b4e__ (the key is the first 12 characters of the SHA-1 of the test id). The profiler captures that frame in every stack it samples while the phase runs, so each sample belongs to exactly one test and phase. There's no clock to synchronise, and it works the same for tachyon, py-spy and every pytest-xdist worker.

Samples outside any test are classified by the pytest hooks on their stack: startup, collection, session teardown or other. The profiler starts before your conftest.py files are imported, so Django setup and slow imports are measured too.

Exact times, sampled reasons

pytest's durations stay the source of truth: a test that took 1.82s took 1.82s. Samples only decide how each phase's time is split. A stack's share of a phase is its sample count divided by the phase's samples, times the phase's duration. Fixture setup times are measured exactly with a pytest_fixture_setup wrapper. Database queries are counted per test through Django's execute_wrapper and SQLAlchemy's engine events.

Overhead

The profiler runs in a separate process and reads the test process's memory from the outside, so the tests themselves barely slow down. That's also why high rates are cheap with tachyon: if short tests get too few samples, raise the rate (--flakytest-rate 5000) and the report tells you when you should.

pytest-xdist

Each worker is sampled on its own and every test records the worker it ran on. One report covers the whole session, and it suggests a better distribution when workers finish far apart.

Categories

Every sample inside a test gets exactly one category: the first rule in this order that matches any frame of its stack. A phase that got no samples counts as unsampled.

Fix next

Suggestions are ranked by the time they'd save per run. Each one comes with the lines of your code that trigger it, the tests it affects, the fix, how to keep the tests correct while making it, and a link to the relevant docs. Severity is high when a fix saves 10% of test time or a minute per run, medium from 3% or 10 seconds, and low otherwise. flakytest checks for:

Warnings

• undersampled: tests long enough to be sampled got no samples, or tests with fewer than 5 samples add up to 5% of test time. The warning recommends a rate that would give the median of those tests 10 samples. Rerun with --flakytest-rate <rate>.
• short_tests: tests shorter than one sample interval got no samples. Expected, and informational only.
• profiler_unavailable: no profiler could attach. Durations, fixture times and query counts are still exact; only the breakdown inside tests is missing.
• sample_errors: the profiler couldn't read a large share of its samples.

Memory

Every test's memory is measured in the process that runs it, from its setup to its teardown, at well under a millisecond per test. Under pytest-xdist each worker is its own process and runs one test at a time, so each worker measures its own tests. For every test the report shows:

• Needed: its peak resident memory (RSS) above what its process had when it started. This is the number to rank by.
• Kept: how much more memory its process had after it than before.

The report lists the tests that needed the most (the Most memory filter), each worker's peak and growth, and, on Linux, all workers added up over time: what the machine needs so CI doesn't run out of memory, and which tests were running at that moment. A process whose memory keeps growing over the run gets a memory_growth warning naming the tests it grew in. Typical causes are module-level caches and lists, lru_cache, signal receivers, and Django's connection.queries with DEBUG=True.

On flakytest.dev the dashboard lists the tests that need the most memory, with a Most memory tab on the tests page. Each run shows its peak and every worker's peak, test pages show what a test needs in each environment, and the environments table shows each environment's peak.

Limitations

• Exact per-test peaks need Linux. There the kernel's peak counter (VmHWM) is reset before every test through /proc/self/clear_refs. On macOS and Windows the peak can't be reset, so a test's peak is only known when it raised its process's peak; a test that needed 300 MB after an earlier one needed 400 MB shows none. Installing psutil adds every test's memory at its start and end there.
• Only the test processes are measured. Memory used by a database server, Redis, a browser or subprocesses isn't counted, and neither is the pytest-xdist controller process, which mostly waits.
• Workers added up is an upper bound. It assumes each worker was at its test's peak for the whole test. On Linux it adds up private memory (RssAnon), so the interpreter and shared libraries every worker maps count once; elsewhere it adds up RSS, which counts them once per worker.
• "Kept" is noisy, peaks are reliable. Python and the C allocator often hold on to freed memory for reuse, so memory a test frees can still look kept, and later tests reuse it. A worker that keeps growing across many tests is the signal to look for.
• It varies between runs. With xdist, tests land on different workers every run, so a test's starting memory differs. flakytest.dev averages each test's peak over its runs.

Coding agents

The reports are written so an agent can act on them. Ask yours to make the suite faster without changing what it tests, and have it check its work:

$ flakytest report                                  # where time goes, ranked fixes with file:line
$ flakytest report --test tests/test_orders.py      # one test, class or file, with its call tree
$ flakytest run && flakytest compare                # faster? and did any test break or stop running?
$ claude mcp add flakytest -- flakytest mcp         # or give it the local MCP server

flakytest compare flags tests whose outcome changed and tests that stopped running, so a speed-up that deletes, skips or breaks tests doesn't go unnoticed. The local MCP server has list_runs, get_report, get_test, list_slow_tests and compare_runs tools.

In CI

Run pytest --flakytest-profile (or set FLAKYTEST_PROFILE=1) in your test job. On GitHub Actions and other Linux runners the profiler can attach without extra setup. In Docker you may need --cap-add SYS_PTRACE. Each CI job becomes its own environment on flakytest.dev. To keep the report, save .flakytest/runs/ as a build artifact. See configure your CI for adding the token as a secret.

flakytest.dev: connect a project

Sign up, create a project and copy its secret token from the project's settings. With FLAKYTEST_SECRET_TOKEN set, every pytest run does two things:

1. Reports results, profiled or not: every test's outcome and duration, for flaky test tracking and muting. Failure text and captured output are sent only for tests that didn't pass.
2. Uploads the profile when the run is profiled, unless FLAKYTEST_UPLOAD=0.

$ export FLAKYTEST_SECRET_TOKEN=your-secret-token
$ pytest --flakytest-profile

The token identifies the project, so keep it in a CI secret or your shell, not in the repository. A project can have a public slug in its settings, which makes its dashboard readable by anyone with the link.

Organizations

Projects belong to an organization, and everyone in it can see and manage all of its projects. Signing up creates one for you, named after your GitHub login or email; rename it, or create more (e.g. one per company), from the switcher at the top. A project's pages are at flakytest.dev/<organization>/<project>/.

• Members see every project, create projects, mute tests and manage tokens. Owners also invite and remove people, change roles, and rename or delete the organization. An organization always has at least one owner.
• Invitations go out by email from the organization's Members page. The link works once, for 14 days; someone without an account signs up and joins in one go.
• Public projects are readable by anyone with the link, which suits open source projects. Turn it on in the project's settings; settings and tokens stay private.

Environments

Runs are grouped by environment, and every page can be filtered by one. Each CI job and each developer machine is its own environment, with its own trends:

• ci:github-actions:<job>, ci:gitlab:<job>, ci:circleci:<job>, ci:buildkite:<label>, ci:jenkins:<job>, or ci on other CI systems that set CI=true.
• local:<user>@<host> on developer machines.

Set FLAKYTEST_ENV to choose the name yourself, for example to keep runs on differently sized CI runners apart or to group a team's laptops.

Dashboard and pages

• Overview: suite wall time and test time per run for each environment (click a point to open the run), where the time goes per environment, the slowest tests and why, the tests that need the most memory, fix next across the latest run of each environment, CI vs local (the tests that are much slower in CI, and the category that explains it), failing and flaky tests, and every environment that reported. Filter by environment and by the last 24 hours, 7, 30 or 90 days.
• Tests: every test with its average duration, its share of the suite and its top category. A memory tab ranks tests by the memory they need, and a flaky tab lists tests by failure classification. Search by name.
• A test: average, median and max per environment, phases and categories, a duration chart per run (failed runs in red), the latest call tree, hot lines, fixtures and database queries, its reliability, and its history with links to each run.
• Runs and profiles: every session's pass/fail results, and every profiled run with its full breakdown, insights and warnings.

Averages and trends are kept per test, environment and day, so pages stay fast however many runs a project has.

Flaky tests

A failure streak is a run of consecutive failing results of one test (skips are ignored). flakytest.dev classifies every streak by how it ended:

• Flaky: the test both failed and passed on the same commit. Same code, different result: noise from timing, ordering or shared state. Runs from a dirty working tree never count as the same commit, so local edit-and-rerun loops aren't mistaken for flakiness.
• Regression caught: the streak ended with a passing run that didn't change the test's file. Application code was fixed: the test did its job.
• Test changed: the streak ended with a passing run that changed the test's file. The test was edited to pass, either an intended behaviour change or a brittle test.
• Ongoing (still failing) and resolved (fixed on another commit by a client too old to report changed files).

Results are counted per environment, so you can see whether a test fails only in CI or only on someone's laptop.

Muting

Mute a test from its page on flakytest.dev, for a week by default. A muted test still runs and is still reported, but its failure doesn't fail your build. At the start of a session the plugin fetches the muted tests, including on pytest-xdist workers. See mute a test.

MCP server

flakytest.dev serves your project's data to coding agents over MCP, authenticated with the project's token:

$ claude mcp add --transport http flakytest https://flakytest.dev/mcp \
    --header "Authorization: Bearer $FLAKYTEST_SECRET_TOKEN"

Tools: list_environments, get_overview, list_slow_tests (with the reason for each), list_memory_tests, get_test (phases, fixtures, hot lines, memory, flakiness), compare_environments, list_flaky_tests, get_insights and list_runs. Ask things like "why is CI slower than my laptop?" or "which flaky tests should I fix first?".

Uploads

• In parts. A profile is uploaded gzip-compressed, split into parts of up to 2,000 tests, so even very large suites never need one huge request. Each part carries the run's summary and only the frames its tests use.
• Processed in parallel. flakytest.dev processes the parts in parallel. Per-test pages update as parts land; the run counts in averages and trends once all its parts are in. A run that's still missing parts an hour after its last one arrived is marked incomplete. It's listed with the tests that did arrive, but left out of suite trends.
• Safe to retry. Uploads are idempotent: the same part again changes nothing, and a changed part replaces the old one. An upload request never waits for processing.
• Your tests aren't affected. Upload failures are logged and never fail your test run. flakytest upload sends a run again later.

What's uploaded

Nothing is sent without FLAKYTEST_SECRET_TOKEN. With it, flakytest.dev receives:

• Results: test ids, outcomes and durations. Keywords, failure text and captured stdout, stderr and logs are sent only for tests that didn't pass, truncated to 50,000 characters each.
• Session: the Python version, installed package names and versions, a fixed list of CI variables (run ids, branch, job name), the git branch, commit and changed file names, and the environment name, which includes your user and host name unless you set FLAKYTEST_ENV.
• Profiles: profile.json: file paths relative to the project, function names and line numbers, the source lines of the hot spots in your code, the normalized SQL of repeated queries, with values replaced by placeholders, and each test's memory figures.

Set FLAKYTEST_UPLOAD=0 to keep profiles local while still tracking flaky tests. See the privacy policy.

Retention

• Uploaded profile documents: 14 days, except the newest run of each environment, which is kept.
• Per-run results and profile rows: 90 days, the longest window the pages offer.
• Daily per-test statistics behind trends: 400 days.
• The latest breakdown of every test in every environment is always kept.

Profile format

profile.json is a documented, versioned format: run metadata, a summary, interned frames and stacks, and per test its phases, samples, categories, segments, hot lines, fixtures and database stats, plus the insights and warnings. Consumers ignore keys they don't know. See docs/profile-format.md.

Troubleshooting

• Start with flakytest doctor. It shows the Python version, whether tachyon and py-spy are available, which one will be used, and on Linux the ptrace setting.
• No breakdown inside tests. The profiler couldn't attach. Sampling profilers read another process's memory: on Linux flakytest lets its own profiler attach (prctl(PR_SET_PTRACER)), which is enough with the default ptrace_scope=1, including GitHub Actions. With ptrace_scope=2 or in Docker, add --cap-add SYS_PTRACE. On macOS profilers need root: sudo -E flakytest run. On Python 3.11–3.14, install py-spy.
• Short tests have no breakdown. Raise the rate, e.g. --flakytest-rate 5000, as the undersampled warning suggests.
• Nothing shows up on flakytest.dev. Check that FLAKYTEST_SECRET_TOKEN is set where pytest runs (CI secrets often aren't passed to forks), that FLAKYTEST_UPLOAD isn't 0, and the end of pytest's output for upload errors. See troubleshooting.
• Still stuck? Email [email protected] or open an issue on GitHub.