flakytest.dev
Sign In Get started

Documentation

Get Started with flakytest

flakytest is a pytest plugin that profiles your test suite and explains, test by test, where the time goes. It works locally without an account; the optional cloud at flakytest.dev compares runs across CI and developer machines and tracks flaky tests. The documentation explains everything in detail.

TL;DR

$ pip install flakytest
$ pytest --flakytest-profile          # profile the run, print the summary
$ flakytest report                    # open the report of the last run
$ flakytest report --format markdown  # the same, for your coding agent

1. Profile your suite locally

Install the plugin into the environment your tests run in (pip install flakytest or uv add --dev flakytest) and run pytest with --flakytest-profile. flakytest run does the same with sensible defaults.

A sampling profiler watches the run: tachyon on Python 3.15+, py-spy on older versions. Each sample is attributed to the test and phase (setup, call, teardown) it was taken in, so every test gets:

  • exact setup, call and teardown times, and the exact setup time of every fixture it ran,
  • its time by category: database, network, password hashing, migrations, factories, sleeps, templates, …
  • the hot lines in your own code, and the biggest pieces of work (fixtures, setUp, the test body),
  • database query counts and repeated queries (N+1) for Django and SQLAlchemy,

and the run gets a ranked fix next list: the changes that save the most time, with the exact setting or line to change.

Very short tests can be missed by a 1000 Hz sampler. If the report warns that tests are undersampled, rerun with the recommended rate, e.g. --flakytest-rate 5000.

2. Send runs to flakytest.dev (optional)

Sign up, then create a project and copy its secret token from the project's settings. With the token set, every run uploads its profile and its pass/fail results: 

$ export FLAKYTEST_SECRET_TOKEN=your-secret-token
$ pytest --flakytest-profile

Runs are grouped by environment: every CI job (ci:github-actions:test) and every developer machine (local:anze@mbp) gets its own trends, so you can see why CI is slower than your laptop. Set FLAKYTEST_ENV to name an environment yourself, and FLAKYTEST_UPLOAD=0 to keep a run local.

Plain pytest (without --flakytest-profile) still reports pass/fail, so flaky tests are tracked and muted tests are honoured on every run.

3. Add it to CI

Add the token as a CI secret and run the profiled suite. See Configure your CI for GitHub Actions and other providers.

4. Connect your coding agent

Locally, flakytest report --format markdown prints the whole report in a form an agent can act on. For the cloud data, add the MCP server with your project's token: 

$ claude mcp add --transport http flakytest https://flakytest.dev/mcp \
    --header "Authorization: Bearer $FLAKYTEST_SECRET_TOKEN"

Any client that speaks MCP over Streamable HTTP works the same way: point it at https://flakytest.dev/mcp with an Authorization: Bearer header. The tools are get_overview, list_slow_tests, get_test, get_insights, compare_environments, list_flaky_tests, list_environments and list_runs.

How failures are classified

  • Flaky: the test failed and passed on the same commit. Runs from a dirty working tree don't count as the same commit.
  • Regression caught: the failures ended with a passing run that didn't change the test's file, so the code was fixed.
  • Test changed: the failures ended with a run that changed the test's file.
  • Still failing: no passing run yet.

Next steps