Flaky tests

How to find flaky tests

Find and reproduce flaky pytest tests: track results across runs, rerun on the same commit, pytest-randomly seeds, pytest-repeat, xdist stress and CI-like CPU limits.

A flaky test passes and fails on the same code, so one run never tells you whether a test is flaky. Finding flaky tests means getting more results on the same code: by collecting the results you already produce, and by deliberately running tests many times, in different orders and under worse conditions. Reproducing one means recreating the order and conditions of the failing run.

Track results across runs

Every CI run and every local run produces results, and most of them are thrown away. Keep them, and flaky tests show up on their own: a test that failed and passed on the same commit is flaky by definition.

With a project token in CI (FLAKYTEST_SECRET_TOKEN), the flakytest plugin reports every test's outcome on flakytest.dev, profiled or not, together with the commit. Every failure streak (consecutive failing results of one test) is classified: flaky when the test also passed on the same commit, otherwise as a caught regression, a changed test, still failing or resolved. Runs from a dirty working tree never count as the same commit, so your local edit-and-rerun loop isn't mistaken for flakiness.

What you get from it:

• The tests page has a flaky tab that lists tests by failure classification.
• Each test's page shows its reliability and history per environment (each CI job and each developer machine is its own), so you can see whether it fails only in CI, and links to the failing runs with their failure output.
• Coding agents can ask the flakytest.dev MCP server (list_flaky_tests, get_test) "which flaky tests should I fix first?".

See the docs for how streaks are classified.

Rerun on the same commit

The cheapest experiment: when a test fails and you suspect flakiness, rerun the job without changing anything. If it passes, you've proved it's flaky (and with result tracking, it's recorded as such). Don't stop there; that's the moment to fix or mute it, not to forget about it.

To find flaky tests before they bite, run the whole suite several times on the same commit, for example nightly:

on:
  schedule:
    - cron: "17 3 * * *"

jobs:
  tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        attempt: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv run pytest
        env:
          FLAKYTEST_SECRET_TOKEN: ${{ secrets.FLAKYTEST_SECRET_TOKEN }}

Five runs of the same commit every night give flaky failures plenty of chances to show up while nobody is waiting on the build.

Shuffle the order with pytest-randomly

Tests that depend on each other pass in the order you always run them and fail in another. pytest-randomly shuffles the order of test files, classes and tests on every run and reseeds random (and Faker, factory_boy and model_bakery) before each test. It's active as soon as it's installed:

$ uv add --dev pytest-randomly
$ pytest
Using --randomly-seed=1553614239
...

That first line is how you reproduce a failure. Rerun with the same seed to get the same order and the same random values:

$ pytest --randomly-seed=1553614239
$ pytest --randomly-seed=last               # the seed of the previous run
$ pytest -p no:randomly                     # turn it off: the usual order
$ pytest --randomly-dont-reorganize         # keep the order, still reseed random

If a test fails with a seed and passes with -p no:randomly, it depends on another test. To find which one, bisect: detect-test-pollution does it automatically.

$ detect-test-pollution --failing-test tests/test_prices.py::test_default_currency --tests ./tests

Test isolation covers the fixes.

Repeat a test many times

Timing problems need many runs of the same test. pytest-repeat repeats tests within one pytest session:

$ pytest --count=200 -x tests/test_worker.py::test_processes_job
$ pytest --count=20 --repeat-scope=session tests/test_worker.py

-x stops at the first failure. With the default --repeat-scope=function each test runs count times in a row; with session the whole selection runs, then runs again, which also catches tests that break the ones after them. pytest-repeat doesn't repeat unittest.TestCase tests (including Django's TestCase): they run once, with a warning.

pytest-flakefinder does the same with a different interface (50 runs by default, with a time limit):

$ pytest --flake-finder --flake-runs=100 -k test_processes_job
$ pytest --flake-finder --flake-max-minutes=10

Check new tests before merging

In the study "An Empirical Analysis of Flaky Tests" (Luo et al., 2014), 78% of flaky tests were flaky from the moment they were written. That makes the pull request that adds a test the cheapest place to catch it. Run the changed test files many times in CI:

$ git diff --name-only --diff-filter=AM origin/main -- tests/ | grep 'test_.*\.py$' | xargs -r pytest --count=50 -x

Stress it: load and parallelism

Many flaky tests only fail when the machine is busy: the thread takes longer to start, the server answers after the sleep is over. Make your machine busy on purpose. Run the suite with more pytest-xdist workers than you have CPUs, and combine it with repeats:

$ pytest -n 16 tests/                                   # on a 4-core laptop
$ pytest -n 8 --count=50 tests/test_worker.py

xdist also changes which tests share a process and in what order they run, so it finds order dependencies and tests that share files or ports. A CPU hog in a second terminal (for example stress-ng --cpu 4) works too.

Recreate CI conditions

When a test fails only in CI, the runner differs from your laptop: fewer CPUs, slower disks, a different timezone, maybe running as root. Recreate the difference:

$ taskset -c 0,1 pytest                                       # Linux: only CPUs 0 and 1
$ docker run --rm --cpus=2 -v "$PWD":/app -w /app ci-image pytest   # the CI image, 2 CPUs
$ TZ=Pacific/Kiritimati pytest                                # UTC+14: local and UTC dates differ most of the day

Django sets the process timezone from TIME_ZONE, so for Django projects change that setting instead of TZ. On flakytest.dev, compare the test's results and durations per environment: a test that only fails in CI and is much slower there usually has a timing assumption.

Reproduce a specific failure

Once you know a test is flaky, rerun it the way it failed:

• The failure output first. Read the traceback and captured output of the failing run (flakytest.dev keeps them for every failing result), not just the test name.
• pytest --lf reruns only the tests that failed last time. If the test passes alone but failed in the full run, it depends on what ran before it, or on timing under load.
• The same order. Use the printed seed (--randomly-seed=...). With pytest-xdist the seed doesn't fix which worker runs which test, so run with -n 0 to get every test in one process in the seeded order.
• The same worker. With pytest -n 4 -v, each line shows the worker ([gw2] PASSED tests/test_api.py::test_list). Pass the tests that ran on the failing worker before the failure, in that order, with -p no:randomly: pytest runs node ids in the order you give them.
• The same time. If it fails around midnight, at month end or on DST changes, freeze time to that moment in the test (see common causes).

When you can make it fail on demand, you can fix it, and check the fix with --count=200.

Further reading

• pytest-randomly
• pytest-repeat
• pytest-flakefinder
• detect-test-pollution
• pytest: rerunning only failures (--lf, --ff)
• flakytest docs: flaky tests

Related guides

• What is a flaky test?: A flaky test passes and fails on the same code. Why flaky tests are expensive (ignored red builds, missed bugs, CI time) and how to detect them across runs.
• Common causes of flaky tests and how to fix each: Why pytest tests pass and fail on the same code: test order, time zones, randomness, sleeps, unordered results, xdist and slow CI. The symptom and fix for each.
• Keep tests independent of each other: Tests should pass alone, in any order and in parallel. Find order-dependent pytest tests with pytest-randomly and pytest-xdist, and fix the shared state behind them.
• Mute flaky tests instead of skipping or retrying them: Mute a flaky pytest test so it can't fail the build while it keeps running and reporting. How muting compares to pytest.mark.skip, xfail, quarantine jobs and retries.

Updated October 18, 2026 · Suggest a change