# How to find flaky tests

> Find and reproduce flaky pytest tests: track results across runs, rerun on the same commit, pytest-randomly seeds, pytest-repeat, xdist stress and CI-like CPU limits.

A [flaky test](https://flakytest.dev/guides/what-is-a-flaky-test/) passes and fails on the same code, so one run never tells you whether a test is flaky. Finding flaky tests means getting more results on the same code: by collecting the results you already produce, and by deliberately running tests many times, in different orders and under worse conditions. Reproducing one means recreating the order and conditions of the failing run.

## Track results across runs

Every CI run and every local run produces results, and most of them are thrown away. Keep them, and flaky tests show up on their own: a test that failed and passed on the same commit is flaky by definition.

With a project token in CI (`FLAKYTEST_SECRET_TOKEN`), the flakytest plugin reports every test's outcome on flakytest.dev, profiled or not, together with the commit. Every *failure streak* (consecutive failing results of one test) is classified: **flaky** when the test also passed on the same commit, otherwise as a caught regression, a changed test, still failing or resolved. Runs from a dirty working tree never count as the same commit, so your local edit-and-rerun loop isn't mistaken for flakiness.

What you get from it:

- The tests page has a flaky tab that lists tests by failure classification.
- Each test's page shows its reliability and history per environment (each CI job and each developer machine is its own), so you can see whether it fails only in CI, and links to the failing runs with their failure output.
- Coding agents can ask the flakytest.dev MCP server (`list_flaky_tests`, `get_test`) "which flaky tests should I fix first?".

See [the docs](https://flakytest.dev/docs/#flaky) for how streaks are classified.

## Rerun on the same commit

The cheapest experiment: when a test fails and you suspect flakiness, rerun the job without changing anything. If it passes, you've proved it's flaky (and with result tracking, it's recorded as such). Don't stop there; that's the moment to fix or [mute](https://flakytest.dev/guides/mute-flaky-tests/) it, not to forget about it.

To find flaky tests before they bite, run the whole suite several times on the same commit, for example nightly:

```yaml
on:
  schedule:
    - cron: "17 3 * * *"

jobs:
  tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        attempt: [1, 2, 3, 4, 5]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv run pytest
        env:
          FLAKYTEST_SECRET_TOKEN: ${{ secrets.FLAKYTEST_SECRET_TOKEN }}
```

Five runs of the same commit every night give flaky failures plenty of chances to show up while nobody is waiting on the build.

## Shuffle the order with pytest-randomly

Tests that depend on each other pass in the order you always run them and fail in another. [pytest-randomly](https://github.com/pytest-dev/pytest-randomly) shuffles the order of test files, classes and tests on every run and reseeds `random` (and Faker, factory_boy and model_bakery) before each test. It's active as soon as it's installed:

```console
$ uv add --dev pytest-randomly
$ pytest
Using --randomly-seed=1553614239
...
```

That first line is how you reproduce a failure. Rerun with the same seed to get the same order and the same random values:

```console
$ pytest --randomly-seed=1553614239
$ pytest --randomly-seed=last               # the seed of the previous run
$ pytest -p no:randomly                     # turn it off: the usual order
$ pytest --randomly-dont-reorganize         # keep the order, still reseed random
```

If a test fails with a seed and passes with `-p no:randomly`, it depends on another test. To find which one, bisect: [detect-test-pollution](https://github.com/asottile/detect-test-pollution) does it automatically.

```console
$ detect-test-pollution --failing-test tests/test_prices.py::test_default_currency --tests ./tests
```

[Test isolation](https://flakytest.dev/guides/test-isolation/) covers the fixes.

## Repeat a test many times

Timing problems need many runs of the same test. [pytest-repeat](https://github.com/pytest-dev/pytest-repeat) repeats tests within one pytest session:

```console
$ pytest --count=200 -x tests/test_worker.py::test_processes_job
$ pytest --count=20 --repeat-scope=session tests/test_worker.py
```

`-x` stops at the first failure. With the default `--repeat-scope=function` each test runs `count` times in a row; with `session` the whole selection runs, then runs again, which also catches tests that break the ones after them. pytest-repeat doesn't repeat `unittest.TestCase` tests (including Django's `TestCase`): they run once, with a warning.

[pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder) does the same with a different interface (50 runs by default, with a time limit):

```console
$ pytest --flake-finder --flake-runs=100 -k test_processes_job
$ pytest --flake-finder --flake-max-minutes=10
```

### Check new tests before merging

In the study "An Empirical Analysis of Flaky Tests" (Luo et al., 2014), 78% of flaky tests were flaky from the moment they were written. That makes the pull request that adds a test the cheapest place to catch it. Run the changed test files many times in CI:

```console
$ git diff --name-only --diff-filter=AM origin/main -- tests/ | grep 'test_.*\.py$' | xargs -r pytest --count=50 -x
```

## Stress it: load and parallelism

Many flaky tests only fail when the machine is busy: the thread takes longer to start, the server answers after the sleep is over. Make your machine busy on purpose. Run the suite with more pytest-xdist workers than you have CPUs, and combine it with repeats:

```console
$ pytest -n 16 tests/                                   # on a 4-core laptop
$ pytest -n 8 --count=50 tests/test_worker.py
```

xdist also changes which tests share a process and in what order they run, so it finds order dependencies and tests that share files or ports. A CPU hog in a second terminal (for example `stress-ng --cpu 4`) works too.

## Recreate CI conditions

When a test fails only in CI, the runner differs from your laptop: fewer CPUs, slower disks, a different timezone, maybe running as root. Recreate the difference:

```console
$ taskset -c 0,1 pytest                                       # Linux: only CPUs 0 and 1
$ docker run --rm --cpus=2 -v "$PWD":/app -w /app ci-image pytest   # the CI image, 2 CPUs
$ TZ=Pacific/Kiritimati pytest                                # UTC+14: local and UTC dates differ most of the day
```

Django sets the process timezone from `TIME_ZONE`, so for Django projects change that setting instead of `TZ`. On flakytest.dev, compare the test's results and durations per environment: a test that only fails in CI and is much slower there usually has a timing assumption.

## Reproduce a specific failure

Once you know a test is flaky, rerun it the way it failed:

- **The failure output first.** Read the traceback and captured output of the failing run (flakytest.dev keeps them for every failing result), not just the test name.
- **`pytest --lf`** reruns only the tests that failed last time. If the test passes alone but failed in the full run, it depends on what ran before it, or on timing under load.
- **The same order.** Use the printed seed (`--randomly-seed=...`). With pytest-xdist the seed doesn't fix which worker runs which test, so run with `-n 0` to get every test in one process in the seeded order.
- **The same worker.** With `pytest -n 4 -v`, each line shows the worker (`[gw2] PASSED tests/test_api.py::test_list`). Pass the tests that ran on the failing worker before the failure, in that order, with `-p no:randomly`: pytest runs node ids in the order you give them.
- **The same time.** If it fails around midnight, at month end or on DST changes, freeze time to that moment in the test (see [common causes](https://flakytest.dev/guides/common-causes-of-flaky-tests/)).

When you can make it fail on demand, you can fix it, and check the fix with `--count=200`.

## Further reading

- [pytest-randomly](https://github.com/pytest-dev/pytest-randomly)
- [pytest-repeat](https://github.com/pytest-dev/pytest-repeat)
- [pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder)
- [detect-test-pollution](https://github.com/asottile/detect-test-pollution)
- [pytest: rerunning only failures (`--lf`, `--ff`)](https://docs.pytest.org/en/stable/how-to/cache.html)
- [flakytest docs: flaky tests](https://flakytest.dev/docs/#flaky)