# Retrying flaky tests isn't a fix > pytest-rerunfailures --reruns and CI job retries hide flaky tests instead of fixing them. Why retries help less than the math suggests, cost CI time and bury real bugs. The most common reaction to a [flaky test](https://flakytest.dev/guides/what-is-a-flaky-test/) is to retry it: `pytest --reruns 3`, `@pytest.mark.flaky(reruns=3)`, `retry: 2` in a GitLab job, or clicking "Re-run failed jobs" in GitHub Actions. The build turns green and everyone moves on. That's the problem. A retry doesn't change why the test fails. It makes CI slower and less predictable, it hides the failure instead of fixing it, and it doesn't even make the build reliable: a test that fails once can fail three times in a row. Meanwhile the test keeps getting flakier, and nobody notices, because the build is green. ## What retries look like [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) reruns a failed test right away, in the same process: ```console $ pytest --reruns 3 # every failing test, up to 3 more times $ pytest --reruns 3 --reruns-delay 2 # wait 2 seconds before each rerun $ pytest --reruns 3 --only-rerun ConnectionError ``` ```python import pytest @pytest.mark.flaky(reruns=3) def test_checkout_sends_receipt(): ... ``` A CI job retry is the coarser version: the whole job runs again, usually on a fresh machine. ## The arithmetic retries rely on Take a test that fails 5% of the time, and give it three attempts (the first run plus `--reruns 2`). If every attempt were an independent coin flip, all three would fail with probability: ```text 0.05 × 0.05 × 0.05 = 0.000125 = 0.0125% ``` That looks like a great deal. With 20 such tests in a suite, the chance that at least one of them fails a build goes from 64% (`1 - 0.95^20`) to about 0.25% (`1 - (1 - 0.000125)^20`). This is the whole case for retries, and it rests on one assumption: that the attempts are independent. For flaky tests they usually aren't. ## Why reality is worse: failures cluster A flaky test fails because of a condition, and retries run under the same condition. The condition rarely goes away in the two seconds between attempts. - **A slow or busy machine.** A test with a timing assumption fails when the CI runner is overloaded: a noisy neighbour, a parallel job, a slow disk. The rerun happens on the same overloaded machine, a few seconds later. - **Test order.** A test that fails because an earlier test left state behind (a module-level cache, a patched setting, a row committed outside a transaction) is rerun by pytest-rerunfailures in the same process, right after it failed. Its fixtures are set up again, but the global state that broke it is still there, so every attempt fails the same way. - **Time of day.** A test that breaks between midnight UTC and midnight local time, on the last day of the month or on a daylight saving change fails on every retry during that window. - **A shared dependency.** A real external service that is down or rate limiting you is still down on the next attempt. Here's an illustration (made-up numbers, not a measurement). Suppose 10% of builds land on an overloaded runner, and there the test fails half the time; everywhere else it never fails. Its overall failure rate is still 5%. But on the overloaded runner, three attempts all fail with probability `0.5^3 = 12.5%`, so the build fails `0.1 × 12.5% = 1.25%` of the time. That's 100 times the 0.0125% the naive calculation promised. For an order-dependent failure, retries in the same process don't help at all: the build fails 5% of the time, with or without `--reruns 2`. So retries buy less than they appear to, and they buy it with a lot of CI time. ## The costs nobody sees ### Every retry adds the test's duration to the build A rerun costs the test's full duration again, plus its fixtures. Flaky tests are often the slow ones: tests that wait for a thread, a browser or a server, and fail when the wait runs out. With `--reruns 3`, a test that waits up to 30 seconds before failing can add 90 seconds to the build. With `--reruns-delay` the delay comes on top. A job-level retry is worse: it reruns everything, so one flaky test costs a whole extra job. Builds that sometimes take 8 minutes and sometimes 14, depending on how many retries happened, are also harder to reason about. "CI got slower" becomes a question with no clear answer. ### The flakiness grows unnoticed With three attempts, a test failing 5% of the time almost never turns a build red. Neither does one failing 20% of the time (0.8% of builds). By the time it fails 40% of the time it fails 6.4% of builds, and 40% of builds pay for at least one extra run of it. Each step looks like the same "occasionally flaky test". The retries hide the trend, and the test goes from easy to fix (one recent change) to hard (months of changes on top). Google's testing blog described the same effect with their own "fail only if it fails 3 times in a row" mechanism: it "reduces false positives, but encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row, which is hardly a perfect solution." Their example: a 15 minute integration test marked flaky and actually broken by a change takes 45 minutes to report the breakage. ### Retries train people to ignore the test Once a test is "the one that needs a retry", its failures stop being read. When it fails for a real reason, that failure gets rerun too, and if the bug is intermittent the rerun may pass. ### A flaky test can be a real bug Sometimes the test is right and the application is flaky: two requests racing to update the same row, a cache invalidated after the response is sent, code that relies on rows coming back in insertion order. In the study "An Empirical Analysis of Flaky Tests" (Luo et al., 2014), 24% of the commits that fixed a flaky test changed the code under test, and 94% of those fixed a bug in it. A retry turns a reproducible signal of a race condition into a green checkmark. Your users get no retries. ## What to do instead ### Fix the test while the failure is fresh The cheapest moment to fix a flaky test is right after its first flaky failure. The failure output is in front of you, the change that introduced it is recent and the person who wrote it remembers how it works. Most causes have a known fix: see [common causes of flaky tests](https://flakytest.dev/guides/common-causes-of-flaky-tests/). To reproduce one, see [how to find flaky tests](https://flakytest.dev/guides/find-flaky-tests/). ### Mute it until it's fixed If you can't fix it today, [mute it on flakytest.dev](https://flakytest.dev/guides/mute-flaky-tests/). A muted test still runs and is still reported, with its failure output, but its failure doesn't fail the build. Compared to a retry: - it costs no extra CI time: the test runs once - its failures are recorded instead of hidden, so you keep collecting evidence and can see whether it's getting better or worse - it needs no code change: no marker, no pull request - it's temporary: a mute lasts a week by default, so it comes back to the team's attention ### If you're already retrying If you have to keep retries for a while (say, end-to-end tests against a third-party sandbox you don't control): - Retry only the failures you expect, with `--only-rerun` (for example `--only-rerun ConnectionError`), never every assertion. - Keep it to one rerun. - Make reruns visible. pytest-rerunfailures marks rerun tests with `R` and counts them in the summary, and its `--fail-on-flaky` option makes the run exit with a custom exit code when a test only passed on a rerun, so a CI step can flag it. - Track which tests needed a retry, and treat each one as a bug to fix, not a setting to tune. ## Further reading - [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) - [Flaky Tests at Google and How We Mitigate Them](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) (Google Testing Blog, 2016) - [An Empirical Analysis of Flaky Tests](https://doi.org/10.1145/2635868.2635920) (Luo et al., FSE 2014) - [flakytest docs: flaky tests and muting](https://flakytest.dev/docs/#flaky)