What is a flaky test?
A flaky test passes and fails on the same code. Why flaky tests are expensive (ignored red builds, missed bugs, CI time) and how to detect them across runs.
A flaky test is a test that passes and fails on the same code. Nothing changed in the application or in the test, yet one run is green and the next one is red. Google's testing blog uses the same definition: a flaky result is "a test that exhibits both a passing and a failing result with the same code."
A flaky failure isn't random in the sense of having no cause. Something the test depends on differs between runs: the order the tests ran in, the time of day, how busy the machine was, a random value, a port that was already taken, how fast a real server answered. The test silently assumes that thing is fixed, and most of the time it is.
Flaky, broken, or a caught regression?
A red test means one of three things:
- A regression. The code changed and broke something. The test did its job. It keeps failing until someone fixes the code.
- A broken test. The test itself is wrong, or it depends on something that changed for good (an expired certificate, a date that's now in the past). It fails every time.
- A flaky test. It fails some of the time on code that also makes it pass.
You can't tell these apart from a single run. You need at least two results on the same commit. A test that failed on a commit and passed when the job was rerun on that same commit is flaky. A test that started failing on one commit and keeps failing until the code changes is a regression.
A test can be flaky in one environment only. A test that fails now and then in CI but never on your laptop is still flaky: the CI machine is slower, has fewer CPUs, runs tests in parallel or uses a different timezone, and the test depends on one of those.
Why flaky tests are expensive
A test suite is only useful if a red build means "something is broken". Every flaky failure is a false alarm, and false alarms cost far more than the minutes it takes to rerun the job.
People stop believing red builds
After a few false alarms, the team's reaction to a red build changes from "what broke?" to "rerun it". Google's post describes exactly this: "It is human nature to ignore alarms when there is a history of false signals coming from a system." Once rerunning is the default reaction, nobody reads the failure output anymore.
Real failures get missed
A real regression in a test known to be flaky looks exactly like the usual noise, so it gets rerun until it passes and merged. Google measured that about 84% of the transitions from pass to fail they observed in post-submit testing involved a flaky test, and wrote that "it is quite common to ignore legitimate failures in flaky tests due to the high number of false-positives."
Flaky tests can be real bugs
Sometimes the test is fine and the code is flaky: a race condition between two requests, a cache that isn't invalidated in time, a query that returns rows in an undefined order that the code relies on. In the 2014 study "An Empirical Analysis of Flaky Tests" (Luo et al., 201 commits that fixed flaky tests in 51 Apache projects), 24% of the fixes changed the code under test, and 94% of those fixed a bug in it. Users hit those bugs too, just less often than your CI does.
CI time and developer time
Every rerun repeats the whole job, or at least the failing tests. Every investigation of a flaky failure costs a developer the time to open the log, decide it's "just flaky" and click retry, and it interrupts whatever they were doing. With a large suite this adds up quickly. Google's example: a project with about 1000 tests and a 1.5% rate of flaky results has around 15 failing tests on any given run, each one needing someone to look at it.
How common it is
Google reported in 2016 that about 1.5% of all their test runs reported a flaky result, and that almost 16% of their tests had some level of flakiness. Your numbers will differ, but the shape is typical: most tests are fine, a minority are occasionally flaky, and across a big suite that minority makes a fully green build surprisingly rare.
Where flakiness comes from
Luo et al. found that the top three root causes in the projects they studied were async waits (not waiting properly for asynchronous work to finish), concurrency and test order dependency. In Python suites the usual suspects are:
- tests that depend on the order they run in, through shared state, caches or database rows
- time:
datetime.now(), midnight, month ends, daylight saving changes and timezones - unseeded randomness and generated data that sometimes collides
time.sleep()used to wait for threads, tasks or servers- results in undefined order: SQL without
ORDER BY, iteration over sets - tests running in parallel with pytest-xdist and sharing files or ports
- real network calls and services that aren't ready yet
Each of these, with an example and the fix, is in common causes of flaky tests.
The same study found that 78% of flaky tests were flaky from the moment they were written. That makes the best time to catch them the pull request that adds them: how to find flaky tests shows how to hammer a new test before it's merged.
How flakytest.dev detects flaky tests
With a project token set in CI (FLAKYTEST_SECRET_TOKEN), the flakytest plugin reports every test's outcome on every run, whether or not the run is profiled, along with the commit and whether the working tree was dirty. flakytest.dev then looks at each test's history.
A failure streak is a run of consecutive failing results of one test (skips are ignored). Each streak is classified by how it ended:
| Classification | Meaning |
|---|---|
| Flaky | The test both failed and passed on the same commit. |
| Regression caught | The streak ended with a passing run that didn't change the test's file: application code was fixed. |
| Test changed | The streak ended with a passing run that changed the test's file: the test was edited to pass. |
| Ongoing | The test is still failing. |
| Resolved | Fixed on another commit by a client too old to report changed files. |
Runs from a dirty working tree never count as the same commit. When you edit code locally and rerun the tests, a fail followed by a pass is you fixing something, not flakiness, and it isn't counted as flaky.
Results are counted per environment (each CI job and each developer machine is its own environment), so a test's page shows whether it fails only in CI or only on someone's laptop. The tests page has a flaky tab that lists tests by failure classification, and coding agents can get the same list through the flakytest.dev MCP server's list_flaky_tests tool. See the docs for details.
What to do with a flaky test
- Fix it as soon as you can. The failure output is fresh, the change that introduced the test is recent and the person who wrote it remembers why it works the way it does. Common causes covers most fixes and finding flaky tests covers reproducing one.
- If you can't fix it today, mute it. A muted test keeps running and reporting, but its failure no longer fails the build. You keep collecting evidence without blocking everyone else.
- Don't paper over it with retries. Rerunning failed tests makes CI slower and teaches everyone to ignore the test, and it doesn't make the build as reliable as the arithmetic suggests. See why retrying flaky tests isn't a fix.
Further reading
- Flaky Tests at Google and How We Mitigate Them (Google Testing Blog, 2016)
- An Empirical Analysis of Flaky Tests (Luo, Hariri, Eloussi, Marinov, FSE 2014)
- flakytest docs: flaky tests
Related guides
- Common causes of flaky tests and how to fix each: Why pytest tests pass and fail on the same code: test order, time zones, randomness, sleeps, unordered results, xdist and slow CI. The symptom and fix for each.
- How to find flaky tests: Find and reproduce flaky pytest tests: track results across runs, rerun on the same commit, pytest-randomly seeds, pytest-repeat, xdist stress and CI-like CPU limits.
- Retrying flaky tests isn't a fix: pytest-rerunfailures --reruns and CI job retries hide flaky tests instead of fixing them. Why retries help less than the math suggests, cost CI time and bury real bugs.
- Mute flaky tests instead of skipping or retrying them: Mute a flaky pytest test so it can't fail the build while it keeps running and reporting. How muting compares to pytest.mark.skip, xfail, quarantine jobs and retries.
Updated October 18, 2026 · Suggest a change