Common causes of flaky tests and how to fix each

Why pytest tests pass and fail on the same code: test order, time zones, randomness, sleeps, unordered results, xdist and slow CI. The symptom and fix for each.

A flaky test depends on something that changes between runs without the test knowing. The list of such things is short, and each has a recognisable symptom. Match the symptom, then apply the fix. To reproduce the failure first, see how to find flaky tests.

Test order and shared state

Symptom: the test passes on its own (pytest tests/test_prices.py::test_default_currency) and fails in the full run, or only with pytest-randomly or pytest-xdist.

An earlier test left something behind: a module-level variable, a patched setting, a singleton, a file, a database row committed outside the test's transaction.

from myapp import pricing


def test_us_customer():
    pricing.set_currency("USD")  # global, never reset
    ...


def test_default_currency():
    assert pricing.get_currency() == "EUR"  # fails if test_us_customer ran first

Fix: never change global state without undoing it. monkeypatch undoes its changes after the test:

def test_us_customer(monkeypatch):
    monkeypatch.setattr(pricing, "_currency", "USD")
    ...

Use Django's override_settings instead of assigning to settings, TestCase instead of committing data, and yield fixtures that clean up after themselves. To find the test that pollutes, see test isolation.

Time, dates and timezones

Symptom: fails at certain times: around midnight, on the last day of the month, on the day daylight saving time changes, or only on machines in a different timezone.

The classic is mixing UTC and local dates. date.today() is the local date; datetime.now(UTC).date() and Django's timezone.now().date() are the UTC date. For several hours every day they differ.

import datetime as dt


def test_report_date():
    report = build_report()  # uses datetime.now(dt.UTC).date()
    assert report.date == dt.date.today()  # local date: wrong for hours every day

Other traps: date(today.year, today.month + 1, 1) breaks in December, "a day" isn't 24 hours on a DST change, and a fixture created "now" compared with "now" a few milliseconds later.

Fix: freeze or travel time to a fixed moment, and pick the edge cases on purpose:

import datetime as dt

import time_machine


@time_machine.travel(dt.datetime(2026, 3, 29, 0, 30, tzinfo=dt.UTC), tick=False)
def test_report_date_just_after_midnight_utc():
    assert build_report().date == dt.date(2026, 3, 29)

In Django code, use timezone.now() and timezone.localdate() consistently. See replacing freezegun with time-machine for the faster option.

Randomness and generated data

Symptom: fails rarely, with data you didn't choose: an empty string, a zero, a duplicate email.

import random


def test_discount():
    amount = random.randint(0, 100)
    assert apply_discount(amount) < amount  # fails when amount is 0

Factories have the same problem. Faker and model_bakery generate random values, and a unique field occasionally gets the same value twice.

Fix: use explicit values for anything the assertion depends on. For unique fields use sequences or Faker's unique proxy:

import factory
from faker import Faker

fake = Faker()
email = fake.unique.email()  # never repeats within this Faker instance


class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = "auth.User"

    email = factory.Sequence(lambda n: f"user{n}@example.com")

Install pytest-randomly: it reseeds random, Faker, factory_boy and model_bakery before every test and prints the seed, so a failure caused by a random value can be replayed with --randomly-seed=<seed>.

Fixed sleeps

Symptom: fails only on slow machines or under load, usually in CI.

def test_cache_expires():
    cache.set("token", "abc", timeout=1)
    time.sleep(1.1)
    assert cache.get("token") is None

sleep() is a guess about how long something takes. On a loaded CI runner the guess is wrong.

Fix: for timeouts and expiry, move time instead of waiting for it (time-machine, or inject a clock). For background work, wait for the condition (next section). See sleeping in tests.

Async code and threads

Symptom: the assertion sometimes runs before the work it checks is done. In the study by Luo et al., not waiting properly for asynchronous work was the most common root cause (45% of the fixes they categorised).

Fix: wait on the thing itself, with a generous timeout that only matters when the test fails:

import threading
import time


def test_worker_processes_job():
    done = threading.Event()
    worker.submit(job, on_done=done.set)
    assert done.wait(timeout=10)


def wait_for(condition, timeout=10.0):
    deadline = time.monotonic() + timeout
    while not condition():
        if time.monotonic() > deadline:
            raise AssertionError("condition not met in time")
        time.sleep(0.01)

Use thread.join(timeout), queue.join(), or await asyncio.wait_for(task, timeout=10) where you can. If the application itself races (two requests updating the same row), the test found a real bug: fix the code, not the test. More in waiting in tests.

Results in undefined order

Symptom: the same items come back in a different order; the assertion compares lists.

SQL without ORDER BY returns rows in whatever order the database likes, which changes with table size, plans and parallel workers. In Python, sets of strings iterate in a different order in every process, because string hashing is randomized (PYTHONHASHSEED).

class ActiveUsersTest(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.alice = baker.make(User, username="alice")
        cls.bob = baker.make(User, username="bob")

    def test_active_users(self):
        users = User.objects.filter(is_active=True)
        self.assertEqual(list(users), [self.alice, self.bob])  # order not guaranteed

Fix: order explicitly (.order_by("username")) when order matters, or compare without order when it doesn't: self.assertQuerySetEqual(users, [self.alice, self.bob], ordered=False), sorted(...), or compare sets. To reproduce a set-order failure, run with a fixed PYTHONHASHSEED=0, but fix it with sorted(). The same applies to @pytest.mark.parametrize over a set: pytest-xdist workers can collect the tests in a different order, which xdist reports as an error. Pass sorted(...) or a list.

Parallel tests with pytest-xdist

Symptom: passes with -n 0, fails with -n auto; errors like "address already in use" or a file with unexpected contents.

Workers are separate processes running at the same time. Anything outside the process is shared: fixed file paths, ports, a Redis database, an S3 bucket.

Fix: give every test or worker its own resource. Use tmp_path instead of fixed paths, bind servers to port 0 and read the assigned port, and namespace external resources with the worker_id fixture (gw0, gw1, or master without xdist). pytest-django already creates one test database per worker. See pytest-xdist.

Real network calls and services that aren't ready

Symptom: timeouts, connection errors, rate limits, or different responses. In CI, connection refused errors at the very start of the job.

Fix: mock external APIs and make unmocked calls fail loudly (see blocking network in tests). For services your tests do need, such as a database container, wait until they're healthy before the tests start:

services:
  postgres:
    image: postgres:17
    env:
      POSTGRES_PASSWORD: postgres
    options: >-
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5

With docker compose, use a healthcheck and depends_on with condition: service_healthy.

Resource leaks

Symptom: tests late in the run fail with "Too many open files", connection pool exhaustion, or interference from a thread a previous test started.

Fix: close what you open (context managers, yield fixtures with teardown), stop threads and servers the test started, and close database connections and clients. Run python -X dev -m pytest to see ResourceWarnings for files and sockets that were never closed.

Caches that outlive a test

Symptom: a test sees a value cached by an earlier test. Like shared state, but harder to spot.

Django's local memory cache persists across tests in the same process, and so does anything decorated with functools.cache or lru_cache, including functions that read settings once.

Fix: clear caches between tests:

import pytest
from django.core.cache import cache

from myapp.config import get_settings


@pytest.fixture(autouse=True)
def clear_caches():
    yield
    cache.clear()
    get_settings.cache_clear()

Floating point comparisons

Symptom: assert 0.30000000000000004 == 0.3, sometimes. Floating point math is deterministic, but the order of operations isn't always: summing over a set, a dict built in a different order or a multithreaded numeric library changes the last digits.

Fix: compare with a tolerance, assert total == pytest.approx(0.3), and use Decimal for money.

Differences in CI

Symptom: never fails locally, fails in CI now and then.

CI runners usually have fewer CPUs, slower disks and noisy neighbours, so every timing assumption gets tighter. They may also run as root (a test expecting PermissionError passes for you and fails there), use UTC, a different locale, or no terminal.

Fix: reproduce the conditions (pin the suite to fewer CPUs, run it in the CI image; see how to find flaky tests) and then fix the underlying cause from this list. On flakytest.dev, results are counted per environment, so a test's page shows whether it fails only in CI.

Related guides

What is a flaky test?: A flaky test passes and fails on the same code. Why flaky tests are expensive (ignored red builds, missed bugs, CI time) and how to detect them across runs.
How to find flaky tests: Find and reproduce flaky pytest tests: track results across runs, rerun on the same commit, pytest-randomly seeds, pytest-repeat, xdist stress and CI-like CPU limits.
Keep tests independent of each other: Tests should pass alone, in any order and in parallel. Find order-dependent pytest tests with pytest-randomly and pytest-xdist, and fix the shared state behind them.
Tests are sleeping: replace time.sleep: Fixed time.sleep and asyncio.sleep calls make pytest suites slow and flaky. Poll for the condition, patch the sleep in retry and backoff code, and move the clock with time-machine.

Updated October 18, 2026 · Suggest a change