# Mute flaky tests instead of skipping or retrying them

> Mute a flaky pytest test so it can't fail the build while it keeps running and reporting. How muting compares to pytest.mark.skip, xfail, quarantine jobs and retries.

The best thing to do with a flaky test is to fix it, as soon as possible, while the failure is fresh: the traceback is in front of you, the change that introduced it is recent and its author remembers how it works. [Common causes of flaky tests](https://flakytest.dev/guides/common-causes-of-flaky-tests/) covers most fixes.

But you can't always fix it today, and leaving it alone isn't an option either. Muting is for the time in between: the test can no longer fail the build, but it keeps running and reporting, so you keep collecting the evidence you need to fix it.

## Why not leave it failing?

Not every test is business critical. When a flaky test in a corner of the admin fails, the whole team shouldn't have to drop everything to fix it right now.

But you can't keep failing tests in CI either. A build that's red because of a known flaky test is a build nobody reads. People start rerunning and merging on red, and the next real failure goes unnoticed in the noise. See [what is a flaky test](https://flakytest.dev/guides/what-is-a-flaky-test/) for what that costs.

So the test has to stop failing the build. The question is how.

## Skipping throws away the evidence

Test frameworks let you skip a test:

```python
import pytest


@pytest.mark.skip(reason="flaky, see #482")
def test_checkout_sends_receipt(): ...
```

This has two problems.

**It needs a code change.** Someone writes the commit, opens a pull request, waits for review and CI, and merges, all to stop a test that is blocking everyone else's pull requests in the meantime. Unskipping it later needs another one.

**The test stops running.** From now on you learn nothing about it. You don't know whether it still fails, how often, in which environment, or whether last week's refactor fixed it by accident. Skipped tests rarely get fixed and re-enabled unless someone tracks them and goes back. When someone finally does, weeks later, nobody remembers why it was skipped, and the logs of the original failure are outdated or gone.

## What muting does

On flakytest.dev you mute a test from its page. A muted test:

- **keeps running** in every build, in every environment that reports to the project
- **keeps reporting**: every result is recorded, and for failures the failure text and captured output too, so its history and flakiness classification keep growing
- **can't fail the build**: when it fails, pytest counts it as muted, not failed
- **needs no code change**: no marker, no pull request, no review. Unmuting is one click too
- **expires**: a mute lasts a week by default, then the test counts again

That's the difference from a skip: when you come back to the test, you have a week or more of fresh failures to look at, not one stale traceback.

## How to mute a test

1. Make sure CI reports to flakytest.dev: set `FLAKYTEST_SECRET_TOKEN` in the test job (see [configure your CI](https://flakytest.dev/ci/)). Muting works for every pytest run with the token, profiled or not.
2. Open the test's page on flakytest.dev (from the flaky tab on the tests page, or from a failing run) and click **Mute**.
3. Pick how long: 1 day, 1 week (the default), 2 weeks or 4 weeks. Write a reason: what fails, a link to the issue, who is on it.
4. Unmute from the same page as soon as the fix is merged, or let the mute expire.

At the start of every pytest session the plugin fetches the project's muted tests, including on pytest-xdist workers. In the progress output a muted test shows as `M` when it failed and `m` when it passed (`MUTE` with `-v`), and the exit code ignores its failure.

A few details worth knowing:

- Muting applies to the test's call phase. If one of its fixtures errors during setup or teardown, that error still fails the build.
- If the plugin can't reach flakytest.dev, or the token isn't set, nothing is muted and the test fails the build as usual. A network problem never hides a failure.
- Muted failures still count as failures in the test's history and in flaky detection, so a muted test that keeps failing on commits where it also passes stays classified as flaky.

## Muting compared to the alternatives

| | Test still runs | Can fail the build | Keeps evidence | Code change | Ends on its own |
| --- | --- | --- | --- | --- | --- |
| `@pytest.mark.skip` | No | No | No | Yes | No |
| `@pytest.mark.xfail` (`strict=False`) | Yes | No, neither XFAIL nor XPASS fails | Only in that run's summary | Yes | No |
| `@pytest.mark.xfail(strict=True)` | Yes | Yes, every time it passes | Only in that run's summary | Yes | No |
| Quarantine job | Yes, in a separate job | No | In that job's logs | Yes (marker and CI config) | No |
| Retries (`--reruns`) | Yes, several times | Only if every attempt fails | Hidden by the passing attempt | No (flag) or yes (marker) | No |
| Mute on flakytest.dev | Yes | No | Every result and failure, per environment | No | Yes, after a week by default |

### xfail

`@pytest.mark.xfail` with the default `strict=False` hides both outcomes: a failure is reported as XFAIL and a pass as XPASS, and neither fails the suite. For a flaky test that's a skip that wastes CI time: it runs, but nobody sees the result.

`xfail(strict=True)` makes XPASS fail the suite. That's right for a test of a known bug that fails every time (you'll notice when the bug is fixed), and wrong for a flaky test: it passes most of the time, so strict xfail turns it into a test that fails the build most of the time.

### Quarantine jobs

A quarantine moves flaky tests to a separate CI job that's allowed to fail:

```python
@pytest.mark.quarantine
def test_checkout_sends_receipt(): ...
```

```yaml
- run: pytest -m "not quarantine"
- run: pytest -m quarantine
  continue-on-error: true
```

The tests keep running, which is better than a skip. But moving a test in or out still needs a code change and a review, the marker has to be registered in your pytest config, nothing makes a test leave the quarantine, and the results of a job that's allowed to fail are rarely read. Google's testing blog noted about their own automatic quarantine that it "could easily mask a real race condition or some other bug in the code being tested."

### Retries

Retries keep the test in the build and make it pass by running it again. They add the test's duration to the build every time they kick in, hide the failure behind the passing attempt and don't stop the test from getting flakier. Because flaky failures cluster (the same slow runner, the same test order, the same time of day), they also make the build less reliable than the math suggests. See [retrying flaky tests isn't a fix](https://flakytest.dev/guides/retrying-flaky-tests/).

## A mute is for days, not forever

A mute is a promise to fix the test soon, not a place to store it. Keep it that way:

- **Keep it short.** A week is the default for a reason. If the fix takes longer, extend the mute on purpose, with an updated reason, rather than picking the longest option up front.
- **Give it an owner.** Write who is fixing it in the reason, and open an issue with a link to the test's page on flakytest.dev.
- **Use the evidence.** While the test is muted, its page collects every failure with its output, per environment. That's often enough to tell a timing problem (fails only in CI, only under load) from an order dependency (fails with some test orders, never alone). [How to find flaky tests](https://flakytest.dev/guides/find-flaky-tests/) covers reproducing it.
- **Unmute after the fix**, and watch the next few runs. If the test stops failing on commits where it also passes, the fix worked.

If a test has been muted for a month and nobody wants to fix it, ask whether it's worth keeping. Deleting a test you don't trust is more honest than keeping it muted forever.

## Further reading

- [flakytest docs: muting](https://flakytest.dev/docs/#mute)
- [flakytest docs: flaky tests](https://flakytest.dev/docs/#flaky)
- [pytest: skip and xfail](https://docs.pytest.org/en/stable/how-to/skipping.html)
- [Flaky Tests at Google and How We Mitigate Them](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) (Google Testing Blog, 2016)