Mute a Tests

Why mute a test?

Not all tests in your application are business-critical, so your whole engineering team shouldn't have to drop everything and fix it immediately when a test fails. On the other hand, you can't have failing tests in your CI because otherwise, your developers will not trust your test suite.

Skipping tests

Test frameworks have ways of marking a test as skipped or ignored. Usually, this requires code changes, pull request reviews, etc. Once a test is skipped or ignored, it will no longer run on your CI, and you will stop gathering information about it.

Skipped tests rarely get fixed and reenabled unless your team tracks the number of skipped tests and goes back to fix them later. If this happens weeks after the initial failure, you might be unable to tell why somebody skipped the test in the first place. The error logs of the initial failure might not be available anymore or be outdated.

Muting tests

We created FlakyTest to help teams deal with this problem. First, you can mute a test with a single click in the UI, so no code changes or pull requests are required!

Secondly, a muted test is different from a skipped one. A muted test will continue to run on your CI and log any errors, but it won't fail your CI build when it fails. This way, you will have more information about the test when you return to it a few weeks later.

Lastly, FlakyTest keeps track of all your muted (and skipped!) tests, so it's easy to set up team metrics around that. We can even send email reports after the number of skipped tests reaches a certain threshold!

If you have any questions or concerns, send us an email [email protected]