Get Started with flakytest.dev

flakytest's pytest plugin enables test tracking and management. This page will show you how to get it up and running.

1. TLDR
2. Set up an example project (optional)
3. Install the pytest plugin
4. Obtain a secret token
5. Send test results to flakytest.dev

TLDR

1. Set up a Python environment with pytest and some tests (see example).
2. Install the flakytest pytest plugin:

   pip install flakytest

3. Obtain a secret token by creating a new project on flakytest.dev. You'll need an account first; create one using either GitHub or Email.
4. Set the FLAKYTEST_SECRET_TOKEN environment variable and run:

   export FLAKYTEST_SECRET_TOKEN={YOUR_SECRET_TOKEN}
pytest

5. See the results of the test run on flakytest.dev.

Set up an example project (optional)

You can skip this step if you have your own project that uses pytest.

We have an example Python project that uses pytest and some tests. Let's set it up.

1. Clone the example project

   git clone https://github.com/anze3db/flakytest-python-example.git

2. Create a new virtual environment

   python -m venv .venv

3. Activate the virtual environment

   source .venv/bin/activate

4. Install the dependencies

   pip install -r requirements.txt

5. And now we can finally run the tests

   pytest

After running the pytest command, you should see the following results:

Don't worry about the failing tests for now. The failures are intentional!

Install the pytest plugin

We will install the flakytest pytest plugin to start gathering test results.

pip install flakytest

Now the flakytest pytest plugin is installed, but we still need to enable it by providing a secret token.

Obtain a secret token

Sign Up and create a flakytest.dev account. If you already have an account, make sure to log in.

To get a secret token, you have to first create a project. You will find the secret token on the project configuration page.

Send test results to flakytest.dev

When the FLAKYTEST_SECRET_TOKEN environment variable is set, the plugin will send test results to flakytest.dev. We can try this out by setting the environment variable and rerunning the pytest command:

export FLAKYTEST_SECRET_TOKEN={YOUR_SECRET_TOKEN}
pytest

We can run the tests multiple times to get more data into flakytest.dev. After 3 runs, the results might look like this:

Next steps

You have configured your project to send test results to flakytest.dev. 🎉

Next steps:

1. Configure flakytest.dev on your CI
2. Mute a test