Configure flakytest in your CI

To configure your CI pipline you have to ensure two things:

1. flakytest pytest plugin has to be installed when you run your tests. This usually means adding flakytest to your development dependencies e.g. dev-requirements.txt.
2. The pytest process has to have access to the FLAKYTEST_SECRET_TOKEN environment variable. This usually means adding the token to your CI environment secrets.

Run pytest with --flakytest-profile to upload a per-test performance profile too. Each CI job becomes its own environment (e.g. ci:github-actions:test, ci:gitlab:unit), so you can compare jobs with each other and with developer machines. GitHub Actions, GitLab CI, CircleCI, Buildkite and Jenkins are detected automatically; set FLAKYTEST_ENV to choose the name yourself.

Setting up GitHub Actions

To set up flakytest in GitHub Actions you have to add a GitHub icon secret, and then expose it to the run command.

To add the secret, go to your repository settings, and then to the Actions secrets and variables page. Add a new repository secret with the name FLAKYTEST_SECRET_TOKEN and the value of the secret token from your project page.

To expose the secret to the run command, you have to add it to the env section of your job. The env section is a dictionary of environment variables that will be available to the run command.

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.14"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run tests
        run: pytest --flakytest-profile
        env:
          FLAKYTEST_SECRET_TOKEN: ${{ secrets.FLAKYTEST_SECRET_TOKEN }}