# Run migrations once with pytest-xdist and CI shards

> With pytest-xdist every worker builds its own Django test database, and every CI shard does too. Migrate once and copy it with PostgreSQL template databases or pg_dump.

Run a Django suite with `pytest -n 8` and pytest-django gives each pytest-xdist worker its own test database: `test_myapp_gw0` to `test_myapp_gw7`. Each worker builds its copy the normal way, by running every migration. Split the suite over four CI runners and each runner does the same again. The same migrations run 8 times, or 32.

`--reuse-db` hides this on your machine, because each worker keeps its database for the next run. CI starts from an empty database service every time, so there every run pays the full price.

## Why the work repeats

Each pytest-xdist worker is its own pytest process with its own session-scoped `django_db_setup` fixture. pytest-django appends the worker id to the test database name so workers don't share a database (transactional tests would wipe each other's data), and every worker then creates and migrates its own.

On a machine with a free core per worker the builds overlap, so you wait roughly as long as one build takes. The rest is still real work: the builds compete with each other for CPU and for the database server, which runs all that DDL several times over, and on CI machines with more workers than cores it turns into wall time. CI runners can't share anything: each one pays for a full build before its first test, in its own runner minutes.

## Django's test runner already migrates once

`manage.py test --parallel` doesn't have this problem. It creates and migrates `test_myapp` once, then clones it for each process (`test_myapp_1`, `test_myapp_2`, ...). How it clones depends on the backend:

| Backend | How Django clones the test database |
|---|---|
| PostgreSQL | `CREATE DATABASE ... TEMPLATE test_myapp` |
| MySQL | `mysqldump` of the test database, loaded into the new one |
| SQLite | copies the database file (an in-memory database is copied into each process) |

`python manage.py test --parallel --timing` prints how long the setup and each clone took. With pytest-xdist you have to set this up yourself.

## PostgreSQL template databases

PostgreSQL can create a database as a copy of another one:

```sql
CREATE DATABASE test_myapp_gw0 TEMPLATE test_myapp;
```

It copies the template's files instead of replaying anything, which for a test database is far cheaper than migrating. One rule: no other session may be connected to the template while it's being copied. From a shell, `createdb --template=test_myapp test_myapp_gw0` does the same.

## pytest-xdist: migrate in one worker, copy for the others

pytest-django lets you replace its `django_db_setup` fixture. This one makes the first worker build `test_myapp` and every worker clone its own database from it, using the same Django functions `--parallel` uses. A file lock (the pattern pytest-xdist documents for running a session fixture once) makes sure only one worker migrates and nobody is connected to the template while a clone is made:

```python
# conftest.py
import pytest
from django.db import connection
from filelock import FileLock


@pytest.fixture(scope="session")
def django_db_setup(django_db_blocker, django_db_keepdb, tmp_path_factory, worker_id):
    if worker_id == "master":
        # Not running under pytest-xdist: one database, built as usual
        with django_db_blocker.unblock():
            old_name = connection.creation.create_test_db(verbosity=0, autoclobber=True, keepdb=django_db_keepdb)
        yield
        if not django_db_keepdb:
            with django_db_blocker.unblock():
                connection.creation.destroy_test_db(old_name, verbosity=0)
        return

    # A directory every worker of this session shares
    shared_tmp = tmp_path_factory.getbasetemp().parent
    with django_db_blocker.unblock():
        with FileLock(shared_tmp / "django_db.lock"):
            # The first worker creates and migrates test_<name>. The others find it
            # already migrated, so migrate has nothing to do.
            old_name = connection.creation.create_test_db(verbosity=0, autoclobber=True, keepdb=True)
            # PostgreSQL: CREATE DATABASE test_<name>_gw0 TEMPLATE test_<name>
            connection.creation.clone_test_db(suffix=worker_id, verbosity=0, autoclobber=True)
        connection.settings_dict.update(connection.creation.get_test_db_clone_settings(worker_id))
        connection.close()
    yield
    with django_db_blocker.unblock():
        connection.creation.destroy_test_db(old_name, verbosity=0)
```

It needs `filelock` (`uv add --dev filelock`) and pytest-xdist installed, since it uses the `worker_id` fixture. Each worker's clone is dropped at the end; the template `test_myapp` stays, like `--reuse-db`, so the next run only applies new migrations to it. Drop it (`dropdb test_myapp`) after editing a migration that was already applied.

Limits: it handles the `default` database only (loop over `connections` for more), it replaces pytest-django's own fixture so `--create-db` and `--nomigrations` no longer apply, and it's meant for PostgreSQL (and MySQL). An in-memory SQLite database can't be shared between processes, so there each worker still migrates.

## CI: migrate once, restore everywhere

Across runners, a file lock can't help. Build the database once in a setup job, dump it with `pg_dump`, and restore the dump on every runner, which is much cheaper than migrating. Cache the dump keyed on a hash of your migrations so it's only rebuilt when they change.

A script that builds the test database exactly the way a test run would:

```python
# scripts/build_test_db.py
import django
from django.db import connection

django.setup()
connection.creation.create_test_db(verbosity=1, autoclobber=True)
```

A GitHub Actions workflow using it, with four shards via pytest-split and four xdist workers per shard:

```yaml
jobs:
  test-db:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10
    env:
      PGHOST: localhost
      PGUSER: postgres
      PGPASSWORD: postgres
      DJANGO_SETTINGS_MODULE: myapp.settings.test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        id: cache
        with:
          path: test-db.dump
          key: test-db-${{ hashFiles('**/migrations/*.py', 'uv.lock') }}
      - if: steps.cache.outputs.cache-hit != 'true'
        uses: astral-sh/setup-uv@v6
      - if: steps.cache.outputs.cache-hit != 'true'
        run: |
          uv run python scripts/build_test_db.py
          pg_dump --format=custom --no-owner --no-acl --file=test-db.dump test_myapp
      - uses: actions/upload-artifact@v4
        with:
          name: test-db
          path: test-db.dump

  test:
    needs: test-db
    runs-on: ubuntu-latest
    strategy:
      matrix:
        group: [1, 2, 3, 4]
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 10
    env:
      PGHOST: localhost
      PGUSER: postgres
      PGPASSWORD: postgres
      DJANGO_SETTINGS_MODULE: myapp.settings.test
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - uses: actions/download-artifact@v4
        with:
          name: test-db
      - run: |
          createdb test_myapp
          pg_restore --no-owner --no-acl --dbname=test_myapp test-db.dump
          for i in 0 1 2 3; do createdb --template=test_myapp test_myapp_gw$i; done
      - run: uv run pytest --reuse-db -n 4 --splits 4 --group ${{ matrix.group }}
```

How the pieces fit:

- `--reuse-db` makes pytest-django use the restored databases. Django still runs `migrate`, which applies only migrations missing from the dump, so a stale dump is slower but never wrong about the schema.
- The `createdb --template` loop makes one database per xdist worker, named the way pytest-django names them. With the conftest fixture above you can skip the loop and `--reuse-db`: restore `test_myapp` and the fixture clones it.
- The cache key includes `uv.lock` because Django's own apps and third-party packages ship migrations too, outside your repository.
- `pg_dump` refuses to dump a server newer than itself. Check that the PostgreSQL client on the runner is at least the version of the `postgres` image, or run `pg_dump` inside the service container.
- Runners that test against a different database version or engine still need their own build.

## Keep the tests correct

- Every copy is built by the same migrations, so a broken migration still fails the run. Under xdist the first worker runs every migration; in CI, any change to a migration changes the cache key, and the setup job runs every migration on an empty database.
- Build the dump with the same settings the tests use. If your settings switch on a `TESTING` flag (`"pytest" in sys.modules`, say), make sure the build script ends up with the same `INSTALLED_APPS` and databases.
- Clones and restored dumps contain whatever the build left in the database, including rows from data migrations, exactly like a freshly migrated one.

## How flakytest finds it

flakytest.dev sees every worker and every runner of a run, which one process can't. It adds two insights next to the plugin's own:

- **Run migrations once, not once per worker**: the `django_db_setup` fixture ran in two or more pytest-xdist workers, and the time spent repeating a build another worker already did is at least 5 seconds.
- **Run migrations once per CI run, not on every runner**: two or more runners of the same CI run each built the test database, and the repeated time (all builds but the slowest) is at least 5 seconds.

Both are high severity from 30 seconds of repeated work. The summary shows the number of builds, the total and the time per build.

## Further reading

- [pytest-django: using a template database for tests](https://pytest-django.readthedocs.io/en/latest/database.html#using-a-template-database-for-tests)
- [pytest-xdist: making session-scoped fixtures execute only once](https://pytest-xdist.readthedocs.io/en/stable/how-to.html#making-session-scoped-fixtures-execute-only-once)
- [Django: `test --parallel`](https://docs.djangoproject.com/en/stable/ref/django-admin/#cmdoption-test-parallel)
- [PostgreSQL: template databases](https://www.postgresql.org/docs/current/manage-ag-templatedbs.html)
- [PostgreSQL: pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) and [pg_restore](https://www.postgresql.org/docs/current/app-pgrestore.html)
- [GitHub Actions: actions/cache](https://github.com/actions/cache)