# JSON, YAML and pydantic serialization in tests

> Tests that parse the same JSON or YAML files again and again, or validate pydantic models in loops, burn CPU. Parse once, use the C YAML loader, reuse TypeAdapters.

Serialization rarely shows up as one slow line. It's spread over the suite: a fixture that reads `tests/data/catalog.yaml` for every test, a helper that round-trips a large payload through `json.dumps` and `json.loads` to "normalize" it, pydantic models validated thousands of times to build test data, a DRF serializer rendering a whole list to check one field. Each call is fast, and in a suite of a few thousand tests it adds up to a real share of the run.

## Why it's slow

- **Parsing the same file again.** A function-scoped fixture that opens and parses a data file does the full parse for every test that uses it, even though the file never changes during the run.
- **Pure-Python YAML.** PyYAML's `yaml.safe_load()` uses the pure-Python `SafeLoader`. PyYAML also has `CSafeLoader`, backed by the libyaml C library, which is much faster on large documents, but you have to ask for it.
- **Validation you don't need.** Building a pydantic model runs every validator. Creating a `TypeAdapter` builds a new validator and serializer each time, so creating one inside a function that's called per test repeats that work.
- **Round-trips.** Dumping to JSON and loading it back to compare two structures, or to deep-copy one, does far more work than comparing or copying them directly.

## The fix

### Parse fixture files once

Cache the parsed result for the whole session, either with `functools.cache` or a session-scoped fixture:

```python
# tests/data.py
import functools
from pathlib import Path

import yaml

DATA_DIR = Path(__file__).parent / "data"
YamlLoader = getattr(yaml, "CSafeLoader", yaml.SafeLoader)


@functools.cache
def load_yaml(name: str):
    with open(DATA_DIR / name) as f:
        return yaml.load(f, Loader=YamlLoader)
```

```python
# tests/conftest.py
import pytest

from tests.data import load_yaml


@pytest.fixture(scope="session")
def catalog():
    return load_yaml("catalog.yaml")
```

`getattr(yaml, "CSafeLoader", yaml.SafeLoader)` falls back to the pure-Python loader when PyYAML was built without libyaml; `yaml.__with_libyaml__` tells you which one you have. The same applies to dumping: `yaml.dump(data, Dumper=yaml.CSafeDumper)`.

### Make cached data immutable, or copy only what changes

A cached object is shared by every test, so a test that modifies it changes what the next test sees. Prefer data that can't be modified: tuples instead of lists, `types.MappingProxyType` for read-only dicts, frozen dataclasses. If a test really needs its own mutable copy, copy the part it changes rather than `copy.deepcopy()` the whole document, which can cost as much as parsing it.

### pydantic: reuse adapters, skip validation for known-good data

```python
# Before: builds a validator on every call
from pydantic import TypeAdapter

from myapp.schemas import Order


def parse_orders(raw: bytes) -> list[Order]:
    return TypeAdapter(list[Order]).validate_json(raw)
```

```python
# After: built once, at import time
from pydantic import TypeAdapter

from myapp.schemas import Order

ORDERS = TypeAdapter(list[Order])


def parse_orders(raw: bytes) -> list[Order]:
    return ORDERS.validate_json(raw)
```

`validate_json()` (and `Model.model_validate_json()`) parses and validates in one step, which pydantic's performance docs recommend over `model_validate(json.loads(raw))`. When a test only needs an instance as input and the data is known to be valid, `Order.model_construct(**fields)` creates it without running validation. It still fills in defaults.

### Compare structures directly

`assert response.json() == expected` compares Python objects; there's no need to normalize both sides through `json.dumps(..., sort_keys=True)` first. With large API responses, assert on the fields the test is about instead of rendering and comparing the whole payload in every test.

## Keep the tests correct

- Return copies of cached data if tests modify it, or make it immutable so a test that tries fails loudly instead of leaking state into the next one.
- `model_construct()` skips validation, so it can create invalid models. Don't use it in tests that are about validation, or for data that comes from user input in the code under test.
- Keep at least one test that goes through the real parsing path (the pure-Python loader if production uses it, the real serializer), so a change in format still gets caught.

## How flakytest finds it

flakytest puts samples in `json`, `pickle`, pydantic, marshmallow, DRF serializers and YAML in the *serialization* category. When serialization takes at least 2 seconds and 5% of test time, it suggests **Serialization is a large share of test time**, with the tests and lines of your code that serialize the most. If PyYAML alone takes half a second or more, the suggestion also tells you to load with `yaml.CSafeLoader`.

## Further reading

- [pydantic: performance tips](https://docs.pydantic.dev/latest/concepts/performance/)
- [pydantic: creating models without validation](https://docs.pydantic.dev/latest/concepts/models/#creating-models-without-validation)
- [PyYAML documentation](https://pyyaml.org/wiki/PyYAMLDocumentation)
- [pytest: fixture scopes](https://docs.pytest.org/en/stable/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session)