JSON, YAML and pydantic serialization in tests

Tests that parse the same JSON or YAML files again and again, or validate pydantic models in loops, burn CPU. Parse once, use the C YAML loader, reuse TypeAdapters.

Serialization rarely shows up as one slow line. It's spread over the suite: a fixture that reads tests/data/catalog.yaml for every test, a helper that round-trips a large payload through json.dumps and json.loads to "normalize" it, pydantic models validated thousands of times to build test data, a DRF serializer rendering a whole list to check one field. Each call is fast, and in a suite of a few thousand tests it adds up to a real share of the run.

Why it's slow

Parsing the same file again. A function-scoped fixture that opens and parses a data file does the full parse for every test that uses it, even though the file never changes during the run.
Pure-Python YAML. PyYAML's yaml.safe_load() uses the pure-Python SafeLoader. PyYAML also has CSafeLoader, backed by the libyaml C library, which is much faster on large documents, but you have to ask for it.
Validation you don't need. Building a pydantic model runs every validator. Creating a TypeAdapter builds a new validator and serializer each time, so creating one inside a function that's called per test repeats that work.
Round-trips. Dumping to JSON and loading it back to compare two structures, or to deep-copy one, does far more work than comparing or copying them directly.

The fix

Parse fixture files once

Cache the parsed result for the whole session, either with functools.cache or a session-scoped fixture:

# tests/data.py
import functools
from pathlib import Path

import yaml

DATA_DIR = Path(__file__).parent / "data"
YamlLoader = getattr(yaml, "CSafeLoader", yaml.SafeLoader)


@functools.cache
def load_yaml(name: str):
    with open(DATA_DIR / name) as f:
        return yaml.load(f, Loader=YamlLoader)

# tests/conftest.py
import pytest

from tests.data import load_yaml


@pytest.fixture(scope="session")
def catalog():
    return load_yaml("catalog.yaml")

getattr(yaml, "CSafeLoader", yaml.SafeLoader) falls back to the pure-Python loader when PyYAML was built without libyaml; yaml.__with_libyaml__ tells you which one you have. The same applies to dumping: yaml.dump(data, Dumper=yaml.CSafeDumper).

Make cached data immutable, or copy only what changes

A cached object is shared by every test, so a test that modifies it changes what the next test sees. Prefer data that can't be modified: tuples instead of lists, types.MappingProxyType for read-only dicts, frozen dataclasses. If a test really needs its own mutable copy, copy the part it changes rather than copy.deepcopy() the whole document, which can cost as much as parsing it.

pydantic: reuse adapters, skip validation for known-good data

# Before: builds a validator on every call
from pydantic import TypeAdapter

from myapp.schemas import Order


def parse_orders(raw: bytes) -> list[Order]:
    return TypeAdapter(list[Order]).validate_json(raw)

# After: built once, at import time
from pydantic import TypeAdapter

from myapp.schemas import Order

ORDERS = TypeAdapter(list[Order])


def parse_orders(raw: bytes) -> list[Order]:
    return ORDERS.validate_json(raw)

validate_json() (and Model.model_validate_json()) parses and validates in one step, which pydantic's performance docs recommend over model_validate(json.loads(raw)). When a test only needs an instance as input and the data is known to be valid, Order.model_construct(**fields) creates it without running validation. It still fills in defaults.

Compare structures directly

assert response.json() == expected compares Python objects; there's no need to normalize both sides through json.dumps(..., sort_keys=True) first. With large API responses, assert on the fields the test is about instead of rendering and comparing the whole payload in every test.

Keep the tests correct

Return copies of cached data if tests modify it, or make it immutable so a test that tries fails loudly instead of leaking state into the next one.
model_construct() skips validation, so it can create invalid models. Don't use it in tests that are about validation, or for data that comes from user input in the code under test.
Keep at least one test that goes through the real parsing path (the pure-Python loader if production uses it, the real serializer), so a change in format still gets caught.

How flakytest finds it

flakytest puts samples in json, pickle, pydantic, marshmallow, DRF serializers and YAML in the serialization category. When serialization takes at least 2 seconds and 5% of test time, it suggests Serialization is a large share of test time, with the tests and lines of your code that serialize the most. If PyYAML alone takes half a second or more, the suggestion also tells you to load with yaml.CSafeLoader.

Related guides

Expensive function-scoped pytest fixtures: A pytest fixture without a scope runs again for every test that uses it. Give expensive fixtures module or session scope, or split them into a shared part and a cheap reset.
setUpTestData deep copies and expensive copy.deepcopy: Django deep-copies attributes set in setUpTestData for every test that reads them. Big lists, prefetched instances and querysets make it slow. What to store instead.
Load Django fixtures once per class, not once per test: Django fixtures (loaddata) save every object one at a time. Load them once per TestCase class or pytest session instead of before every test, or use factories.
Faster factory_boy and model_bakery factories: Slow factory_boy and model_bakery factories? Create shared data once, use build() or baker.prepare() instead of saving, bulk create rows, reuse one Faker instance and tune Hypothesis.

Updated October 18, 2026 · Suggest a change