Profiles / #2

Passed Run of Oct 18, 16:02

CI github-actions:test · 0b902f6 main dirty · pass/fail report

Some tests got too few samples to explain their time. Rerun with pytest --flakytest-profile --flakytest-rate 1000.

Wall time

21.64 s

1 worker

Test time

17.61 s

sum of test durations

Tests

234

234 passed

Profiler

py-spy

500 Hz · 7,300 samples · Python 3.14.8 · flakytest 2026.0b1

Peak memory

171 MB

test process

Where the time goes

Test time by category

Session timeline

Other

1.14 s

Tests

16.34 s

Startup

212 ms

Collection

779 ms

Between tests

2.48 s

Session teardown

593 ms

Test phases

Setup 5.15 s Call 12.34 s Teardown 112 ms

Fix next

Ranked by the time each fix saves per run

Use plain static file storage in tests

Static file storage/manifest handling costs 6.0s.

High App code

5.97 s per run

Fix: In test settings use STORAGES["staticfiles"] = {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"} so templates don't need a manifest.

Details

Where:

apps/orgs/tests.py:30 test_signing_up_creates_an_organization · 394 ms
apps/api/tests.py:487 upload · 304 ms
ft/tests.py:205 test_change_password · 298 ms
apps/mcp/tests.py:66 post · 278 ms
ft/tests.py:161 test_signup · 175 ms
ft/tests.py:96 test_troubleshooting · 161 ms
apps/projects/urls.py:8 <module> · 143 ms
apps/projects/tests.py:88 test_index_page · 136 ms
ft/tests.py:200 test_change_password · 136 ms
apps/orgs/views.py:32 org_shell · 104 ms

Most affected tests:

ft/tests.py::TestAccountSettings::test_change_password
apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
ft/tests.py::TestAccountPages::test_signup
apps/api/tests.py::TestApiIndex::test_index
apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
ft/tests.py::TestTrubleshootingPage::test_troubleshooting
apps/projects/tests.py::IndexPageTest::test_index_page

Disable django-debug-toolbar in tests

django-debug-toolbar is installed in the test settings and costs 3.4s.

High App code

3.42 s per run

Fix: Only add debug_toolbar to INSTALLED_APPS and MIDDLEWARE when not testing (e.g. `if DEBUG and not TESTING:`).

Details

Where:

apps/orgs/tests.py:30 test_signing_up_creates_an_organization · 390 ms
ft/tests.py:205 test_change_password · 298 ms
ft/tests.py:161 test_signup · 175 ms
apps/projects/urls.py:8 <module> · 143 ms
ft/tests.py:200 test_change_password · 136 ms
apps/orgs/views.py:32 org_shell · 104 ms
apps/api/urls.py:72 wrapped_view · 103 ms
apps/projects/analytics.py:257 latest_profiles · 88 ms
apps/projects/urls.py:41 can_manage_project · 75 ms
apps/projects/urls.py:50 find_project · 73 ms

Most affected tests:

ft/tests.py::TestAccountSettings::test_change_password
apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
apps/api/tests.py::TestApiIndex::test_index
ft/tests.py::TestAccountPages::test_signup
apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
apps/api/tests.py::ProfileUploadTest::test_upload_adds_migrations_per_worker_insight
apps/projects/tests.py::ProjectPagesTest::test_projects_page

Documentation

Repeated queries (N+1) in the code under test

26 tests run the same query 10+ times, e.g. 180x in test_seed_demo. N+1 queries slow the tests and production alike.

High Database

3.07 s per run

Fix: Django: fetch related rows in one query with select_related() for foreign keys and prefetch_related() for reverse/many-to-many relations, then lock the count in with assertNumQueries (or pytest-django's django_assert_num_queries).

Details

This changes application code, so it speeds up production too; the tests stay as they are.

180x in test_seed_demo: SELECT "projects_test"."id", "projects_test"."name", "projects_test"."project_id", "projects_test"."slug", "projects_test"."date_last_run", "projects_test"."date_first_seen", "projects_test"."first_commit_id", "projects_test"."muted" FROM "projects_test" WHERE ("projects_test"."name" = %s AND "proj…
22x in test_dashboard_filters: SELECT "projects_environment"."id", "projects_environment"."project_id", "projects_environment"."name", "projects_environment"."kind", "projects_environment"."provider", "projects_environment"."details", "projects_environment"."date_first_seen", "projects_environment"."date_last_seen" FROM "project…
21x in test_configure_page: SELECT "projects_profile"."id", "projects_profile"."project_id", "projects_profile"."number", "projects_profile"."environment_id", "projects_profile"."session_id", "projects_profile"."run_id", "projects_profile"."commit_id", "projects_profile"."ingest_id", "projects_profile"."external_id", "project…
20x in test_index: SELECT "django_content_type"."model" AS "model" FROM "django_content_type" WHERE "django_content_type"."app_label" = %s
20x in test_daily_stats_count_a_profile_once_every_part_is_in: SELECT "projects_profilepart"."id", "projects_profilepart"."profile_id", "projects_profilepart"."index", "projects_profilepart"."count", "projects_profilepart"."checksum", "projects_profilepart"."status", "projects_profilepart"."error", "projects_profilepart"."tests_count", "projects_profilepart"."…

Most affected tests:

apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
apps/projects/tests.py::ProjectPagesTest::test_configure_page
apps/api/tests.py::TestApiIndex::test_index
apps/api/tests.py::ProfileUploadTest::test_daily_stats_count_a_profile_once_every_part_is_in
apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
apps/api/tests.py::IngestStartEndpointTest::test_details_are_kept_for_failures_only
apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress

Documentation

Stop re-running migrations on every test run

Migrations take 2.7s per run, before the first test can start.

High Migrations

2.46 s per run

Fix: Run with --reuse-db (add it to addopts in pytest.ini/pyproject) so the test database survives between runs; pass --create-db when migrations change. If no test depends on data migrations, --nomigrations builds the schema straight from the models, which is faster still.

Details

CI should still build the database from scratch (no --reuse-db) so broken migrations are caught.

Where:

apps/projects/migrations/0025_backfill_slugs_and_numbers.py:35 forwards · 2 ms
apps/projects/migrations/0014_backfill_session_counts.py:23 backfill · 2 ms
apps/projects/migrations/0022_backfill_organizations.py:28 forwards · 2 ms
apps/projects/migrations/0012_backfill_session_environments.py:30 backfill · 2 ms

Most affected tests:

apps/api/tests.py::TestApiIndex::test_index

Documentation

Make the test database server skip durability work

Tests spend 10.7s (61%) in the database. Test databases don't need to survive a crash, so the server can skip flushing to disk.

High Database

2.14 s per run

Fix: For a PostgreSQL test server (Docker/CI service) start it with -c fsync=off -c synchronous_commit=off -c full_page_writes=off, and put its data directory on tmpfs; for MySQL use --innodb-flush-log-at-trx-commit=0 --skip-log-bin. Never on a server with data you need.

Details

Only affects crash safety of the throwaway test server; query behaviour is identical.

Where:

apps/api/tests.py:76 join_workers · 132 ms
apps/projects/profiles.py:458 write_perfs · 130 ms
apps/projects/tests.py:953 setUp · 119 ms
apps/projects/profiles.py:313 write_snapshots · 116 ms
apps/projects/models.py:466 save · 105 ms
apps/orgs/views.py:32 org_shell · 104 ms
apps/api/urls.py:72 wrapped_view · 103 ms
apps/projects/environments.py:63 get_or_create_environment · 100 ms
apps/projects/models.py:230 upsert · 97 ms
apps/api/tests.py:73 setUp · 90 ms

Most affected tests:

apps/api/tests.py::TestApiIndex::test_index
apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
apps/api/tests.py::IngestStartEndpointTest::test_ingest_start_progress_end
apps/projects/tests.py::ProfilePagePaginationTest::test_slowest_tests_first_then_the_next_page
apps/api/tests.py::IngestStartEndpointTest::test_results_are_counted_and_the_payload_dropped
apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress
apps/api/tests.py::IngestStartEndpointTest::test_ingest_progress_before_start

Documentation

Use a fast password hasher in tests

1.4s (8% of test time) is spent hashing passwords. Slow hashing protects real passwords; in tests it only burns CPU.

Medium Password hashing

1.31 s per run

Fix: In your test settings set PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"] (currently PBKDF2PasswordHasher).

Details

Only change the hasher in test settings. Tests that assert on a specific hash format need updating.

Where:

ft/tests.py:205 test_change_password · 298 ms
apps/orgs/tests.py:30 test_signing_up_creates_an_organization · 197 ms
ft/tests.py:161 test_signup · 165 ms
ft/tests.py:230 test_projects_only_lists_your_own_even_for_superusers · 162 ms
apps/orgs/tests.py:194 test_superusers_can_open_any_project_but_dont_list_them · 150 ms
ft/tests.py:213 test_change_password · 142 ms
ft/tests.py:200 test_change_password · 134 ms
ft/tests.py:175 setUpTestData · 133 ms

Most affected tests:

ft/tests.py::TestAccountSettings::test_change_password
apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
ft/tests.py::TestAccountPages::test_signup
ft/tests.py::TestAccountSettings::test_projects_only_lists_your_own_even_for_superusers
apps/orgs/tests.py::SuperuserTest::test_superusers_can_open_any_project_but_dont_list_them

Documentation

Use TestCase instead of TransactionTestCase

900ms goes to flushing every table after tests in 11 TransactionTestCase / transaction=True / live_server tests.

Medium DB schema

810 ms per run

Fix: Use django.test.TestCase (or @pytest.mark.django_db without transaction=True): it wraps each test in a transaction and rolls back instead of truncating every table. Keep TransactionTestCase only for tests that need real commits (on_commit hooks can use TestCase.captureOnCommitCallbacks).

Details

Tests that rely on commit behaviour, multiple connections or a live server still need a transactional test case.

Most affected tests:

apps/api/tests.py::IngestStartEndpointTest::test_details_are_kept_for_failures_only
apps/api/tests.py::IngestStartEndpointTest::test_ingest_multiple_sessions
apps/api/tests.py::IngestStartEndpointTest::test_results_are_counted_and_the_payload_dropped
apps/api/tests.py::IngestStartEndpointTest::test_ingest_valid
apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress
apps/api/tests.py::IngestStartEndpointTest::test_ingest_progress_before_start
apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
apps/api/tests.py::IngestStartEndpointTest::test_ingest_start_progress_end

Documentation

Tests open new database connections over and over

Connecting to the database takes 670ms.

Medium Database

536 ms per run

Fix: Reuse connections: create the engine once per session (not per test) and keep the default pool (NullPool reconnects on every checkout); in Django avoid closing connections in tests (connection.close()/close_old_connections).

Details

Where:

apps/api/tests.py:76 join_workers · 110 ms
apps/api/tests.py:73 setUp · 46 ms
apps/api/tasks.py:266 warm_reports · 44 ms
apps/api/urls.py:72 wrapped_view · 41 ms
apps/api/tests.py:928 setUp · 13 ms
apps/api/tests.py:331 test_results_are_counted_and_the_payload_dropped · 11 ms
apps/api/tests.py:975 setUp · 11 ms
apps/api/tests.py:287 test_all_test_status_types · 10 ms
apps/api/tests.py:212 test_ingest_start_progress_end · 9 ms
apps/api/tests.py:988 test_upload_is_processed_by_the_worker · 7 ms

Most affected tests:

apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
apps/api/tests.py::IngestStartEndpointTest::test_results_are_counted_and_the_payload_dropped
apps/api/tests.py::IngestStartEndpointTest::test_ingest_start_progress_end
apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress
apps/api/tests.py::IngestStartEndpointTest::test_ingest_progress_before_start
apps/api/tests.py::IngestStartEndpointTest::test_details_are_kept_for_failures_only
apps/api/tests.py::ProfileUploadQueueTest::test_upload_is_processed_by_the_worker
apps/projects/tests.py::LegacyUrlTest::test_by_id

Create shared test data in setUpTestData instead of setUp

setUp() runs before every test and takes 556ms in total (db 542ms, library 11ms, factories 3ms) in 13 classes: SettingsTest (3 tests), ProjectsPageTest (5 tests), ProjectEditPage (6 tests), ProjectDeletePage (4 tests).

Low Database

389 ms per run

Fix: Move object creation that tests only read from setUp() into `@classmethod def setUpTestData(cls)`: it runs once per class and each test still sees the original rows (TestCase rolls back to a savepoint).

Details

Since Django 3.2 attributes assigned in setUpTestData are deep-copied per test, so in-memory changes don't leak between tests. Keep objects a test modifies in the database (or re-fetches) as before.

Where:

apps/projects/tests.py:953 setUp · 124 ms
apps/api/tests.py:73 setUp · 94 ms
apps/projects/tests.py:100 setUp · 63 ms
apps/projects/tests.py:142 setUp · 48 ms
apps/projects/tests.py:268 setUp · 39 ms
apps/orgs/tests.py:167 setUp · 32 ms
apps/projects/tests.py:187 setUp · 24 ms
apps/projects/tests.py:337 setUp · 23 ms
ft/tests.py:179 setUp · 22 ms
apps/projects/tests.py:218 setUp · 21 ms

Most affected tests:

apps/api/tests.py::IngestStartEndpointTest::test_details_are_kept_for_failures_only
apps/projects/tests.py::ProjectEditPage::test_project_submit
apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
apps/projects/tests.py::ProjectRunsPage::test_not_logged_in
apps/api/tests.py::ProfileUploadQueueTest::test_upload_is_processed_by_the_worker
apps/projects/tests.py::ProjectsPageTest::test_project
apps/projects/tests.py::ProjectsPageTest::test_no_project
apps/projects/tests.py::ProjectPagesTest::test_dashboard_without_data_shows_setup

Documentation

Tests

201–234 of 234, slowest first · click a column to sort this page

Test	Outcome	Duration	Setup	Call	Teardown	Mostly	Queries
apps/projects/tests.py::AnalyticsTest::test_ci_vs_local	passed	9 ms	0.33 ms	8 ms	0.09 ms	App code	13
apps/projects/tests.py::FlakyReportTest::test_window	passed	8 ms	0.39 ms	8 ms	0.37 ms	Database	13
apps/projects/tests.py::FlakyReportTest::test_single_test	passed	8 ms	0.40 ms	7 ms	0.50 ms	Database	11
apps/api/tests.py::ProfileUploadTest::test_processing_failure_is_recorded	passed	8 ms	0.40 ms	7 ms	0.10 ms	Unsampled	15 3 repeated
apps/projects/tests.py::SlugsAndNumbersTest::test_upsert_keeps_slugs_and_makes_them_unique	passed	8 ms	0.35 ms	7 ms	0.41 ms	Unsampled	16 6 repeated
apps/projects/tests.py::StorageTest::test_slow_tests_come_from_daily_stats	passed	7 ms	0.42 ms	7 ms	0.12 ms	Database	11
apps/projects/tests.py::AnalyticsTest::test_compare_environments	passed	7 ms	0.39 ms	7 ms	0.09 ms	Database	11
apps/projects/tests.py::AnalyticsTest::test_slow_tests	passed	6 ms	0.32 ms	6 ms	0.11 ms	Database	10
apps/projects/tests.py::AnalyticsTest::test_fix_next	passed	6 ms	0.31 ms	6 ms	0.08 ms	Database	8
apps/projects/tests.py::AnalyticsTest::test_resolve_scope	passed	6 ms	0.40 ms	6 ms	0.09 ms	Database	11 6 repeated
apps/projects/tests.py::StorageTest::test_snapshot_renders_like_the_whole_document	passed	6 ms	0.32 ms	5 ms	0.36 ms	Database	9
apps/projects/tests.py::AnalyticsTest::test_environment_rows	passed	6 ms	0.37 ms	5 ms	0.09 ms	Database	8
apps/projects/tests.py::RealProfileDocumentTest::test_entry_details	passed	4 ms	0.33 ms	4 ms	0.17 ms	Unsampled	5
apps/projects/tests.py::AnalyticsTest::test_ci_vs_local_needs_both	passed	4 ms	0.32 ms	4 ms	0.08 ms	Unsampled	9
apps/projects/tests.py::AnalyticsTest::test_suite_trend	passed	4 ms	0.38 ms	3 ms	0.37 ms	Database	6
apps/projects/tests.py::StorageTest::test_daily_stats_sum_the_days_results	passed	4 ms	0.34 ms	3 ms	0.11 ms	Unsampled	8
apps/projects/tests.py::ClassifyStreaksTest::test_ongoing	passed	3 ms	0.34 ms	2 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::RealProfileDocumentTest::test_flat_stacks_still_work	passed	2 ms	0.31 ms	2 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_flaky_when_failed_and_passed_on_same_commit	passed	2 ms	0.29 ms	1 ms	0.08 ms	Database	5
apps/projects/tests.py::ClassifyStreaksTest::test_test_changed_when_the_test_file_changed	passed	2 ms	0.35 ms	0.88 ms	0.29 ms	Unsampled	5
apps/projects/tests.py::MigrationInsightsTest::test_per_runner	passed	2 ms	1 ms	0.16 ms	0.07 ms	Imports	–
apps/projects/tests.py::SlugsAndNumbersTest::test_test_slugs_read_like_their_node_ids	passed	1 ms	0.43 ms	0.95 ms	0.09 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_environment_kinds_are_counted	passed	1 ms	0.32 ms	0.93 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_multiple_streaks_in_order	passed	1 ms	0.37 ms	0.87 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_regression_caught_when_app_code_fixed_it	passed	1 ms	0.35 ms	0.87 ms	0.07 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_flaky_even_if_the_pass_came_first_and_elsewhere	passed	1 ms	0.37 ms	0.82 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_test_changed_matches_paths_relative_to_the_repository	passed	1 ms	0.28 ms	0.90 ms	0.07 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_no_failures	passed	1 ms	0.34 ms	0.83 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::ClassifyStreaksTest::test_resolved_without_changed_files	passed	1 ms	0.35 ms	0.81 ms	0.08 ms	Unsampled	5
apps/projects/tests.py::MigrationInsightsTest::test_per_worker	passed	0.64 ms	0.42 ms	0.15 ms	0.07 ms	Unsampled	–
apps/projects/tests.py::MigrationInsightsTest::test_per_worker_ignores_cheap_setups	passed	0.56 ms	0.37 ms	0.10 ms	0.09 ms	Unsampled	–
apps/projects/tests.py::MigrationInsightsTest::test_per_worker_needs_repeated_builds	passed	0.54 ms	0.37 ms	0.10 ms	0.07 ms	Unsampled	–
apps/projects/tests.py::MigrationInsightsTest::test_per_runner_needs_several_runners	passed	0.54 ms	0.29 ms	0.14 ms	0.12 ms	Unsampled	–
apps/projects/tests.py::MigrationInsightsTest::test_per_worker_tolerates_bad_fixture_data	passed	0.51 ms	0.33 ms	0.11 ms	0.07 ms	Unsampled	–

Page 2 of 2 Previous

Memory

The test process peaked at 197 MB

Needed the most memory	Needed	Kept after
apps/api/tests.py::TestApiIndex::test_index	32 MB	32 MB
apps/projects/tests.py::ProfilePagePaginationTest::test_slowest_tests_first_then_the_next_page	3 MB	3 MB
apps/projects/tests.py::RealProfileDocumentTest::test_call_tree	2 MB	2 MB
apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization	1 MB	1 MB
apps/api/tests.py::ProfileUploadTest::test_missing_parts_mark_the_profile_incomplete	1 MB	1 MB
apps/orgs/tests.py::SuperuserTest::test_superusers_can_open_any_project_but_dont_list_them	1 MB	1 MB
apps/projects/tests.py::ProjectTestsPage::test_not_logged_in	920 KB	920 KB
apps/projects/tests.py::ProjectRunPage::test_not_logged_in	888 KB	888 KB
apps/api/tests.py::ProfileUploadTest::test_reject_other_formats	860 KB	860 KB
apps/orgs/tests.py::AccessTest::test_public_projects_are_readable_by_anyone	860 KB	860 KB

Slowest fixtures

Exact setup time across the session

Fixture	Scope	Calls	Total	Max
_django_setup_unittest	class	49	3.59 s	3.58 s
django_db_setup	session	1	3.58 s	3.58 s
_unittest_setUpClass_fixture_RealProfileDocumentTest	class	1	230 ms	230 ms
_unittest_setUpClass_fixture_ProjectPagesTest	class	1	181 ms	181 ms
_unittest_setUpClass_fixture_TestAccountSettings	class	1	140 ms	140 ms
_unittest_setUpClass_fixture_AnalyticsTest	class	1	124 ms	124 ms
_unittest_setUpClass_fixture_McpTest	class	1	122 ms	122 ms
_unittest_setUpClass_fixture_ProfilePagePaginationTest	class	1	111 ms	111 ms
_unittest_setUpClass_fixture_StorageTest	class	1	76 ms	76 ms
_unittest_setUpClass_fixture_MemoryTest	class	1	58 ms	58 ms
_unittest_setUpClass_fixture_FlakyReportTest	class	1	48 ms	48 ms
_unittest_setUpClass_fixture_LegacyUrlTest	class	1	40 ms	40 ms
_unittest_setUpClass_fixture_FlakyReportCacheTest	class	1	32 ms	32 ms
_unittest_setUpClass_fixture_TestProjectTestPage	class	1	15 ms	15 ms
_unittest_setUpClass_fixture_ProjectRunPage	class	1	15 ms	15 ms