flakytest.dev anze3db flakytest
All environments · last 7 days

Test suite performance

Suite wall time
28.25 s
avg over 4 profiled runs
Total test time
23.07 s
sum of test durations per run
Tests
258
in 1 environment
Failure rate
0.0%
0 of 1,019 results
Flaky tests
0

Suite wall time per run

Click a point to open the run

Where the time goes

Avg test time per run
CI github-actions:test
23.07 s
Database Test code Password hashing App code Factories Migrations Other (Libraries, DB schema, Templates, Imports, ...)

Slowest tests

All tests →
Test and why it's slow Avg Share of suite Runs
apps/api/tests.py::TestApiIndex::test_index
82% migrations · fixture django_db_setup (session scope) pytest_django/fixtures.py:198
3.31 s
14.3%
4
apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
59% database · test body apps/projects/tests.py:1161
1.69 s
7.3%
4
ft/tests.py::TestAccountSettings::test_change_password
96% password hashing · test body ft/tests.py:210
970 ms
4.2%
4
apps/api/tests.py::ProfileUploadTest::test_missing_parts_mark_the_profile_incomplete
61% templates · test body apps/api/tests.py:830
656 ms
2.8%
4
apps/projects/tests.py::ProfilePagePaginationTest::test_slowest_tests_first_then_the_next_page
63% database · setUpClass django/test/testcases.py:1439
321 ms
1.4%
4
apps/projects/tests.py::ProjectPagesTest::test_configure_page
78% database · setUpClass django/test/testcases.py:1439
321 ms
1.4%
4
apps/orgs/tests.py::SuperuserTest::test_superusers_can_open_any_project_but_dont_list_them
80% password hashing · test body apps/orgs/tests.py:194
315 ms
1.4%
4
apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
72% password hashing · test body apps/orgs/tests.py:28
284 ms
1.2%
4
apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress
49% database · test body apps/api/tests.py:246
273 ms
1.2%
4
apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
85% database · test body apps/projects/tests.py:970
264 ms
1.1%
4

Fix next

From the latest run per environment, by time saved
1

Use plain static file storage in tests

Static file storage/manifest handling costs 9.2s.

High App code ci:github-actions:test
9.16 s per run
Fix: In test settings use STORAGES["staticfiles"] = {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"} so templates don't need a manifest. Read the guide
Details

Where:

  • apps/api/tests.py:487 ProfileUploadTest.upload · 491 ms
  • apps/mcp/tests.py:67 McpTest.post · 398 ms
  • ft/tests.py:210 TestAccountSettings.test_change_password · 327 ms
  • apps/guides/content.py:112 render · 319 ms
  • apps/orgs/tests.py:28 SignupTest.test_signing_up_creates_an_organization · 225 ms
  • ft/tests.py:166 TestAccountPages.test_signup · 181 ms
  • ft/tests.py:205 TestAccountSettings.test_change_password · 162 ms
  • apps/projects/tests.py:320 ProjectRunPage.test_other_peoples_project · 156 ms
  • apps/api/urls.py:72 api_request.<locals>.decorator.<locals>.wrapped_view · 153 ms
  • apps/projects/urls.py:692 <module> · 149 ms

Most affected tests:

  • apps/api/tests.py::ProfileUploadTest::test_missing_parts_mark_the_profile_incomplete
  • ft/tests.py::TestAccountSettings::test_change_password
  • apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
  • apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
  • apps/api/tests.py::TestApiIndex::test_index
  • ft/tests.py::TestAccountPages::test_signup
  • apps/api/tests.py::ProfileUploadTest::test_reupload_split_differently
  • apps/projects/tests.py::ProjectRunPage::test_other_peoples_project
2

Disable django-debug-toolbar in tests

django-debug-toolbar is installed in the test settings and costs 5.8s.

High App code ci:github-actions:test
5.83 s per run
Fix: Only add debug_toolbar to INSTALLED_APPS and MIDDLEWARE when not testing (e.g. `if DEBUG and not TESTING:`). Read the guide
Details

Where:

  • ft/tests.py:210 TestAccountSettings.test_change_password · 327 ms
  • apps/guides/content.py:112 render · 319 ms
  • apps/orgs/tests.py:28 SignupTest.test_signing_up_creates_an_organization · 213 ms
  • ft/tests.py:166 TestAccountPages.test_signup · 181 ms
  • ft/tests.py:205 TestAccountSettings.test_change_password · 162 ms
  • apps/api/urls.py:72 api_request.<locals>.decorator.<locals>.wrapped_view · 153 ms
  • apps/projects/urls.py:692 <module> · 149 ms
  • apps/projects/urls.py:498 project_profile · 136 ms
  • apps/projects/urls.py:40 can_manage_project · 125 ms
  • apps/orgs/views.py:33 org_shell · 115 ms

Most affected tests:

  • ft/tests.py::TestAccountSettings::test_change_password
  • apps/api/tests.py::ProfileUploadTest::test_missing_parts_mark_the_profile_incomplete
  • apps/orgs/tests.py::SignupTest::test_signing_up_creates_an_organization
  • apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
  • apps/api/tests.py::TestApiIndex::test_index
  • ft/tests.py::TestAccountPages::test_signup
  • apps/api/tests.py::ProfileUploadTest::test_upload_adds_migrations_per_worker_insight
  • apps/projects/tests.py::ProjectPagesTest::test_dashboard

Documentation

3

Make the test database server skip durability work

Tests spend 11.8s (59%) in the database. Test databases don't need to survive a crash, so the server can skip flushing to disk.

High Database ci:github-actions:test
2.36 s per run
Fix: For a PostgreSQL test server (Docker/CI service) start it with -c fsync=off -c synchronous_commit=off -c full_page_writes=off, and put its data directory on tmpfs; for MySQL use --innodb-flush-log-at-trx-commit=0 --skip-log-bin. Never on a server with data you need. Read the guide
Details

Only affects crash safety of the throwaway test server; query behaviour is identical.

Where:

  • ft/db/postgresql/base.py:39 DatabaseWrapper.get_new_connection · 743 ms
  • apps/projects/models.py:466 Profile.save · 291 ms
  • apps/projects/profiles.py:352 write_snapshots · 155 ms
  • apps/api/urls.py:72 api_request.<locals>.decorator.<locals>.wrapped_view · 151 ms
  • apps/projects/environments.py:63 get_or_create_environment · 134 ms
  • apps/projects/urls.py:40 can_manage_project · 125 ms
  • apps/projects/models.py:230 TestManager.upsert · 123 ms
  • apps/projects/tests.py:954 ProjectPagesTest.setUp · 122 ms
  • apps/projects/rollups.py:88 add_perfs · 121 ms
  • apps/orgs/views.py:33 org_shell · 115 ms

Most affected tests:

  • apps/api/tests.py::TestApiIndex::test_index
  • apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
  • apps/projects/tests.py::ProjectPagesTest::test_configure_page
  • apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
  • apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
  • apps/api/tests.py::IngestStartEndpointTest::test_ingest_start_progress_end
  • apps/api/tests.py::IngestStartEndpointTest::test_ingest_progress_before_start
  • apps/api/tests.py::IngestStartEndpointTest::test_results_are_counted_and_the_payload_dropped

Documentation

4

Repeated queries (N+1) in the code under test

27 tests run the same query 10+ times, e.g. 180x in test_seed_demo. N+1 queries slow the tests and production alike.

High Database ci:github-actions:test
2.25 s per run
Fix: Django: fetch related rows in one query with select_related() for foreign keys and prefetch_related() for reverse/many-to-many relations, then lock the count in with assertNumQueries (or pytest-django's django_assert_num_queries). Read the guide
Details

This changes application code, so it speeds up production too; the tests stay as they are.

  • 180x in test_seed_demo: SELECT "projects_test"."id", "projects_test"."name", "projects_test"."project_id", "projects_test"."slug", "projects_test"."date_last_run", "projects_test"."date_first_seen", "projects_test"."first_commit_id", "projects_test"."muted" FROM "projects_test" WHERE ("projects_test"."name" = %s AND "proj…
  • 22x in test_dashboard_filters: SELECT "projects_environment"."id", "projects_environment"."project_id", "projects_environment"."name", "projects_environment"."kind", "projects_environment"."provider", "projects_environment"."details", "projects_environment"."date_first_seen", "projects_environment"."date_last_seen" FROM "project…
  • 21x in test_configure_page: SELECT "projects_profile"."id", "projects_profile"."project_id", "projects_profile"."number", "projects_profile"."environment_id", "projects_profile"."session_id", "projects_profile"."run_id", "projects_profile"."commit_id", "projects_profile"."ingest_id", "projects_profile"."external_id", "project…
  • 20x in test_index: SELECT "django_content_type"."model" AS "model" FROM "django_content_type" WHERE "django_content_type"."app_label" = %s
  • 20x in test_daily_stats_count_a_profile_once_every_part_is_in: SELECT "projects_profilepart"."id", "projects_profilepart"."profile_id", "projects_profilepart"."index", "projects_profilepart"."count", "projects_profilepart"."checksum", "projects_profilepart"."status", "projects_profilepart"."error", "projects_profilepart"."tests_count", "projects_profilepart"."…

Most affected tests:

  • apps/projects/tests.py::SeedDemoCommandTest::test_seed_demo
  • apps/projects/tests.py::ProjectPagesTest::test_dashboard_filters
  • apps/projects/tests.py::ProjectPagesTest::test_configure_page
  • apps/api/tests.py::TestApiIndex::test_index
  • apps/api/tests.py::ProfileUploadTest::test_daily_stats_count_a_profile_once_every_part_is_in
  • apps/api/tests.py::IngestStartEndpointTest::test_all_test_status_types
  • apps/api/tests.py::IngestStartEndpointTest::test_details_are_kept_for_failures_only
  • apps/api/tests.py::IngestStartEndpointTest::test_ingest_end_after_progress

Documentation

CI vs local

Tests that are much slower in CI, and why

Upload profiles from CI and from a developer machine to compare them.

Locally: FLAKYTEST_SECRET_TOKEN=… pytest --flakytest-profile

Failing and flaky tests

No failures in this window.

Environments

Every CI job and developer machine that reported in the last 7 days
Environment Profiled runs Sessions Avg wall time Peak memory Last seen
CI github-actions:test 4 4 28.25 s 183 MB 10 hours ago
CI github-actions 0 0 – – 2 years, 1 month ago