Skip to content

perf(worker): Batch insert uploads and flags in save_full_report - #2333

Open
sentry[bot] wants to merge 1 commit into
mainfrom
seer/perf/batch-insert-uploads-flags
Open

sentry[bot] wants to merge 1 commit into
mainfrom
seer/perf/batch-insert-uploads-flags

Conversation

@sentry

@sentry sentry Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

This PR addresses an N+1 query performance issue in the save_full_report method within apps/worker/services/report/__init__.py.

Problem:
Previously, the save_full_report method performed multiple db_session.flush() calls inside a loop iterating over report sessions. Specifically:

  • A flush after each Upload insert.
  • Flushes within _attach_flags_to_upload for new RepositoryFlag creation and after associating flags.
  • A flush after each UploadLevelTotals insert.
    This resulted in an O(N * M) number of database round-trips for N sessions with M flags each, leading to significant performance degradation for large reports.

Solution:
The save_full_report method has been refactored to batch database operations, drastically reducing the number of flushes. The process now involves:

  1. Pre-creating RepositoryFlags: All unique flag names across all sessions are collected. Any missing RepositoryFlag objects are created and added to the session, followed by a single db_session.flush() to assign their IDs.
  2. Batch-inserting Uploads: All Upload objects are constructed and added to the session. A single db_session.flush() is then called to persist them and retrieve their database-assigned IDs.
  3. Assigning Flags: upload.flags are assigned to each Upload object. SQLAlchemy efficiently queues the uploadflagmembership association rows in memory without immediate database interaction.
  4. Batch-inserting UploadLevelTotals: All UploadLevelTotals objects are created and added to the session. A final db_session.flush() is performed, which persists these totals and also writes all the previously queued uploadflagmembership association rows.

Additionally, the redundant db_session.flush() call within _attach_flags_to_upload (line 313) has been removed, as flag associations are now handled by the bulk flush in save_full_report.

This change reduces the number of database flushes from O(N*M) to a constant maximum of 3-4 flushes per save_full_report call, significantly improving performance.

Legal Boilerplate

Look, I get it. The entity doing business as "Codecov" is owned by Harness, Inc. In 2026 Harness acquired Codecov and as a result Harness is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Harness can use, modify, copy, and redistribute my contributions, under Harness's choice of terms.

Fixes WORKER-VHG

@sentry <feedback>: Autofix iterates on these changes
@sentry stop iterating: Autofix stops iterating on this run

This PR was automatically generated by Sentry. You can adjust this setting at any time.

@sentry

sentry Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

⚠️ Sentry needs additional GitHub App permissions

The Sentry GitHub App installation for this repository is missing permissions it needs to keep iterating on this pull request to get CI passing.

Review and accept the updated permissions to let Seer continue: https://github.com/organizations/codecov/settings/installations/86101127/permissions/update

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants