Skip to content

feat(hotpath): fill sessions ingest/transcript/backfill gaps - #776

Draft
ScriptedAlchemy wants to merge 2 commits into
codex/tracedecay-total-redesign-plan-reopenedfrom
cursor/hotpath-gap-fill-f2b0
Draft

feat(hotpath): fill sessions ingest/transcript/backfill gaps#776
ScriptedAlchemy wants to merge 2 commits into
codex/tracedecay-total-redesign-plan-reopenedfrom
cursor/hotpath-gap-fill-f2b0

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Summary

Gap-fill hotpath coverage in crates/tracedecay-sessions only (HP-sessions scope). 42 new #[hotpath::measure] sites across 10 files, all on persist / ingest / recall / backfill / IO paths. Purely additive: no existing measure sites, gauges, or program logic were touched (diff is 58 inserted attribute lines, 0 deletions). No Cargo.toml changes — the crate's hotpath / hotpath-alloc / hotpath-cpu passthroughs already exist and none of them are reachable from default or production. Macros no-op without --features hotpath.

Rebased onto the current base tip. During the rebase, the base had independently instrumented the four hermes sweep entry points (sessions.hosts.hermes.ingest_projects / ingest_project / ingest_user / ingest_legacy); those base labels were kept and this PR's duplicate attributes for the same functions were dropped. This PR still adds the two hermes discovery-walk measures below.

Verified after rebase: cargo check -p tracedecay-sessions --features hotpath and cargo check -p tracedecay-sessions both pass; no duplicate labels crate-wide; npm run lint:commit passes. Base is codex/tracedecay-total-redesign-plan-reopened (#707, not merged by this PR).

New measure sites

File Function New label
runtime/store_access/transcript.rs get_parse_offset (free fn) sessions.transcript.offset_read
runtime/store_access/transcript.rs set_parse_offset (free fn) sessions.transcript.offset_write
runtime/store_access/transcript.rs upsert_session sessions.transcript.session_upsert
runtime/store_access/transcript.rs get_session_result sessions.transcript.session_read
runtime/store_access/transcript.rs upsert_session_message_in_existing_tx sessions.transcript.message_upsert
runtime/store_access/transcript.rs persist_transcript_offset_result sessions.transcript.offset_commit
runtime/store_access/transcript.rs upsert_transcript_batches_inner sessions.transcript.batch_persist
runtime/store_access/transcript.rs advance_parse_offset_result sessions.transcript.offset_advance
runtime/store_access/transcript.rs replace_parse_offset_result sessions.transcript.offset_replace
runtime/store_access/transcript.rs replace_parse_offset_pair_result sessions.transcript.offset_replace_pair
runtime/hosts/hermes/ingest.rs all_profile_sources sessions.hosts.hermes.discover_profiles
runtime/hosts/hermes/ingest.rs candidate_state_dbs sessions.hosts.hermes.discover_candidates
runtime/workflow/workflow_index.rs ensure_workflow_index_schema sessions.workflow_index.ensure_schema
runtime/workflow/workflow_index.rs read_ingest_watermark sessions.workflow_index.watermark_read
runtime/workflow/workflow_index.rs upsert_run sessions.workflow_index.run_upsert
runtime/workflow/workflow_index.rs upsert_agent sessions.workflow_index.agent_upsert
runtime/workflow/workflow_index.rs runs_for_session sessions.workflow_index.runs_for_session
runtime/workflow/workflow_index.rs run_for_id sessions.workflow_index.run_read
runtime/workflow/workflow_index.rs agents_for_run sessions.workflow_index.agents_for_run
runtime/workflow/workflow_index.rs runs_for_git_scope sessions.workflow_index.runs_for_git_scope
runtime/git_correlation/backfill/history_progress.rs install_final_schema sessions.git_correlation.history_schema
runtime/git_correlation/backfill/history_progress.rs read_progress sessions.git_correlation.history_progress_read
runtime/git_correlation/backfill/history_progress.rs read_oldest_progress sessions.git_correlation.history_progress_oldest
runtime/git_correlation/backfill/history_progress.rs insert_progress sessions.git_correlation.history_progress_insert
runtime/git_correlation/backfill/history_progress.rs compare_and_swap_progress sessions.git_correlation.history_progress_cas
runtime/git_correlation/backfill/history_progress.rs upsert_segment sessions.git_correlation.history_segment_upsert
runtime/git_correlation/backfill/history_progress.rs read_pending_page sessions.git_correlation.history_pending_page
runtime/git_correlation/backfill/bounded.rs run_bounded_history_index_page sessions.git_correlation.backfill.bounded_page
runtime/git_correlation/backfill/bounded.rs stream_git_evidence sessions.git_correlation.backfill.stream_evidence
runtime/git_correlation/backfill/bounded.rs resume_git_evidence sessions.git_correlation.backfill.resume_evidence
runtime/git_correlation/backfill/bounded.rs dry_run_native_history sessions.git_correlation.backfill.dry_run
runtime/git_correlation/backfill/bounded.rs persist_frontier sessions.git_correlation.backfill.frontier_persist
runtime/hosts/cursor_composer/ingest.rs drain_composer_projection_queue sessions.hosts.cursor.composer_projection_drain
runtime/hosts/cursor_composer/ingest.rs advance_composer_coverage sessions.hosts.cursor.composer_coverage_advance
runtime/hosts/cursor_composer/ingest.rs ingest_with_context sessions.hosts.cursor.composer_sweep
runtime/hosts/cursor_composer/ingest.rs ingest_state_vscdb sessions.hosts.cursor.composer_state_vscdb
runtime/hosts/cursor_composer/ingest.rs ingest_chat_store_dbs sessions.hosts.cursor.composer_chat_stores
runtime/hosts/cursor_composer/ingest.rs ingest_one_store_db sessions.hosts.cursor.composer_store_db
runtime/ingest/scheduler.rs read_ingest_frontier sessions.ingest.frontier_read
runtime/ingest/scheduler.rs write_ingest_frontier sessions.ingest.frontier_write
runtime/ingest/scheduler.rs read_codex_discovery_frontier sessions.ingest.codex_frontier_read
runtime/ingest/scheduler.rs write_codex_discovery_frontier sessions.ingest.codex_frontier_write
runtime/hosts/cursor.rs (add-missing) try_ingest_cursor_user_transcript_event_capped_with_admission sessions.hosts.cursor.ingest_user_event_capped
runtime/hosts/cursor.rs (add-missing) admit_cursor_sweep_observations_with_session_ids sessions.hosts.cursor.sweep_admit
runtime/hosts/cursor.rs (add-missing) CursorSweepSource::transcript_paths sessions.hosts.cursor.sweep_discover
runtime/hosts/cursor.rs (add-missing) dispatch_model_for_agent sessions.hosts.cursor.dispatch_model_scan
runtime/source.rs (add-missing) load_transcript_cursor sessions.source.cursor_load
runtime/source.rs (add-missing) read_host_provider_coverage sessions.source.coverage_read
runtime/source.rs (add-missing) persist_host_provider_coverage sessions.source.coverage_persist
runtime/source.rs (add-missing) preflight_strict_jsonl sessions.source.preflight_jsonl
runtime/source.rs (add-missing) read_changed_file sessions.source.read_changed
runtime/source.rs (add-missing) read_changed_with_companion sessions.source.read_changed_companion
runtime/observation/jsonl_observation_admission.rs (add-missing) build_shared_jsonl_page sessions.observation.jsonl_page_build
runtime/observation/jsonl_observation_admission.rs (add-missing) prepare_shared_jsonl_window_with_background_cpu sessions.observation.jsonl_window_prepare
runtime/observation/jsonl_observation_admission.rs (add-missing) shared_jsonl_page_with_cancellation sessions.observation.jsonl_page_lookup
runtime/observation/jsonl_observation_admission.rs (add-missing) advance_coverage sessions.observation.jsonl_coverage_advance
runtime/observation/jsonl_observation_admission.rs (add-missing) capture sessions.observation.jsonl_capture
runtime/observation/jsonl_observation_admission.rs (add-missing) capture_window sessions.observation.jsonl_capture_window

Conventions followed

  • Labels use the crate's established prefixes (sessions.transcript.*, sessions.hosts.hermes.*, sessions.hosts.cursor.*, sessions.git_correlation.*, sessions.workflow_index.*, sessions.ingest.*, sessions.source.*, sessions.observation.*); no duplicate labels crate-wide.
  • Async fns use future = true; sync FS-walk/scan fns use plain #[hotpath::measure].
  • Skipped getters, Display impls, row decoders, thin wrappers, #[cfg(test)] code, and planning-only pure functions.
Open in Web Open in Cursor 

@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d6325c5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Performance Comparison codex/tracedecay-total-redesign-plan-reopenedcursor/hotpath-gap-fill-f2b0

Total Elapsed Time: 5.11s → 5.23s (+2.4%)
CPU Baseline: 87.72µs → 92.66µs (+5.6%)
Benchmark ID: index-bench-timing

timing - Execution duration of functions.

+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| Function                                 | Calls                      | Avg                             | P95                             | Total                            | % Total                      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| tracedecay-index-bench                   | 1 → 1 (+0.0%)              | 5.11s → 5.23s (+2.3%)           | 5.11s → 5.23s (+2.3%)           | 5.11s → 5.23s (+2.3%)            | 100.00% → 100.00% (+0.0%)    |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.append_pages              | 13 → 13 (+0.0%)            | 189.35ms → 193.58ms (+2.2%)     | 263.06ms → 266.86ms (+1.4%)     | 2.46s → 2.52s (+2.4%)            | 48.19% → 48.12% (-0.1%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.sqlite              | 13 → 13 (+0.0%)            | 131.70ms → 140.20ms (+6.5%)     | 181.01ms → 194.90ms (+7.7%)     | 1.71s → 1.82s (+6.4%)            | 33.52% → 34.85% (+4.0%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| CodeIndexProductionOwnerV1::extract_file | 276 → 276 (+0.0%)          | 4.41ms → 4.30ms (-2.5%)         | 6.73ms → 6.57ms (-2.4%)         | 1.22s → 1.19s (-2.5%)            | 23.84% → 22.69% (-4.8%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| code_index.workers.install               | 83 → 83 (+0.0%)            | 15.09ms → 14.30ms (-5.2%)       | 51.68ms → 50.33ms (-2.6%)       | 1.25s → 1.19s (-4.8%)            | 24.53% → 22.69% (-7.5%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.postings            | 13 → 13 (+0.0%)            | 87.93ms → 87.50ms (-0.5%)       | 124.32ms → 120.00ms (-3.5%)     | 1.14s → 1.14s (+0.0%)            | 22.38% → 21.75% (-2.8%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| code_index.build.and_publish             | 2 → 2 (+0.0%)              | 474.01ms → 470.90ms (-0.7%)     | 478.41ms → 472.38ms (-1.3%)     | 948.02ms → 941.81ms (-0.7%)      | 18.56% → 18.01% (-3.0%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| domain.canonical.sha256                  | 114526 → 114526 (+0.0%)    | 8.20µs → 8.11µs (-1.1%)         | 13.46µs → 13.50µs (+0.3%)       | 938.59ms → 928.34ms (-1.1%)      | 18.37% → 17.75% (-3.4%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.finalization.advance_wake | 14 → 14 (+0.0%)            | 55.73ms → 59.51ms (+6.8%)       | 244.58ms → 250.74ms (+2.5%)     | 780.18ms → 833.12ms (+6.8%)      | 15.27% → 15.93% (+4.3%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.commit              | 13 → 13 (+0.0%)            | 40.40ms → 49.34ms (+22.1%) ⚠️   | 55.18ms → 70.91ms (+28.5%) ⚠️   | 525.16ms → 641.40ms (+22.1%) ⚠️  | 10.28% → 12.27% (+19.4%)     |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.prepare_pages             | 13 → 13 (+0.0%)            | 49.67ms → 45.40ms (-8.6%)       | 71.17ms → 67.08ms (-5.7%)       | 645.65ms → 590.23ms (-8.6%)      | 12.64% → 11.29% (-10.7%)     |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| code_index.chunk.index_file              | 276 → 276 (+0.0%)          | 2.10ms → 2.02ms (-3.8%)         | 3.10ms → 2.93ms (-5.5%)         | 580.09ms → 557.10ms (-4.0%)      | 11.36% → 10.65% (-6.2%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| code_index.extract.parser_artifact       | 276 → 276 (+0.0%)          | 1.88ms → 1.86ms (-1.1%)         | 2.85ms → 2.82ms (-1.1%)         | 517.76ms → 512.35ms (-1.0%)      | 10.14% → 9.80% (-3.4%)       |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| query.artifact.batch.parallel_prepare    | 13 → 13 (+0.0%)            | 40.20ms → 35.98ms (-10.5%)      | 58.92ms → 53.44ms (-9.3%)       | 522.55ms → 467.79ms (-10.5%)     | 10.23% → 8.95% (-12.5%)      |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| 🆕 code_index.build.assemble             | 0 → 2 (+100.0%) ⚠️         | 0.00ns → 172.72ms (+100.0%) ⚠️  | 0.00ns → 183.89ms (+100.0%) ⚠️  | 0.00ns → 345.45ms (+100.0%) ⚠️   | 0.00% → 6.61% (+100.0%) ⚠️   |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+
| 🗑️ code_index.chunk.build                | 276 → 0 (-100.0%) 🚀       | 1.29ms → 0.00ns (-100.0%) 🚀    | 1.86ms → 0.00ns (-100.0%) 🚀    | 356.98ms → 0.00ns (-100.0%) 🚀   | 6.99% → 0.00% (-100.0%) 🚀   |
+------------------------------------------+----------------------------+---------------------------------+---------------------------------+----------------------------------+------------------------------+

Generated with hotpath-rs

Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
@cursor
cursor Bot force-pushed the cursor/hotpath-gap-fill-f2b0 branch from 089ef04 to efb8b56 Compare August 31, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants