Skip to content

[Test] Cache the hash-graph debug binary in integration tests - #9987

Closed
lunelson wants to merge 1 commit into
mainfrom
ln/speed-up-playwright-ci
Closed

lunelson wants to merge 1 commit into
mainfrom
ln/speed-up-playwright-ci

Conversation

@lunelson

@lunelson lunelson commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is a test PR, not ready to merge. It measures whether caching the hash-graph debug binary is safe and worthwhile. Enabling this caching is now tracked in SRE-1174 (internal); this PR is the experiment behind it.

🌟 Purpose

The Playwright integration job took 18–25 minutes on pull requests. Most of that was not installing dependencies or Playwright (the browser install takes about 8 seconds). Each run compiles the hash-graph binary before any service starts, and hash-graph#compile is cache: false. That compile took 4.5–5 minutes even though sccache hits about 99% of the time. The hash-graph-integration, hash-graph-benches, @tests/hash-graph-http and @tests/hash-backend-integration jobs pay the same cost. #10005 measured where those minutes go.

Three changes on main since this PR was opened address parts of that at the source:

This PR has been rebased onto that main (0879fe517c) and now contains only what remains on top: turning caching on, with the binary as an output. With all three in, a cold compile takes about 3 minutes, and caching the binary cuts that to 11–16 seconds (see the results below).

10019 (SRE-1171) is complementary: it moves hash-graph's code into a library crate that sccache can cache, which should make the cold compile faster still. This PR saves the whole compile when the graph's Rust inputs are unchanged; #10019 speeds up the compile when they aren't.

Timings from six PR runs of Integration (@tests/hash-playwright) before any of these changes:

Step Time
Start background tasks (compile hash-graph, then wait for graph, API and frontend) 430–815s
Run tests 210–420s
Clean up disk 50–136s
Start Docker services ~65s
yarn install ~60s
Install tools ~33s
turbo prune ~32s
Install Playwright browser ~8s

🔍 Changes

hash-graph#compile is cached, with target/debug/hash-graph as its only output. The build itself is unchanged. Its hash covers:

  • every workspace crate the binary links, through the ^manifest dependency it inherits from the root turbo.json (SRE-1173: Depend on ^manifest in compile and compile:release #10007);
  • the external crates in Cargo.lock. Turborepo's Cargo workspace support already hashes these per package; I confirmed it by changing tokio's version, which changed the hash;
  • added here: the root Cargo.toml, which holds the profiles and the workspace dependency versions. On main an edit to it doesn't change the compile's hash, which only becomes a problem once the binary is cached;
  • added here: CARGO_PROFILE_* and RUSTFLAGS, which change the binary if anyone sets them;
  • added here: RUNNER_OS and RUNNER_ARCH. Turborepo doesn't hash the platform, so without these a binary from another OS or architecture could be restored.

apps/hash-graph/docs/task-dependencies.json is regenerated with turbo run //#doc:task-dependencies. The root compile: { cache: false } default stays for other packages, and compile:release is unchanged.

An earlier version of this PR also set CARGO_PROFILE_DEV_DEBUG=line-tables-only. It was removed: it is a separate trade-off that hadn't been discussed. Debuggers lose variable and type information, and sccache would store a second copy of every crate, because other jobs still build with full debug info.

❓ Open questions for infra

  1. Does this match what SRE-1174 intends? SRE-1173: Depend on ^manifest in compile and compile:release #10007 explains that an earlier attempt failed in the merge queue because a cache hit restored no binary. Declaring target/debug/hash-graph as the output is meant to fix exactly that. Alternatives would be building the binary once per workflow and sharing it as an artifact, or running the graph from its Docker image.
  2. Cache hits skip side effects. SRE-1061: Move the crates' package.json scripts into turbo.json commands #9755 keeps side-effect tasks uncached for this reason. Declaring the binary as an output covers the binary, but cargo build also fills target/ for later Cargo commands in the same job. With a cache hit, cargo test in the hash-graph-integration and hash-graph-benches jobs no longer gets those artifacts. This only matters when test:integration misses the Turborepo cache, and sccache still covers it. Is that trade-off acceptable?
  3. Remote cache size. With full debug info the binary is 2.9 GB (measured in [Test] Measure where the hash-graph compile spends its time #10005), and every change to Rust code uploads a new one to the Vercel remote cache. It uploads and restores without errors, and a restore takes 13–16 seconds (see the results below), so the question is only storage and bandwidth. Is there a budget to respect? If this is too large, the options I can see are:
    • compressing the debug sections only for the final link;
    • keeping the debug info out of the binary in separate files;
    • reducing debug info for this compile (line-tables-only brings it to 1.0 GB), with the drawbacks described above.
  4. Inputs that might be missing. Are there inputs the hash doesn't see that could make a cached binary stale? For example, environment variables read by build scripts, or C libraries that -sys crates link from the system.
  5. A later cache miss now costs a full rebuild. In the warm re-run on current main, the @tests/hash-backend-integration job restored the binary in the "Start background tasks" step, but its "Run tests" step ran turbo again and got a cache miss for the same hash (a6de408921a564de). Cargo then rebuilt the whole binary in 3m48s (with sccache hits). On main today the earlier compile leaves Cargo's build state in target/, so a second compile finishes in seconds; with this PR only the binary is restored, so Cargo starts from nothing. This happened in 1 of the 6 backend-integration runs I have logs for. I couldn't find the cause because the workflow sets TURBO_GLOBAL_WARNING_DISABLED=1; a failed or timed-out download of the 2.9 GB artifact, which Turborepo reports as a miss, seems most likely. Possible fixes:
    • stop later steps from re-checking compile, for example by running the test step with --only;
    • make the artifact smaller, which runs into the debug-info trade-off in question 3;
    • turn Turborepo warnings back on so the cause shows up next time.

🧪 How this is being tested

The current commit e3018a6c43 passed a cold run and a warm re-run against main at 0879fe517c, which includes #10006, #10007 and #10010 (see "Results on current main" below). The checks below ran earlier on commit b2c22ea6c5, against main at 615fca1e76, before #10010 merged.

  1. ✅ First run (cold cache): all test jobs passed. hash-graph#compile missed the cache in every graph job and compiled in 3m26s–4m06s. sccache missed only 2 crates, down from 13, so BE-981: Update temporalio-client and temporalio-common to 1.1.0 #10006 fixed the miss chain. This compile time is what caching saves on each run, before SRE-1172: Link a cached prebuilt AWS-LC in CI instead of compiling it every run #10010.
  2. ✅ Re-run of the same commit (warm cache): every graph job restored the binary from the remote cache in 11.4–14.5 seconds, sccache executed no compiles, and all tests passed.
  3. ✅ Stale-binary check: test commit a7054110a6 changes only a doc comment in libs/@local/graph/store/rust/src/lib.rs, a crate hash-graph depends on. All six graph jobs missed the cache, recompiled hash-graph-store and the binary (3m41s–4m38s), and passed. That commit will be reverted before this PR leaves draft.

📊 Results on current main

Test run 38395647308 on commit e3018a6c43, with #10006, #10007 and #10010 on main: attempt 1 ran on a cold cache, and attempt 2 re-ran every job on a warm cache. Both passed. In attempt 1 sccache missed only 2 crates and the prebuilt AWS-LC was used. In attempt 2 every graph job reported 1 cached, 1 total for hash-graph#compile. The Playwright job wasn't selected, because the commit doesn't touch any file its tests depend on.

"Start background tasks" step, in seconds:

Job Attempt 1 (cold) Attempt 2 (warm) Compile, cold → warm
hash-graph-integration 233 40 3m10s → 12.4s
hash-graph-benches 245 41 3m14s → 14.7s
@tests/hash-graph-http 272 64 3m20s → 16.4s
@tests/hash-backend-integration 360 188 2m57s → 11.2s
  • The backend job's warm 188 seconds is mostly not compilation. After the 11-second restore, hash-api ran its data migrations for about 2 minutes 10 seconds before it started listening. That happens with or without this PR.
  • The same job's "Run tests" step missed the cache and rebuilt the binary in 3m48s, as described in open question 5. The other three jobs had no misses.

📊 Results after the first rebase

These came from commit b2c22ea6c5, before #10010 was on main.

Test run 38388077399: attempt 1 ran on a cold cache, and attempt 2 re-ran every job on a warm cache. The Playwright job wasn't selected in this run, because the commit doesn't touch any file its tests depend on.

"Start background tasks" step, in seconds:

Job Attempt 1 (cold) Attempt 2 (warm) Compile, cold → warm
hash-graph-benches 327 38 4m06s → 11.4s
@tests/hash-graph-http 267 39 3m26s → 12.8s
hash-graph-integration 274 56 3m40s → 13.4s
@tests/hash-backend-integration 412 178 3m45s → 14.5s

📊 Results before the rebase

These came from commit 9ea01d7c90, before #10006 and #10007 were on main, with full debug info. The cache behaviour should carry over; the compile time it saves is expected to be smaller now that #10006 is in. Test run 38280632661: attempt 1 ran on a cold cache, attempt 2 re-ran every job on a warm cache.

Every job that compiles the graph restored the binary from the remote cache in attempt 2. All five reported 1 cached, 1 total for hash-graph#compile, and sccache executed no compiles. Restoring the 2.9 GB binary took 12.8–16 seconds. In attempt 1 the Playwright job compiled it in 4m39s (a cache miss), as expected.

"Start background tasks" step, in seconds (compile hash-graph, then wait for the services):

Job Before this PR Attempt 1 (cold) Attempt 2 (warm)
@tests/hash-playwright 430–815 444 158
@tests/hash-backend-integration 844 (one sample) 440 185
hash-graph-integration 269–549 382 64
hash-graph-benches 284–537 338 57
@tests/hash-graph-http 487 (one sample) 378 43

The whole Playwright job took 774s in attempt 2, against 1,073–1,494s before this PR.

  • "Run tests" also dropped in the Rust jobs, but not because of this PR. On a re-run of the same commit their test:integration tasks hit the Turborepo cache too. Only the "Start background tasks" column measures this change.
  • What remains in the Playwright job's step is mostly hash-api starting (about 1 minute 45 seconds in an earlier run). It is the next candidate to look at, outside this PR.

@lunelson lunelson added area/infra Relates to version control, CI, CD or IaC (area) meta/DO NOT MERGE DO NOT MERGE. May be used in combination with `state/blocked` labels Oct 9, 2026
@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Oct 13, 2026 12:17pm UTC
petrinaut Ready Ready Preview Oct 13, 2026 12:17pm UTC
petrinaut-docs Ready Ready Preview Oct 13, 2026 12:17pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Oct 13, 2026 12:17pm UTC

Request Review

@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.97%. Comparing base (136b588) to head (e3018a6).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9987   +/-   ##
=======================================
  Coverage   67.97%   67.97%           
=======================================
  Files        1946     1946           
  Lines      201776   201776           
  Branches     8317     8317           
=======================================
  Hits       137164   137164           
  Misses      62987    62987           
  Partials     1625     1625           
Flag Coverage Δ
hash-graph 21.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Oct 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ 6 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

✅ 98 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚠️ as_constant < 1 ns < 1 ns N/A
⚠️ constant_equal < 1 ns < 1 ns N/A
⚠️ constant_not_equal < 1 ns < 1 ns N/A
⚠️ access < 1 ns < 1 ns N/A
⚠️ runtime_equal < 1 ns < 1 ns N/A
⚠️ runtime_not_equal < 1 ns < 1 ns N/A

Comparing ln/speed-up-playwright-ci (1360eec) with main (51755b6)

Open in CodSpeed

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark results

hash-graph-benches – Integrations

policy_resolution_large

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2002 $$18.8 \mathrm{ms} \pm 156 \mathrm{μs}\left({\color{red}7.19 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$2.22 \mathrm{ms} \pm 17.4 \mathrm{μs}\left({\color{gray}2.66 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 1002 $$8.55 \mathrm{ms} \pm 79.3 \mathrm{μs}\left({\color{gray}1.76 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 3314 $$26.8 \mathrm{ms} \pm 237 \mathrm{μs}\left({\color{red}5.94 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$8.61 \mathrm{ms} \pm 79.8 \mathrm{μs}\left({\color{gray}0.199 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 1527 $$15.3 \mathrm{ms} \pm 122 \mathrm{μs}\left({\color{red}8.81 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 2078 $$20.3 \mathrm{ms} \pm 220 \mathrm{μs}\left({\color{red}11.2 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.39 \mathrm{ms} \pm 13.8 \mathrm{μs}\left({\color{gray}-0.735 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 1033 $$9.35 \mathrm{ms} \pm 77.0 \mathrm{μs}\left({\color{gray}4.72 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_medium

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 102 $$2.41 \mathrm{ms} \pm 19.4 \mathrm{μs}\left({\color{gray}1.91 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.95 \mathrm{ms} \pm 15.7 \mathrm{μs}\left({\color{gray}3.27 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 52 $$2.15 \mathrm{ms} \pm 13.0 \mathrm{μs}\left({\color{gray}-1.382 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 269 $$3.33 \mathrm{ms} \pm 23.2 \mathrm{μs}\left({\color{gray}1.46 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$2.33 \mathrm{ms} \pm 21.7 \mathrm{μs}\left({\color{gray}0.658 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 108 $$2.68 \mathrm{ms} \pm 21.3 \mathrm{μs}\left({\color{gray}3.94 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 133 $$2.89 \mathrm{ms} \pm 27.7 \mathrm{μs}\left({\color{gray}-0.560 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$2.23 \mathrm{ms} \pm 16.0 \mathrm{μs}\left({\color{gray}3.22 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 63 $$2.68 \mathrm{ms} \pm 19.5 \mathrm{μs}\left({\color{red}6.72 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_none

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 2 $$1.81 \mathrm{ms} \pm 16.3 \mathrm{μs}\left({\color{gray}4.87 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.82 \mathrm{ms} \pm 28.6 \mathrm{μs}\left({\color{red}8.29 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 2 $$1.82 \mathrm{ms} \pm 15.8 \mathrm{μs}\left({\color{gray}-1.703 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 8 $$1.98 \mathrm{ms} \pm 13.7 \mathrm{μs}\left({\color{gray}2.86 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$1.86 \mathrm{ms} \pm 11.9 \mathrm{μs}\left({\color{gray}2.37 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 3 $$2.04 \mathrm{ms} \pm 15.9 \mathrm{μs}\left({\color{gray}1.36 \mathrm{\%}}\right) $$ Flame Graph

policy_resolution_small

Function Value Mean Flame graphs
resolve_policies_for_actor user: empty, selectivity: high, policies: 52 $$1.98 \mathrm{ms} \pm 16.0 \mathrm{μs}\left({\color{gray}2.28 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: low, policies: 1 $$1.78 \mathrm{ms} \pm 13.4 \mathrm{μs}\left({\color{gray}-3.910 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: empty, selectivity: medium, policies: 26 $$1.87 \mathrm{ms} \pm 12.7 \mathrm{μs}\left({\color{gray}-2.054 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: high, policies: 94 $$2.22 \mathrm{ms} \pm 14.6 \mathrm{μs}\left({\color{gray}-1.450 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: low, policies: 1 $$1.91 \mathrm{ms} \pm 9.18 \mathrm{μs}\left({\color{gray}-3.850 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: seeded, selectivity: medium, policies: 27 $$2.15 \mathrm{ms} \pm 13.2 \mathrm{μs}\left({\color{gray}3.74 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: high, policies: 66 $$2.15 \mathrm{ms} \pm 15.4 \mathrm{μs}\left({\color{gray}-4.357 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: low, policies: 1 $$1.95 \mathrm{ms} \pm 12.0 \mathrm{μs}\left({\color{gray}-0.278 \mathrm{\%}}\right) $$ Flame Graph
resolve_policies_for_actor user: system, selectivity: medium, policies: 29 $$2.07 \mathrm{ms} \pm 19.6 \mathrm{μs}\left({\color{gray}-0.434 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_complete

Function Value Mean Flame graphs
entity_by_id;one_depth 1 entities $$22.5 \mathrm{ms} \pm 99.5 \mathrm{μs}\left({\color{gray}0.577 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 10 entities $$47.6 \mathrm{ms} \pm 323 \mathrm{μs}\left({\color{gray}2.47 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 25 entities $$23.8 \mathrm{ms} \pm 115 \mathrm{μs}\left({\color{gray}-2.574 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 5 entities $$28.6 \mathrm{ms} \pm 189 \mathrm{μs}\left({\color{gray}2.87 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;one_depth 50 entities $$27.9 \mathrm{ms} \pm 173 \mathrm{μs}\left({\color{gray}2.30 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 1 entities $$24.2 \mathrm{ms} \pm 153 \mathrm{μs}\left({\color{gray}0.294 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 10 entities $$272 \mathrm{ms} \pm 1.40 \mathrm{ms}\left({\color{gray}-0.319 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 25 entities $$55.7 \mathrm{ms} \pm 501 \mathrm{μs}\left({\color{red}5.53 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 5 entities $$58.9 \mathrm{ms} \pm 554 \mathrm{μs}\left({\color{gray}-1.062 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;two_depth 50 entities $$168 \mathrm{ms} \pm 835 \mathrm{μs}\left({\color{gray}-2.316 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 1 entities $$7.02 \mathrm{ms} \pm 32.3 \mathrm{μs}\left({\color{gray}-1.087 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 10 entities $$7.14 \mathrm{ms} \pm 37.6 \mathrm{μs}\left({\color{gray}-2.864 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 25 entities $$7.29 \mathrm{ms} \pm 39.3 \mathrm{μs}\left({\color{gray}0.999 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 5 entities $$7.41 \mathrm{ms} \pm 66.1 \mathrm{μs}\left({\color{gray}1.58 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id;zero_depth 50 entities $$7.23 \mathrm{ms} \pm 48.3 \mathrm{μs}\left({\color{gray}0.115 \mathrm{\%}}\right) $$ Flame Graph

read_scaling_linkless

Function Value Mean Flame graphs
entity_by_id 1 entities $$7.02 \mathrm{ms} \pm 31.9 \mathrm{μs}\left({\color{gray}-1.447 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$7.30 \mathrm{ms} \pm 50.9 \mathrm{μs}\left({\color{gray}4.49 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 100 entities $$7.06 \mathrm{ms} \pm 34.4 \mathrm{μs}\left({\color{gray}-3.328 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1000 entities $$7.15 \mathrm{ms} \pm 50.7 \mathrm{μs}\left({\color{gray}0.111 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10000 entities $$7.27 \mathrm{ms} \pm 42.1 \mathrm{μs}\left({\color{gray}-3.086 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity

Function Value Mean Flame graphs
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1 $$7.44 \mathrm{ms} \pm 49.7 \mathrm{μs}\left({\color{gray}-1.063 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1 $$7.53 \mathrm{ms} \pm 46.3 \mathrm{μs}\left({\color{gray}1.61 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1 $$7.53 \mathrm{ms} \pm 46.1 \mathrm{μs}\left({\color{gray}1.32 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1 $$7.53 \mathrm{ms} \pm 53.0 \mathrm{μs}\left({\color{gray}1.75 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2 $$7.54 \mathrm{ms} \pm 67.0 \mathrm{μs}\left({\color{gray}1.16 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1 $$7.50 \mathrm{ms} \pm 59.0 \mathrm{μs}\left({\color{gray}2.03 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1 $$7.32 \mathrm{ms} \pm 32.6 \mathrm{μs}\left({\color{gray}-2.306 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1 $$7.43 \mathrm{ms} \pm 41.7 \mathrm{μs}\left({\color{gray}-0.426 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1 $$7.54 \mathrm{ms} \pm 59.3 \mathrm{μs}\left({\color{gray}0.793 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity_type

Function Value Mean Flame graphs
get_entity_type_by_id Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba $$5.54 \mathrm{ms} \pm 30.6 \mathrm{μs}\left({\color{gray}-2.636 \mathrm{\%}}\right) $$ Flame Graph

representative_read_multiple_entities

Function Value Mean Flame graphs
entity_by_property traversal_paths=0 0 $$39.6 \mathrm{ms} \pm 354 \mathrm{μs}\left({\color{gray}2.50 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$76.5 \mathrm{ms} \pm 417 \mathrm{μs}\left({\color{gray}2.71 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$49.3 \mathrm{ms} \pm 298 \mathrm{μs}\left({\color{red}16.5 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$57.8 \mathrm{ms} \pm 467 \mathrm{μs}\left({\color{red}16.8 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$63.3 \mathrm{ms} \pm 611 \mathrm{μs}\left({\color{red}9.61 \mathrm{\%}}\right) $$
entity_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$61.1 \mathrm{ms} \pm 465 \mathrm{μs}\left({\color{gray}-2.398 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=0 0 $$29.8 \mathrm{ms} \pm 291 \mathrm{μs}\left({\color{gray}-0.924 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=255 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true $$47.7 \mathrm{ms} \pm 382 \mathrm{μs}\left({\color{gray}-2.874 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false $$33.2 \mathrm{ms} \pm 250 \mathrm{μs}\left({\color{lightgreen}-7.254 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true $$38.9 \mathrm{ms} \pm 277 \mathrm{μs}\left({\color{gray}-4.539 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true $$40.7 \mathrm{ms} \pm 227 \mathrm{μs}\left({\color{gray}-2.226 \mathrm{\%}}\right) $$
link_by_source_by_property traversal_paths=2 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true $$40.4 \mathrm{ms} \pm 232 \mathrm{μs}\left({\color{gray}-3.974 \mathrm{\%}}\right) $$

scenarios

Function Value Mean Flame graphs
full_test query-limited $$80.5 \mathrm{ms} \pm 461 \mathrm{μs}\left({\color{gray}1.66 \mathrm{\%}}\right) $$ Flame Graph
full_test query-unlimited $$86.3 \mathrm{ms} \pm 495 \mathrm{μs}\left({\color{red}5.06 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-limited $$12.1 \mathrm{ms} \pm 153 \mathrm{μs}\left({\color{lightgreen}-14.332 \mathrm{\%}}\right) $$ Flame Graph
linked_queries query-unlimited $$341 \mathrm{ms} \pm 1.66 \mathrm{ms}\left({\color{gray}0.529 \mathrm{\%}}\right) $$ Flame Graph

@lunelson

Copy link
Copy Markdown
Contributor Author

Closing without merging. Caching only the binary is too fragile: in the last warm re-run a later turbo step missed the cache for the same hash and did a full rebuild, because Cargo's build state isn't restored. It also leaves Docker reuse, architecture skew and change detection open (see open question 5). The direction instead is to make the compile itself fast enough, through #10006, #10010 and #10019, and to run steps in parallel. The measurements in the description are a baseline for that work: a cold compile on main at 0879fe517c takes about 3 minutes.

@lunelson lunelson closed this Oct 13, 2026
@lunelson
lunelson deleted the ln/speed-up-playwright-ci branch October 13, 2026 13:49

This branch was successfully deployed

5 active (2 outdated) deployments
Preview – petrinaut-docs — e3018a6c Deployed Oct 13, 2026 by vercel[bot]
Preview – hash — e3018a6c Deployed Oct 13, 2026 by vercel[bot]
Preview – petrinaut — e3018a6c Deployed Oct 13, 2026 by vercel[bot]
pull-request — a7054110 Deployed Oct 13, 2026 by lunelson via Sourcemaps (@apps/hash-integration-worker) #39637
Preview – hashdotdesign-tokens — 9ea01d7c Deployed Oct 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-graph area/apps meta/DO NOT MERGE DO NOT MERGE. May be used in combination with `state/blocked` type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

1 participant