[Test] Measure where the hash-graph compile spends its time

[lunelson]

Note

This is a throwaway measurement PR. It will be closed without merging. Other workflow runs on it are cancelled on purpose. Only hash-graph compile timings matters.

🌟 Purpose

In the Playwright integration job, compiling hash-graph takes 4.5–5 minutes even though sccache hits 99% of the time. sccache's statistics account for only part of that:

• 1,212 cache hits, adding up to 209 seconds of downloading;
• 13 misses, adding up to 96 seconds of compiling.

It doesn't time the roughly 176 crates it can't cache, such as proc macros, build scripts and the binary crate itself, and it doesn't time linking. This PR measures where the rest of the time goes. It supports the discussion on #9987.

🔍 Changes

A temporary workflow, hash-graph-compile-timings.yml, runs the same compile as hash-graph#compile on the same runner type and sccache bucket. It runs two variants side by side:

• default: today's full debug info, as on main;
• line-tables-only: the setting [Test] Cache the hash-graph debug binary in integration tests #9987 uses.

Each variant:

1. zeroes sccache's statistics, then runs cargo build --bin hash-graph --all-features --timings and prints the statistics and the binary's size;
2. touches main.rs and recompiles only the binary crate with -Z time-passes, to separate code generation from linking;
3. uploads the --timings HTML report, the pass timings and sccache's debug log as artifacts.

Results will be posted here as a comment.

[lunelson]

Results

Run 38278761076. The default variant reproduces the Playwright job's baseline: 1,214 sccache hits, 13 misses, and a compile of 290s (284s in Cargo's own report). The line-tables-only variant isn't a fair speed comparison, because sccache wasn't warm for it (79 misses). The link-timing step failed (cargo rustc needs -p hash-graph in a virtual workspace), but the --timings report covers the same ground.

Where the 284 seconds go (full debug info, as on main)

Time	Duration	What runs
0–44s	44s	Dependency crates, mostly sccache hits
44–142s	98s	The aws-lc-sys build script compiles the AWS-LC C library. Everything that uses TLS (rustls, reqwest, tonic, the AWS SDK, and so the whole graph stack) waits for it
142–226s	84s	A chain of crates that miss sccache on every run: temporalio-common → temporalio-client → hash-temporal-client → hash-graph-store → hash-graph-postgres-store (14s) → hash-graph-atlas (38s), plus hash-graph-api
226–284s	57s	The hash-graph binary crate: compiling it and linking a 2.9 GB binary. sccache never caches this

Why the same 13 crates always miss sccache

temporalio-common 0.5.0 generates code in its build script by iterating over a std::collections::HashSet (build.rs, line 342). Rust randomises hash-set iteration order per process, so the generated payload_visitor_impl.rs lists the same impls in a different order on every build. I confirmed this locally: three clean builds produced three different files with identical lines.

The crate's source therefore changes on every build, so sccache misses it. Every crate that depends on it misses too, because sccache's key includes the dependency's compiled output. That accounts for 12 of the 13 misses, and for most of the 84 seconds above. It affects every CI job that compiles these crates, not just this one.

temporalio-common 1.1.0 fixes it by sorting before iterating. 1.0.0 doesn't have the fix. HASH is on 0.5.0, so this needs a major-version upgrade of temporalio-client and temporalio-common. 1.1.0 was published on 2026-10-06, so global-min-publish-age = "7 days" blocks it until 2026-10-13. Patching 0.5.0 locally with the same one-line sort is an alternative if the upgrade is large.

Binary size

Debug info	Binary size	Binary crate (compile and link)
Full (main)	2.9 GB	57s
line-tables-only (#9987)	1.0 GB	32s

line-tables-only saves about 25s on the binary crate. Even at that setting, #9987 stores a binary of about 1.0 GB in the Turborepo remote cache for each Rust change, which is relevant to its open question about remote cache size.

What this suggests

1. Make temporalio-common deterministic (upgrade to 1.1.0 or patch 0.5.0). That should turn about 80s of recompiling into sccache hits in every job that builds these crates.
2. Look at aws-lc-sys. Its build script is 98s on the critical path even with sccache serving the C objects. The workspace already selects tls-ring for temporalio-client, so it's worth finding what still pulls in aws-lc-rs and whether it can use ring too.
3. Keep line-tables-only for the CI compile: it saves about 25s and shrinks the binary to about a third.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Oct 12, 2026 5:47pm UTC
hashdotdesign-tokens	Ready	Preview	Oct 12, 2026 5:47pm UTC
petrinaut	Ready	Preview	Oct 12, 2026 5:47pm UTC
petrinaut-docs	Ready	Preview	Oct 12, 2026 5:47pm UTC

[lunelson]

Closing; this was a throwaway measurement PR. Its findings (the temporalio miss chain and the aws-lc-sys build script) were addressed in #10006 and #10010, and the remaining compile time is being tackled in #10019.