feat(scratch): mount the cache, lock the slot, set the env var - #412
Closed
raphaelvigee wants to merge 2 commits into
Closed
raphaelvigee wants to merge 2 commits into
raphaelvigee wants to merge 2 commits into
Conversation
This was referenced Aug 29, 2026
Makes a scratch declaration do something. A referenced cache is now materialized into the sandbox, guarded against concurrent use, and handed to the tool through the variable its declaration names — so `scratch = ["//build:gocache"]` is the whole of what a consumer writes. Mounting is one `symlink(2)` per target pointing out of the sandbox at the canonical slot directory. That is the entire mechanism, and it is why a scratch costs an inode per target rather than a copy: teardown removes the link, not the tree, since `remove_dir_all` does not follow symlinks — the same property read-only input staging already relies on for the Go SDK. Measurements on a Go corpus are what rule out the copy: seeding a warm cache into each sandbox cut `go list` CPU by 60% and moved wall time by exactly zero, because the cost was never CPU but the half-thousand filesystem entries created and destroyed per sandbox. The link target is the *canonical* path, not something sandbox-local. Tools bake absolute paths into their cache entries, so if every consumer saw its own path the cache would restore and be inert — present, and useless. The symlink is created by the bridge rather than the engine because the bridge owns sandbox creation: the FUSE path may redirect the package dir into a mount, so there is no earlier moment at which the directory reliably exists. Putting it there also covers both sandbox modes at once. A scratch that would land where an input already did is a hard error. This is the one way a scratch can cause a *wrong build* rather than a slow one — the target would read cache contents where it believes it reads a declared dependency, bytes no `hashin` describes — so it must fail rather than silently win. Locking is a keyed cross-process reader/writer lock per slot, `access` deciding which guard. It ships in the same change as `access` deliberately: an `exclusive` that does not serialize is a silent lie. Guards are taken in sorted slot order so two targets naming the same pair in opposite orders cannot deadlock, and the acquire sits after dependency resolution but before the worker permit — after deps, or a dep needing the same slot could never get it; before the permit, so a target queued on a contended slot holds no worker, which also makes the wait provably bounded rather than circular. Slot identity is `(addr, version, platform-components)`. `platform = "any"` therefore gives one slot for every machine, which is what will let a portable cache travel between a Linux CI runner and a macOS laptop. ABI: `RunRequest`/`ManagedRunRequest` gain `repeated ScratchMount scratch`, and `ABI_SEMVER` goes 0.5.0 -> 0.6.0. Additive and cold-path — a prost wire field, not a vtable change — so an old plugin decodes a new host's request and ignores the mounts. Its targets then run without a scratch, which costs a cold cache and never a wrong build: the lock is keyed on a declaration an old plugin cannot see, so there is no shared directory for it to race either. The cdylib path is not optional here, since plugin-go is a cdylib and is the reason this exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
raphaelvigee
force-pushed
the
raphaelvigee/scratch-refs
branch
from
August 29, 2026 15:34
3af14f3 to
4680aba
Compare
raphaelvigee
force-pushed
the
raphaelvigee/scratch-mount
branch
from
August 29, 2026 15:34
4573c40 to
1625864
Compare
raphaelvigee
marked this pull request as ready for review
August 29, 2026 15:34
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of four (stack #408), on top of #407. This is where a scratch declaration
starts doing something: a referenced cache is materialized into the sandbox,
guarded against concurrent use, and handed to the tool through the variable its
declaration names.
After this, the whole surface a consumer writes is:
Mounting is one symlink
Per target, pointing out of the sandbox at the canonical slot directory. That is
the entire mechanism, and it is why a scratch costs an inode rather than a copy:
teardown removes the link, not the tree, since
remove_dir_alldoes not followsymlinks — the same property read-only input staging already relies on for the
11k-file Go SDK.
The measurements are what rule out the obvious alternative. Seeding a warm cache
into each sandbox cut
go listCPU 778s → 309s and moved wall time byexactly zero (interleaved A/B: 217.3s vs 219.5s), because the cost was never
CPU — it was the ~500 filesystem entries created and destroyed per sandbox.
The link target is the canonical path, not something sandbox-local. Tools bake
absolute paths into their cache entries (Go's action IDs are the standing
example), so if every consumer saw its own path the cache would restore and be
inert — present, and useless.
The bridge creates it, not the engine. The bridge owns sandbox creation: the
FUSE path may redirect the package dir into a mount, so there is no earlier moment
at which the directory reliably exists. Putting it there covers both sandbox modes
at once, and
sandboxfuseimplementsmkdir/symlinkthrough to the upper dir.The one wrong-build guard
A scratch that would land where an input already did is a hard error. This is
the only way a scratch can cause a wrong build rather than a slow one: the target
would read cache contents where it believes it reads a declared dependency — bytes
no
hashindescribes. Silently winning either way is not an option.Locking ships with
access, deliberatelyA keyed cross-process reader/writer lock per slot,
accesschoosing the guard. Itis in this PR and not a later one because an
exclusivethat does not serialize isa silent lie.
Two ordering rules, both mirroring what
executealready does:opposite orders cannot deadlock.
After deps, or a dep needing the same slot could never get it — the same diamond
the worker permit is already ordered to avoid. Before the permit, so a target
queued on a contended slot holds no worker; and since the lock is always taken
first, no permit holder is ever blocked on a slot, which makes the wait provably
bounded rather than circular.
Slot identity
(addr, version, platform-components). Soplatform = "any"gives one slot forevery machine, which is what will let a portable cache travel between a Linux CI
runner and a macOS laptop once the remote lands.
Compatibility:
ABI_SEMVER0.5.0 → 0.6.0RunRequest/ManagedRunRequestgainrepeated ScratchMount scratch. Additive andcold-path — a prost wire field, not a vtable change — so an old plugin decodes a
new host's request and ignores the mounts. Its targets then run without a scratch,
which costs a cold cache and never a wrong build: the lock is keyed on a
declaration an old plugin cannot see, so there is no shared directory for it to
race either. Minor, not major: no plugin must be rebuilt, but one must be to
mount a scratch.
The cdylib path is not optional here —
plugin-gois a cdylib and is the reasonthis feature exists.
An earlier revision of the design claimed mounting needed no ABI surface at all, on
the theory that the resolved mount would ride on
RunInput.annotations. It cannot:link.rs:36filtersruntime: falseinputs out ofLinkedTargetDef, which isexactly what makes a scratch a pure graph edge — and therefore also why it never
becomes a
RunInput. Corrected indocs/SCRATCH.md.Tests
12 more, 54 total across the stack. The load-bearing one is
a_scratch_carries_state_between_runs: run 1 sees an empty cache and writes amarker, run 2 (a genuine re-execute, not a hit) reads it back — through the real
sandbox, symlink and env var.
Worth knowing if you write another test here: an
EResultholds a riding readlock on its addr, so keeping one alive across
reopen()deadlocks the secondengine's write lock against it. It only bites when the second run re-executes; a
cache hit takes a read lock and coexists. It cost ~30 minutes of hang and surfaced
as an unrelated
ENOENTon a lock file.Stack created with GitHub Stacks CLI • Give Feedback 💬