Skip to content

feat(scratch): mount the cache, lock the slot, set the env var - #412

Closed
raphaelvigee wants to merge 2 commits into
raphaelvigee/scratch-refsfrom
raphaelvigee/scratch-mount
Closed

raphaelvigee wants to merge 2 commits into
raphaelvigee/scratch-refsfrom
raphaelvigee/scratch-mount

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 22, 2026 •

Copy link
Copy Markdown
Member

Third of four (stack #408), on top of #407. This is where a scratch declaration
starts doing something: a referenced cache is materialized into the sandbox,
guarded against concurrent use, and handed to the tool through the variable its
declaration names.

After this, the whole surface a consumer writes is:

target(name = "server", driver = "bash", scratch = ["//build:gocache"],
       run = ["go build ./..."])          # GOCACHE is set; nothing else to wire

Mounting is one symlink

Per target, pointing out of the sandbox at the canonical slot directory. That is
the entire mechanism, and it is why a scratch costs an inode rather than a copy:
teardown removes the link, not the tree, since remove_dir_all does not follow
symlinks — the same property read-only input staging already relies on for the
11k-file Go SDK.

The measurements are what rule out the obvious alternative. Seeding a warm cache
into each sandbox cut go list CPU 778s → 309s and moved wall time by
exactly zero (interleaved A/B: 217.3s vs 219.5s), because the cost was never
CPU — it was the ~500 filesystem entries created and destroyed per sandbox.

The link target is the canonical path, not something sandbox-local. Tools bake
absolute paths into their cache entries (Go's action IDs are the standing
example), so if every consumer saw its own path the cache would restore and be
inert — present, and useless.

The bridge creates it, not the engine. The bridge owns sandbox creation: the
FUSE path may redirect the package dir into a mount, so there is no earlier moment
at which the directory reliably exists. Putting it there covers both sandbox modes
at once, and sandboxfuse implements mkdir/symlink through to the upper dir.

The one wrong-build guard

A scratch that would land where an input already did is a hard error. This is
the only way a scratch can cause a wrong build rather than a slow one: the target
would read cache contents where it believes it reads a declared dependency — bytes
no hashin describes. Silently winning either way is not an option.

Locking ships with access, deliberately

A keyed cross-process reader/writer lock per slot, access choosing the guard. It
is in this PR and not a later one because an exclusive that does not serialize is
a silent lie.

Two ordering rules, both mirroring what execute already does:

  • Guards are taken in sorted slot order, so two targets naming the same pair in
    opposite orders cannot deadlock.
  • The acquire sits after dependency resolution and before the worker permit.
    After deps, or a dep needing the same slot could never get it — the same diamond
    the worker permit is already ordered to avoid. Before the permit, so a target
    queued on a contended slot holds no worker; and since the lock is always taken
    first, no permit holder is ever blocked on a slot, which makes the wait provably
    bounded rather than circular.

Slot identity

(addr, version, platform-components). So platform = "any" gives one slot for
every machine
, which is what will let a portable cache travel between a Linux CI
runner and a macOS laptop once the remote lands.

Compatibility: ABI_SEMVER 0.5.0 → 0.6.0

RunRequest/ManagedRunRequest gain repeated ScratchMount scratch. Additive and
cold-path — a prost wire field, not a vtable change — so an old plugin decodes a
new host's request and ignores the mounts. Its targets then run without a scratch,
which costs a cold cache and never a wrong build: the lock is keyed on a
declaration an old plugin cannot see, so there is no shared directory for it to
race either. Minor, not major: no plugin must be rebuilt, but one must be to
mount a scratch.

The cdylib path is not optional here — plugin-go is a cdylib and is the reason
this feature exists.

An earlier revision of the design claimed mounting needed no ABI surface at all, on
the theory that the resolved mount would ride on RunInput.annotations. It cannot:
link.rs:36 filters runtime: false inputs out of LinkedTargetDef, which is
exactly what makes a scratch a pure graph edge — and therefore also why it never
becomes a RunInput. Corrected in docs/SCRATCH.md.

Tests

12 more, 54 total across the stack. The load-bearing one is
a_scratch_carries_state_between_runs: run 1 sees an empty cache and writes a
marker, run 2 (a genuine re-execute, not a hit) reads it back — through the real
sandbox, symlink and env var.

Worth knowing if you write another test here: an EResult holds a riding read
lock on its addr, so keeping one alive across reopen() deadlocks the second
engine's write lock against it. It only bites when the second run re-executes; a
cache hit takes a read lock and coexists. It cost ~30 minutes of hang and surfaced
as an unrelated ENOENT on a lock file.


Stack created with GitHub Stacks CLI • Give Feedback 💬

raphaelvigee and others added 2 commits August 29, 2026 17:31
Makes a scratch declaration do something. A referenced cache is now materialized
into the sandbox, guarded against concurrent use, and handed to the tool through
the variable its declaration names — so `scratch = ["//build:gocache"]` is the
whole of what a consumer writes.

Mounting is one `symlink(2)` per target pointing out of the sandbox at the
canonical slot directory. That is the entire mechanism, and it is why a scratch
costs an inode per target rather than a copy: teardown removes the link, not the
tree, since `remove_dir_all` does not follow symlinks — the same property
read-only input staging already relies on for the Go SDK. Measurements on a Go
corpus are what rule out the copy: seeding a warm cache into each sandbox cut
`go list` CPU by 60% and moved wall time by exactly zero, because the cost was
never CPU but the half-thousand filesystem entries created and destroyed per
sandbox.

The link target is the *canonical* path, not something sandbox-local. Tools bake
absolute paths into their cache entries, so if every consumer saw its own path
the cache would restore and be inert — present, and useless.

The symlink is created by the bridge rather than the engine because the bridge
owns sandbox creation: the FUSE path may redirect the package dir into a mount,
so there is no earlier moment at which the directory reliably exists. Putting it
there also covers both sandbox modes at once.

A scratch that would land where an input already did is a hard error. This is the
one way a scratch can cause a *wrong build* rather than a slow one — the target
would read cache contents where it believes it reads a declared dependency,
bytes no `hashin` describes — so it must fail rather than silently win.

Locking is a keyed cross-process reader/writer lock per slot, `access` deciding
which guard. It ships in the same change as `access` deliberately: an
`exclusive` that does not serialize is a silent lie. Guards are taken in sorted
slot order so two targets naming the same pair in opposite orders cannot
deadlock, and the acquire sits after dependency resolution but before the worker
permit — after deps, or a dep needing the same slot could never get it; before
the permit, so a target queued on a contended slot holds no worker, which also
makes the wait provably bounded rather than circular.

Slot identity is `(addr, version, platform-components)`. `platform = "any"`
therefore gives one slot for every machine, which is what will let a portable
cache travel between a Linux CI runner and a macOS laptop.

ABI: `RunRequest`/`ManagedRunRequest` gain `repeated ScratchMount scratch`, and
`ABI_SEMVER` goes 0.5.0 -> 0.6.0. Additive and cold-path — a prost wire field,
not a vtable change — so an old plugin decodes a new host's request and ignores
the mounts. Its targets then run without a scratch, which costs a cold cache and
never a wrong build: the lock is keyed on a declaration an old plugin cannot see,
so there is no shared directory for it to race either. The cdylib path is not
optional here, since plugin-go is a cdylib and is the reason this exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-refs branch from 3af14f3 to 4680aba Compare August 29, 2026 15:34
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-mount branch from 4573c40 to 1625864 Compare August 29, 2026 15:34
@raphaelvigee
raphaelvigee marked this pull request as ready for review August 29, 2026 15:34
@raphaelvigee

Copy link
Copy Markdown
Member Author

Folded into #403 (declare/reference/mount) and #431 (lineages, store management, remote) — the split was finer than the review needed.

@raphaelvigee
raphaelvigee deleted the branch raphaelvigee/scratch-refs August 29, 2026 15:51
@raphaelvigee
raphaelvigee deleted the raphaelvigee/scratch-mount branch August 29, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant