feat(scratch): reference a scratch cache by addr from a consuming target - #407
Closed
raphaelvigee wants to merge 5 commits into
Closed
feat(scratch): reference a scratch cache by addr from a consuming target#407raphaelvigee wants to merge 5 commits into
raphaelvigee wants to merge 5 commits into
Conversation
This was referenced Aug 29, 2026
…ctory Adds the declaration half of scratch caching: a target kind that names a mutable, non-hermetic directory persisting across runs, which many targets can share. Nothing mounts, locks or stores anything yet — a declaration is inert on its own, and the engine work that gives it meaning lands on top of this. Declaring it as a target rather than inline at each use site is the point. Settings live in exactly one place, so two consumers cannot disagree about a slot's `access` or `version` — that whole class of error becomes inexpressible rather than something to detect and report. The addr is the cache's identity, so packages give namespacing for free (`//go:cache` and `//rust:cache` need no agreed prefix convention) along with visibility and `heph query revdeps`. It is a builtin driver shaped exactly like `plugingroup`, so no new Starlark global is introduced: `target(driver = "scratch", …)` is the surface, and `#[derive(Spec)]` supplies both the parser and the BUILD-file LSP schema. Two settings are author assertions heph cannot check, and both are documented as such with conservative defaults: - `access = "shared"` says the tool is safe under concurrent access. Go's build cache is the motivating case — forcing it exclusive would serialize a whole Go build. Defaults to `exclusive`. - `platform = "any"` says the contents are portable, which lets one slot serve every machine. It asserts two things, not one: no host dependence *and* no embedded absolute paths. Defaults to `os_arch`, because restoring a host-specific cache onto the wrong host is the one mistake here that is not merely slow. Mount-path validation lives on the declaration rather than at each use, so a bad path is reported once at its source. A mount is a symlink out of the sandbox, so absolute and `..`-escaping paths are rejected; so is mounting at the workspace root, which would replace the consumer's whole tree. Design doc: docs/SCRATCH.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Adds the reference half: `scratch = ["//build:gocache"]` on an exec target, resolved against the declaration and validated. Still nothing mounts or locks — the edge exists in the graph and the errors land, which is what the storage work builds on. A reference is an Input with `hashed: false, runtime: false` — the one combination nothing else uses, and both halves are load-bearing. It materializes no artifacts because a declaration has none, and it must not touch the consumer's cache key because a target's outputs are required to be identical whether its scratch is warm, cold, or absent. That second half is the property most likely to be "fixed" later into a bug, so it is asserted three ways: the def hash is byte-identical with and without a reference, referencing a scratch does not change what a target produces, and bumping a scratch's `version` leaves every consumer's cached result a hit. The tempting alternative — fold the declaration into the consumer's hashin so a `version` bump rebuilds users of the cache — is an over-hash: a fresh slot changes nothing about the outputs, so the rebuild is pure waste, and if it *did* change them the target is already broken and rebuilding is not the fix. Nothing new crosses the plugin ABI. The reference rides on `Input.annotations`, host-visible and already the producer→host channel (`read_only`, `stage_per_file` are the precedent), and the settings are read from the declaration's spec config rather than travelling on the edge. So: no proto change, no `ABI_SEMVER` bump, and a third-party driver participates without being recompiled. Validation lives in the engine rather than the driver, because the properties that matter are only visible across a *set* of resolved references — and putting them there means every driver gets the checks, not just pluginexec: - a referenced addr that is not a `scratch` target, naming both ends and suggesting `deps`, since that is the likely intent - two references claiming one environment variable, which would silently make one of two real caches unreachable; the fix is on a declaration, so the error says so - two mounts overlapping, which would write one cache through the other. Compared component-wise, not by string prefix, so `.cache/go` and `.cache/golang` are correctly not a collision - the same scratch referenced twice, rather than quietly collapsing it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
`link.rs` filters `runtime: false` inputs out of `LinkedTargetDef`, which is what makes a scratch reference a pure graph edge — and is also why it never becomes a `RunInput`, so the resolved mount cannot travel on `RunInput` annotations as §5.3 claimed. Declaration and reference stay ABI-free (and are implemented that way); mounting will need an additive `RunRequest` field. Also sharpens the testing section: asserting `hashout` does not prove a target's key did not move, because a target whose `hashin` changed still produces identical bytes. The def hash has to be compared directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
A target whose `hashin` moved still produces identical bytes, so comparing `hashout` passes while the cache misses on every run — the assertion looked like it guarded §6.3 and did not. Compare the def hash instead, for both the reference and the `version` bump. Adds a precondition that `def_hash` distinguishes two genuinely different targets, so the equality assertions cannot pass vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
raphaelvigee
force-pushed
the
raphaelvigee/scratch-driver
branch
from
August 29, 2026 15:34
0ba1157 to
16a872f
Compare
raphaelvigee
force-pushed
the
raphaelvigee/scratch-refs
branch
from
August 29, 2026 15:34
3af14f3 to
4680aba
Compare
raphaelvigee
marked this pull request as ready for review
August 29, 2026 15:34
raphaelvigee
force-pushed
the
raphaelvigee/scratch-driver
branch
from
August 29, 2026 15:51
16a872f to
b174355
Compare
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of four (stack #408), on top of #403. That one added the declaration;
this one adds the reference —
scratch = ["//build:gocache"]on a consumingtarget — and every check that does not need storage. Mounting is #412.
Nothing observable happens yet beyond
heph query. That is deliberate: thevalidation lands before anything can write.
The reference is
hashed: false, runtime: falseThe one combination nothing else in the tree uses, and both halves are
load-bearing:
runtime: false— it materializes no artifacts, because a declaration hasnone.
link.rs:36filters these out ofLinkedTargetDef, so a scratch neverreaches the sandbox as an input.
hashed: false— it must not touch the consumer's cache key, because atarget's outputs are required to be identical whether its scratch is warm,
cold, or absent.
That second half is the property most likely to be "fixed" into a bug later, so
it is asserted three ways: the def hash is byte-identical with and without a
reference; referencing a scratch does not change what a target produces; and
bumping a scratch's
versionleaves every consumer's cached result a hit.The tempting alternative is to fold the declaration into the consumer's
hashin,so bumping
versionrebuilds everything using the cache. That is an over-hash. Ifoutputs really are identical warm-or-cold, a fresh slot changes nothing and the
rebuild is pure waste; if they are not, the target is already broken and
rebuilding is not the fix. So a
versionbump gives every consumer a fresh emptyslot and invalidates nothing — which is exactly what you want when the reason for
bumping it is "the old cache had gone bad".
The edge still exists in the graph, which buys
heph query revdeps //build:gocache— "who shares this cache?", the question every serialization surprise starts with
— and turns a bad addr into an ordinary
TargetNotFoundErrorrather than abespoke "unknown scratch name".
Compatibility: nothing new crosses the plugin ABI
No proto message, no
TargetDeffield, noABI_SEMVERbump. The reference rideson
Input.annotations— host-visible, and already the producer→host channel(
READ_ONLY_ANNOTATIONandSTAGE_PER_FILE_ANNOTATIONare the precedent) — andthe settings are read from the declaration's spec config rather than travelling on
the edge. A third-party driver participates without being recompiled.
The cost is stringly-typed settings on the wire, which is the trade
read_onlyalready made.
driver-support/src/scratch.rsholds the key constants so they haveone definition.
(Mounting does need ABI surface — see #412. Declaration and reference do not.)
Validation is in the engine, not the driver
A driver sees its own target's config and nothing else. The properties that matter
are only visible across a resolved set of references, which is a host operation —
and putting the checks there means every driver gets them, not just pluginexec:
scratchtarget — names both ends, and suggestsdeps, since that is the likely intentof two real caches unreachable. The fix is on a declaration, so the error says
so; there is nothing to change at the use site
component-wise, not by string prefix, so
.cache/goand.cache/golangarecorrectly not a collision — a
starts_withwould emit an error the authorcannot act on
Tests
22, across pluginexec parse, the engine's resolver, and e2e. The def-hash ones are
the load-bearing set — note they compare the def hash and not
hashout: atarget whose key moved still produces identical bytes, so a hashout comparison
would pass while the cache missed on every run. There is a precondition asserting
the comparison discriminates at all, so the equality checks cannot pass vacuously.
Stack created with GitHub Stacks CLI • Give Feedback 💬