Skip to content

feat(scratch): reference a scratch cache by addr from a consuming target - #407

Closed
raphaelvigee wants to merge 5 commits into
raphaelvigee/scratch-driverfrom
raphaelvigee/scratch-refs
Closed

feat(scratch): reference a scratch cache by addr from a consuming target#407
raphaelvigee wants to merge 5 commits into
raphaelvigee/scratch-driverfrom
raphaelvigee/scratch-refs

Conversation

@raphaelvigee

@raphaelvigee raphaelvigee commented Aug 22, 2026

Copy link
Copy Markdown
Member

Second of four (stack #408), on top of #403. That one added the declaration;
this one adds the referencescratch = ["//build:gocache"] on a consuming
target — and every check that does not need storage. Mounting is #412.

Nothing observable happens yet beyond heph query. That is deliberate: the
validation lands before anything can write.

The reference is hashed: false, runtime: false

The one combination nothing else in the tree uses, and both halves are
load-bearing:

  • runtime: false — it materializes no artifacts, because a declaration has
    none. link.rs:36 filters these out of LinkedTargetDef, so a scratch never
    reaches the sandbox as an input.
  • hashed: false — it must not touch the consumer's cache key, because a
    target's outputs are required to be identical whether its scratch is warm,
    cold, or absent.

That second half is the property most likely to be "fixed" into a bug later, so
it is asserted three ways: the def hash is byte-identical with and without a
reference; referencing a scratch does not change what a target produces; and
bumping a scratch's version leaves every consumer's cached result a hit.

The tempting alternative is to fold the declaration into the consumer's hashin,
so bumping version rebuilds everything using the cache. That is an over-hash. If
outputs really are identical warm-or-cold, a fresh slot changes nothing and the
rebuild is pure waste; if they are not, the target is already broken and
rebuilding is not the fix. So a version bump gives every consumer a fresh empty
slot and invalidates nothing — which is exactly what you want when the reason for
bumping it is "the old cache had gone bad".

The edge still exists in the graph, which buys heph query revdeps //build:gocache
— "who shares this cache?", the question every serialization surprise starts with
— and turns a bad addr into an ordinary TargetNotFoundError rather than a
bespoke "unknown scratch name".

Compatibility: nothing new crosses the plugin ABI

No proto message, no TargetDef field, no ABI_SEMVER bump. The reference rides
on Input.annotations — host-visible, and already the producer→host channel
(READ_ONLY_ANNOTATION and STAGE_PER_FILE_ANNOTATION are the precedent) — and
the settings are read from the declaration's spec config rather than travelling on
the edge. A third-party driver participates without being recompiled.

The cost is stringly-typed settings on the wire, which is the trade read_only
already made. driver-support/src/scratch.rs holds the key constants so they have
one definition.

(Mounting does need ABI surface — see #412. Declaration and reference do not.)

Validation is in the engine, not the driver

A driver sees its own target's config and nothing else. The properties that matter
are only visible across a resolved set of references, which is a host operation —
and putting the checks there means every driver gets them, not just pluginexec:

  • a referenced addr that is not a scratch target — names both ends, and suggests
    deps, since that is the likely intent
  • two references claiming one environment variable, which would silently make one
    of two real caches unreachable. The fix is on a declaration, so the error says
    so; there is nothing to change at the use site
  • two mounts overlapping, which would write one cache through the other. Compared
    component-wise, not by string prefix, so .cache/go and .cache/golang are
    correctly not a collision — a starts_with would emit an error the author
    cannot act on
  • the same scratch referenced twice, rather than quietly collapsing it

Tests

22, across pluginexec parse, the engine's resolver, and e2e. The def-hash ones are
the load-bearing set — note they compare the def hash and not hashout: a
target whose key moved still produces identical bytes, so a hashout comparison
would pass while the cache missed on every run. There is a precondition asserting
the comparison discriminates at all, so the equality checks cannot pass vacuously.


Stack created with GitHub Stacks CLIGive Feedback 💬

raphaelvigee and others added 5 commits August 29, 2026 17:30
…ctory

Adds the declaration half of scratch caching: a target kind that names a
mutable, non-hermetic directory persisting across runs, which many targets can
share. Nothing mounts, locks or stores anything yet — a declaration is inert on
its own, and the engine work that gives it meaning lands on top of this.

Declaring it as a target rather than inline at each use site is the point.
Settings live in exactly one place, so two consumers cannot disagree about a
slot's `access` or `version` — that whole class of error becomes inexpressible
rather than something to detect and report. The addr is the cache's identity, so
packages give namespacing for free (`//go:cache` and `//rust:cache` need no
agreed prefix convention) along with visibility and `heph query revdeps`.

It is a builtin driver shaped exactly like `plugingroup`, so no new Starlark
global is introduced: `target(driver = "scratch", …)` is the surface, and
`#[derive(Spec)]` supplies both the parser and the BUILD-file LSP schema.

Two settings are author assertions heph cannot check, and both are documented as
such with conservative defaults:

- `access = "shared"` says the tool is safe under concurrent access. Go's build
  cache is the motivating case — forcing it exclusive would serialize a whole Go
  build. Defaults to `exclusive`.
- `platform = "any"` says the contents are portable, which lets one slot serve
  every machine. It asserts two things, not one: no host dependence *and* no
  embedded absolute paths. Defaults to `os_arch`, because restoring a
  host-specific cache onto the wrong host is the one mistake here that is not
  merely slow.

Mount-path validation lives on the declaration rather than at each use, so a bad
path is reported once at its source. A mount is a symlink out of the sandbox, so
absolute and `..`-escaping paths are rejected; so is mounting at the workspace
root, which would replace the consumer's whole tree.

Design doc: docs/SCRATCH.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Adds the reference half: `scratch = ["//build:gocache"]` on an exec target,
resolved against the declaration and validated. Still nothing mounts or locks —
the edge exists in the graph and the errors land, which is what the storage work
builds on.

A reference is an Input with `hashed: false, runtime: false` — the one
combination nothing else uses, and both halves are load-bearing. It materializes
no artifacts because a declaration has none, and it must not touch the
consumer's cache key because a target's outputs are required to be identical
whether its scratch is warm, cold, or absent.

That second half is the property most likely to be "fixed" later into a bug, so
it is asserted three ways: the def hash is byte-identical with and without a
reference, referencing a scratch does not change what a target produces, and
bumping a scratch's `version` leaves every consumer's cached result a hit. The
tempting alternative — fold the declaration into the consumer's hashin so a
`version` bump rebuilds users of the cache — is an over-hash: a fresh slot
changes nothing about the outputs, so the rebuild is pure waste, and if it *did*
change them the target is already broken and rebuilding is not the fix.

Nothing new crosses the plugin ABI. The reference rides on `Input.annotations`,
host-visible and already the producer→host channel (`read_only`,
`stage_per_file` are the precedent), and the settings are read from the
declaration's spec config rather than travelling on the edge. So: no proto
change, no `ABI_SEMVER` bump, and a third-party driver participates without
being recompiled.

Validation lives in the engine rather than the driver, because the properties
that matter are only visible across a *set* of resolved references — and putting
them there means every driver gets the checks, not just pluginexec:

- a referenced addr that is not a `scratch` target, naming both ends and
  suggesting `deps`, since that is the likely intent
- two references claiming one environment variable, which would silently make
  one of two real caches unreachable; the fix is on a declaration, so the error
  says so
- two mounts overlapping, which would write one cache through the other.
  Compared component-wise, not by string prefix, so `.cache/go` and
  `.cache/golang` are correctly not a collision
- the same scratch referenced twice, rather than quietly collapsing it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
`link.rs` filters `runtime: false` inputs out of `LinkedTargetDef`, which is
what makes a scratch reference a pure graph edge — and is also why it never
becomes a `RunInput`, so the resolved mount cannot travel on `RunInput`
annotations as §5.3 claimed. Declaration and reference stay ABI-free (and are
implemented that way); mounting will need an additive `RunRequest` field.

Also sharpens the testing section: asserting `hashout` does not prove a target's
key did not move, because a target whose `hashin` changed still produces
identical bytes. The def hash has to be compared directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
A target whose `hashin` moved still produces identical bytes, so comparing
`hashout` passes while the cache misses on every run — the assertion looked like
it guarded §6.3 and did not. Compare the def hash instead, for both the
reference and the `version` bump.

Adds a precondition that `def_hash` distinguishes two genuinely different
targets, so the equality assertions cannot pass vacuously.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FArWjycMDyWeSfHHtpgtoU
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-driver branch from 0ba1157 to 16a872f Compare August 29, 2026 15:34
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-refs branch from 3af14f3 to 4680aba Compare August 29, 2026 15:34
@raphaelvigee
raphaelvigee marked this pull request as ready for review August 29, 2026 15:34
@raphaelvigee
raphaelvigee force-pushed the raphaelvigee/scratch-driver branch from 16a872f to b174355 Compare August 29, 2026 15:51
@raphaelvigee

Copy link
Copy Markdown
Member Author

Folded into #403 (declare/reference/mount) and #431 (lineages, store management, remote) — the split was finer than the review needed.

@raphaelvigee
raphaelvigee deleted the raphaelvigee/scratch-refs branch August 29, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant