Skip to content

docs cache: reclaim digest-addressed content reuse without reopening the publish race #18

Description

@amondnet

Split out of the review of #17, where reusing a digest-named content directory was declined with a cited harm rather than applied.

What was declined

publish names each content directory <tag>.content-<sha12>-<uuid>. The digest is provenance; the random suffix makes the name private to one publication. cubic asked for the suffix to go, so that a --refresh producing unchanged bytes reuses the tree already on disk instead of extracting a second copy.

The cost is real: one extra documentation tree per --refresh (2.4 MB for Boot 3.5.16, 3.6 MB for 4.1.1), held until the leftover sweep reclaims it an hour later.

Why it was not applied

A deterministic name is shared by every publisher of those bytes, so publish has to decide what to do about whatever already occupies it, and both answers were worse than the duplicate:

  • Reuse when usable, delete and re-extract when not. The gap between the check and the delete is a TOCTOU window. A publisher that filled the name and swapped the link into place in that window has its live tree deleted by the loser. This is the exact race that the greptile thread ("Concurrent publication can fail") and the local gpt review both found, and the unique name is what removed it.
  • Trust whatever is there. A partial tree left by a killed run gets published as if it were complete.

mkdirSync would claim the name atomically (EEXIST tells you who won), and POSIX rename can replace an empty directory, so the winner could still move its tree in with one rename. Windows has no equivalent, so that path forks the implementation in the one place this cache most needs to behave identically.

What would change the answer

A claim-then-fill sequence that is atomic on both POSIX and Windows. With one, the digest name can come back and --refresh over unchanged bytes becomes free again.

Scope

  • publish / contentName in scripts/docs.ts
  • The duplicate is bounded: explicit --refresh only (a cache hit never reaches publication), one tree, TTL-reclaimed.

Not urgent — the current behavior is correct, and the cost is bounded disk for at most an hour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions