Skip to content

pgw#991: a bare-hex blob address is REFUSED, not guessed into a namespace - #520

Merged
PaulFidika merged 1 commit into
masterfrom
991-blob-digest-qualified
Aug 7, 2026
Merged

pgw#991: a bare-hex blob address is REFUSED, not guessed into a namespace#520
PaulFidika merged 1 commit into
masterfrom
991-blob-digest-qualified

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Closes the SDK half of the th#1641 pair. The hub route landed as 0ce09518; this is the address the SDK was building for it.

The defect

request_context._download_blob_by_digest promoted any digest lacking a : to blake3:<hex>, and its docstring asserted the server "indexes all CAS content by blake3 digest". Neither has been true since th#1303:

  • the repo-CAS is sha256internal/storage/cas_paths.go says so in a comment naming the lane: "the repo-CAS read path no longer speaks blake3 at all";
  • blake3 survives for exactly one consumer, the dataset-CAS.

So a caller holding a bare sha256 hex — the form hub listings and snapshot manifests quote — was silently addressed into the namespace its blob is not in, and the miss read as missing DATA rather than as a malformed address.

The fix: refuse, don't re-point

The guess is deleted, not changed to sha256:. Two live namespaces disagree on the algorithm, so promoting bare hex is a coin flip whichever constant it picks — and the hub does not offer that choice. storage.ParseDigest answers bare hex with:

cas ref: not algorithm-tagged (bare hex is refused; write "sha256:<hex>")

and validateDigestParam fronts it on every by-digest route, including getBlobContentByDigest. _parse_cas_digest is that function transcribed — algorithm-tagged, a supported algorithm (sha256/blake3, the exact content of casSupportedAlgos), hex of that algorithm's declared width, lowercased — so the worker refuses exactly what the hub refuses and accepts exactly what it accepts, and it happens before the socket is opened.

The refusal is typed by th#1259 provenance, like every other address fault:

origin raises maps to
REF_ORIGIN_PAYLOAD (caller) BlobDigestMalformedError (blob_digest_malformed, a PayloadRefError) JOB_STATUS_INVALID — a tenant's typo is never model-health evidence
REF_ORIGIN_PLATFORM RuntimeError fatal — the platform emitting a malformed address IS a statement about the release

Test

tests/test_blob_digest_qualified_pgw991.py runs the real fetch path against a real HTTP server whose by-digest route runs a transcription of storage.ParseDigest and answers 400 invalid_digest exactly where the hub does — so a regression that re-introduces the guess fails on the hub's own rule, not on a worker-side assertion about it. Parity is asserted in both directions over a table of addresses (nothing the hub accepts is refused; nothing it refuses is sent), and the rig records every address that reaches the wire, which is what proves a malformed one reaches nothing.

  • RED, by restoring the old blake3: line under the new tests: 11 failed, 4 passed.
  • GREEN with the fix: 15/15.

Gates (local; run in this worktree)

mypy clean (233 source files) · ruff check src/gen_worker clean · lint_http_timeouts, lint_unreached_surface, lint_config_reads all exit 0 · full tests/ + tests_v2/ run reported in a follow-up comment.

No version bump and no CHANGELOG.md edit — changelog.d/pgw991.md is this lane's fragment, folded in at the cut.

…pace

`_download_blob_by_digest` promoted any digest without a `:` to
`blake3:<hex>`, and its docstring asserted the hub "indexes all CAS content by
blake3 digest". Neither has been true since th#1303. The repo-CAS is sha256;
blake3 survives in tensorhub's `casSupportedAlgos` for exactly one consumer,
the dataset-CAS. So a caller holding a bare sha256 hex — the form hub listings
and snapshot manifests quote — was silently addressed into the namespace its
blob is not in, and got back a miss that reads as missing DATA rather than as
a malformed address. th#1641 (`0ce09518`) fixed the ROUTE's absence; this is
the other half the filing named.

THE GUESS IS DELETED, NOT RE-POINTED AT `sha256:`. Two live namespaces
disagree on the algorithm, so promoting bare hex is a coin flip whichever
constant it picks, and the hub does not offer the choice: `storage.ParseDigest`
answers bare hex with "not algorithm-tagged (bare hex is refused; write
\"sha256:<hex>\")", and `validateDigestParam` fronts it on every by-digest
route. `_parse_cas_digest` is that function transcribed — algorithm-tagged, a
supported algorithm, hex of that algorithm's declared width, lowercased — so
the worker refuses exactly what the hub refuses and accepts exactly what it
accepts, and the refusal happens before the socket is opened.

The refusal is typed by th#1259 PROVENANCE like every other address fault. A
caller-supplied address raises `BlobDigestMalformedError`
(`blob_digest_malformed`, a `PayloadRefError` -> `JOB_STATUS_INVALID`), so a
tenant's typo is never model-health evidence; a platform-produced one stays a
fatal `RuntimeError`, because the platform emitting a malformed address is a
statement about the release.

Test: `tests/test_blob_digest_qualified_pgw991.py`, the real fetch path against
a real HTTP server whose by-digest route runs a TRANSCRIPTION of
`storage.ParseDigest` and answers `400 invalid_digest` exactly where the hub
does — so a regression that re-introduces the guess fails on the hub's own
rule, not on a worker-side assertion about it. Parity is asserted in both
directions over a table of addresses, and the rig records every address that
reaches the wire, which is what proves a malformed one reaches nothing.
RED-verified by restoring the old `blake3:` line under the new tests: 11
failed, 4 passed. GREEN with the fix: 15/15.
@PaulFidika
PaulFidika force-pushed the 991-blob-digest-qualified branch from 624e23a to 4b3c2f6 Compare August 7, 2026 06:01
@PaulFidika
PaulFidika merged commit dd07570 into master Aug 7, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 991-blob-digest-qualified branch August 7, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant