pgw#991: a bare-hex blob address is REFUSED, not guessed into a namespace - #520
Merged
Conversation
…pace `_download_blob_by_digest` promoted any digest without a `:` to `blake3:<hex>`, and its docstring asserted the hub "indexes all CAS content by blake3 digest". Neither has been true since th#1303. The repo-CAS is sha256; blake3 survives in tensorhub's `casSupportedAlgos` for exactly one consumer, the dataset-CAS. So a caller holding a bare sha256 hex — the form hub listings and snapshot manifests quote — was silently addressed into the namespace its blob is not in, and got back a miss that reads as missing DATA rather than as a malformed address. th#1641 (`0ce09518`) fixed the ROUTE's absence; this is the other half the filing named. THE GUESS IS DELETED, NOT RE-POINTED AT `sha256:`. Two live namespaces disagree on the algorithm, so promoting bare hex is a coin flip whichever constant it picks, and the hub does not offer the choice: `storage.ParseDigest` answers bare hex with "not algorithm-tagged (bare hex is refused; write \"sha256:<hex>\")", and `validateDigestParam` fronts it on every by-digest route. `_parse_cas_digest` is that function transcribed — algorithm-tagged, a supported algorithm, hex of that algorithm's declared width, lowercased — so the worker refuses exactly what the hub refuses and accepts exactly what it accepts, and the refusal happens before the socket is opened. The refusal is typed by th#1259 PROVENANCE like every other address fault. A caller-supplied address raises `BlobDigestMalformedError` (`blob_digest_malformed`, a `PayloadRefError` -> `JOB_STATUS_INVALID`), so a tenant's typo is never model-health evidence; a platform-produced one stays a fatal `RuntimeError`, because the platform emitting a malformed address is a statement about the release. Test: `tests/test_blob_digest_qualified_pgw991.py`, the real fetch path against a real HTTP server whose by-digest route runs a TRANSCRIPTION of `storage.ParseDigest` and answers `400 invalid_digest` exactly where the hub does — so a regression that re-introduces the guess fails on the hub's own rule, not on a worker-side assertion about it. Parity is asserted in both directions over a table of addresses, and the rig records every address that reaches the wire, which is what proves a malformed one reaches nothing. RED-verified by restoring the old `blake3:` line under the new tests: 11 failed, 4 passed. GREEN with the fix: 15/15.
PaulFidika
force-pushed
the
991-blob-digest-qualified
branch
from
August 7, 2026 06:01
624e23a to
4b3c2f6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the SDK half of the th#1641 pair. The hub route landed as
0ce09518; this is the address the SDK was building for it.The defect
request_context._download_blob_by_digestpromoted any digest lacking a:toblake3:<hex>, and its docstring asserted the server "indexes all CAS content by blake3 digest". Neither has been true since th#1303:internal/storage/cas_paths.gosays so in a comment naming the lane: "the repo-CAS read path no longer speaks blake3 at all";So a caller holding a bare sha256 hex — the form hub listings and snapshot manifests quote — was silently addressed into the namespace its blob is not in, and the miss read as missing DATA rather than as a malformed address.
The fix: refuse, don't re-point
The guess is deleted, not changed to
sha256:. Two live namespaces disagree on the algorithm, so promoting bare hex is a coin flip whichever constant it picks — and the hub does not offer that choice.storage.ParseDigestanswers bare hex with:and
validateDigestParamfronts it on every by-digest route, includinggetBlobContentByDigest._parse_cas_digestis that function transcribed — algorithm-tagged, a supported algorithm (sha256/blake3, the exact content ofcasSupportedAlgos), hex of that algorithm's declared width, lowercased — so the worker refuses exactly what the hub refuses and accepts exactly what it accepts, and it happens before the socket is opened.The refusal is typed by th#1259 provenance, like every other address fault:
REF_ORIGIN_PAYLOAD(caller)BlobDigestMalformedError(blob_digest_malformed, aPayloadRefError)JOB_STATUS_INVALID— a tenant's typo is never model-health evidenceREF_ORIGIN_PLATFORMRuntimeErrorTest
tests/test_blob_digest_qualified_pgw991.pyruns the real fetch path against a real HTTP server whose by-digest route runs a transcription ofstorage.ParseDigestand answers400 invalid_digestexactly where the hub does — so a regression that re-introduces the guess fails on the hub's own rule, not on a worker-side assertion about it. Parity is asserted in both directions over a table of addresses (nothing the hub accepts is refused; nothing it refuses is sent), and the rig records every address that reaches the wire, which is what proves a malformed one reaches nothing.blake3:line under the new tests: 11 failed, 4 passed.Gates (local; run in this worktree)
mypyclean (233 source files) ·ruff check src/gen_workerclean ·lint_http_timeouts,lint_unreached_surface,lint_config_readsall exit 0 · fulltests/+tests_v2/run reported in a follow-up comment.No version bump and no
CHANGELOG.mdedit —changelog.d/pgw991.mdis this lane's fragment, folded in at the cut.