fix(ci): the sovereign-ci pin was two commits behind the one that makes gx10 work

[noahgift]

Closes #3223

ci / test, ci / coverage and ci / lint have been failing on gx10 with:

##[error]Docker pull failed with exit code 1

which names neither the architecture nor the tag. By hand on gx10:

no matching manifest for linux/arm64/v8 in the manifest list entries

The pin, not the registry

This repo pinned sovereign-ci.yml@4453399ee (2026-09-05), whose every job runs

image: localhost:5000/sovereign-ci:stable@sha256:66016caa…

an amd64-only index. paiml/.github fixed exactly this on 2026-09-10 in
70e51ec04 — "sovereign-ci: pin a two-architecture image index (amd64 +
arm64)" — and this repo, which pins by SHA, never moved.

gx10 carries four clean-room runners and the workflow's default runs_on
is ["self-hosted","clean-room"] — arch-neutral, so the router sends work there
by design (#3100 / PMAT-3138). Routing and image disagreed for three days.

Measured on gx10, both digests

pull @sha256:66016caa…  →  no matching manifest for linux/arm64/v8
pull @sha256:70f2cdc9…  →  Status: Image is up to date

The delta is two commits, one file, +20/-11 — both additive

commit	change
23916c00d	adds an optional runs_on input, default ["self-hosted","clean-room"] — exactly the label set this repo already gets
70e51ec04	swaps the image digest on every job to the two-arch index

No input this repo passes changes meaning. Full 40-char SHA, as before.

How it was found

The job log does not say which version of a reusable workflow ran. The run's
referenced_workflows field does:

paiml/.github/.github/workflows/sovereign-ci.yml@4453399ee3794714800ff8db316ea7e1d3705a00

against a main tip of 70e51ec04. That is the field to read when a reusable
workflow behaves like an older version of itself.

Separately, and still worth having

The :stable tag in the registry was also amd64-only, so any
docker pull localhost:5000/sovereign-ci:stable — the mutants job does
exactly that — failed on gx10 too. I re-pointed it at the existing multi-arch
index (same amd64 digest, so no x86 consumer changed). Neither fix covers the
other: the tag fix does nothing for a digest-pinned container:, and this
bump does nothing for a tag-based pull.

Verified

check_workflow_env_defined, check_no_hosted_runners, check_runner_labels — green.

🤖 Generated with Claude Code

[github-actions]

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3225 head=a55fae23f5864e3bca37bd8f4ef44fc591c69711 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

[noahgift]

Measured impact on the merge queue, since this is now the thing the queue is waiting on.

Across the last 25 merge-group runs, 9 failed. Classifying every failing job by runner:

failing jobs	on gx10	shape
4	4	ci / coverage, ci / lint, ci / gate, gate
4	4	ci / coverage, ci / test, ci / gate, gate
5	4	+ workspace-test (the separate SIGSEGV, #3226)
4	2	mixed
1 ×5	0	present alone — not a required check

22 failing jobs, 14 of them (64%) on gx10. The five single-job failures are all present, which the ruleset does not require, so they are noise. The four multi-job failures are this bug: a container job lands on gx10, Initialize containers dies on no matching manifest for linux/arm64/v8, ci fails, and gate — which is required — fails with it.

Three container jobs per merge group against ~4 gx10 runners in a ~23-runner clean-room pool puts roughly 44% of merge groups on this path.

What I deliberately did not do. The obvious lever is to drop the clean-room label from gx10's runners until this merges, which would stop container jobs landing there. I measured the cost and left it alone: it removes gx10 from all clean-room selection — including guard-tree, guard-cargo and gate, which pass there — for the several hours the queue needs, against a standing instruction to keep gx10 loaded. The queue is slow, not deadlocked; three PRs merged today through exactly these conditions.

So this sits in the queue like everything else. The measurement is here so the cost of waiting is visible rather than assumed.

🤖 Generated with Claude Code

[noahgift]

Reproduced on gx10 itself, with the exact command the runner runs. Earlier in this PR I was reasoning from the workflow diff; this is the metal.

gx10$ curl -s http://localhost:5000/v2/sovereign-ci/tags/list
{"name":"sovereign-ci","tags":["stable-arm64","stable","multi"]}

# the digest the STALE pin resolves to -- present in the registry, HTTP 200
gx10$ D=sha256:66016caace1f07b57c2ab84aa91eba7c01e3abe348c36daf4504c51f43f66bf0
gx10$ curl -s .../manifests/$D | jq '[.manifests[].platform]'
platforms: [{'architecture': 'amd64', 'os': 'linux'}, {'architecture': 'unknown', 'os': 'unknown'}]

# what :stable resolves to NOW, after the re-tag
gx10$ curl -s .../manifests/stable | jq '[.manifests[].platform]'
platforms: [{'architecture': 'arm64', 'os': 'linux'}, {'architecture': 'amd64', 'os': 'linux'}]

gx10$ docker pull "localhost:5000/sovereign-ci:stable@$D"
no matching manifest for linux/arm64/v8 in the manifest list entries

So the two facts that matter:

1. The digest is in gx10's registry. This was never a missing-image or empty-registry problem, and the :stable re-tag I did earlier cannot fix it — tag@digest resolves by digest and ignores the tag. A digest is content-addressed, so there is no registry-side repair. Only this PR fixes it.
2. The :stable index on gx10 is already two-arch, which is why guard-cargo (it runs docker run against the tag from inside the step) passes on gx10 while every job using a container: (which pulls the pinned digest) fails there.

And the fix is measured, not argued — on this PR's own head a55fae23f, run 34764156346:

ci / lint      success  runner=gx10-pool2      <- the arch that was failing
ci / coverage  success  runner=yoga-build3
ci / test      success  runner=intel-clean-room-9

ci / lint passing on gx10-pool2 is the whole claim.

Blast radius while this waits

main has been red since the SHA pin merged at 10:09 (11c235016). Current main run 34763934924:

ci / coverage  failure  runner=gx10-pool1
ci / test      failure  runner=gx10-build
ci / lint      failure  runner=gx10-pool2
ci / gate      failure  runner=gx10-pool3
gate           failure  runner=intel-clean-room-9

all five with Docker pull failed with exit code 1 / Value cannot be null. (Parameter 'ContainerId').

Merges into main: nine between 06:38 and 11:01, one at 13:34, none since. Merge groups are still building — they are not deadlocked — but any group whose container jobs land on a gx10 clean-room runner fails and ejects its head PR. #2838 was ejected that way at 15:18 (run 34765202728) and is back to OPEN CLEAN.

I tried to move this PR to the front of the queue so it could heal the fleet sooner; dequeuePullRequest is refused for this token (NOT_FOUND), and the alternatives (draft/undraft, close/reopen) cost more than they save. So it waits its turn from position 9.

🤖 Generated with Claude Code