CI: move every job to the org self-hosted fleet — scry was the last holdout - #205
Draft
avrabe wants to merge 1 commit into
Draft
CI: move every job to the org self-hosted fleet — scry was the last holdout#205avrabe wants to merge 1 commit into
avrabe wants to merge 1 commit into
Conversation
…oldout
On 2026-09-02 a PR sat 85 minutes with 0 of 13 jobs started, surviving a
full cancel + re-run, while 5 runners in the pulseengine fleet sat IDLE.
The cause is not contention:
scry 19 of 19 jobs on GitHub-hosted `ubuntu-latest`
rivet/witness/meld heavy jobs already on
[self-hosted, linux, x64, light|rust-cpu]
org fleet 12 runners, ALL online, 7 busy / 5 idle
The org migrated to a Hetzner self-hosted fleet and scry never followed,
so it is the one repo fully exposed when hosted capacity does not
materialise. rivet even ships a job literally named "Traceability (hosted
fallback)", which is what engineering around this failure mode looks
like.
Mapping mirrors rivet's: `light` for checks that do no heavy Rust compile
(Format, WIT round-trip, AADL, claim-check, required-checks, rivet
validate/delta/traceability, Pages), `rust-cpu` for anything that builds
or proves (Clippy, Test, Bazel, cargo-deny, MC/DC, Rocq, Verus, publish,
release, MC/DC evidence).
JOB NAMES ARE UNCHANGED, so all 12 required contexts (scry#130) keep
reporting — `runs-on` does not affect a check's context name. Verified
with check-required-checks.py: 12 contexts, 13 jobs, PASS.
CORRECTED A COMMENT THIS CHANGE FALSIFIED. rocq-proofs.yml said "Stays on
ubuntu-latest: requires Nix + Bazel for the hermetic Rocq 9.0 toolchain".
The Nix and Bazel requirement is satisfied by the install STEPS in that
job, not by the runner image, so it was never a reason to stay hosted —
and leaving the comment would have been the same drift as the "carry no
verifies link by construction" line repaired in #198.
THE RISK IS RECORDED RATHER THAN DISCOVERED. `bazel-build` and `test`
install Nix via cachix/install-nix-action, because Bazel resolves
@rocq_toolchains through rules_nixpkgs and fails with "nix-build not
found in PATH" without it. That install has to work on self-hosted too.
ci.yml's header now says so, so a failure on a Nix step points at the
runner class rather than sending someone into the build. Rocq and Verus
are `continue-on-error`, making them the cheapest early signal.
Refs: FEAT-093
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc
avrabe
marked this pull request as draft
September 3, 2026 05:58
Contributor
Author
|
Parked as draft — the diagnosis holds, the execution does not, and the urgency evaporated. What this PR got rightThe exposure is real and worth fixing eventually: scry is the only pulseengine repo still on GitHub-hosted What it got wrong5 jobs fail, and 4 of them fail on the same step:
That is exactly the risk this PR's own scry needs Nix because Bazel resolves And the urgency is goneThe stall that motivated this was transient — long, but transient. #204's re-run completed green with all 12 jobs after roughly nine hours, and rivet/meld have since completed successfully too. So this is no longer blocking the v3.3.0 release, which proceeds on the existing configuration. What would make it landableOne of:
I have not picked one because (2) is an infrastructure call and (1) needs a runner I can test against. Left as a draft with the evidence rather than closed, since the underlying exposure has not gone away — it just stopped being today's problem. |
📐 rivet artifact deltaPR: #205 Base SHA: Validationhead — `rivet validate` resultbase — `rivet validate` result (for comparison)Artifact stats
full stats — headDiff (base → head)AADL model — headPosted by the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-09-02 a PR sat 85 minutes with 0 of 13 jobs started, surviving a full cancel + re-run, while 5 runners in the pulseengine fleet sat idle. This is not contention:
ubuntu-latest[self-hosted, linux, x64, light|rust-cpu]The org migrated to a Hetzner self-hosted fleet and scry never followed, so it is the one repo fully exposed when hosted capacity doesn't materialise. rivet even ships a job named "Traceability (hosted fallback)" — that is what engineering around this failure mode looks like.
Mapping (mirrors rivet's ci.yml)
light— no heavy Rust compile: Format · WIT round-trip · AADL (spar) · claim-check · Required-checks · Rivet validate / delta / traceability · Pagesrust-cpu— builds or proves: Clippy · Test · Bazel build · cargo-deny · MC/DC · Rocq · Verus · Publish · Release · MC/DC evidenceRequired contexts are unaffected
Job names are unchanged, and
runs-ondoes not affect a check's context name. Verified rather than assumed —check-required-checks.py --against-file: 12 contexts, 13 jobs, PASS.Corrected a comment this change falsified
rocq-proofs.ymlsaid:The Nix and Bazel requirement is satisfied by the install steps in that job, not by the runner image — so it was never a reason to stay hosted. Leaving it would have been the same drift as the "carry no verifies link by construction" line repaired in #198.
The risk, recorded rather than discovered
bazel-buildandtestinstall Nix viacachix/install-nix-action, because Bazel resolves@rocq_toolchainsthroughrules_nixpkgsand fails with"nix-build not found in PATH"without it. That install has to work on self-hosted too.ci.yml's header now says so, so a failure on a Nix step points at the runner class rather than sending someone into the build. Rocq and Verus arecontinue-on-error, making them the cheapest early signal.This PR is its own test: if the jobs start, the diagnosis was right.
Refs: FEAT-093
🤖 Generated with Claude Code
https://claude.ai/code/session_01KkNzkNYzPh7366DkNijeNc