ci(2183): the R8 manifest check lived in a job nobody has to wait for - #755
Conversation
`Static analysis` (installer-tests.yaml's `static` job) is a required status check on no branch — measured against develop, staging, main and both rulesets. Four guards lived there: the R8 installer-manifest check, cross-OS facts single-sourcing, style/terminology, and PSScriptAnalyzer. A PR could change an installer sub-script, leave scripts/manifest.sha256 stale, and merge on its required checks — shipping a manifest that rejects the scripts it covers. They could also be silenced. On five runs of #752 the job's ShellCheck step ate the whole `timeout-minutes: 10` on `apt-get install shellcheck`; all four guards reported `skipped` while every required check went green. The apt-get was never needed — shellcheck is on the ubuntu-latest image. No branch-protection change and no new required context: `Source-of-truth drift` is already required on develop and main, and its header already declares itself the home for guards that must gate. The Makefile's `drift` target already ran R8 + check-facts + check-style while that job ran check-drift + env-vocabulary, so each side gated what the other did not. One list now, in the Makefile. - drift-checks.yaml runs `make drift` and holds no guard list of its own - make drift gains check-drift.sh (five guards), and runs ALL of them before failing, so one red guard no longer hides the rest — it still blocks - Lint (required) absorbs the deleted job's bash -n + shellcheck, which restated Lint's own 19-file SHELLCHECK_FILES sweep at the same severity, plus its PSScriptAnalyzer; `make lint-warnings` keeps the advisory SC2034 pass - apt-get in Lint is now conditional on shellcheck being absent - the k3s-cuda paths move to drift-checks.yaml with the check-facts guard (#616) - docs/SUPPLY_CHAIN.md and .cursor/BUGBOT.md both asserted the gate existed; corrected to name the required job Mutation-proved: changing cluster.sh without regenerating the manifest turns `make drift` red (anchor asserted: blob 87080bd -> 0b74919c); a stale manifest plus a style violation reports both and exits non-zero. bats 1125/1125 green. Refs tracebloc/backend#2183 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0172f7e. Configure here.
|
Findings from a review pass. No verdict attached. The thesis holds up — I read branch protection independently rather than taking the body on trust, and every claim in it checks out (see the bottom section). The findings are about two things the move carries along with it, plus the collision with #754. #755 × #754 — the dangerous breakage merges without a conflict markerVerified by running a real 3-way The problem is what merges cleanly. #754 deletes The other order is worse: #754's diff replaces the whole Even in the better order (#754 first), this PR's rebase needs Findings1.
|
develop (client#747) made telemetry-vocabulary-agreement.sh a REQUIRED guard (backend#1907) via individual steps + the enumerated lists this PR collapses. Folded it into DRIFT_GUARDS so `make drift` (the required Source-of-truth check) still runs it; verified `make drift` green including that guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…review) Both findings from Arturo's review. Built on top of b4ab27b (a concurrent session's develop merge) rather than a second resolution of my own -- see the note at the end, which is the more useful half of this message. FIX 1 -- `make parse` failed open, and #753 had just promoted it to be the entire content of the required `Lint` check. `find scripts ... | xargs -0 -n1 bash -n` loses find's status across the pipe (recipes run under dash, no pipefail) and xargs with no input runs nothing and exits 0, so in a tree with no scripts/ dir it printed find: scripts: No such file or directory all shell scripts parse and exited 0, under both shells. It now materialises the list, refuses a zero count, and prints the derived count. Proven four ways: missing dir -> non-zero, zero .sh files -> non-zero, real parse error -> non-zero, clean tree -> "all 39 shell scripts parse". The first attempt at that third anchor was INERT -- `if [ -z "$x" ; then` is valid bash, the missing bracket being a runtime not a parse error -- so it was redone with an unterminated `if`, after checking that bash -n rejects it standalone. FIX 2 -- the superset claim was true set-wise and false scope-wise. The deleted jobs swept their 34 files on every run; `quality / shellcheck` reads only the PR diff, its caller passes all-files: false and declares no schedule. So #753 as written left NO CI job doing a whole-tree error-severity sweep -- demonstrable on #754 itself, where that job is green in 8s reading "Shell files to check: 0". `Lint` therefore runs `make lint` (parse + derived whole-tree shellcheck) rather than `make parse`. Costs no install, shellcheck being preinstalled is the whole finding, and it restores `make lint` == CI `Lint` parity (backend#1850). The two survivors are complementary: whole-tree here, diff-scoped in the org job, neither installing anything, neither holding a file list. Verified green on the 44-file derived set under the RUNNER's shellcheck 0.9.0 via koalaman/shellcheck:v0.9.0, not merely a local 0.11.0, before arming it on a required check. WHY THIS SITS ON b4ab27b. I had resolved the same develop conflict locally and resolved the Makefile with `git checkout --ours`, which takes the whole file and so silently DROPPED develop's non-conflicting addition of `scripts/tests/telemetry-vocabulary-agreement.sh` to the `drift` target. Nothing would have failed: the guard would simply have stopped running. b4ab27b kept it, so that resolution is the base and mine was discarded. This is the same shape as the #755 hazard flagged on the PR -- the damage from a merge is in what disappears without a marker, not in what conflicts -- and it caught me one hour after I wrote that warning. The conflict itself is the thesis: to add one script, #747 had to hand-edit the enumerated list in three places. The derivation picked it up with no edit, 42 -> 44 on its own. Refs #753
|
Sequencing note from #754's review (Arturo's finding, independently reproduced). This PR and #754 break each other in a way that merges without a conflict marker. Real 3-way
Different offsets, so git merges them silently. I resolved only the visible seconds comment and ran the result:
Reverse order is worse: #754's diff replaces the whole Suggested order: #754 first, then rebase this one with
Either way the second PR needs a real rebase and re-review. Please don't resolve this through GitHub's conflict editor — the breakage is outside the markers, so a clean-looking resolution is exactly what produces the silent no-op. I have not touched this branch. Flagging only. For the record, this bit me too within the hour: resolving #754's own conflict with develop I used One scoping correction: this PR is not an alternative remedy to backend#2182 / #2183. Those are about making the R8 manifest step a gate, which #754 deliberately leaves alone. The only overlap is where shellcheck lives and whether its file list is enumerated. |
saadqbal
left a comment
There was a problem hiding this comment.
The gate is real now — Source-of-truth drift and Lint keep their exact names, both run on every PR with no paths: filter, nothing carries continue-on-error, and no needs: pointed at the job you delete. R8 ends up behind a context that actually blocks. Good change.
One thing I'd fix first: make drift can print drift: all guards green and exit 0 having run zero guards. Inline.
Also note the head moved to 74eeb05 (develop merged in) and brought a sixth guard, telemetry-vocabulary-agreement.sh. Neither prose copy of the guard list followed it — which is the exact divergence this PR exists to stop, so worth catching on the way past.
Not inline because the lines aren't in the diff: Makefile:19 and Makefile:151 still describe SHELLCHECK_FILES and lint in terms of the static job you delete here.
On whether the check still detects the right failures: gen-manifest.sh --check diffs a full regeneration against the committed manifest, so a changed digest and a manifest that has gained or lost a line both fail, and it cross-checks its FILES/WINDOWS_FILES against both bootstraps' arrays with a bats test asserting the manifest lists every fetched file. What nothing checks is a new scripts/lib/foo.sh that install-k8s.sh sources but that never got added to install.sh's FILES — that file is neither fetched nor covered, and every guard stays green. Pre-existing, not yours.
Coordination with #754, which is open on the same three files: it deletes SHELLCHECK_FILES, which your new lint-warnings expands. If it lands first, make lint-warnings becomes a bare shellcheck with no operands behind a || true — permanently green, silently. Whoever rebases second should hand-check that and the Lint step block rather than trust a clean merge.
Ran make drift against this head in a scratch copy: 6/6 green, 2.6 s. Local bats didn't finish inside 10 minutes so I dropped it; CI's bats (bash unit, mocked) is green.
Pre-merge, since I can't read branch protection: confirm Static analysis really is in no required-check list before this merges — if it is required anywhere, deleting the job leaves those PRs waiting on a context that will never report.
Skipping what aptracebloc already raised (the PSGallery install, the missing timeout-minutes, if: always() vs !cancelled()).
aptracebloc
left a comment
There was a problem hiding this comment.
Re-reviewed at 74eeb059. The branch has only merged develop since my last pass — no findings addressed — and there's a new fail-open in the required check this PR creates. The thesis is still right; the implementation needs another turn.
First, correcting myself from my earlier comment: I described this and #754 as opposite remedies for the same gap. That was wrong. backend#2182/#2183 are about making the R8 step a gate, which #754 explicitly leaves alone. This PR is a different answer on where shellcheck lives, not the opposite of that one.
N3 (blocking) — make drift can report green having run nothing
@saadqbal found this; I reproduced both shapes independently against this head:
$ make drift DRIFT_GUARDS=
drift: all guards green → exit 0
$ make drift "DRIFT_GUARDS=bash -c 'exit 1'|scripts/gen-manifest.sh --check"
/bin/sh: 1|scripts/gen-manifest.sh --check: No such file or directory
drift: all guards green → exit 0
@guards='$(DRIFT_GUARDS)' interpolates inside single quotes, so any future guard containing a ' — bash -c '…', python3 -c '…' — terminates the assignment and the list collapses to zero iterations. The newly-required Source-of-truth drift then goes green having run nothing.
The asymmetry is what makes this worth blocking on rather than filing: parse and shellcheck in #754 both carry exactly the zero-count guard drift lacks — same author, same day, same class of defect, opposite outcome. And this PR's whole argument is that a guard which can't fail isn't a gate. Fix: count iterations, compare against the number of |-separated entries, and stop interpolating into a quoted string.
Four findings from my last pass, all untouched
Unconditional Install-Module PSScriptAnalyzer (standard-checks.yml:117) — still unconditional, still inside a required job. The run log shows the module is already on the image (version '1.25.0' … currently in use) and the step took 38s of 59s. Note the pair interaction: #754 adds timeout-minutes: 10 to that exact job, so if #754 lands first this degrades from a 360-minute pending to a bounded 10-minute red. Better, but it's still a network fetch that doesn't need to happen.
Lint has no timeout-minutes — unaddressed here; #754 supplies it. So the pair fixes it, this PR alone doesn't.
drift-checks.yaml:95-97 still reads "No timeout-minutes here, deliberately." The reasoning doesn't survive this PR's own change: guards reported skipped harmlessly only because Static analysis wasn't required. Once Source-of-truth drift is required, a timeout-kill sets it cancelled, which branch protection treats as not-success — red and blocked, which is what you want. timeout-minutes: 20 gives both properties against guards that measure ~1s. The real hang risk isn't make drift anyway; it's azure/setup-helm at :61, which downloads a binary before any guard runs.
docs/SUPPLY_CHAIN.md:106-107 still names branches/{develop,main}/protection. I re-hit the 404 today: that path returns {"message":"Not Found","status":"404"} at non-admin level, indistinguishably from "branch is not protected", while repos/tracebloc/client/branches/develop → .protection.required_status_checks.contexts works. A reader following the runbook concludes the opposite of the truth, which defeats the note's own purpose.
N4 (nit) — the count has already drifted
The job comment enumerates five guards and the Makefile says "All five" and "the five drift guards"; DRIFT_GUARDS has been six since the develop merge. Also confirming @saadqbal's sub-point: "all five are pure local file comparisons and all have a --write / regenerate mode" isn't true of check-drift.sh, which the very next sentence admits shells out to helm template. Worth fixing mainly because the job header eight lines above says this job "holds no list of its own" — and this comment is a list of its own that has already drifted.
The $(SHELLCHECK_FILES) no-op is still live
Merged 9d811b39 × 74eeb059, resolved only the one visible Makefile conflict (a timing comment), ran it with an argv-logging stub:
$ make lint-warnings
STUB shellcheck argv: --severity=warning --shell=bash
No files specified.
EXIT=0
SHELLCHECK_FILES has no definition in the merged Makefile — only a mention inside a comment — while lint-warnings still references it. Zero operands, || true swallows the usage error, exit 0. A permanent green no-op, with a benchmark-number conflict as the only warning. make lint on the same tree is healthy (39 parsed, 44 shellcheck operands), so this is specifically the advisory sweep going dark.
Good news on the other half: the PSScriptAnalyzer-drop hazard I flagged earlier is gone. #754 changing run: make parse → run: make lint made that step byte-identical on both sides, so the standard-checks.yml conflict shrank to the Install shellcheck region and this PR's advisory-shellcheck and PSScriptAnalyzer steps both merge in cleanly. Verified in the merged file.
Suggested path
Land #754 first — it's green, self-contained, and leaves the R8-gate question open for this PR. Then rebase this onto develop and, while in there:
- drop
Ensure shellcheck(standard-checks.yml:59); preinstalled is #754's whole basis - keep
run: make lintas-is — now identical on both sides - rewrite
lint-warningsonto the derived set with the same zero-count guard, or drop|| true. Best shape: factor the classifier out ofshellcheck:into one place both targets read, so the warning sweep can't silently diverge from the error sweep again - fix N3's zero-iteration fail-open and N4's counts
- settle the drift-checks timeout and the SUPPLY_CHAIN endpoint
Resolve the three conflicts hunk-by-hunk, not whole-file and not in GitHub's conflict editor — both the $(SHELLCHECK_FILES) no-op and #754's new bounded bats step sit outside the markers.
— drafted with Claude Code
…753) (#754) shellcheck ran in three places. Two apt-installed it and hung 35+ minutes on 2026-08-19; the third, the org reusable `quality / shellcheck`, passed in ~5 s because shellcheck is preinstalled on ubuntu-latest (0.9.0). The comment justifying the install -- "shellcheck itself is not on the runner image" -- was false. Measured before deleting: the two enumerated lists were byte-identical to each other, expanded to 34 files, and were a strict subset of the derived set. The enumeration had drifted past eight real scripts, and a parse error planted in index-invariants.sh was invisible to it. Both local copies are gone; the Makefile now derives the set instead of listing it. Scope is stated honestly rather than overclaimed: `quality / shellcheck` reads only the PR diff, so `Standard checks / Lint` runs `make lint` over the whole tree. Two complementary required gates, neither installing anything, neither holding a file list. Verified green under the runner's 0.9.0, not just a local 0.11.0. Also: `make parse` now fails closed (it exited 0 on a missing scripts/ dir while being the entire content of a required check), both bats installs are bounded and retried after one stalled on this very PR, every job in standard-checks.yml has a timeout, and four comments that had gone false were fixed -- file counts removed from prose entirely, since they rotted twice in one day. Review by @aptracebloc and @saadqbal found three real defects in this PR, all fixed here. Follow-ups deliberately left open: backend#2182 / #2183 (make the R8 manifest gate required) and #755's rebase, whose `lint-warnings` will silently no-op against the deleted SHELLCHECK_FILES if it is merged without one. Closes #753
…ings #754 landed on develop, so the sequencing Arturo recommended is satisfied: it removed shellcheck's third copy and explicitly left the R8-as-a-gate question to this PR. Three files conflicted; resolved hunk-by-hunk, taking develop's derived classifier and this PR's deletion of the `static` job. N3 (blocking, Asad + Arturo) — `make drift` could report green having run zero guards, in the very check this PR makes required: $ make drift DRIFT_GUARDS= -> green, exit 0 $ make drift "DRIFT_GUARDS=bash -c 'exit 1'|scripts/..." -> green, exit 0 `guards='$(DRIFT_GUARDS)'` expanded inside single quotes, so the first guard containing a `'` terminated the assignment and the list collapsed. Fixed at the root by exporting the list and reading it from the environment — no quote in a guard can collapse an env var — plus an empty-list refusal and an iteration count asserted against the number of declared entries. Both shapes now exit non-zero; the quoted case runs both guards and reports the failing one. The comment that claimed "this is NOT the fail-open shape" is now three machine checks instead of a claim (backend#1729 rule 7). The `$(SHELLCHECK_FILES)` no-op (Arturo) — #754 deleted that variable while this PR's new `lint-warnings` still referenced it, so the advisory sweep ran with no operands and `|| true` swallowed the usage error: permanently green. Confirmed on the merged tree. Both sweeps now read ONE derivation, scripts/sh-files.sh, which exits non-zero on a zero-file classification — so a broken derivation is loud in both instead of silent in either. Mutation-proved via SH_FILES= stub: both targets go red. Also from the reviews: - drift-checks.yaml: deleted the five-guard enumeration. It was already wrong (six since telemetry-vocabulary-agreement.sh arrived) eight lines under a header claiming the job "holds no list of its own". `make drift` prints the live set and count. - drift-checks.yaml: `timeout-minutes: 20`. The "no timeout, deliberately" argument didn't survive this PR — `skipped` guards were harmless only while the job was required nowhere; now a kill is `cancelled`, which blocks. The hang risk is azure/setup-helm, not the guards. - standard-checks.yml: dropped `Ensure shellcheck` (preinstalled — #754's basis). - standard-checks.yml: PSScriptAnalyzer installs only if absent. 1.25.0 ships on the image and was already loaded while the step spent 38s of 59s re-fetching it inside a required check. - standard-checks.yml: `if: always()` -> `if: !cancelled()`, so a lost job stops spending its budget. - docs/SUPPLY_CHAIN.md: the runbook said to read `branches/{b}/protection`, which is admin-only and 404s indistinguishably from "not protected" — a non-admin following it concluded the opposite of the truth. Now names `branches/develop` -> `.protection.required_status_checks.contexts`. - Makefile header: names drift-checks.yaml, and `drift` as the second target this file defines rather than copies. Answering Asad's pre-merge ask: `Static analysis` is in NO required-check list — develop, staging and main all read, plus both rulesets. Deleting the job leaves no PR waiting on a context that will never report. Verified: make check green; shellcheck 45 files (the new script included, and mutation-proved covered — a planted parse error reddens the gating sweep); lint-warnings reports 45 and real warnings instead of "No files specified"; drift 6/6 green. bats runs past the local 10-minute budget, as Asad also found; CI is the authority. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both reviews addressed at #754 has merged, so the sequencing @aptracebloc recommended is satisfied rather than skipped: develop now carries the derived classifier, and #754's own comment hands the R8-as-a-gate question to this PR. I merged develop and resolved the three conflicts hunk-by-hunk (not whole-file, not in the web editor) — and both of the things you warned sit outside the markers were there. N3 — fixed at the root, not just detectedThe interpolation was the mechanism, so a count alone would have papered over it. The list is now The SHELLCHECK_FILES no-op — confirmed and fixedReproduced on the merged tree exactly as described: no operands, Mutation-proved with a stub (
The new script is itself classified (45 vs 44) and mutation-proved covered: a planted parse error reddens the gating sweep. It caught a defect in itself on the way in — a comment line beginning with the linter's name is inline-directive syntax, so it was an SC1073 error. Exactly the trap The other four, all addressed
@saadqbal's pre-merge ask — answered with the reads
So deleting the job leaves no PR waiting on a context that will never report. Two things I am deliberately not fixing here@saadqbal's uncovered-lib gap — a new Pester still gates nothing —
|
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 75b042f. Configure here.
Summary
The R8 installer-manifest supply-chain check could not block a merge, and could be silenced by an unrelated
apt-get.Static analysis(installer-tests.yaml'sstaticjob) is a required status check on no branch — verified againstdevelop,staging,mainand both rulesets. Four guards lived there: R8 manifest, cross-OS facts single-sourcing (#435/#616), style/terminology, PSScriptAnalyzer. So a PR could change an installer sub-script, leavescripts/manifest.sha256stale, and merge on its required checks — publishing a manifest that rejects the very scripts it ships.They could also be silenced outright. On five consecutive runs of #752 the job's ShellCheck step consumed the whole
timeout-minutes: 10onapt-get update && apt-get install shellcheck:Every required check on that sha was green. And the
apt-getwas never needed: shellcheck ships onubuntu-latest— the org'squality / shellcheckjob installs nothing and prints its version 3s after checkout.Approach
No branch-protection change, no new required context.
Source-of-truth driftis already required ondevelopandmain, and its own header already declares itself "the home for every duplicated-declaration guard in this repo, because it is the one that GATES".The two lists had already diverged in the way that hides:
make driftran R8 + check-facts + check-style, the required job ran check-drift + env-vocabulary — each side gated exactly what the other did not. A stale manifest was caught by the pre-push hook and not at the merge gate. So there is now one list, in the Makefile, and the required job runs it.gen-manifest.sh --check(R8)Static analysis— required nowhereSource-of-truth drift— required on develop + maincheck-facts.sh --checkStatic analysisSource-of-truth driftcheck-style.shStatic analysisSource-of-truth driftbash -n+ shellcheckStatic analysis, restatingLint's own 19-file sweepLintviamake lint(already there)Static analysisLint--severity=warningStatic analysis, advisorymake lint-warnings, still advisoryThe
staticjob is deleted — every part of it either moved to a required context or was already duplicated by one.Guards no longer share a fate.
make driftruns all five even when one fails, then exits non-zero. One red guard no longer hides four others, and it still blocks — the guards are independent in their reporting, not in whether they gate. Notimeout-minuteson the drift job, deliberately: a required job that hangs blocks the merge, whereas one killed mid-way reports guards asskipped. Prefer the slow red.Mutation proof
Changed
scripts/lib/cluster.shwithout regenerating the manifest, asserting the anchor actually applied:Then a stale manifest and a style violation together: all five guards ran, both failures reported (
!! FAILED×2), exit non-zero.Docs that asserted the gate existed
docs/SUPPLY_CHAIN.mdclaimed "A CI gate ininstaller-tests.yaml… fails any PR that changes a sub-script without regenerating the committed manifest". That was false for as long as it was written..cursor/BUGBOT.mdsaid the same. Both now name the required job, and SUPPLY_CHAIN.md carries a note telling the next person to move the check to a job whose context is inbranches/{develop,main}/protection.Test plan
make drift— 5/5 guards green; mutation-proved red twice (above)make lint,make lint-warnings,make check— greenbats scripts/tests/*.bats— 1125/1125 passyaml.safe_load100755in git, so the deletedchmod +xsteps were redundantclient/**change, so noChart.yamlbump neededChecklist
developDeliberately not fixed
Pester does not gate.
Pester (windows-latest)/Pester (ubuntu-latest)are required on no branch, soUnit testsgates bats only and the PowerShell installer has no gating unit suite. Making it required means a required check on a windows-latest runner — a cost/flake decision, not a cleanup. Flagged instandard-checks.ymland in tracebloc/backend#2183.stagingrequires neitherLint,Unit testsnorSource-of-truth drift— onlygate / gate+ the fourquality / *. Promotions reach staging via the release train off develop where all three are required, so it is not a hole today, but it is not what the other branches look like.Refs tracebloc/backend#2183
🤖 Generated with Claude Code
Note
Medium Risk
Changes which CI contexts block merges and centralizes guard logic in the Makefile; mistakes could false-block PRs or briefly miss a guard, but behavior is documented and fail-closed by design.
Overview
Moves R8 manifest, cross-OS facts, style, and related drift guards from installer-tests’ non-required
Static analysisjob into the already-requiredSource-of-truth driftjob, which now runs a singlemake driftstep instead of enumerating guards in YAML.Deletes the entire
staticjob frominstaller-tests.yaml(bash-n, shellcheck, gen-manifest, check-facts, check-style, PSScriptAnalyzer). PSScriptAnalyzer and advisory shellcheck warnings land on requiredLint(standard-checks.yml); PSScriptAnalyzer reuses the image’s module when present.Makefile becomes the sole list of drift guards (
DRIFT_GUARDS, includingcheck-drift.sh), with fail-closed execution (empty list, quote-collapse, run-all-then-fail).scripts/sh-files.shunifies shell file discovery forshellcheckandlint-warnings. drift-checks adds a 20-minute timeout and push paths for k3s-cuda/Makefile; docs/BUGBOT note the real gating job.Reviewed by Cursor Bugbot for commit 75b042f. Bugbot is set up for automated code reviews on this repo. Configure here.