release-train: develop -> staging - #243
Merged
Merged
Conversation
#239) * feat(ci): watch for PRs bricked by a required check that never reports A required status check that never runs leaves its context at "Expected - waiting for status to be reported". The PR is then approved, has nothing red to point at, and is permanently unmergeable -- the one CI failure mode with no red signal at all. Nobody is notified and no reviewer sees a problem, because there is no failure, only an absence. Nothing inside a PR can detect that, so it needs a watcher. EMPIRICAL, NOT STATIC. Reasoning about "can this check ever report here" means modelling path filters x matrix expansion x reusable inputs x their defaults x `if:` expressions; run across 19 repos that produced three separate classes of false positive before it produced anything true. Comparing a branch's required contexts against the contexts actually PRESENT on the PR needs none of it: required-but-absent is bricked, whatever the cause. Required contexts come from caller-drift.py's `read_protection`, imported rather than reimplemented -- it already unions classic protection with rulesets, and a ruleset-only branch 404s on the classic API (backend#1276). A second copy is how one of them silently stops reading half the picture. THE DISTINCTION THAT MAKES IT READABLE, learned the hard way here: an absent context and a not-yet-started one are identical at a glance. The first version reported release-train#67 bricked three minutes after a push, and it had every context minutes later. A head younger than 60 minutes is not judged, and an unreadable age counts as young -- a false brick is what makes a report ignorable, which is the failure mode of the always-red check one repo over. FOUND A FOURTH CAUSE ON ITS FIRST REAL RUN, one the ticket does not list: a CONFLICTED PR. GitHub cannot compute a merge commit, so pull_request workflows never run and every required context stays absent. release-train#67, 82 minutes after its last push, had 0 workflow runs on its head sha. The fix is a rebase, not a protection change, so the report names that cause separately. Fails closed throughout: an unreadable branch, age or PR list is reported as "could not audit" and exits 2. Reporting "0 bricked" while part of the fleet was never read is the shape this exists to remove. Evidence: 10-case offline selftest passes; full-fleet run over all 20 inventory repos takes 94s and reports exactly one finding (release-train#67, above). Closes tracebloc/backend#1721 * fix(ci): the watcher's own fail-open paths (Bugbot, .github#239) Two findings, both the shape this watcher exists to report, in the watcher. A CAPPED PR LIST READ AS CLEAN. `gh pr list` truncates at --limit and says nothing about it, so a repo with more open PRs than the cap was audited PARTIALLY and reported clean for the ones it never saw. The cap is now 200 -- high enough that no repo here approaches it, the busiest having ~10 open PRs against one base -- and reaching it raises "could not audit" instead of returning a partial view. THE GRACE WINDOW USED THE WRONG CLOCK. A commit's committer date is when it was WRITTEN. A force-push can put a long-dated commit on a branch a second ago, which would then be judged immediately -- producing exactly the false "bricked" the window exists to prevent. The honest clock is when CI first saw the head: the OLDEST check suite on the sha, since GitHub creates a suite per app as soon as it has work for that head. The commit date remains as a fallback for a head with no suite at all, which is the conflicted case -- there, no suite will ever exist and the commit date is the only clock there is. Also drops an unused `timedelta` import that failed `quality / ruff`. 13 selftest cases, up from 10. The two new ones needed the real functions put back first: earlier cases swap `open_prs` and `head_age_minutes` out wholesale, so a later case that forgets silently tests the previous case's stub -- this file's own subject matter, and it caught me on the first run. * fix(ci): an unreadable check-suites read is young, not the commit date A failed check-suites API read (502/403/rate-limit) fell through to the commit committer date, which a force-push can set to any past instant -- so an unreadable CI clock plus an old commit still reported BRICKED, the exact false-brick the grace window exists to prevent. Distinguish a failed read (return None -> caller treats the head as young and skips) from a genuinely empty one (conflicted PR, no suite will ever exist -> commit date is the only clock). Adds a selftest case for the failed-read path. Bugbot, .github#239. --------- Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com>
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
tracebloc-release-train
Bot
added
the
gate-nudge
tracebloc-release-train
Bot
removed
the
gate-nudge
…ger exists (#237) `CURRENT_COL` is read from the BOARD, whose column is "Staging (agent review)". The guard's list carried only the pre-rename "Staging (human review)", so an issue hand-closed while sitting in the agent-review column did not match, `Done` overwrote its deploy state, and kanban-archive.yml then hid the card entirely -- precisely the sequence this guard was added to prevent (Bugbot, .github#126), reopened by a string that quietly stopped being true. kanban-reconcile.yml's equivalent guard already lists both names, which is what makes this a drift rather than a design gap: one list was updated at the rename and its copy was not. The guard also stranded silently. Refusing is right -- the two real cases needed OPPOSITE answers (backend#1493 had shipped via cli#452 and belonged in Prod; data-ingestors#488 was reverted and belonged in Done), so no default is correct and only the person closing the issue knows which. But a `::notice::` in a run log is invisible by the time anyone looks at the board, so the card sat in a deploy state with nothing anywhere saying why. It now comments on the issue with the two options, and warns if even that fails. Verified: YAML parses, the embedded shell parses, house-rules clean, the version-bump-gate selftest still 51/0. Refs tracebloc/backend#1846 Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com>
…run at all (#244) Bugbot HIGH on the staging promotion PR #243. `bricked-prs.yml` runs `pip install --quiet pyyaml` on `ubuntu-latest` with no `actions/setup-python`. Ubuntu 24.04 marks its system Python externally managed (PEP 668), so pip refuses, the step fails, and THE AUDIT NEVER RUNS. The dependency is real, not incidental: bricked-prs.py loads caller-drift.py for the protection reader, and that module hard-fails without PyYAML by design. So the install cannot simply be dropped. Worse than a broken job. This audit's whole purpose is to find PRs blocked by a required check that never reports -- a silence. A guard that cannot start produces exactly the same silence as a clean fleet, so its failure looks like its success. Nothing else would have said so. bricked-prs-selftest.yml already sets up Python 3.12 before the identical install, so this is that shape rather than a new decision -- the audit and its selftest now agree, which is also why the selftest never caught this. Checked the class rather than the instance: this was the ONLY workflow in the repo with a `pip install` and no `setup-python`. actionlint clean; YAML parses; setup-python pinned by SHA with its version comment per the action-pins gate.
…245) Three sites still WROTE `Staging (human review)`, a column that no longer exists on the board, and two fallbacks quietly translated it to `FR on staging`. Behaviour today is correct because of those fallbacks -- and both are labelled "removed in step 3 of #1592". So step 3 as written is an outage: delete the fallbacks while three writers still emit the dead name and every staging promotion fails its option lookup and exits 1. The order is the whole fix. Writers emit the live name; only then is the fallback dead code, and it goes in the same change so nothing is left half-done. kanban-closure-router.yml:71 pull_request writer kanban-closure-router.yml:156 issue-closure writer advance-deploy-env.yml:71 push writer Found by @shujaatTracebloc reviewing .github#237, which fixed the same stale-string class on the GUARD path. This is the WRITE path. WHAT THIS DELIBERATELY DOES NOT DO: it does not start writing `Staging (agent review)`. Nothing writes that column anywhere, by design -- fr-gate ranks it 7 and says so ("READ-ONLY for now; nothing writes this value yet, #1578 does that in a LATER hop"). A card landing in `FR on staging` on a staging merge is the documented transition, not a skipped stage. Writing the agent column here would move cards into a column nothing advances, which is backend#1846's dead end, created on purpose. READERS are left alone. `case` arms that accept both names match values read FROM the board, which can never be the dead one -- that is tolerance, not a dependency, and it costs nothing. `fr-pass-comment.yml` needs no change either: it resolves the column by PROBING the board and fails loudly when neither name exists, which is the shape the other two should have had. Verified: both files parse as YAML, every embedded `run:` block parses under `bash -n`, house-rules clean, and no writer of the dead name remains. Refs tracebloc/backend#1592 Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com>
Contributor
Author
|
bugbot run |
tracebloc-release-train
Bot
added
the
gate-nudge
tracebloc-release-train
Bot
removed
the
gate-nudge
This was referenced Aug 13, 2026
… of warning (#246) Bugbot HIGH on the staging promotion PR #243. An unresolvable Status option logged `::warning::`, set SKIP_STATUS=1 and let the run finish GREEN. So the board silently stops advancing while every push reports success -- the failure and the success are indistinguishable, and the only signal is a warning in a log nobody opens. This was the one asymmetry in the family. Twelve lines above, the Deploy environment lookup aborts on exactly this condition; kanban-closure-router.yml aborts on it too (":230 Could not resolve Status option - aborting; exit 1"). Only this path degraded, and Status is the half that drives the pipeline view. WHAT I DID NOT CHANGE, having checked it: the finding also suggests the writers lost their rename-window fallback while the resolvers kept it. They did, but that is not live -- the board's Status options today are `Staging (agent review)` and `FR on staging` (read from the live project), and the writers emit `FR on staging`, so every lookup resolves. `opt_either('Staging (human review)', 'FR on staging')` in kanban-reconcile.yml is now vestigial: its own comment says it is the backend#1592 rename shim, "removed in step 3". Finishing that retirement is a five-file sweep and a separate change from this abort; filed rather than folded in during a release freeze. Also corrected the comment above the block, which described the removed behaviour ("degrades gracefully ... rather than failing the whole workflow") and would have read as the contract to the next person. actionlint clean; YAML parses.
…it (#247) * feat(kanban): assert the board has every Status column the writers emit The board's column names and the workflows that write them are two systems that must agree, and nothing checked that they did. #1592 carried the `FR on staging` rename by hand across three PRs; in between, the writers emitted a column the board did not have and only a fallback made it work. That gap is exactly what .github#243 reported and what #245 closed by hand. This makes it un-driftable instead of remembered. The failure it prevents is a silent one. A written name that does not resolve means the card is not moved -- and until #246 that was a `::warning::` on a green run, so a frozen board and a working board looked identical. SCOPE IS DELIBERATELY NARROW: * WRITERS are asserted. A name that is written must exist or the write cannot land. `advance-deploy-env.yml` and `kanban-closure-router.yml`, listed rather than globbed -- `STATUS=` is a common shell variable and other workflows use it for "ok"/"absent"/"unreadable", which are not column names. * RESOLVERS are NOT asserted. They ask for names on both sides of a rename on purpose: `opt_either("Staging (human review)", "FR on staging")` is the #1592 shim and its first argument is SUPPOSED to be absent today. Enforcing there would make the tolerance impossible to express. * `STATUS_NAME="$OVERRIDE"` is skipped: a per-repo .kanban.yml value cannot be known here. Runs on PRs touching the writers, and DAILY -- the board can be renamed in the UI at any time with no PR to hang a check on, which is the failure this is really for. Fails closed throughout: an unreadable board, an unparseable response, zero options returned, or a literal pattern that matches nothing are all errors. The last one matters most -- a stale regex would make the check pass vacuously, which is the failure mode of every guard it is modelled on. Verified: passes on today's board (5 names); reproducing the pre-#245 state (writer emitting "Staging (human review)") fails and names the exact file and value; selftest 6/6 with the board stubbed, covering absent names, a stale name among good ones, and a case near-miss. * fix(kanban): gate the conformance check on the selftest job Bugbot (Medium): the check job ran in parallel with selftest and had no needs: selftest. If the checker failure paths regress while the board still matches, "Selftest the checker" goes red but "Written Status names exist on the board" can still report success -- a conformance-named green from a broken guard. caller-drift + standards-sync gate their audit on selftest for the same reason. Add needs: selftest so the conformance verdict is only trusted once the checker is proven able to fail. .github#247. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
bugbot run |
tracebloc-release-train
Bot
added
the
gate-nudge
tracebloc-release-train
Bot
removed
the
gate-nudge
#248) * fix(kanban): the column check missed three writers, and could not tell Two Bugbot findings on the staging promotion PR #243, both on code I added today. MEDIUM — the conformance check from #247 was itself incomplete, which is the exact defect it exists to prevent. `WRITERS` named two files and `LITERAL` matched one idiom, so it never saw: set-pr-status.yml `echo "status_name=In progress"` (UNQUOTED value) fr-pass-comment.yml `NEXT="Ready for prod"` Three column names -- In progress, Code review, Ready for prod -- were invisible to a guard built to make a rename impossible to miss. It reported a clean sweep of a subset, which reads exactly like a clean sweep of everything. Widening the hand-list alone would rot the same way, so the names are now derived a SECOND, independent way and the two are compared: any board column name ASSIGNED in a writer file that no idiom matched means the idiom list is stale, and that is an error rather than a smaller answer. Assignment sites only, whole values -- so comments, the rank `case` arms and prose are ignored, and `Ready` does not match inside `Ready for prod`. It earned that immediately: the cross-check caught my own first attempt at the widened regex, which required a trailing quote and silently stopped matching `STATUS_NAME="FR on staging" ;;`. Without it I would have shipped a check that found fewer names than before while printing "all names exist". Coverage went 5 names / 2 files -> 8 names / 4 files. LOW — the abort message added in #246 said "The Deploy environment write may already have happened". It cannot have: that check runs at :190 and the first `update_field` is at :314, so on that path nothing was written at all. Corrected to say so, since the point of the message is to tell an operator where they stand. Selftest 8/8 (adds: an assigned-but-unmatched name is reported; a comment mentioning a column is not). caller-drift selftest 162/0. ruff + actionlint clean. * fix(kanban): the PR trigger must cover every writer, and now cannot drift Bugbot Medium on this PR. I broadened `WRITERS` to four workflows and left `kanban-columns.yml`'s `paths:` filter listing the original two — so a PR touching only `set-pr-status.yml` or `fr-pass-comment.yml` never ran this check and a bad column write could merge until the next cron. Same shape as the defect this PR is already fixing: the guard is present, reports green, and is not watching the thing it names. Two files apart this time instead of two variables. The filter now lists all four. But listing them is what went wrong once already, so the selftest DERIVES the requirement: every entry in WRITERS must appear in the workflow's `paths:` block, read out of the YAML rather than restated. Adding a fifth writer without wiring its trigger is now a red test, not a silent gap. Verified by removing `set-pr-status.yml` from the filter: the assertion fails. Selftest 10/10 with it restored. ruff + actionlint clean. This is the third instance today of "fixed the code, left the trigger config" -- e2e#81 was the same bug in a tsc dry-run paths filter. Recorded in the message because the pattern is worth more than the fix. * style(selftest): split a combined import (ruff E401) It was in the block I added two commits ago and I claimed ruff clean without reading the exit code -- the `&&` swallowed it. Checked properly this time.
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ed77044. Configure here.
tracebloc-release-train
Bot
added
gate-nudge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
Medium Risk
Changes core deploy/kanban promotion behavior (fail-closed status resolution and column renames) and adds org-wide cron jobs that depend on PROJECTS_KANBAN_TOKEN and GitHub APIs; misconfiguration will surface as red runs rather than silent drift.
Overview
Hardens org kanban automation so misnamed Status columns and silent skips stop looking like success, and adds fleet guards for stuck PRs and writer/board drift.
Kanban writers now target
FR on stagingfor staging merges (replacingStaging (human review)) and drop the temporary rename fallbacks inadvance-deploy-env.ymlandkanban-closure-router.yml.advance-deploy-envaborts the run when the Status option cannot be resolved (no more green runs that skip Status updates).kanban-closure-routerextends the Done guard toStaging (agent review)and comments on the issue when a hand-close is blocked because the card is still in a deploy column.New
kanban-columns-check(daily cron + PR paths) verifies every Status literal emitted by the four writer workflows exists on the engineer board, with selftests that keepWRITERSand the workflowpaths:filter in sync.New
bricked-prs(every 4h) scans inventory repos for open PRs whose required contexts are still absent after a grace window (distinguishing conflicted vs never-reported), reusingcaller-driftfor protection; scheduled workflow installs Python/PyYAML like the selftest.Reviewed by Cursor Bugbot for commit ed77044. Bugbot is set up for automated code reviews on this repo. Configure here.