fix(notify): page once per breakage, and name what a redirect hit (ENG-2324)

[lucas-koontz]

User story

As an engineer on call for MindsHub
I want one Slack message per breakage, and an alert that names what actually broke
So that the channel stays worth reading and the first message is enough to start diagnosing

Why this matters

Public web probe merged at 19:46Z on 2026-09-04 and began firing on its */5 cron seven minutes later. It is correct: production console.mindshub.ai/cowork returns 301 to /cowork/ which returns 403, and /assets/ returns 403. Both are real, both are ENG-2317, and the fix for them is on staging waiting for a release.

The problem is what the notifier does with that. Nothing suppressed a repeated failure, so the same two routes re-paged every run. Measured cadence is 6.5 minutes once GitHub throttles the cron, which is about 220 Slack messages a day for one already-known bug, into #monitoring-prod, the channel that also carries Cloudflare's paging 60-second checks. That is how a real page gets scrolled past.

Two more defects surfaced while confirming it:

• The prior-run lookup read the branch's fifty most recent runs and filtered by workflow afterwards. At a five-minute cadence those fifty slots span about four hours, so any other main workflow whose previous run was older found no evidence of itself and silently dropped its recovery message. The probe was degrading every other pipeline's alerting in this repo.
• format_alert_label raised an uncaught ValueError once failure identities outgrew the label cap, in the one code path whose job is to report that something is down.

What changed

flowchart TB
    subgraph Before
        R1["Run finishes red"] --> P1["post = true, unconditionally"]
        P1 --> S1["Slack alert"]
        S1 --> N1["Next run, 5 min later"]
        N1 --> R1
    end

Loading

flowchart TB
    subgraph After
        R2["Run finishes"] --> L2["Look up the previous conclusive run<br/>of THIS workflow on this branch"]
        L2 --> D2{"status vs predecessor"}
        D2 -->|"failed, predecessor green"| A2["Alert"]
        D2 -->|"failed, no evidence"| A2
        D2 -->|"failed, predecessor also failed"| Q2["Quiet, already reported"]
        D2 -->|"recovered, predecessor failed"| G2["Recovery"]
        D2 -->|"recovered, predecessor green"| Q3["Quiet, routine green"]
    end

Loading

A three-day outage now costs about 70 messages instead of about 660, and never fewer than one per hour while it lasts.

Collapsed is not muted, and that distinction is the point. A run's conclusion records that it failed, never what failed, so two dead routes and a total console outage are both failure. Suppressing every repeat would let the second grow behind an alert already sent for the first — with /cowork red, the entire console going down would have posted nothing. A still-failing pipeline therefore repeats itself every repeat-alert-after-minutes (default 60), measured from the oldest failure in the current streak so it lands once per window at any cadence.

The two directions fail open in opposite ways on purpose. With no evidence an alert posts and a recovery does not, because an unreported failure costs more than a duplicate one. A caller missing the actions: read grant therefore degrades to today's behaviour rather than to silence.

Acceptance criteria

• A failure whose predecessor also failed posts nothing within the reminder window; the first failure still posts.
• A still-failing pipeline reports itself again once per repeat-alert-after-minutes, and only once per window.
• A missing or unparseable streak clock posts rather than staying quiet.
• repeat-alert-after-minutes of zero, a negative, or a non-number falls back to the default and cannot mute the channel.
• A failure with no evidence (prev_conclusion empty, the refused-lookup case) still posts.
• cancelled and skipped are not evidence the channel was already told, so they let the alert through.
• --force-post overrides the deduplication, so the reusable's own smoke test still posts.
• The freeze veto still silences both directions, and still outranks --force-post.
• The prior-run lookup runs for failures as well as recoveries, and still skips when nothing it could find would change the outcome.
• The lookup asks the per-workflow runs endpoint, so a frequent cron cannot crowd another workflow out of its own history.
• A caller whose workflow_ref names another repo keeps the branch scan and its name filter.
• An unreadable history still degrades to "no evidence" and exits 0, never failing the job.
• A failing redirect names its destination, bounded to 64 characters.
• An oversized alert label degrades to a count instead of raising.
• Each fix has a test that fails without it, proven by mutation rather than asserted.

How to test

1. uv run --with pyyaml --with pytest --python 3.12 -m pytest tests/ -q and actionlint -color.
2. Confirm the live evidence the dedupe will read:

gh api "repos/mindsdb/github-actions/actions/workflows/console-route-probe.yml/runs?branch=main&status=completed&per_page=10" \
  --jq '[.workflow_runs[] | select(.conclusion=="failure" or .conclusion=="success")][0].conclusion'

Expect failure, which is what makes the next probe run stay quiet once this merges.

3. Mutation-check the dedupe: change post in decide back to an unconditional True and re-run tests/test_notify_decision.py. Expect 5 failures. Restore.
4. Mutation-check the lookup: re-add inputs.status == 'recovered' && to the prev step's if: and re-run tests/test_notify_pipeline_status.py. Expect 2 failures. Restore.
5. After merge, watch #monitoring-prod. The probe keeps running every five minutes and keeps failing until ENG-2317 reaches production, and it should post nothing until it recovers.

Notes for the reviewer

• This does not silence the alert. The breakage still fails the run, still shows red in the Actions tab, still pages on the transition into failure and out of it, and still repeats hourly while it lasts. What is removed is the per-run repeat.
• The first version of this PR did have a real hole, caught on review: unbounded deduplication meant a worsening outage was masked by an alert already sent for a smaller one. 2fed976 adds the bounded reminder and the tests that fail without it. Worth reviewing that commit specifically. All 21 endpoints keep their contract, and no endpoint was dropped from config/console-route-probe.json to get the run green.
• Dropping /cowork and /assets/ was the tempting fix and is the wrong one. They are the only checks that noticed the production bug, and removing them breaks 7 contract tests that pin the endpoint count. The contract is right; the notifier was wrong.
• The lookup change has a blast radius beyond this workflow. Every caller of notify-pipeline-status now gets per-workflow history. That is strictly better for recovery detection, and it is what stops the probe degrading the rest of this repo's alerting, but it does change which run counts as "previous" for any workflow that shares a branch with a busier one.
• per_page drops from 50 to 10 on the per-workflow path because the filter is now server-side. The branch-scan fallback keeps 50.
• The redirect target is origin-controlled, so it is compacted and capped at 64 characters before it reaches the label, the same treatment network errors already get.
• Production is still broken on those two routes. This PR makes the alerting sane; ENG-2317 fixes the routes.

Verified locally

Check	Result
pytest tests/ -q	214 passed
actionlint -color	Passed, exit 0
workflow_graph.py --allow-external-reusables	13 workflows, every local reusable composes
git diff --check	Passed
Mutation: dedupe reverted to unconditional post	5 tests failed, as intended
Mutation: location capture removed	2 tests failed, as intended
Mutation: label degradation reverted to raise	2 tests failed, as intended
Mutation: lookup re-gated on recovered, per-workflow endpoint removed	2 tests failed, as intended
Mutation: reminder removed (unbounded dedupe)	5 tests failed, as intended
Live prev step executed against the real API, per-workflow path	prev_conclusion=failure, streak of 7, dated to the first probe failure at 19:53:09Z
Live prev step executed against the real API, branch-scan fallback	prev_conclusion=success, streak 0
Simulated 24h of the */5 probe against a standing breakage	23 messages, versus ~220 before and 1 with unbounded dedupe
Live per-workflow endpoint + jq against mindsdb/github-actions	Returned failure, the value that will suppress the next duplicate
Live branch-scan fallback + jq	Returned success, so the cross-repo path still composes
Live production reproduction	/cowork 301 to /cowork/, /cowork/ 403, /assets/ 403; all 7 other probed routes 200 with the SPA marker

Ships with

Production fix for the routes this alert is reporting: mindsdb/mindshub_frontend#1529

Merge order: independent. This one only changes how the channel is told.