mindsdb · mindsdb-devops · Sep 4, 2026 · Sep 4, 2026 · Sep 4, 2026 · Sep 4, 2026
@@ -96,6 +96,10 @@ on:
         description: "Post a 'recovered' message with no prior failure to recover from. Only the smoke-test dispatch sets this; a real pipeline leaves it off, or every green run reports a recovery."
         type: boolean
         default: false
+      repeat-alert-after-minutes:
+        description: "How long a still-failing pipeline stays quiet before it reports the failure again. Repeats are collapsed so a standing breakage does not page every run, but never indefinitely: a conclusion says that a run failed, not what failed, so a suppressed repeat can hide an outage that grew."
+        type: number
+        default: 60
       runs-on:
         description: "Runner label for the notify job"
         type: string
@@ -177,5 +181,6 @@ jobs:
           freeze-token: ${{ steps.freeze-token.outputs.token }}
           github-token: ${{ github.token }}
           force-post: ${{ inputs.force-post }}
+          repeat-alert-after-minutes: ${{ inputs.repeat-alert-after-minutes || 60 }}
           slack-channel-id: ${{ secrets.SLACK_ENG_CHANNEL_ID }}
           slack-bot-token: ${{ secrets.GH_ACTIONS_SLACK_BOT_TOKEN }}
@@ -40,8 +40,21 @@ sends no failure alert. An endpoint that fails twice makes the run red and passe
 failure to `notify-main-failure.yml`, which posts to the engineering Slack channel. That message
 names each failing environment, route, and observed result. A `200` response without the configured
 marker is reported as either `status 200 missing SPA marker` or
-`status 200 missing website marker`. The next successful run uses that reusable workflow's
-existing recovery lookup, so routine green runs stay silent.
+`status 200 missing website marker`. A redirect names its destination, as in
+`production /cowork status 301 to /cowork/`, because the status alone says a route moved but not
+where to, and the destination is usually the diagnosis.
+
+**One breakage pages once, then hourly.** A failing endpoint alerts on the run that breaks it, then
+goes quiet, then repeats itself once an hour for as long as it lasts, because `notify-pipeline-status`
+collapses a failure whose predecessor also failed. At this cadence the alternative is roughly 220
+messages a day for a single unfixed route, which is how a real page gets scrolled past. Routine green
+runs stay silent through the same lookup.
+
+The hourly repeat is not a formality. A run's conclusion records **that** it failed, never **what**
+failed, so two dead routes and a total console outage are both `failure`. Suppressing every repeat
+would let the second grow behind an alert already sent for the first. The repeat bounds how long a
+change in the failing set can go unmentioned; raise `repeat-alert-after-minutes` for a known noisy
+breakage, but do not try to disable it.
 
 The reviewable source of truth is [`config/console-route-probe.json`](config/console-route-probe.json).
 The Console matrix crosses these environments:
@@ -188,11 +201,36 @@ single job runs unless the workflow was cancelled and derives the outcome from
 
 `recovered` is not the same as green: the reusable looks up the previous
 conclusive run of the same workflow on the same branch and stays silent unless it
-failed, so a routine green merge posts nothing. That lookup needs `actions: read`
-on this job, because the default workflow token carries contents + packages read
-only and a called workflow can never hold more than its caller grants. Without
-it the lookup is refused and the job stays silent (it never fails the run), so a
-missing recovery message is the symptom to look for.
+failed, so a routine green merge posts nothing.
+
+**The same lookup collapses repeated failures.** A red run whose predecessor was
+also red posts nothing, so a standing breakage pages once rather than on every
+run. The two directions fail open in opposite ways on purpose: with no evidence,
+an alert posts and a recovery does not, because an unreported failure costs more
+than a duplicate one.
+
+**Collapsed is not muted.** A conclusion says that a run failed, not what failed,
+so a suppressed repeat could hide an outage that grew behind an alert already
+sent. A still-failing pipeline therefore reports itself again every
+`repeat-alert-after-minutes` (default 60), measured from the oldest failure in
+the current streak so the repeat lands once per window at any cadence. An
+unreadable or missing streak clock posts rather than staying quiet, and setting
+the interval to zero or a non-number falls back to the default instead of
+muting the channel.
+
+That lookup needs `actions: read` on this job, because the default workflow token
+carries contents + packages read only and a called workflow can never hold more
+than its caller grants. Without it the lookup is refused and the job stays silent
+(it never fails the run), so a missing recovery message is the symptom to look
+for — and, since the deduplication reads the same evidence, a caller missing the
+grant also re-pages every run of a standing failure.
+
+The lookup asks the per-workflow runs endpoint rather than paging the branch's
+recent runs, so one frequent cron cannot crowd another workflow out of its own
+history. It used to read fifty branch-wide runs and filter afterwards, which the
+five-minute public web probe compressed to about four hours of coverage; any
+workflow whose previous run was older than that found no evidence and silently
+dropped its recovery message.
 
 ### Scoping staging alerts to the freeze window
 
@@ -241,8 +279,8 @@ Requires two org secrets, reaching the workflow via `secrets: inherit`:
 must be a member of that channel.
 
 `workflow_dispatch` on the reusable itself is a smoke test: it posts a sample
-message in either style, skipping the prior-run lookup so `recovered` always
-posts.
+message in either style, skipping the prior-run lookup so both `recovered` and a
+repeated `failed` always post.
 
 ### The one failure it cannot see: a run that never started
 

@@ -63,13 +63,17 @@ inputs:
     required: false
     default: ""
   github-token:
-    description: "Token for the prior-run lookup behind the recovery message. Needs actions:read."
+    description: "Token for the prior-run lookup that both suppresses a duplicate alert and licenses a recovery. Needs actions:read. Without it every run of a standing failure re-pages."
     required: false
     default: ""
   force-post:
-    description: "'true' to post a recovery with no prior failure to recover from. The smoke test, and nothing else: a release-train wrapper must leave this alone or every manual run reports a recovery that did not happen."
+    description: "'true' to post with no prior run to compare against, in either direction. The smoke test, and nothing else: a release-train wrapper must leave this alone or every manual run reports a recovery that did not happen."
     required: false
     default: "false"
+  repeat-alert-after-minutes:
+    description: "How long a still-failing pipeline stays quiet before it reports the failure again. Repeats are collapsed so a standing breakage does not page every run, but never indefinitely: a run's conclusion says that it failed, not what failed, so a suppressed repeat can hide an outage that grew. Raise it for a noisy known breakage; do not disable it."
+    required: false
+    default: "60"
   branch:
     description: "Branch to name in the message. Defaults to the running ref, which is wrong for a `workflow_run` caller, where GITHUB_REF is the default branch rather than the branch whose pipeline finished."
     required: false
@@ -123,23 +127,31 @@ runs:
         python3 "${SHARED}/scripts/freeze_state.py" read \
           --repo "${REPO}" --ruleset-name "${RULESET_NAME}" --on-error escalate
 
-    # For the recovery message, find out how the PREVIOUS conclusive run of this
-    # same workflow on this same branch ended. This step only gathers evidence;
-    # `notify_decision.py` below decides what it means.
+    # Find out how the PREVIOUS conclusive run of this same workflow on this same
+    # branch ended. This step only gathers evidence; `notify_decision.py` below
+    # decides what it means.
+    #
+    # BOTH directions need that evidence. A failure used to post unconditionally
+    # and skip this lookup, so a standing breakage re-paged on every run:
+    # ENG-2324's five-minute probe turned one unfixed production route into about
+    # 220 Slack messages a day. The conclusion that suppresses a duplicate alert
+    # is the same conclusion that licenses a recovery, so one lookup serves both
+    # and the two directions fail open opposite ways — no evidence pages, and no
+    # evidence withholds a recovery.
     #
-    # Skipped when nothing it could find would change the outcome: a failure posts
-    # regardless, `force-post` posts regardless, and a freeze-scoped caller outside
-    # the window posts nothing either way. `steps.freeze.outputs.frozen` is empty
-    # when the freeze step did not run, which is not 'false', so an unscoped caller
-    # still reaches this.
+    # Still skipped when nothing it could find would change the outcome:
+    # `force-post` posts regardless, and a freeze-scoped caller outside the window
+    # posts nothing either way. `steps.freeze.outputs.frozen` is empty when the
+    # freeze step did not run, which is not 'false', so an unscoped caller still
+    # reaches this.
     #
     # BRANCH is deliberately the running ref rather than `inputs.branch`. For a
     # `workflow_run` caller those differ: the run is attributed to the default
     # branch, so that is where its own history lives, while the message wants the
     # branch whose pipeline finished.
     - name: Check how the previous run ended
       id: prev
-      if: inputs.status == 'recovered' && inputs.force-post != 'true' && steps.freeze.outputs.frozen != 'false'
+      if: inputs.force-post != 'true' && steps.freeze.outputs.frozen != 'false'
       shell: bash
       env:
         GH_TOKEN: ${{ inputs.github-token }}
@@ -174,24 +186,69 @@ runs:
         WF_PATH="${WORKFLOW_REF%%@*}"
         export WF_PATH="${WF_PATH#"${REPO}/"}"
 
-        # Most recent CONCLUSIVE run on this branch, excluding the current one:
-        # cancelled and skipped runs are not evidence either way.
-        if ! PREV=$(gh api "repos/${REPO}/actions/runs?branch=${BRANCH}&status=completed&per_page=50" \
-          --jq '[ .workflow_runs[]
-                  | select(.path == env.WF_PATH or .name == env.WORKFLOW)
-                  | select((.id|tostring) != env.RUN_ID)
-                  | select(.conclusion == "success" or .conclusion == "failure"
-                           or .conclusion == "timed_out" or .conclusion == "startup_failure")
-                ][0].conclusion // ""' 2>&1); then
+        # Ask for THIS workflow's history rather than the branch's most recent
+        # fifty runs. A frequent cron on the same branch crowds every other
+        # workflow out of a branch-wide page: at ENG-2324's five-minute cadence
+        # fifty slots span about four hours, so an hourly workflow found no
+        # evidence of itself and silently dropped its recovery message. The
+        # per-workflow endpoint takes the file name and cannot be crowded out.
+        #
+        # A caller whose `workflow_ref` points outside this repo has no such
+        # endpoint here, so that case keeps the branch scan and its name filter.
+        ENDPOINT="repos/${REPO}/actions/runs?branch=${BRANCH}&status=completed&per_page=50"
+        CALLER='select(.path == env.WF_PATH or .name == env.WORKFLOW)'
+        case "$WF_PATH" in
+          .github/workflows/*)
+            ENDPOINT="repos/${REPO}/actions/workflows/${WF_PATH##*/}/runs?branch=${BRANCH}&status=completed&per_page=10"
+            CALLER='.'
+            ;;
+        esac
+
+        # Most recent CONCLUSIVE run of this workflow on this branch, excluding
+        # the current one: cancelled and skipped runs are not evidence either way.
+        #
+        # Also measure the CURRENT FAILURE STREAK and when it started, which is
+        # what lets a repeated failure be quiet without being silent. The streak
+        # is the leading run of failures in this history; its oldest member dates
+        # the breakage. A page that truncates the streak understates its age,
+        # which makes the reminder fire sooner rather than later.
+        read -r -d '' RUNS_JQ <<'JQ' || true
+        [ .workflow_runs[]
+          | CALLER_FILTER
+          | select((.id|tostring) != env.RUN_ID)
+          | select(.conclusion == "success" or .conclusion == "failure"
+                   or .conclusion == "timed_out" or .conclusion == "startup_failure")
+        ]
+        | map({conclusion, created_at})
+        | (map(.conclusion == "failure" or .conclusion == "timed_out"
+               or .conclusion == "startup_failure") | index(false)) as $break
+        | (if $break == null then length else $break end) as $streak
+        | [ (.[0].conclusion // ""),
+            (.[0].created_at // ""),
+            ($streak | tostring),
+            (if $streak > 0 then .[$streak - 1].created_at else "" end)
+          ]
+        | @tsv
+        JQ
+        RUNS_JQ="${RUNS_JQ//CALLER_FILTER/${CALLER}}"
+
+        if ! ROW=$(gh api "$ENDPOINT" --jq "$RUNS_JQ" 2>&1); then
           # Almost always a missing `actions: read` grant on the caller job. Report
           # no evidence rather than failing: an unreadable history has to read as
           # "not a recovery", because a notify step must never redden a green run.
-          echo "Could not read run history, so there is no evidence either way: ${PREV}"
+          # It also has to read as "not a duplicate", so the alert still pages.
+          echo "Could not read run history, so there is no evidence either way: ${ROW}"
           exit 0
         fi
 
-        echo "prev_conclusion=${PREV}" >> "$GITHUB_OUTPUT"
+        IFS=$'\t' read -r PREV PREV_STARTED STREAK STREAK_STARTED <<<"${ROW}"
+        {
+          echo "prev_conclusion=${PREV}"
+          echo "prev_started_at=${PREV_STARTED}"
+          echo "streak_started_at=${STREAK_STARTED}"
+        } >> "$GITHUB_OUTPUT"
         echo "Previous conclusive run on ${BRANCH} concluded '${PREV:-none}'."
+        echo "Consecutive failures before this run: ${STREAK:-0}, oldest at ${STREAK_STARTED:-n/a}."
 
     # One decision, one place, with tests. See scripts/notify_decision.py for the
     # three outcomes and why `force-post` is an input rather than an event check.
@@ -205,9 +262,15 @@ runs:
         FROZEN: ${{ steps.freeze.outputs.frozen }}
         FORCE_POST: ${{ inputs.force-post }}
         PREV_CONCLUSION: ${{ steps.prev.outputs.prev_conclusion }}
+        PREV_STARTED_AT: ${{ steps.prev.outputs.prev_started_at }}
+        STREAK_STARTED_AT: ${{ steps.prev.outputs.streak_started_at }}
+        REPEAT_AFTER_MINUTES: ${{ inputs.repeat-alert-after-minutes }}
       run: |
         set -euo pipefail
         python3 "${SHARED}/scripts/notify_decision.py" \
+          --streak-started-at "${STREAK_STARTED_AT}" \
+          --prev-started-at "${PREV_STARTED_AT}" \
+          --repeat-alert-after-minutes "${REPEAT_AFTER_MINUTES}" \
           --status "${STATUS}" \
           --freeze-scoped "${FREEZE_SCOPED}" \
           --frozen "${FROZEN}" \