feat(monitoring): probe public web endpoints (ENG-2324)

[lucas-koontz]

User story

As an engineer on call for MindsHub
I want the public Console routes, production and staging Cowork, and website checked through the edge
So that route, DNS, certificate, proxy, and website failures alert me before customers report them

Why this matters

Production and staging have returned redirects and 403 responses on routes that Kubernetes still considered healthy. Cloudflare covers four representative public paths every minute, while this independent runner checks the broader 21-endpoint public matrix every five minutes.

Acceptance criteria

• A workflow runs every five minutes and supports manual dispatch.
• It checks nine reviewed routes on production and staging Console, both Cowork roots, and https://mindshub.ai/, for 21 exact public endpoints.
• Console and Cowork responses require exact status 200 and <div id="root"></div>; the website requires exact status 200 and its og:site_name marker.
• Redirects, 200 error pages, and a body carrying the wrong endpoint marker fail.
• The first attempt covers all 21 endpoints, then only failures retry after 15 seconds.
• A recovered retry remains green and silent; persistent failures make the run red.
• One notifier label names every persistent endpoint and observed result. The next green run uses the existing recovery path.
• The endpoint list, markers, cadence, activation gate, and excluded authenticated journey are documented and contract-tested.

How to test

1. Run uv run pytest, Ruff, and actionlint .github/workflows/console-route-probe.yml.
2. After ENG-2317 reaches production, run uv run --python 3.12 scripts/probe_console_routes.py --retry-delay-seconds 0. All 21 endpoints must pass.
3. Dispatch the workflow after that gate and confirm a healthy run posts nothing.
4. In an isolated exercise, make one endpoint return 403 or 301 twice. Confirm the run fails and one message identifies the endpoint and result.
5. Restore the endpoint and confirm the next run posts recovery.
6. Return 200 with the wrong configured marker for a Console route, Cowork, and the website. Confirm all three fail.

Notes for the reviewer

• ENG-2317 is a hard activation gate. The current read-only probe still sees production /cowork return 301 and /assets/ return 403. Do not merge, promote, or dispatch this workflow until ENG-2317 is deployed and all 21 endpoints pass.
• Staging Cowork is the twentieth endpoint; the website is the twenty-first. Staging Cowork uses the same root route and SPA marker as production Cowork. The configuration, tests, counts, and README now agree.
• No live workflow dispatch was run. A dispatch inherits the real notifier secrets and can send a failure or recovery message.
• The retry is a debounce. Only first-attempt failures repeat after 15 seconds, which keeps transient failures silent without delaying healthy endpoints.
• Cloudflare is complementary. Terraform #203 checks four representative public paths every minute and keeps three non-paging diagnostics. This workflow runs outside Cloudflare and covers the complete route matrix.
• The authenticated 15-minute journey remains excluded. It needs its own interaction contract and test plan before it can provide an actionable signal.

Verified locally

Check	Result
uv run pytest	183 passed
uv run pytest tests/test_probe_console_routes.py	35 passed
Ruff check and format check	Passed
actionlint .github/workflows/console-route-probe.yml	Passed
Staging Cowork public request	HTTP 200 with the exact SPA marker
Full read-only public probe	19 of 21 passed; only the two recorded ENG-2317 production routes failed
Stale-count regression	The prior head failed five contracts after the endpoint was added; the updated contracts pass
All-endpoint network outage regression	Every endpoint and result remains in the bounded alert label; verbose details remain in the run log
git diff --check	Passed

Ships with

Merged sibling: mindsdb/terraform#203

Merge order: Apply the merged terraform#203 revision first. Merge this PR only after ENG-2317 reaches production and all 21 public endpoints pass locally.