Skip to content

fix(llm): raise one exception type per MindsHub billing stop (ENG-813) - #505

Merged
lucas-koontz merged 3 commits into
stagingfrom
fix/eng-813-billing-stops-name-the-limit
Sep 27, 2026
Merged

lucas-koontz merged 3 commits into
stagingfrom
fix/eng-813-billing-stops-name-the-limit

Conversation

@lucas-koontz

@lucas-koontz lucas-koontz commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

User story

As a Cowork user whose task just stopped over MindsHub billing
I want the stop message, the settings screens and the console to agree on which limit stopped me
So that I know whether to wait, switch to MindsHub Air, add funds, or ask my admin

Linear: ENG-813

Why this matters

On hosted web Cowork, every spent free Air allowance tells the user "Your balance ran out", and a tripped free-Air fuse reads as "the provider is overloaded". The reason is lost inside anton. anton raises one TokenLimitExceeded for every MindsHub billing refusal, and the hosted worker forwards only "TypeName: message", so cowork-server cannot tell the reasons apart. anton also retries the fuse three times against a limit that stays shut until the end of the UTC day.

What should happen

Each MindsHub gate refusal raises its own type, and the type name is the part that survives a hosted turn:

Gate reason Status anton raises
wallet_empty 402 WalletEmptyError
included_allowance_exhausted 429 AllowanceExhaustedError
free_air_daily_spend_fuse_exceeded 429 FreeServingPausedError, never retried
permission_denied + X-MindsHub-Deny-Detail: model_restricted 403 ModelRestrictedError

All three billing types subclass TokenLimitExceeded, and ModelRestrictedError subclasses ModelUnavailableError, so every existing catch keeps working.

Acceptance criteria

  • Each gate billing code raises its own subclass, in both SDK dialects and on both mid-stream lanes, with the gate's reason, status_code and a validated reset_at read off X-MindsHub-Reset-At.
  • A hosted turn_failed frame from a billing stop with a reset_at carries it as its own key; every other failure's frame is unchanged.
  • The fuse makes one session attempt and is never classified as transient; a velocity rate_limited 429 stays transient. The SDK's own retries stop on the gateway's x-should-retry: false (mindshub_inference#592).
  • A hosted turn_failed string starts with the subclass name and fits the 300-character cap.
  • One typed parse of the provider error body replaces the hand-split reads, with each reader's field order unchanged (checked against the previous code on 60,000 randomized bodies).
  • A 403 with the model_restricted deny detail (header or error.deny_detail) raises ModelRestrictedError naming the model; a plain 403 permission_denied is unchanged.
  • Negative: a provably third-party origin never produces any of these types or a reset_at. The legacy 429-detail branch and OpenAI insufficient_quota still raise plain TokenLimitExceeded.
  • Every change has a regression test that fails with the change reverted.

How to test

  1. uv run --group dev pytest tests/test_status_error_mapper.py tests/test_transient_retry.py tests/test_transient_retry_e2e.py tests/test_curated_errors.py tests/test_cloud_turn_entrypoint.py. All pass.
  2. Revert mindshub_billing_stop in anton/core/llm/provider.py to raise plain TokenLimitExceeded. The type tests, the cloud_turn wire-name test and the e2e tests fail.
  3. Remove free_air_daily_spend_fuse_exceeded from _MINDSHUB_BILLING_STOPS. The no-retry session test and the classify_transient fuse rows fail.

Notes for the reviewer

This must reach workers after cowork-server's type table, in each environment. Merging to staging pushes the minds-anton-scratchpad:staging image, and merging to main pushes :production with no approval step. Worker pods pull the moving tag at their next start, so no scratchpad-controller deploy holds it back. cowork-server's current remote_turn_error matches only the name TokenLimitExceeded, so if these names reach a worker first, hosted billing stops fall to the generic "An unexpected error occurred." Merge to staging only after cowork-server#577 is deployed on staging, and promote to main only after it is live in production. "On main" is not enough: cowork-server's production deploy waits for an approval after its main merge.

Desktop gets this from a stable release. Packaged desktops resolve anton-agent from PyPI and skip staging rc pre-releases, so they pick up this change after the main release, through the desktop's anton-only update. Until then a desktop fuse is still retried by the older session loop.

The copy changed on purpose. The request-time message uses a colon instead of an em-dash, and the allowance copy says "you have no free MindsHub Air allowance left" instead of "included token allowance is exhausted", which is true both for an org that spent its allowance and for one that never had one. No copy names the allowance's size or window.

One helper owns the origin gate. mindshub_billing_stop and mindshub_model_restriction replace four hand-copied branches. The provably-foreign check is unchanged: an unknown origin stays trusted, as before.

Left alone for other tickets. insufficient_quota stays plain TokenLimitExceeded for ENG-2263, whose ProviderBillingError must not subclass TokenLimitExceeded. Expect adjacent-line conflicts in CURATED_PROVIDER_ERRORS.

The gateway error body is typed now (review round). ProviderErrorBody parses either dialect once; classify_transient keeps its envelope-or-top fallback explicitly.

Verified locally

Check Result
Targeted suites (5 files), after rebase onto staging 374 passed
uv run --group dev pytest tests/ --ignore=tests/e2e 4235 passed, 60 skipped
pytest tests/e2e/ 42 passed
uv lock --check clean, lockfile unchanged
Review round: uv run --group dev pytest tests/ --ignore=tests/e2e 4244 passed, 60 skipped
Review round: pytest tests/e2e/ 42 passed
Regression proofs each wrong implementation planted, tests failed, file restored byte for byte
Pre-PR sweep diff integrity clean, no provenance or debug leftovers

Ships with

Anchor: mindsdb/cowork#1027 carries the deploy label; its environment pr-cowork-1027 runs cowork#1027's own build and the staging worker image, so the hosted type names and the hosted reset_at arrive there only after anton and scratchpad-controller merge to staging.

Merge order:

  1. mindsdb/auth#587 and mindsdb/mindshub_inference#592, in either order. Both are additive and safe alone.
  2. mindsdb/mindshub_frontend#1601, at any point.
  3. mindsdb/cowork#1027. Preferred before cowork-server's new codes deploy on web, so free_serving_paused and model_restricted get their cards. Not enforceable on desktop, where the sidecar updates from PyPI on its own schedule; an older app shows the server's sentence without the Add funds button.
  4. mindsdb/cowork-server#577.
  5. mindsdb/scratchpad-controller#67, at any point. It forwards reset_at only when a worker sends it.
  6. mindsdb/anton#505, last, with two gates. Merge to staging only after cowork-server#577 is deployed on staging. Promote anton to main only after cowork-server#577 is live in production. The minds-anton-scratchpad:staging and :production tags move on those merges, and worker pods pull the moving tag at their next start (imagePullPolicy: Always), so no controller deploy holds the new exception names back. If they reach a worker before cowork-server knows them, every hosted billing stop reads "An unexpected error occurred."

- WalletEmptyError, AllowanceExhaustedError and FreeServingPausedError subclass
  TokenLimitExceeded, so a hosted turn's "TypeName: message" names the limit.
- The free-Air daily fuse is a billing stop, never retried as a rate limit.
- ModelRestrictedError for a 403 carrying X-MindsHub-Deny-Detail model_restricted.
- One shared helper for the OpenAI, mid-stream and Anthropic mappers; the
  MindsHub-origin gate is unchanged, and reset_at is read off X-MindsHub-Reset-At.

Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console.

Refs: ENG-813
Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console.

Refs: ENG-813
@lucas-koontz lucas-koontz self-assigned this Sep 25, 2026
…r body once (ENG-813)

- A hosted turn_failed frame from a MindsHub billing stop that knows when
  its limit lifts now carries reset_at beside the error string, so the web
  fuse and allowance cards can name the time. Every other failure is
  unchanged.
- The allowance stop says "you have no free MindsHub Air allowance left",
  true for an org that spent its allowance and one that never had one.
- One typed parse of the provider error body (ProviderErrorBody) replaces
  the hand-split top-level and envelope reads in the OpenAI and Anthropic
  mappers, wallet_denial_code, mindshub_model_restriction and
  classify_transient, with each reader's field order unchanged.
- The origin-gate docstring states cowork-server's check order correctly;
  ragged comments are re-flowed.

Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console.

Refs: ENG-813
@lucas-koontz
lucas-koontz merged commit 94b3ef0 into staging Sep 27, 2026
10 checks passed
@lucas-koontz
lucas-koontz deleted the fix/eng-813-billing-stops-name-the-limit branch September 27, 2026 06:21
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant