fix(llm): raise one exception type per MindsHub billing stop (ENG-813) - #505
Merged
Merged
Conversation
- WalletEmptyError, AllowanceExhaustedError and FreeServingPausedError subclass TokenLimitExceeded, so a hosted turn's "TypeName: message" names the limit. - The free-Air daily fuse is a billing stop, never retried as a rate limit. - ModelRestrictedError for a 403 carrying X-MindsHub-Deny-Detail model_restricted. - One shared helper for the OpenAI, mid-stream and Anthropic mappers; the MindsHub-origin gate is unchanged, and reset_at is read off X-MindsHub-Reset-At. Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console. Refs: ENG-813
Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console. Refs: ENG-813
…r body once (ENG-813) - A hosted turn_failed frame from a MindsHub billing stop that knows when its limit lifts now carries reset_at beside the error string, so the web fuse and allowance cards can name the time. Every other failure is unchanged. - The allowance stop says "you have no free MindsHub Air allowance left", true for an org that spent its allowance and one that never had one. - One typed parse of the provider error body (ProviderErrorBody) replaces the hand-split top-level and envelope reads in the OpenAI and Anthropic mappers, wallet_denial_code, mindshub_model_restriction and classify_transient, with each reader's field order unchanged. - The origin-gate docstring states cowork-server's check order correctly; ragged comments are re-flowed. Lucas Koontz, ENG-813: Make every billing stop name the limit that fired, in Cowork and the console. Refs: ENG-813
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User story
As a Cowork user whose task just stopped over MindsHub billing
I want the stop message, the settings screens and the console to agree on which limit stopped me
So that I know whether to wait, switch to MindsHub Air, add funds, or ask my admin
Linear: ENG-813
Why this matters
On hosted web Cowork, every spent free Air allowance tells the user "Your balance ran out", and a tripped free-Air fuse reads as "the provider is overloaded". The reason is lost inside anton. anton raises one
TokenLimitExceededfor every MindsHub billing refusal, and the hosted worker forwards only"TypeName: message", so cowork-server cannot tell the reasons apart. anton also retries the fuse three times against a limit that stays shut until the end of the UTC day.What should happen
Each MindsHub gate refusal raises its own type, and the type name is the part that survives a hosted turn:
wallet_emptyWalletEmptyErrorincluded_allowance_exhaustedAllowanceExhaustedErrorfree_air_daily_spend_fuse_exceededFreeServingPausedError, never retriedpermission_denied+X-MindsHub-Deny-Detail: model_restrictedModelRestrictedErrorAll three billing types subclass
TokenLimitExceeded, andModelRestrictedErrorsubclassesModelUnavailableError, so every existing catch keeps working.Acceptance criteria
reason,status_codeand a validatedreset_atread offX-MindsHub-Reset-At.turn_failedframe from a billing stop with areset_atcarries it as its own key; every other failure's frame is unchanged.rate_limited429 stays transient. The SDK's own retries stop on the gateway'sx-should-retry: false(mindshub_inference#592).turn_failedstring starts with the subclass name and fits the 300-character cap.model_restricteddeny detail (header orerror.deny_detail) raisesModelRestrictedErrornaming the model; a plain 403permission_deniedis unchanged.reset_at. The legacy 429-detail branch and OpenAIinsufficient_quotastill raise plainTokenLimitExceeded.How to test
uv run --group dev pytest tests/test_status_error_mapper.py tests/test_transient_retry.py tests/test_transient_retry_e2e.py tests/test_curated_errors.py tests/test_cloud_turn_entrypoint.py. All pass.mindshub_billing_stopinanton/core/llm/provider.pyto raise plainTokenLimitExceeded. The type tests, the cloud_turn wire-name test and the e2e tests fail.free_air_daily_spend_fuse_exceededfrom_MINDSHUB_BILLING_STOPS. The no-retry session test and theclassify_transientfuse rows fail.Notes for the reviewer
This must reach workers after cowork-server's type table, in each environment. Merging to
stagingpushes theminds-anton-scratchpad:stagingimage, and merging tomainpushes:productionwith no approval step. Worker pods pull the moving tag at their next start, so no scratchpad-controller deploy holds it back. cowork-server's currentremote_turn_errormatches only the nameTokenLimitExceeded, so if these names reach a worker first, hosted billing stops fall to the generic "An unexpected error occurred." Merge tostagingonly after cowork-server#577 is deployed on staging, and promote tomainonly after it is live in production. "Onmain" is not enough: cowork-server's production deploy waits for an approval after itsmainmerge.Desktop gets this from a stable release. Packaged desktops resolve anton-agent from PyPI and skip staging
rcpre-releases, so they pick up this change after themainrelease, through the desktop's anton-only update. Until then a desktop fuse is still retried by the older session loop.The copy changed on purpose. The request-time message uses a colon instead of an em-dash, and the allowance copy says "you have no free MindsHub Air allowance left" instead of "included token allowance is exhausted", which is true both for an org that spent its allowance and for one that never had one. No copy names the allowance's size or window.
One helper owns the origin gate.
mindshub_billing_stopandmindshub_model_restrictionreplace four hand-copied branches. The provably-foreign check is unchanged: an unknown origin stays trusted, as before.Left alone for other tickets.
insufficient_quotastays plainTokenLimitExceededfor ENG-2263, whoseProviderBillingErrormust not subclassTokenLimitExceeded. Expect adjacent-line conflicts inCURATED_PROVIDER_ERRORS.The gateway error body is typed now (review round).
ProviderErrorBodyparses either dialect once;classify_transientkeeps its envelope-or-top fallback explicitly.Verified locally
staginguv run --group dev pytest tests/ --ignore=tests/e2epytest tests/e2e/uv lock --checkuv run --group dev pytest tests/ --ignore=tests/e2epytest tests/e2e/Ships with
Anchor: mindsdb/cowork#1027 carries the
deploylabel; its environmentpr-cowork-1027runs cowork#1027's own build and the staging worker image, so the hosted type names and the hostedreset_atarrive there only after anton and scratchpad-controller merge tostaging.Merge order:
free_serving_pausedandmodel_restrictedget their cards. Not enforceable on desktop, where the sidecar updates from PyPI on its own schedule; an older app shows the server's sentence without the Add funds button.reset_atonly when a worker sends it.stagingonly after cowork-server#577 is deployed on staging. Promote anton tomainonly after cowork-server#577 is live in production. Theminds-anton-scratchpad:stagingand:productiontags move on those merges, and worker pods pull the moving tag at their next start (imagePullPolicy: Always), so no controller deploy holds the new exception names back. If they reach a worker before cowork-server knows them, every hosted billing stop reads "An unexpected error occurred."