Skip to content

Report Grok usage with recorded and estimated spend - #3135

Closed
olddonkey wants to merge 47 commits into
steipete:mainfrom
olddonkey:feat/grok-real-token-usage
Closed

olddonkey wants to merge 47 commits into
steipete:mainfrom
olddonkey:feat/grok-real-token-usage

Conversation

@olddonkey

@olddonkey olddonkey commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem and resulting behavior

Grok's local fallback reads ending context occupancy from signals.json, which is not completed-turn usage, and publishes no local cost. This PR reads completed turns from bounded native CLI logs and carries recorded-versus-estimated cost through the menu and Usage & Spend.

  • Read both session/update and _x.ai/session/update records in updates.jsonl, with daily, model, and request breakdowns.
  • Prefer a positive outer usage.costUsdTicks / 1e10 as the authoritative turn total. Count it once. Show nested model dollars only when all nested ticks exist and reconcile to that total; otherwise retain model tokens with unknown model dollars. Missing recorded cost falls back to disclosed public xAI list prices.
  • Keep useful remote data during billing failures while rescanning local sessions. Consumers select the newer publication for the current configuration and preserve account override isolation.
  • Retain cost provenance after menu and dashboard window/day filtering. Populated surfaces disclose Grok CLI-recorded spend, list price where unrecorded · not a bill.
  • Refresh the xAI models.dev catalog for Grok-only installs and keep its pricing fingerprint independent from native Codex pricing.

OpenCodex integration

With the existing Include OpenCodex usage logs switch enabled (off by default), Usage & Spend also includes reported physical Grok OAuth attempts. The producer contract originally proposed in OpenCodex #3642 has now landed in dev through the attributed carry #3762, merged as f00f2bcaea251ebe7ad4de9e38337b4be0ccee47. It writes attempts[].credentialSource from the resolved upstream transport. Compatibility is verified against the pinned producer commit in the evidence below; availability in a released OpenCodex version is not claimed.

The landed carry's immutable source contract has additionally been inspected directly, including resolved-adapter stamping, persistence normalization, and OAuth/API-key regression assertions. The executable capture remains pinned to its documented producer revision; this source audit does not claim a new capture or released-version coverage.

Only provider: "xai", credentialSource: "grok-oauth" attempts with an upstream send and reported token usage qualify. API-key traffic, historic rows, locally answered requests, unknown sources, and estimated/unreported token counts stay excluded. Current configuration and top-level credential metadata cannot retroactively classify usage.

Combo requests contribute each qualifying attempt's own token counts, never the parent aggregate. Duplicate request IDs are resolved before grouping; duplicate attempt ordinals are rejected. SQLite cache schema 3 retains attempt metadata and rebuilds older derived caches from the log. OpenCodex dollars use list prices and remain estimates, including when combined with native CLI-recorded spend. Missing token classes and unknown prices retain tokens without inventing a dollar value. The Grok menu continues to use native CLI logs.

Producer-to-dashboard evidence and review fixes

The committed evidence report and raw ledger fixture contain output from OpenCodex's unmodified production handlers and durable usage writer at 146ed679c9633e5d68726217fcadc8e0b107339b. Two localhost HTTP requests exercised OAuth 401 replay through Responses and native Chat API-key dispatch. Upstream responses and credentials used isolated fixtures; this is production-path capture evidence, not live vendor authentication or billing evidence. The capture helper rejects unexpected external fetches and is reproducible against that pinned checkout.

CodexBar imports the exact captured bytes through the production disk loader, with no injected entries or loader closure, and reopens the persisted cache:

producer_capture_sha256=ef6d8758b40910f6e5993d5b5a105a2ad2834c6c1bd0565ab87b61cf091c4978
producer_log_rows=2 total_reported_tokens=10 grok_oauth_tokens=5
producer_import_dashboard_tokens=5 cache_reopen_bytes=0
producer_api_key_only_subscription_rows=0

Standalone reports retain explicit xAI custom-price estimates from either the caller overlay or the application overlay, including known zero. Raw xAI records without an explicit price remain token-only, and the subscription fan-out still excludes API-key and historic records. opencode and opencode-free retain their existing catalog and custom prices.

Application overlays first match the original model name before the provider-qualified catalog name. Bare keys retain precedence when both keys exist; incomplete rates stay unknown, cached-input accounting is preserved, and caller-supplied custom pricing still takes precedence over the application overlay. New regressions cover these cases through the application-overlay parameter and the standalone disk/cache loader. Coverage includes known-zero overrides, incomplete rates, cache accounting, and both snapshot and application overlays.

Both async native Grok scan entry points propagate the executor's cancellation callback through discovery, JSONL reads, and aggregation. An in-flight regression cancels after parsing begins, observes CancellationError, and confirms the next queued scan runs within one second. Cancelled partial parses are uncacheable and cannot establish complete history. The earlier c3919a224 serial proof stopped at 4,874/40,000 decoded records and released the queue after 0.001864583 seconds. The evidence report also retains the initial measurement.

Native numeric safety

Native token conversion rejects booleans, fractions, negative values, and out-of-range numbers without trusting NSNumber's clamping integer bridge. An absent token class retains the established zero default; a malformed count remains unknown. Explicit valid totals retain precedence, so decoding never adds input and output unnecessarily.

Checked addition carries unknown or overflowed values through model, day, and window aggregation, including after later valid records and cached-log rereads. Valid neighboring token classes and CLI-recorded spend remain available. Incomplete token accounting does not establish full history coverage; estimated pricing requires representable inputs. Production-scanner regressions cover Int.max plus one within a turn, sums across turns and days, nested models, invalid JSON number types, explicit totals, snapshots, and cache reuse.

The same checked, complete-count aggregation now covers remote-backed Grok menu projection, rolling-window requests, comparison summaries, and menu/Widget fallback totals. Unknown daily counts cannot be dropped into a plausible partial sum. An end-to-end scanner regression covers remote-backed and fallback projections for cross-day overflow and an unknown day followed by a valid day; menu, Widget, and dashboard totals stay unavailable, while narrowing to a valid one-day window restores its count.

Bounds and compatibility

Native scans run on the dedicated executor with limits of 64 MiB / 20,000 turns per file, 1 MiB per record, 256 sessions / 256 MiB / 100,000 turns per scan, and 4,096 discovery entries. The process cache retains at most 64 files or 50,000 turns. Cancellation and truncated history cannot publish complete coverage.

Merged main 8b254dbec11ddd5c5547878d9640e4e965306c71, retaining its checked token aggregation, daily spend ledger, native parser-revision migration, and Grok terminal-billing-failure work avoidance. The local-summary injection seam now invokes the PR's pricing-aware 365-day scanner only after the upstream billing/identity gate permits a snapshot. Failed billing still refreshes local spend through the app's existing independent fallback path.

OpenCodex token counts retain upstream's safe numeric conversion and truncation policy. Attempt ordinals and send counts remain exact integers: booleans, fractional counts, and out-of-range values cannot qualify an OAuth attempt. Aggregation retains overflow as unavailable while keeping valid neighboring token classes and Grok estimate coverage. The upstream cursor parser version invalidates legacy numeric caches; schema 3 still preserves request-time attempt provenance.

Regenerated the native parser hash from the merged source: a8559238a5fc0480. Current-main hash f5fdba377006d7be and prior-PR hash c3a879df4eff7187 join the compatible predecessors. Stored rows and checkpoints are retained, while upstream's per-file parser revisions schedule bounded reparsing where needed. SQLite adoption tests cover the current-main and prior-PR hashes, and upstream whitespace/subagent migration tests remain in the validation set. The architecture gate keeps its exact provider-reference checks at their updated source locations. Grok release notes are under 0.59.1 — Unreleased; all published sections match main.

Merge with main (2026-09-19)

Head: eb1647cc8ed68e39f5ed90eb8730ed3d182ee67d. Three merge commits bring the branch onto main at 0f8489dd8 (the third, eb1647cc8, only re-anchors the gatekeeper allowlist after three more main commits). The first two: the first (e7b2b2203, main at 88aab2ba6, 136 commits since the previous rebase) resolves eleven conflicting files; the second (32a73cd90, 25 further commits) only re-resolves the parser-hash family after main regenerated its hash again.

Main's provider-aware pricing replaced the Codex-only lookup this branch had extended, so the xAI behaviour is re-expressed on top of it rather than alongside it:

  • codexModelsDevPricingTargets keeps main's resolver-backed form, widened to xAI routes with this branch's grok -build alias fallback. The duplicate copy in CostUsagePricing+CodexResolver.swift is dropped; only the route union remains there.
  • The OpenCodex aggregator prices through main's providerCostUSD, with this branch's standalone-xAI gate kept ahead of it: raw xAI rows without Grok OAuth provenance stay token-only unless explicit custom prices exist.
  • OpenCodexUsageLog adopts main's TestProcessSafety path lookup; the branch's private test-detection helper is gone.
  • The live-consumer token projection follows main's relocation of the menu-card model into UsageStore+MenuCardModel.swift; the override-card regression now uses the store API.
  • Window summaries keep this branch's complete token sum and main's CheckedSum request total. ProviderRow keeps both incompleteRequestCount and the Grok cost disclaimer.
  • Parser hash regenerated to b004d0cf7d471304; main's 6d48baf0ed980828 and 6a4df886696f4ab5 are in the compatible predecessor list and the exact-list assertion. Gatekeeper anchors were re-pointed at the merged sources, and the merged openai route cluster and the xAI alias fallback carry design markers.

Validation:

  • First merge head e7b2b2203: make check clean; make test 114 groups, 11,456 tests, 0 failures, 0 retries (792 s).
  • Final head 32a73cd90: make check 0 violations in 2,431 files with the parser-hash check clean; focused Grok, OpenCodex, CostUsage, gatekeeper, spend-dashboard and menu-card suites pass, with the concurrency and bounded-progress suites confirmed green in isolation. make test on this head: 116 groups, 11,577 tests, 0 failures, 0 retries (702 s).

Merge with main (2026-09-21)

Head: 75945a360f90abe5abac262c6c80d4cf65ec2326, on main at d8d0f3394 (46 commits since the previous merge). Seven files conflicted:

  • PreferencesSpendDashboardPane.swift: main's feat(spend): group usage details by provider #3353 moved the per-provider rows into SpendDashboardProviderBreakdown, so the Grok cost disclosure moved with them — on the source row when a provider lists its sources, and under the provider header when its single native source is folded into it.
  • UsageStore.swift / UsageStore+TokenCost.swift: main relocated resetTokenUsageState into the token-cost extension; the Grok scan cancellation moved with it. The live-consumer selection this branch adds and main's localized no-data message both remain.
  • Parser hash regenerated to 611a82b6eb2b5ffc; main's 03e43d1217789d16 and the previous PR head b004d0cf7d471304 join the compatible predecessor list, the exact-list assertion, and the SQLite adoption arguments. Gatekeeper anchors re-pointed at the merged sources.
  • Release notes moved under 0.63.1 — Unreleased; published sections are identical to main.

Live-corpus window defect (2026-09-21)

A user ran this branch's own live proof on a 547M-token corpus and got a correct scanner snapshot but an empty dashboard row. The shape my corpora never had: a window mixing priced days with days the scanner could not price at all — turns attributed to no model with no costUsdTicks, which the scanner declares as unpriced requests. Two consumers treated such a day as making the window unknown:

  • The generic window narrowing publishes a window cost only when every retained day carries one. Both Grok projections now sum the priced days through one shared helper.
  • The dashboard cost-consistency rule only knew Codex's and Cursor's explicitly unpriceable rows. A Grok day whose requests all went unpriced is now an explicit gap; the recorded total survives and the coverage counts disclose the unpriced requests.

The regression fails on the reported assertions with either fix reverted. The live proof now prints the coverage, active-day, unpriced-day and unpriced-request figures per window and asserts the documented no-total contract for a bounded scan, so a failing run says which shape it hit.

The standalone xAI gate in the OpenCodex aggregator now follows the resolved billing route rather than the model namespace, so router-hosted xai/... rows keep their recorded-provider price (#3676) while directly billed rows stay token-only without Grok OAuth provenance.

Validation

Verified on exact head 75945a360f90abe5abac262c6c80d4cf65ec2326, on main at d8d0f3394:

  • make check: passed — SwiftFormat clean, SwiftLint 0 violations in 2530 files, provider/package/documentation gates green.
  • make test: passed — 1299/1299 selections, 119/119 groups successful on the first pass, 0 failed groups, 0 retries, 0 timeouts (785.8 s).
  • The exact CI group that failed (GeminiStatusProbe…GrokCostUsagePricing, 8 suites, --no-parallel) passes on this head; GrokAccountContextTests 8/8.
  • git diff --check clean; 0 behind main.

#3805 (merged into main during the previous run) added a fallback regression whose fixture expected signals.json occupancy to count as tokens; it is re-expressed on a completed turn here, since this branch counts completed turns only. Earlier heads' validation is recorded in the PR comments; the historical native-corpus proof and the pinned OpenCodex producer capture are unchanged and remain supplemental.

Maintainer decision

Please revisit the 2026-08-21 Grok cost ruling before merge. It preferred public-card pricing based partly on my incorrect 1e9 divisor; #3345 established 1e10, and the corrected measurement explains the difference. This branch uses recorded spend with public-card fallback.

Whether existing Grok users should receive this disclosed dollar surface by default remains an owner decision. The accounting corrections and source labeling do not override that decision. Maintainer approval is still required.

@clawsweeper

clawsweeper Bot commented Aug 22, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

olddonkey added a commit to olddonkey/CodexBar that referenced this pull request Aug 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09cf7edb0f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/UsageStore+Refresh.swift Outdated
Comment thread Sources/CodexBarCore/Providers/Grok/GrokLocalSessionScanner.swift
@clawsweeper clawsweeper Bot added merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Aug 22, 2026
@clawsweeper

clawsweeper Bot commented Aug 22, 2026 •

Copy link
Copy Markdown

Codex review: blocked before merge. Reviewed September 21, 2026, 6:54 AM ET / 10:54 UTC (Revision 66).

ClawSweeper review

What this changes

Replace Grok context-occupancy totals with completed-turn usage, display recorded spend with labeled price estimates, and import eligible OpenCodex OAuth usage into Usage & Spend.

Merge readiness

⛔ Blocked before merge - 3 items remain

This remains useful work absent from main and the latest release. Earlier correctness findings are addressed, and the runtime evidence is sufficient; the revised default dollar-display policy remains an owner decision.

Priority: P2
Reviewed head: 75945a360f90abe5abac262c6c80d4cf65ec2326
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) Substantial runtime evidence and focused regression coverage support a sound patch with no remaining actionable code finding.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): Real native logs exercised the scanner and live menu/dashboard projection with recorded totals and disclosures; pinned producer output exercised disk import, OAuth/API-key separation, and cache reopening. Historical screenshots support the UI evidence; current regressions cover the independently reported mixed-day defect without claiming a new run of that user's corpus.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): Real native logs exercised the scanner and live menu/dashboard projection with recorded totals and disclosures; pinned producer output exercised disk import, OAuth/API-key separation, and cache reopening. Historical screenshots support the UI evidence; current regressions cover the independently reported mixed-day defect without claiming a new run of that user's corpus.
Evidence reviewed 12 items Policy and reviewed identities: Read the full root AGENTS.md; no nested AGENTS.md or maintainer-notes directory was found. Reviewed the introduced delta from d8d0f33 to 75945a3. Raw test-merge records contain the pinned main parent followed by the exact PR head, and its tree equals the head tree.
Still necessary on main and released code: Current main still sums totalTokensBeforeCompaction and contextTokensUsed from signals.json and publishes nil local costs. Inspection of v0.63.0's release commit found the same occupancy-based implementation. Neither implements this branch's completed-turn accounting.
Release comparison: The supplied latest release, v0.63.0, retains signals.json occupancy accounting and nil costs; this review does not identify a shipped fixing release.
Findings None None.
Security None None.

How this fits together

CodexBar turns local provider logs into usage and spending summaries for its menu and dashboard. This change processes Grok completed turns and optional OpenCodex attempt records while retaining cost provenance and incomplete-history information.

flowchart TD
  A[Grok CLI logs] --> B[Bounded completed-turn scanner]
  C[Recorded spend and price catalog] --> B
  B --> D[Daily usage and cost provenance]
  E[Optional OpenCodex logs] --> F[Reported OAuth attempt filter]
  F --> D
  D --> G[Window and coverage projection]
  G --> H[Menu and spend dashboard]
Loading

Decision needed

Question Recommendation
Should existing Grok users with cost tracking enabled receive CLI-recorded spend with disclosed list-price fallback by default? Approve recorded-first display: Accept the proposed default for cost-tracking users with the existing provenance and non-bill disclosures.

Why: The revised accounting evidence supports recorded-first costs, but the owner has not approved this default-display change and the changes-requested review remains outstanding.

Before merge

  • Resolve merge risk (P2) - Existing Grok users with cost tracking enabled automatically gain recorded dollars and list-price fallback after upgrading; owner approval of this revised display policy remains unresolved.
  • Complete next step (P2) - Have steipete approve or narrow the revised Grok dollar-display default and resolve the outstanding changes-requested review.
  • Resolve maintainer decision - Resolve the maintainer decision shown above before merge.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test growth Production +1,788/−206 lines; tests +3,388/−155 lines The growth supports bounded parsing, cost provenance, and cache compatibility, with more added test coverage than production code.

Merge-risk options

Maintainer options:

  1. Accept the revised display default (recommended)
    Approve recorded spend with disclosed estimates for existing cost-tracking users after reconsidering the outstanding owner review.
  2. Gate dollars behind opt-in
    Preserve token accounting for existing users and add an explicitly approved opt-in with fresh-install and upgrade coverage.

Technical review

Best possible solution:

Use completed-turn accounting with recorded-first costs, visible estimate and coverage disclosures, and explicitly owner-approved upgrade defaults.

Do we have a high-confidence way to reproduce the issue?

Yes: main visibly sums context occupancy rather than completed-turn consumption, corroborated by independent real-session measurements; this review did not execute a reproduction.

Is this the best way to solve the issue?

Yes for the accounting approach: bounded completed-turn parsing and recorded-cost precedence fit the observed data, while the default dollar-display choice still requires owner approval.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against d8d0f3394989.

Labels

Label justifications:

  • P2: This improves Grok usage and spend accuracy without evidence of an urgent application-wide outage.
  • merge-risk: 🚨 compatibility: Upgrading changes the dollar information shown to existing Grok cost-tracking users before the revised default has owner approval.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Real native logs exercised the scanner and live menu/dashboard projection with recorded totals and disclosures; pinned producer output exercised disk import, OAuth/API-key separation, and cache reopening. Historical screenshots support the UI evidence; current regressions cover the independently reported mixed-day defect without claiming a new run of that user's corpus.
  • proof: sufficient: Contributor real behavior proof is sufficient. Real native logs exercised the scanner and live menu/dashboard projection with recorded totals and disclosures; pinned producer output exercised disk import, OAuth/API-key separation, and cache reopening. Historical screenshots support the UI evidence; current regressions cover the independently reported mixed-day defect without claiming a new run of that user's corpus.

Evidence

What I checked:

  • Policy and reviewed identities: Read the full root AGENTS.md; no nested AGENTS.md or maintainer-notes directory was found. Reviewed the introduced delta from d8d0f33 to 75945a3. Raw test-merge records contain the pinned main parent followed by the exact PR head, and its tree equals the head tree. (AGENTS.md:1, 75945a360f90)
  • Still necessary on main and released code: Current main still sums totalTokensBeforeCompaction and contextTokensUsed from signals.json and publishes nil local costs. Inspection of v0.63.0's release commit found the same occupancy-based implementation. Neither implements this branch's completed-turn accounting. (Sources/CodexBarCore/Providers/Grok/GrokLocalSessionScanner.swift:123, d8d0f3394989)
  • Release comparison: The supplied latest release, v0.63.0, retains signals.json occupancy accounting and nil costs; this review does not identify a shipped fixing release. (Sources/CodexBarCore/Providers/Grok/GrokLocalSessionScanner.swift:123, f3e718c897d5)
  • Maintainer review and outstanding scope decision: The owner's August 25 changes-requested review supports completed-turn accounting and requires standalone xAI isolation, populated disclosures, continued fallback refreshes, separate pricing fingerprints, and bounded discovery. Those repairs are present. The complete PR body still explicitly requests approval for recorded-first dollars with list-price fallback for existing users; the earlier willingness to merge predates that revised policy. (Sources/CodexBarCore/Providers/Grok/GrokProviderDescriptor.swift:87, 75945a360f90)
  • Review continuity: The direct-xAI pricing exception now follows the billing provider, preserving OpenRouter prices and explicit custom overrides. Both changelog entries are under 0.63.1 Unreleased. Checked arithmetic, bounded discovery, fresh-catalog initialization, and failed-billing rescans address the retained earlier findings. The latest introduced commit adds a completed-turn fixture to the upstream RPC fallback regression. (Sources/CodexBarCore/Vendored/OpenCodexUsage/OpenCodexUsageAggregator.swift:321, 75945a360f90)
  • Native behavior proof and independent feedback: Inspected both downloaded screenshots from Report Grok usage with recorded and estimated spend #3135 (comment) as historical UI evidence. Later real-session runs exercised the production scanner and live menu/dashboard models, reporting 98,631,812 tokens and $12.94957366 with recorded provenance. The independent 547M-token run exposed mixed priced/unpriced-day projection failures; the current shared window-cost helper and dashboard gap handling address that shape, with a focused regression. No post-repair rerun of that independent corpus is claimed. (Tests/CodexBarTests/GrokWindowProvenanceProofTests.swift:116, 75945a360f90)

Likely related people:

  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Chipagosfinest: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • olddonkey: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (65 earlier review cycles; latest 8 shown)
  • reviewed 2026-09-12T05:29:15.595Z sha 6d3d5af :: blocked before merge. :: [P2] Preserve overflow safety through the Grok menu projection
  • reviewed 2026-09-12T05:39:46.066Z sha a417361 :: blocked before merge. :: none
  • reviewed 2026-09-12T05:51:31.245Z sha a417361 :: blocked before merge. :: none
  • reviewed 2026-09-19T08:34:41.831Z sha e7b2b22 :: blocked before merge. :: [P1] [P1] Scope the xAI pricing exception to the billing provider | [P3] [P3] Move the Grok entries out of the published release section
  • reviewed 2026-09-19T08:44:00.414Z sha 32a73cd :: blocked before merge. :: [P1] [P1] Scope the xAI pricing exception to the billing provider | [P3] [P3] Move the Grok entries into the current unreleased section
  • reviewed 2026-09-19T08:56:08.241Z sha 32a73cd :: blocked before merge. :: [P1] [P1] Scope the xAI pricing exception to the billing provider | [P3] [P3] Move the Grok entries into the current unreleased section
  • reviewed 2026-09-19T15:35:05.884Z sha eb1647c :: blocked before merge. :: [P1] [P1] Scope the xAI pricing exception to the billing provider | [P3] [P3] Move the Grok entries into the current unreleased section
  • reviewed 2026-09-21T09:12:34.113Z sha 16be323 :: blocked before merge. :: none

@olddonkey
olddonkey force-pushed the feat/grok-real-token-usage branch from 08360b5 to e3cd3b9 Compare August 22, 2026 06:21
@olddonkey olddonkey changed the title Report real Grok token usage and list-price cost from CLI session logs Report real Grok token usage and list-price cost, from the CLI logs and OpenCodex alike Aug 22, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Aug 22, 2026
@olddonkey

Copy link
Copy Markdown
Contributor Author

Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against 09cf7edb0, which no longer exists — the branch has since been rebased onto 27c7f334e and the head is now 03e5a25dc, so I'm summarising here rather than replying in a stale diff.

P1 — Preserve the Grok fallback on repeated probe failures

Fixed in 923193ec0, Sources/CodexBar/UsageStore+Refresh.swift. The guard had been hoisted into the if provider == .grok, publication == nil condition, so a Grok failure with a publication fell through to the generic else if tokenCostRequiresProviderSnapshot { clearTokenSnapshot } branch. Grok now owns its branch outright and can never reach the clear:

if provider == .grok {
    if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
        Task { @MainActor [weak self] in
            await self?.scanAndPublishGrokLocalTokenSnapshot(...)
        }
    }
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
    self.clearTokenSnapshot(for: provider)
}

Regression coverage is in missing remote snapshot scans and publishes local tokens then clears empty data. Per the review's request it now drives two consecutive failing refreshes (03e5a25dc) rather than one — which matters here, because the first failure is what publishes through the fallback scan and only the second arrives with a publication in place, i.e. the failure that used to wipe the row. Both iterations assert the row still reads 77 tokens and that no redundant rescan ran.

P2 — Refresh pricing before scanning Grok sessions

Correct, and thank you — this was a genuine gap and not one the local tests would have surfaced. refreshPricingIfAllowed is gated to Codex and Claude, and Grok never reaches it at all because its snapshot comes from the provider probe rather than CostUsageFetcher.loadTokenSnapshot. On a machine with Codex or Claude also enabled the shared cache is already populated, so the failure is invisible there; enable only Grok and the catalog never appears and the Cost row shows tokens with no money, permanently.

Fixed in 744677e68. The Grok scan paths now request ModelsDevPricingPipeline.refreshIfNeeded through a summarizeRequestingPricingRefresh wrapper, called from all four scan sites (GrokStatusProbe, both branches in GrokProviderDescriptor, and UsageStore.scanAndPublishGrokLocalTokenSnapshot). It is detached rather than awaited, matching how the Codex and Claude paths already treat it — pricing availability must not delay or fail a local scan — and it is safe to call repeatedly, since it returns immediately unless the cache is stale and serialises through its own coordinator. summarize itself stays synchronous and side-effect free.

Note the inline comment still points at GrokLocalSessionScanner.swift:662; that line is the unchanged pricing lookup, and the fix is upstream of it in the new wrapper, so the anchor looks live even though it is addressed.

Coverage: absent models dev cache requests a background refresh, stale models dev cache requests a background refresh, and fresh models dev cache skips the background refresh. All three assert whether a refresh was requested through an injected transport — no test touches the network.

Real-session evidence

CODEXBAR_LIVE_GROK_CATALOG_PROOF=1 swift test --filter GrokXAISpendCatalogTests, against real local Grok CLI sessions, through the shipped code path:

catalog_source=grok
today_tokens=5043749
last_30_days_tokens=52696354
today_cost_usd=3.3471699999999993
window_cost_usd=49.353424
cost_provenance=listPriceEstimate
history_days=365
priced_days=4
token_days=4
daily_buckets=4
available_sources=grok

The same corpus on main reports 653K tokens and no cost. history_days=365 shows the requested window is honoured (it was pinned to 30). priced_days == token_days shows no day was silently left unpriced. The gated proof was extended in e3cd3b9ce to print cost, provenance and priced-day coverage, since tokens alone cannot evidence the half of this change that is about money.

Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent.

Merge risk / branch state

Rebased onto current main (27c7f334e); the branch reports clean. Full suite on the head: 77/77 groups, 922 selections, 0 failures. swiftformat --lint and swiftlint --strict clean. Upstream CI green on the previous head including all three Linux builds.

One thing deliberately left undone: no CHANGELOG.md entry. 0.54.1 was finalized and there is no open Unreleased section, so I did not invent a version heading — happy to add one wherever you prefer.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Aug 22, 2026
@olddonkey

olddonkey commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Both new findings addressed at 44d79a95a.

P1 — Do not map every xAI log record to the Grok subscription

Agreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own xai provider precisely to keep those apart, so the old behaviour crossed a boundary the app deliberately maintains.

The usage log carries no per-record credential evidence; I checked every field emitted for xai rows (requestId, timestamp, provider, model, requestedModel, resolvedModel, usage, usageStatus, status, routeDecision, …) and there is nothing about auth, account or key. The signal that does exist is ~/.opencodex/config.json, which records authMode per provider.

So attribution now requires positive OAuth evidence:

  • xai routes to .subscription(.grok) only when its configured authMode is OAuth. Anything else returns .tokenOnly — the spend is real, it just belongs to no tracked subscription — rather than .unknown, which would read as "unrecognised provider".
  • Fail closed. A missing or malformed config, a providers block without xai, or an entry without authMode all count as no evidence and keep the records off the Grok row.
  • OpenCodexRouteDispatcher stays a pure function. The set of OAuth-backed provider ids is threaded in from the caller (OpenCodexUsageFanOut → SpendDashboardSource), so the routing site never touches the filesystem and every existing caller and test that does not care about auth keeps working.
  • The gate applies only to xai. openai, kimi-coding, deepseek and opencode-go are untouched — changing them would be an unreviewed behaviour change for other providers — and a test pins that they ignore xAI auth state entirely.

Coverage: xai OAuth config routes to Grok, xai non OAuth config stays token only (parameterised over several non-OAuth values), xai routing fails closed without readable complete OAuth config, non xai subscription routes ignore xai auth state, plus fan-out cases proving the same entries land on the Grok row under an OAuth config and are absent under an API-key one. No test reads the developer's real ~/.opencodex; the home directory is injected.

docs/grok.md no longer claims this path cannot distinguish OAuth from API-key traffic, because it now can.

P2 — Republish the Grok snapshot after a missing catalog refreshes

I looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree:

The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every summarize. The next Grok scan therefore prices against the refreshed catalog with no extra machinery, bounding the unpriced window to a single refresh cycle. That is the same behaviour Codex and Claude already have: refreshPricingIfAllowed dispatches into Task.detached and their current scan does not wait for it either.

The alternative — plumbing a completion signal back across the actor boundary into the @MainActor publication path — buys one refresh cycle of latency on first run, at the cost of a new cross-actor completion path in code that publishes user-visible spend. That trade looked disproportionate, and inconsistent with how the two established providers behave. I have recorded the reasoning as a comment at the call site rather than leaving it implicit, so the next reader does not have to re-derive it.

Happy to build it if you would rather have it.

Evidence

The attribution itself only becomes visible in the app: SpendDashboardSource.mergingOpenCodexInputs is what merges the fan-out into provider rows, and the CLI's cost command reports OpenCodex as its own source rather than routing it, so terminal output cannot show this path. The figures below are read off the freshly packaged build running against real local data, on a machine whose ~/.opencodex/config.json has "xai": { "authMode": "oauth" }; screenshots of both panes follow.

The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is grok-4.6-build, while OpenCodex's records resolve to the bare grok-4.6 / grok-4.5 / grok-4.3. Both sit under the Grok provider.

model row source shown
grok-4.6-build Grok CLI session logs $50.52 · 54M
grok-4.6 OpenCodex $161.21 · 182M
grok-4.5 OpenCodex $4.54 · 5.5M
grok-4.3 OpenCodex $0.50 · 201K

Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex xai records. The CLI half is reproducible by anyone on their own machine through the gated proof test (CODEXBAR_LIVE_GROK_CATALOG_PROOF=1), whose output is in the PR body.

The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config.

State

Full suite on 44d79a95a: 77/77 groups, 922 selections, 0 failures. swiftformat --lint and swiftlint --strict clean. Rebased on 27c7f334e.

image image

@clawsweeper clawsweeper Bot added proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. proof: sufficient Contributor real behavior proof is sufficient. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Aug 22, 2026
@olddonkey olddonkey changed the title Report real Grok token usage and list-price cost, from the CLI logs and OpenCodex alike Report real Grok token usage and list-price cost from CLI logs Aug 23, 2026
@olddonkey

Copy link
Copy Markdown
Contributor Author

Addressed both current findings in 211e1977d and resolved the two review threads.

  • P1 / historical xAI attribution: removed current-config-based xAI → Grok routing. usage.jsonl has no request-time credential provenance, so xAI records now remain token-only until the producer can persist that evidence. Removed the config reader/plumbing and added dispatcher/fan-out regressions.
  • P2 / first pricing publication: when no models.dev artifact exists, the first Grok scan now awaits the initial best-effort refresh attempt before summarizing. A successful refresh prices the first returned snapshot; stale catalogs still price immediately and refresh in the background. Added a regression that writes the catalog during refresh and asserts the first summary is priced.
  • Updated the PR title/body and docs/grok.md so they no longer claim OpenCodex xAI traffic is merged into the Grok subscription row.

Validation on the exact pushed head:

  • focused Grok/OpenCodex suites: 32 tests passed
  • make check: passed
  • make test: 922 selections, 77/77 groups, 0 failed groups, 0 retries
  • branch is based on current main (27c7f334e) and the merge-tree is clean

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 23, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. proof: sufficient Contributor real behavior proof is sufficient. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. labels Aug 23, 2026
@olddonkey

Copy link
Copy Markdown
Contributor Author

Final validation for c35734ad5a747a83c46b3844c8ce33fd46dbefe1:

  • All nine GitHub CI checks passed, including both macOS shards and all Linux builds.
  • Local make check passed with zero violations in 2,143 files.
  • Local make test passed all 1,036 selections in 87 groups on the first attempt, with zero failures, retries, or timeouts.
  • Cache-adoption, pricing/provenance regressions and the 41-test architecture gate passed. The generated parser hash is c3a879df4eff7187, with current-main and previous-PR cache compatibility retained.
  • GitHub reports this head mergeable/clean; there are no unresolved inline threads. The latest ClawSweeper review reports no actionable code findings.

The PR body now has the final validation results and the immutable landed-producer source audit. The owner display-policy decision and older changes-requested review remain unresolved. The review environment's inability to retrieve the later producer carry is also recorded; the supplied source audit does not expand the pinned executable capture into a claim about a released producer version.

@olddonkey

Copy link
Copy Markdown
Contributor Author

Merged latest main 8b254dbec11ddd5c5547878d9640e4e965306c71 into this branch and pushed 40f9e47f7373d69f7738c582a9dc288231f329fd. GitHub now reports the branch mergeable.

The conflict resolution preserves upstream's checked numeric aggregation, parser-revision migrations, and Grok terminal-billing work gate while retaining completed-turn accounting, recorded-versus-estimated disclosure, custom-price precedence, and per-attempt OAuth attribution. The generated parser hash is a8559238a5fc0480; current-main and prior-PR SQLite adoption are covered. Release notes now sit under 0.59.1 Unreleased, with published sections identical to main.

Validation on this head:

  • make check: zero violations in 2,198 Swift files.
  • Focused tests: 433 tests in 35 suites passed, including nine upstream billing-failure scenarios and the added OAuth overflow/estimate-coverage regression.
  • make test: 1,080 selections in 90 groups passed on the first attempt, with zero failures, retries, or timeouts.
  • The pinned producer ledger still imports five OAuth tokens, excludes API-key traffic, and reopens the cache without rereading the log.
  • GitHub CI is running: https://github.com/steipete/CodexBar/actions/runs/34674108255

The PR body records the current results and distinguishes them from the historical native-corpus proof. The owner display-policy decision remains open.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@olddonkey

Copy link
Copy Markdown
Contributor Author

Addressed the newly reported native overflow finding in 6d3d5af1bb28aeaeb2bdb13cfcc70cdd4d0b2cc7.

The native parser now validates NSNumber conversions, preserves unknown counts, and uses checked addition across model/day/window totals. Later valid records and cache rereads cannot clear an unknown aggregate. Explicit valid totals retain precedence, valid neighboring token classes and recorded spend remain available, and incomplete token accounting does not establish full coverage. The added production-scanner regression suite covers per-record and cross-record overflow, multiple days, nested models, malformed number types, explicit totals, snapshot projection, and cache reuse.

Validation:

  • Final make check: zero violations in 2,199 files.
  • Focused regression run: 326 tests in 30 suites passed.
  • make test: 1,081 selections in 91 groups passed on the first attempt, zero failures/retries/timeouts. Only test-call formatting changed afterward; it was rebuilt and rechecked by the final proof run.
  • Final serial proof: 31 tests in 6 suites passed, including fresh local native logs through menu/dashboard projections. The 30-day window retains 98,631,812 tokens and $12.94957366 with recorded provenance; empty 1-day/7-day windows remain unknown. No provider authentication or live billing is involved.
  • The pinned producer ledger again imports five OAuth tokens, excludes API-key traffic, and reopens the cache without log reads; cancellation releases the scan queue within one second.

The PR body has the current results. The new CI run is pending; the owner dollar-display decision remains open.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@olddonkey

Copy link
Copy Markdown
Contributor Author

Addressed the remaining menu-projection overflow finding in a41736133e7a2d922db7bbc5b0e18e7e729e855d.

A checked, complete-count helper now covers the remote-backed Grok window projection, window request totals, comparison summaries, and menu/Widget fallback totals. Unknown daily values are retained as unknown instead of being dropped into a partial sum. This complements the native scanner repair in the preceding commit.

The new end-to-end regression writes native completed-turn logs and exercises both remote-backed and fallback live consumers. It covers individually representable Int.max/1 totals across days and an already-unknown day followed by a valid day. The menu, Widget, and dashboard keep the aggregate unavailable; narrowing to the unaffected one-day window restores its known count. All four scenarios passed.

Current-head validation: make check has zero violations in 2,199 files; 888 focused tests in 110 suites passed; 40 serial native-window/producer-import/projection/cancellation proof tests passed. The fresh 30-day native window still reports 98,631,812 tokens and $12.94957366 with recorded provenance and disclosures. Empty short windows remain unknown. The full suite and new GitHub CI are running, with current results distinguished in the PR body.

The owner's default dollar-display decision remains open.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@olddonkey

Copy link
Copy Markdown
Contributor Author

Final author-side validation for a41736133e7a2d922db7bbc5b0e18e7e729e855d is complete:

  • make check: zero violations in 2,199 files.
  • make test: all 1,081 selections in 91 groups passed on the first attempt, with zero failures, retries, or timeouts. No source changed afterward.
  • 888 focused tests and 40 current-source native/producer/projection/cancellation proof tests passed.
  • GitHub lint, changes, GitGuardian, and Linux x64/ARM64/musl builds/tests passed: https://github.com/steipete/CodexBar/actions/runs/34675986313
  • Both macOS CI shards remain queued waiting for runners, so I am not claiming a complete GitHub CI pass.
  • GitHub reports the head mergeable; there are no unresolved inline threads. The latest exact-head review reports Platinum 4/6 and no actionable findings.

The PR body now has the final local validation and exact CI limitation. The default-dollar display policy and the owner's outstanding changes-requested review remain maintainer decisions.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 12, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

olddonkey and others added 2 commits September 19, 2026 01:28
Resolve the 136-commit drift since the last rebase. Provider-aware pricing on
main replaced the Codex-only lookup this branch extended, so the xAI-specific
behaviour is re-expressed on top of it instead of alongside it:

- codexModelsDevPricingTargets keeps main's resolver-backed form, widened to xAI
  routes with the branch's grok `-build` alias fallback; the duplicate copy in
  CostUsagePricing+CodexResolver is dropped and only the route union remains.
- The OpenCodex aggregator uses main's providerCostUSD path, with the branch's
  standalone-xAI gate kept ahead of it: raw xAI rows without Grok OAuth
  provenance stay token-only unless explicit custom prices exist.
- OpenCodexUsageLog adopts main's TestProcessSafety-based path lookup.
- The menu-card live-consumer projection moves with main's relocation of the
  card model into UsageStore+MenuCardModel; the override-card test uses the
  store API.
- Window summaries keep the branch's complete token sum and main's CheckedSum
  request total; ProviderRow keeps both incompleteRequestCount and the Grok
  cost disclaimer.
- Parser hash regenerated; main's 6d48baf0ed980828 joins the compatible
  predecessor list. Gatekeeper anchors re-pointed at the merged sources, and
  the merged openai route cluster and xAI alias fallback carry design markers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main regenerated its parser hash again after the previous merge. Keep
main's predecessor entry for 6d48baf0ed980828, add its new 6a4df886696f4ab5
as the current-main predecessor, and regenerate this branch's hash.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@olddonkey

Copy link
Copy Markdown
Contributor Author

@steipete requesting re-review on 32a73cd90, now merged onto main at 0741047cb (details in the "Merge with main (2026-09-19)" section of the description).

Status of the four P1s and the P2 from your review:

  1. xAI history token-only in the standalone aggregator — OpenCodexUsageAggregator.listPriceUSD returns nil for xAI rows without Grok OAuth provenance unless explicit custom prices exist; only the fan-out's per-attempt Grok OAuth rows are priced. Re-expressed on top of main's providerCostUSD path in this merge.
  2. "Not a bill" on populated surfaces — GrokProviderDescriptor carries menuHintLines: [.estimate] and an estimateDisclaimer; SpendDashboardModel.ProviderRow.costDisclaimer labels dashboard rows, rendered in PreferencesSpendDashboardPane.
  3. Failed-billing fallback rescans — every remote billing failure schedules scanAndPublishGrokLocalTokenSnapshot (UsageStore+Refresh.swift), so completed turns keep advancing; covered by GrokFailedBillingWorkTests.
  4. Separate xAI fingerprint — CostUsagePricing.xaiModelsDevProviderIDs is fingerprinted apart from the Codex set; GrokCostUsagePricingTests pins that an xAI catalog change leaves Codex caches alone.
  • P2 bounded discovery — GrokLocalSessionScanner caps discovery at maximumDiscoveryEntries (4096) before the 256-session cap.

All review threads are resolved. On this head: make check 0 violations, make test 116 groups / 11,577 tests / 0 failures; GitHub lint, changes, and Linux builds are green, macOS shards pending.

Only the gatekeeper allowlist conflicted; its anchors are re-pointed at the
merged sources and the Grok refresh cluster entry is kept.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Chipagosfinest

Copy link
Copy Markdown
Contributor

Chiming in as an affected Grok user on the outstanding default-display question, plus a source that resolves the last bit of uncertainty on the divisor.

On cost_in_usd_ticks / 1e10: this is confirmed in xAI's own docs, not just reverse-engineered — docs.x.ai/developers/cost-tracking states cost_in_usd_ticks divided by 1e10 is the actual billed USD amount. Credit to @olddonkey and the #3345 thread for nailing the divisor empirically across two independent corpora before this citation existed; this just removes the remaining doubt.

On the default-display question: as a user currently seeing signals.json-derived numbers that undercount real spend by 50-80x, I'd rather have accurate recorded-spend-with-disclaimer by default than a quiet undercount. The estimateDisclaimer / "recorded spend, not a bill" labeling in this PR already covers the accuracy-vs-liability concern.

Happy to independently verify against my own ~/.grok/sessions/ corpus if that's useful evidence toward unblocking review.

@Chipagosfinest

Copy link
Copy Markdown
Contributor

Ran the PR's own `GrokWindowProvenanceProofTests` live-corpus test (`CODEXBAR_LIVE_GROK_CATALOG_PROOF=1`) against my real `~/.grok/sessions` (547M+ tokens across 30 days, larger/messier than the corpora referenced in the PR description).

Good news — the core fix works. The scanner correctly produced real recorded spend:

  • 30-day: 547,877,256 tokens / $142.8059227, provenance mixed, 12/30 days priced
  • 7-day: 142,923,685 tokens / $47.09328332, provenance vendorMetered, 2/7 days priced
  • 1-day: no priced day in window (correctly nil, not a fabricated number)

But the test itself failed — 6 assertion failures, all downstream of the scanner. The raw snapshot (selected.last30DaysCostUSD, .costProvenance) is correct, but it doesn't propagate to the menu/dashboard row on this corpus:

  • row.costDisclaimer → nil (expected "Grok CLI-recorded spend, list price where unrecorded · not a bill.")
  • row.totalCost → nil (expected 47.09 / 142.81)
  • group.provenance → .unknown (expected .vendorMetered / .mixed)

So on my account, the live-consumer projection (tokenSnapshotForLiveProviderConsumer → UsageMenuCardView.Model) breaks the chain somewhere between the correct scanner output and the display row — the numbers compute right but wouldn't render for me if this shipped as-is today. Full failure output available if useful; didn't dig into root cause tonight to keep this a clean data point rather than a half-finished fix.

olddonkey and others added 3 commits September 21, 2026 01:36
A real corpus mixes priced days with days the scanner could not price at all:
turns the CLI attributed to no model and recorded no ticks for. The scanner
declares those as unpriced requests, but two consumers treated such a day as
making the whole window unknown, so the dashboard row showed no total, no
disclosure, and no provenance while the scanner's snapshot was correct.

- The generic window narrowing publishes a window cost only when every retained
  day carries one. Both Grok projections now sum the days that were priced
  through one shared helper; a Grok day without a cost is a disclosed gap, not an
  unknown total.
- The dashboard's cost-consistency rule only knew Codex's and Cursor's ways of
  declaring a row unpriceable. A Grok day whose requests all went unpriced is now
  an explicit gap as well, so the recorded total survives and the coverage counts
  disclose the unpriced requests.

The live proof now prints the coverage, active-day, unpriced-day and unpriced-
request figures for each window, and asserts the documented no-total contract
when a bounded scan cannot establish coverage, so a failing run says which gap
it hit instead of only that the row was empty.

Also cite xAI's cost-tracking documentation for the tick divisor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The gate that keeps raw xAI rows token-only without Grok OAuth provenance keyed
on the model namespace, so a router-hosted `xai/...` row billed by OpenRouter
lost the catalog price its recorded provider entitles it to under steipete#3676. It now
follows the resolved billing route: the `xai` provider, or the legacy OpenAI
transport label carrying an xAI route. Router-hosted xAI rows keep their
recorded-provider price; directly billed rows stay token-only as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Seven files conflicted. Main's steipete#3353 moved the per-provider dashboard rows
into SpendDashboardProviderBreakdown, so the Grok cost disclosure moves with
them: on the source row when a provider lists its sources, under the provider
header when its single native source is folded into it. Main relocated
resetTokenUsageState into UsageStore+TokenCost; the Grok scan cancellation
moves with it, and the live-consumer selection this branch adds keeps main's
localized no-data message beside it.

Parser hash regenerated to 611a82b6eb2b5ffc; main's 03e43d1217789d16 and the
previous PR head b004d0cf7d471304 join the compatible predecessor list, the
exact-list assertion, and the SQLite adoption arguments. Gatekeeper anchors
re-pointed at the merged sources; the shared dashboard construct now carries
both main's and this branch's Grok references. Window narrowing moves into
CostUsageModels+Narrowing.swift to keep CostUsageModels.swift within the file
length limit. Release notes moved under 0.63.1 — Unreleased.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@olddonkey

Copy link
Copy Markdown
Contributor Author

@Chipagosfinest thank you — that run found a real defect, and I could reproduce the dashboard half of it from your numbers alone. Fixed on exact head 16be323d7f8b056139eaa50b31393f4585094855.

What broke

Your scanner output was right; the two consumers downstream of it were not, and they failed on a corpus shape my own proofs never had: a window that mixes priced days with days the scanner could not price at all. That happens when a turn is attributed to no model and carries no costUsdTicks — nothing to read, nothing to reconstruct — and the scanner declares it through unpricedRequestCount rather than inventing a number.

  1. The generic window narrowing publishes a window cost only when every retained day carries one, so one unpriced day inside the 30-day window nilled last30DaysCostUSD on the live-consumer path. Both Grok projections now sum the days that were priced through one shared helper; a costless Grok day is a disclosed gap, not an unknown total.
  2. The dashboard's cost-consistency rule only knew Codex's and Cursor's ways of declaring a row unpriceable, so a Grok day whose requests all went unpriced made it declare the whole history inconsistent — no total, no disclosure, .unknown provenance. Exactly the three assertions you saw, twice. A Grok day whose requests are all unpriced is now an explicit gap too; the recorded total survives and the coverage counts disclose the unpriced requests.

The regression writes one recorded day and one unpriced day, projects the 30-day window through tokenSnapshotForLiveProviderConsumer, and asserts the row total, the disclosure, the provenance, and coverage.unpriced == 1. With either fix reverted it fails on the same expectations you reported.

One thing your run can still tell me

verifySurfaces asserted a dashboard total unconditionally, but there is a second shape that legitimately leaves the dashboard empty: a scan that hit one of the production bounds (256 sessions, 4,096 directory entries, 100,000 turns, 256 MiB) cannot establish coverage, and the dashboard deliberately shows no total for it rather than presenting a partial history as complete — the menu still shows the scanned window. On a corpus your size that is possible. The live proof now prints live_full_history_coverage_established, live_window_unpriced_days, and live_window_unpriced_requests per window and asserts the documented contract for each shape, so if you have a minute to re-run it on this head, those three lines will say which one you hit. If coverage was not established for you, the fix here is not the whole story and I will look at the bound.

The docs.x.ai citation for the 1e10 divisor is now in the divisor's comment and in docs/grok.md — thanks for that too.

@olddonkey

Copy link
Copy Markdown
Contributor Author

Both findings from the 2026-09-19 review are fixed on exact head 16be323d7f8b056139eaa50b31393f4585094855, which also merges current main and fixes a live-corpus defect reported below by @Chipagosfinest.

[P1] Scope the xAI pricing exception to the billing provider

You were right: the gate keyed on the model namespace, so a router-hosted xai/... row billed by OpenRouter lost the catalog price its recorded provider entitles it to under #3676. It now follows the resolved billing route — the xai provider, or the legacy OpenAI transport label carrying an xAI route — and nothing else. Two regressions: openrouter + xai/grok-fixture prices at the recorded-provider rate, while xai + grok-fixture and openai + xai/grok-fixture stay token-only without Grok OAuth provenance, as before.

[P3] Move the Grok entries into the current unreleased section

Both entries now sit under 0.63.1 — Unreleased; the published sections are identical to main.

Live-corpus defect (not in your review, found by a user running the PR's own proof)

A window mixing priced days with days the scanner could not price at all — turns attributed to no model with no costUsdTicks — lost its total on the dashboard: the generic window narrowing nilled the window cost, and the dashboard consistency rule had no Grok notion of an explicitly unpriced day. Both are fixed with a regression that fails on the reported assertions when either fix is reverted; details in my reply to that report.

Gate on this head

  • make check: passed — SwiftFormat clean, SwiftLint 0 violations in 2531 files, provider/package/documentation gates green.
  • make test: passed — 1298/1298 selections, 119/119 groups successful on the first pass, 0 failed groups, 0 retries, 0 timeouts (742.5 s).
  • Focused: ProviderArchitectureGatekeeperTests 41/41; the Grok scanner, pricing, projection, menu-fallback and dashboard suites, the OpenCodex pricing and fan-out suites, and the cost-store adoption suite — 211 tests in 14 suites. git diff --check clean.
  • The three main commits that landed while the suite ran (bb62d3553, dc2e01ea9, d8d0f3394) merge without conflict; GitHub reports the head mergeable.

The owner decision is unchanged: whether existing Grok users should receive CLI-recorded spend with disclosed list-price fallback by default. Your recommendation to approve it after the pricing and release-note findings were fixed is noted; both are.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

olddonkey and others added 2 commits September 21, 2026 03:27
steipete#3805 added a regression that keeps local Grok tokens through the RPC-to-proxy
fallback, with a fixture that carries only `signals.json` and expects its
context-window occupancy to surface as 42 tokens. This branch deliberately
stops counting occupancy — `signals.json` is session metadata only, and local
history comes from completed turns — so on the merged tree the fallback kept
no history and the test failed.

The fixture now records one completed 42-token turn beside the same
`signals.json`, which keeps the test's subject (the fallback retains local
history) and demonstrates the branch's contract (occupancy is not counted).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@olddonkey

Copy link
Copy Markdown
Contributor Author

The red swift-test-macos (1, 2) on 16be323d7 was a real interaction with main, not a flake, and it is fixed on exact head 75945a360f90abe5abac262c6c80d4cf65ec2326.

CI runs the test merge with main, and main gained #3805 while this head's suite was running. #3805 adds missing RPC billing retains local tokens through the proxy fallback, whose fixture carries only a signals.json and expects its context-window occupancy (40 + 2) to surface as 42 local tokens. That is the accounting this PR deliberately stops doing — signals.json is session metadata only, and local history comes from completed turn_completed records — so on the merged tree the fallback kept no history and the test failed on both passes. The exact CI group passes locally on this branch alone; it fails only once #3805 is merged in, which I reproduced before changing anything.

The fixture now records one completed 42-token turn beside the same signals.json. That keeps the test's subject — the RPC-to-proxy fallback retains local history — and demonstrates this branch's contract at the same time, since the occupancy fields contribute nothing to the 42. Every other assertion in that test (unpriced, .unknown provenance, share-stats tokens, live-only history omitted from the usage JSON) holds unchanged.

Also merged the three main commits that landed during the previous run (bb62d3553, dc2e01ea9, d8d0f3394); no conflicts.

Gate on this head

  • make check: passed — SwiftFormat clean, SwiftLint 0 violations in 2530 files, provider/package/documentation gates green.
  • make test: passed — 1299/1299 selections, 119/119 groups successful on the first pass, 0 failed groups, 0 retries, 0 timeouts (785.8 s).
  • The exact CI group that failed (GeminiStatusProbe…GrokCostUsagePricing, 8 suites, --no-parallel) passes on this head; GrokAccountContextTests 8/8.
  • git diff --check clean; 0 behind main.

@steipete

Copy link
Copy Markdown
Owner

Thanks @olddonkey. At fetched head eb1647cc8ed68e39f5ed90eb8730ed3d182ee67d, the four historical P1s have corresponding source repairs: raw xAI pricing is gated, populated surfaces have disclosure hooks, failed billing schedules fresh scans, and xAI pricing has a separate fingerprint. This is not a fresh full validation of the rewrite. The patch spans 51 files and adds 1,529 net production lines, beyond the current bounded/non-growth landing policy. The default recorded-first dollar display also remains an owner decision. I recommend closing this broad PR and retaining its accounting evidence for a separately scoped implementation. No code from it was imported in this lane.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants