feat(probably-decisions): add Probably-judged Sentinel decision benchmark

[talos]

TL;DR

Adds probably_decisions, a benchmark that runs a fixed Probably decision program over redacted Sentinel ban-candidate dossiers with the model under test as the semantic judge, so judgment quality and research quality can be scored separately.

What changed?

• probably-decisions/probably/: a self-contained Probably interpreter (lexer, parser, runtime) with a Provider seam (judge, write) and a recorded effect trace per run.
• programs.ts: sentinel_case_v1, the SCANNER_SPEC decision path encoded in Probably (compromised-key gate, static-lead-only rejection, load-vs-abuse, materiality, remedy choice, hold when uncertain). Also derives per-action branch expectations for scoring.
• judge.ts: two JudgeFn implementations. chatJudge prompts a chat model for a JSON label distribution. decisionsJudge sends one Decisions choice question whose criteria are the labels and maps the returned probabilities back to labels.
• providers/decisions-client.ts: a small client for POST /api/alpha/decisions (origin derived from baseUrl, bearer auth, session and trace headers, provider preferences, zod-validated response, generation id recording, rate-limit retry and timeout). isDecisionsModel routes typesafe/* ids (e.g. ~typesafe/jev-latest) to it.
• solver.ts: two modes. judgment feeds the dossier plus evidence straight into input(). research first runs a bounded tool loop (list_evidence_sections, read_evidence_section, submit_dossier) and then judges the submitted dossier. Chat judges go through the harness ModelService, so caching, session IDs and generation IDs work unchanged. In research mode a Decisions model as model fails at startup with an explanatory error, since the tool loop needs a chat model; put Jev in judgeModel instead.
• scorer.ts: action accuracy, enactment agreement, branch agreement, Brier score on judge probabilities, evidence coverage and gold-fact recall (research mode), per-action recall, hold precision, run failure rate.
• dataset.ts + schema.ts: JSONL record schema, streaming loader with range paging, stable recordToSample.
• __fixtures__/sentinel-sample.jsonl: 102 redacted, human-reviewed records used by default (all denied and reverted cases, plus a seeded stratified sample of approved ones across the five enacted actions and hold).
• Config schema, metadata, registry and CLI wiring, README with provenance.

Why?

Sentinel scanners and other agents make a chain of semantic judgments before enacting a remedy. Probably makes each judgment an explicit feels/match effect with a probability, and input() is a hard seam between evidence gathering and judgment. Holding the program fixed means a wrong final action attributes cleanly to either research or judgment.

How to test

bun test src/benchmarks/probably-decisions
bun run bench --benchmark probably_decisions --model openai/gpt-5-mini --limit 6 --concurrency 3 --reasoning-effort low
bun run bench --benchmark probably_decisions --model ~typesafe/jev-latest --limit 6 --concurrency 2 --reasoning-effort none
bun run bench --benchmark probably_decisions --model openai/gpt-5-mini --limit 2 --reasoning-effort low --solver-config '{"mode":"research","maxResearchSteps":8}'

Expected: sample metadata carries predicted, gold, judgeCount, branchAgreement, brier, and in research mode evidenceCoverage and factRecall. Run-level metrics include action_accuracy, enactment_agreement, branch_agreement, brier, run_failure_rate.

Benchmark impact

New benchmark, no existing scores change. Smoke on openai/gpt-5-mini (reasoning low): judgment mode 2/6 correct on the first six fixture records with run_failure_rate 0; research mode 2/2 on the first two records with fact_recall 1 and evidence_coverage 0.9. Judgment mode makes 2 to 4 judge calls per sample and cost about $0.014 for six samples.

The dataset is derived from Sentinel ban-candidate cases and their human review outcomes (approved, denied, reverted). Gold action is the enacted remedy for approved cases and hold for denied or reverted ones. Records are redacted (no emails, names, account IDs, IPs, keys, links, Slack refs, raw UUIDs) and dataset.test.ts asserts that over every fixture record. The fixture is internal-only and must not leave the organization. Slack scanner summaries were used only as informational context, not as labels.

Temperature is fixed at 0 and maxTokens is never set.

Jev smoke, judgment mode (--model ~typesafe/jev-latest --limit 6): 2/6 correct, run_failure_rate 0, branch_agreement 0.79, brier 1.03, total cost $0.0017 for six samples (about 8x cheaper than gpt-5-mini on the same records).

Reviewer focus

• programs.ts sentinel_case_v1: does the encoded decision path match SCANNER_SPEC (two independent signals, static leads are not proof, compromised key is a hard stop, uncertainty means hold)?
• scorer.ts Brier and branch-agreement derivation from the effect trace.
• decisions-client.ts request shape against the Decisions alpha schema, and whether typesafe/ prefix routing is the right capability signal.
• Whether the 12-per-action cap on approved records in the fixture is the right balance against the 42 hold records.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included (fixture is redacted and marked internal-only in the README)
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/4d197bca9d774b71a2ddc6b7188f0ea0
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/4d197bca9d774b71a2ddc6b7188f0ea0?variant=devin
Requested by: @talos

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from John

SYSTEM:
<latest_message>
John Krauss (U09M1SL591A) [ts=1789707155.716799]: implement <https://probably-lang.southpolesteve.workers.dev/|probably-lang.southpolesteve.workers.dev> on top of <https://openrouter.ai/docs/client-sdks/typescript/sdks/decisions/README|openrouter.ai/docs/…/README>
</latest_message>

=== BEGIN THREAD HISTORY ===
John Krauss (U09M1SL591A) [ts=1789707155.716799]: implement <https://probably-lang.southpolesteve.workers.dev/|probably-lang.southpolesteve.workers.dev> on top of <https://openrouter.ai/docs/client-sdks/typescript/sdks/decisions/README|openrouter.ai/docs/…/README>
=== END THREAD HISTORY ===
Channel ID: D09MKU1TRPG
Thread URL: https://openrouter.slack.com/archives/D09MKU1TRPG/p1789707155716799?thread_ts=1789707155.716799&amp;cid=D09MKU1TRPG

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.