From 41cc160c06e1047f1cd38f35f1d37c82f0915a5d Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 17:34:58 -0700 Subject: [PATCH 01/19] =?UTF-8?q?feat(lint):=201/7=20=E2=80=94=20prose=20b?= =?UTF-8?q?udget=20ratchet=20and=20one=20home=20for=20rationale?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `tests/lint/prose_budget.py`: a stdlib-only measurement of the essay-shaped prose in `src/coder_eval` — docstrings over 150 words (Typer commands exempt) plus comment runs of 3+ lines — gated against a module baseline of 79,754 words. It also resolves every `Rationale: § ` pointer, and under `--assert-code-unchanged ` proves a commit moved prose only, by comparing the docstring-stripped AST and the multiset of functional directive comments. The gate is wired in both seams: `make docs-budget` / `make verify`, and the `quality-gate` job in pr-checks.yml — CI restates every step and never invokes `make`, so the Makefile line alone would gate nothing on a PR. Relocates the 12 sections of `.claude/architecture-notes.md` verbatim into `.claude/notes/.md` and deletes it, so rationale has exactly one home and the tree has exactly one index. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2 --- .claude/architecture-notes.md | 128 ------------ .claude/notes/README.md | 68 +++++++ .claude/notes/agents.md | 13 ++ .claude/notes/contracts.md | 9 + .claude/notes/isolation.md | 7 + .claude/notes/orchestration.md | 21 ++ .claude/notes/permissions.md | 7 + .claude/notes/persistence.md | 4 + .claude/notes/reporting.md | 15 ++ .claude/notes/timing.md | 4 + .github/workflows/pr-checks.yml | 3 + CLAUDE.md | 18 +- Makefile | 6 +- tests/lint/prose_budget.py | 348 ++++++++++++++++++++++++++++++++ tests/test_prose_budget.py | 255 +++++++++++++++++++++++ 15 files changed, 771 insertions(+), 135 deletions(-) delete mode 100644 .claude/architecture-notes.md create mode 100644 .claude/notes/README.md create mode 100644 .claude/notes/agents.md create mode 100644 .claude/notes/contracts.md create mode 100644 .claude/notes/isolation.md create mode 100644 .claude/notes/orchestration.md create mode 100644 .claude/notes/permissions.md create mode 100644 .claude/notes/persistence.md create mode 100644 .claude/notes/reporting.md create mode 100644 .claude/notes/timing.md create mode 100644 tests/lint/prose_budget.py create mode 100644 tests/test_prose_budget.py diff --git a/.claude/architecture-notes.md b/.claude/architecture-notes.md deleted file mode 100644 index 4c6188b9f..000000000 --- a/.claude/architecture-notes.md +++ /dev/null @@ -1,128 +0,0 @@ -# Architecture Notes (long form) - -Design rationale moved out of `CLAUDE.md` so that file stays a working reference -rather than a changelog. Nothing here is deleted history: every paragraph below was -previously a bullet in `CLAUDE.md`'s "Key Architectural Patterns" section. - -**This file is NOT auto-loaded into context.** Read it when you touch one of the -subsystems below — each entry explains *why* the design is shaped the way it is, and -most of them are written around a specific shipped defect. - -Authoritative sources, when this file and the code disagree: the code wins, then the -lint rule docstrings in `tests/lint/rules/`, then the guides under `docs/`. - ---- - -## Contents - -- [Config merging and CLI overrides](#config-merging-and-cli-overrides) -- [Datasets and aggregation](#datasets-and-aggregation) -- [Token accounting and the reconciliation message](#token-accounting-and-the-reconciliation-message) -- [Reference solutions and the anti-cheat window](#reference-solutions-and-the-anti-cheat-window) -- [Harness run-limit parity](#harness-run-limit-parity) -- [Execute vs. run: the grading switch](#execute-vs-run-the-grading-switch) -- [Detached grading and `Sandbox.adopt`](#detached-grading-and-sandboxadopt) -- [`--resume` is command-relative](#--resume-is-command-relative) -- [Published rates and run-time caps](#published-rates-and-run-time-caps) -- [Early stop on criterion](#early-stop-on-criterion) -- [The CE lint-rule catalogue](#the-ce-lint-rule-catalogue) -- [Plugin and GitHub Action layout](#plugin-and-github-action-layout) - ---- - -## Config merging and CLI overrides - -- **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly. - -- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only. - - ---- - -## Datasets and aggregation - -- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.) - -- **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure. - - ---- - -## Token accounting and the reconciliation message - -- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - -- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. - - ---- - -## Reference solutions and the anti-cheat window - -- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half. - - ---- - -## Harness run-limit parity - -- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. **Pi** is the same shape — the CLI (`pi -p --mode json`) streams a real multi-step loop per `communicate()` (`turn_start`/`turn_end`), so `max_turns: N` counts native `turn_start` steps; Pi retries transient/provider errors INTERNALLY (`agent_end.willRetry`), and the reducer finalizes once at `agent_settled`/EOF (not the first `agent_end`), folding the retry cycles into one turn. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex, Antigravity, and Pi (all run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), `allowed_tools`/`disallowed_tools` on Pi (its built-in tool names are lowercase — `bash`/`read`/… — and cannot map to the Claude-namespaced config default, so forwarding them would strip the agent of ALL tools; warned+ignored like the three agents above) — Pi DOES enforce `system_prompt` (`--append-system-prompt`, a small win over OpenCode) and DOES honor `plugins` for skills (each resolved skills dir → a `--skill ` arg via the shared `_plugin_skill_dirs` resolver, recorded as `pi_skill_paths`, so it CAN run activation suites) but does NOT read `system_prompt_file`, and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md. - - ---- - -## Execute vs. run: the grading switch - -- **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` with grading removed — the agent runs and the full trajectory is captured, but no criterion is checked, `weighted_score` is `None` (never `0.0`, which would be indistinguishable from "graded and scored zero"), and the row finalizes as **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are facts about the *run*, not about grading, and still apply (so `execute` still exits non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog check, and post-failure diagnostics); it crosses the docker boundary in `context.json` (defaulting to `True` in-container, so a host predating `execute` keeps grading). It is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because a task YAML must never declare itself ungraded; only the invoking command decides. `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code path. Three things are refused rather than degraded: `--junit-xml` (a report of verdicts, and there are none — though `reports_junit` still emits `` for an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades nothing), and simulation tasks (their turn-continuation logic reads criteria results, so an ungraded dialog would silently change its own stopping behavior). `stop_early:` blocks are inert under `execute` for the same reason the kill switch exists: the full trajectory is the deliverable. Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and grades with its own tests. - - ---- - -## Detached grading and `Sandbox.adopt` - -- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses is ONE parameter (`grade_in_place`, resolved by `_gate_scope_for_grade`), not two: it shipped beside an `include_setup_phase` every caller passed as its exact complement, and a future caller setting one and forgetting the other would silently drop half of a SECURITY gate. Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and `_assert_regrade_honored` refuses a returned row whose `started_at` moved, because an image predating this change ignores the unknown `regrade` key and RUNS THE AGENT, which the host would otherwise fold back as the recorded row's verdict (the exact sibling of `_assert_grade_honored`, one release later). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. `grant_container_access` now RETURNS what it widened and `run()` restores it in the same `finally`: the two staging dirs are disposable, but the graded workspace is the caller's tree — an operator-supplied `--workspace` was left world-writable permanently. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: every container is launched `TELEMETRY_ENABLED=false` under the invariant "container silent, host emits once", and the grading path had inherited only the silent half. The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `git clone` now passes `--` before the URL (argv position 2, so a value beginning with `-` was parsed as an option). `Sandbox.resolve_files` is containment-checked for the same reason: criterion paths were the one task-authored path skipping `_resolve_within_sandbox`, and `Path(root) / '/etc/passwd'` is `/etc/passwd`. An escaping LITERAL now raises `CheckerMisuseError` rather than resolving to `[]`: returning no match books an eval-CONFIG error as an agent failure — a gating 0.0 reading "file does not exist" for a file that plainly does exist and that no agent behaviour could place inside the sandbox (CE039's exact distinction). `tasks/byod_smoke_test.yaml` was broken that way for several commits, checking `/opt/byod_marker` baked into the BYOD image, with only a task-log warning to show for it; it now asserts on the container with `run_command: test -f …`, which is what a claim about the IMAGE rather than about the agent's workspace should look like. The guard keys on the escaping path EXISTING, so a merely-absent absolute path stays an ordinary failing verdict, and the GLOB branch still warns-and-drops, since filtering some matches out of a search is its normal behaviour. A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (the one legitimate rewrite — we are already inside the container the driver asked for), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. - - ---- - -## `--resume` is command-relative - -- **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate ` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`. - - ---- - -## Published rates and run-time caps - -- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)`. The guard originally shipped on `RunSummary` alone, so the same 10-task `execute` run with one crash rendered "Pass Rate: n/a" in `run.md` and "Pass Rate: 0.0%" in `experiment.md`. `measured` is **counted evidence** (`tasks_measured` / `rows_measured` — rows carrying a `weighted_score`), never a bucket count: the first version tested `tasks_succeeded + tasks_failed == 0`, but `TIMEOUT` and the two budget stops are category `failed` and reachable under `execute` (`_check_run_limits` still runs on the ungraded branch), so ONE timed-out row in a 100-task ungraded night read as "measured" and published `pass_rate: 0.0` — a real 0% point on the evalboard trend for a run that graded nothing. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender. - -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. - - ---- - -## Early stop on criterion - -- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. - - ---- - -## The CE lint-rule catalogue - -Each rule's authoritative rationale lives in its own module docstring under -`tests/lint/rules/` (or, for the doc-surface and whole-tree rules, in the -corresponding `@pytest.mark.lint` class in `tests/test_custom_lint.py`). The prose -summary below is kept for orientation only — when it disagrees with a rule file, -the rule file is correct. - -Recent additions, each traceable to a shipped defect: **CE064** (in `src/coder_eval/agents/`, a module that imports `TurnClock` must pass an explicit `timestamp=` to `AgentStartEvent` and `AgentEndEvent` — the turn's OUTER bounds, which no other rule looks at, since CE058-CE061 all scope to `AssistantMessage` and the bracket is not one. `timing.decompose_turn` produces `harness_startup_ms` / `harness_teardown_ms` by subtracting a generation-window bound from a bracket timestamp, so the two must share a basis; all three clocked harnesses derived their bounds from the `TurnClock` and let the bracket fall back to `StreamEvent.timestamp`'s `default_factory=datetime.now`, putting a monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact split `TurnClock` exists to remove, reintroduced at the one seam the clock did not own. Measured on a live antigravity turn: an `AgentEndEvent` stamped **17 us BEFORE its own last message finished**, which cannot happen (the event is constructed strictly after the final flush), and `decompose_turn` clamped that negative and published `0.0` — "measured, and instant", the CE058 confusion arrived at from the other direction — for a harness whose real tail is ~0.1 ms; it now records 0.035 ms. It surfaced on one harness only because the drift is tens of microseconds and antigravity is the only one that holds its process across turns, so nothing happens between its last flush and its end event; every other harness books a tail of 7-543 ms, where the drift is invisible rather than absent — which is why the fix is at every clocked site rather than at that one. SCOPE IS DERIVED, never a harness list: codex and opencode take their spans from the CLI's own epoch stamps, deliberately have no `TurnClock`, and are correctly invisible to the rule — a raw bracket is CONSISTENT with their bounds — and the day either adopts a clock the rule starts applying with no edit. BLIND SPOT, in the rule's docstring: presence, not correctness. It cannot tell `self.clock.now()` from a `datetime.now()` spelled out at the call site, because the three harnesses legitimately reach their clock three ways; the guard for the SOURCE is behavioural (`tests/_bracket_clock.py` injects a stand-in anchored a year from real time, so a reverted argument fails by a year rather than by the microseconds that separate the two clocks), which is the division of labour CE060 states — a rule removes the SILENT case, a default nobody chose), **CE063** (no module in `src/coder_eval/agents/` may import `busy_ms` — tool execution comes out of a generation window in exactly ONE place, `timing.py::subtract_tool_time`. Five reducers used to do it themselves while the head and tail were already computed centrally at the same seam, and that asymmetry is where every timing defect on this branch lived — none of them in the arithmetic, all of them in the bookkeeping AROUND it: when to reset a per-step span list (clearing it at `step_start` wiped a span before the flush could subtract it, a 100% overstatement of that window), when to clear a spent start stamp (a second flush with no intervening start republished the previous span — 3000 ms of generation for a 2000 ms turn), when to advance the mark. A sixth harness reaching for `busy_ms` rebuilds that, and its tool time is then subtracted TWICE — by the reducer and again by the collector — under-reporting generation on one harness only, which takes a corpus comparison to notice. A separate id from CE061 rather than a rebody: CE061 asks where a window's ARITHMETIC came from and four reducers still call `close_window`, so its property is live and unsuperseded; this asks whether a reducer subtracts at all. It deliberately does NOT reuse CE061's `_imports_the_helper`, whose bare-module-import branch exists so `timing.close_window(...)` counts as reaching the helper — inverted into a ban that branch flags four of the five reducers. CE061 is now **exemption-free**: claude-code was its one permanent `# noqa` and, with the subtraction moved, calls the shrunken `close_window` like the other four), **CE060** (in `src/coder_eval/agents/`, every `AssistantMessage(...)` must pass `message_id` explicitly — an identity invariant, which is why it is its own id rather than a second arm of CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field defaulted to `None` on every message it ever recorded, and the evalboard — which groups assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock gap when either side lacks one — collapsed a whole turn's generations into ONE timeline row as soon as the harness's generation windows became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on the day they were written — a snapshot is regenerated from whatever the code currently does, so it catches a later change and never an initial omission. The damage was not confined to the timeline, which is why "only granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the 10 s slow-generation bar were per-turn too. Unlike its two siblings it **derives its constructor set from each module's own `coder_eval.models` imports** instead of hardcoding the spelling, which closes exactly the blind spot CE058's clause below concedes: `claude_code_agent.py` binds only `AssistantMessage as AssistantMessageTelemetry`, so a name list guards that file's two construction sites purely by coincidence, and an arbitrary `as Msg` is missed outright. Widening CE058/CE059 the same way is recorded in `.claude/harness-candidates.md`. BLIND SPOT, in the rule's docstring: the runtime `None` — the kwarg must be PRESENT, not statically non-`None`, because OpenCode's `messageID` and Pi's `responseId` legitimately evaluate to `None` when the CLI omits them, and passing a fallback expression *is* deciding), **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). - - ---- - -## Plugin and GitHub Action layout - -plugins/coder-eval/ # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills//SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `-` (`lint-tasks`, `check-skill`). Never `-`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling. - -action.yml # Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the `v` tag + cuts the Release, so nothing consumer-visible moves before the wheel is on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing. diff --git a/.claude/notes/README.md b/.claude/notes/README.md new file mode 100644 index 000000000..55bffc8df --- /dev/null +++ b/.claude/notes/README.md @@ -0,0 +1,68 @@ +# Architecture notes + +Design rationale moved out of `CLAUDE.md` and out of `src/` docstrings, so both stay +working references rather than changelogs. Nothing here is deleted history. + +**These notes are NOT auto-loaded into context.** Read the file for a subsystem when you +touch it — each entry explains *why* the design is shaped the way it is, and most of them +are written around a specific shipped defect. + +Authoritative sources, when a note and the code disagree: the code wins, then the lint +rule docstrings in `tests/lint/rules/`, then the guides under `docs/`. + +## Contents + +- [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity +- [contracts.md](contracts.md) — criteria, datasets, aggregation, judging +- [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading +- [orchestration.md](orchestration.md) — config merge, resume, early stop, execute vs. run +- [permissions.md](permissions.md) — the chmod window and the reference anti-cheat +- [persistence.md](persistence.md) — atomic writes and judge persistence +- [reporting.md](reporting.md) — reports, pricing, harbor, telemetry, the plugin and Action +- [timing.md](timing.md) — the turn clock and the single subtraction seam + +## What belongs here, and what stays in the source + +Every paragraph in a docstring or comment sorts into exactly one bucket: + +| Bucket | Test | Action | +|---|---|---| +| **CONTRACT** | A caller must know it to call correctly: what it returns, what it mutates, what it raises, an invariant they must maintain. | **Keep** in the source, compressed to its claim. | +| **HAZARD** | A future editor breaks something if they do not know it. Reads as "do not X without Y" — a coupling between two distant places. | **Keep** in the source, 1–3 lines, stating the coupling only. | +| **RATIONALE** | Why the design is this shape; what was considered and cut; what defect motivated it; what was verified experimentally. | **Move** here, to `.md`. | +| **HISTORY** | "used to", "no longer", "previously", "an earlier revision", "shipped once as". | **Delete.** Git holds it. | +| **CROSS-MODULE CLAIM** | Asserts a current property of a different module or harness. | **Delete**, replaced by a link to that module's SSOT. | + +## The pointer line + +Moved rationale leaves exactly one line behind, at the end of the docstring or comment +block it came from: + +``` +Rationale: .claude/notes/timing.md § subtract_tool_time +``` + +Path relative to the repo root, then `§`, then the target `##` heading text verbatim. +There is no other accepted form: `tests/lint/prose_budget.py` parses this one and fails +`make docs-budget` when the file or the heading does not exist. + +## The prose budget is one number, not a lint rule + +`make docs-budget` reports the standing total and fails when it grows. It is deliberately +**not** a `CE` rule: `tests/lint/rules/` polices per-pattern invariants one AST at a time, +while this is a single whole-tree total. Making it a rule would mean a rule class, a rule +test and a catalogue entry to enforce one integer — enlarging the harness the budget +exists to shrink. Do not "fix" this by promoting it. + +Nothing here states how many `CE` rules exist. `tests/lint/rules/` owns that count, and a +number written down anywhere else is a second declaration that will be wrong. + +## The CE lint-rule catalogue + +Each rule's authoritative rationale lives in its own module docstring under +`tests/lint/rules/` (or, for the doc-surface and whole-tree rules, in the +corresponding `@pytest.mark.lint` class in `tests/test_custom_lint.py`). The prose +summary below is kept for orientation only — when it disagrees with a rule file, +the rule file is correct. + +Recent additions, each traceable to a shipped defect: **CE064** (in `src/coder_eval/agents/`, a module that imports `TurnClock` must pass an explicit `timestamp=` to `AgentStartEvent` and `AgentEndEvent` — the turn's OUTER bounds, which no other rule looks at, since CE058-CE061 all scope to `AssistantMessage` and the bracket is not one. `timing.decompose_turn` produces `harness_startup_ms` / `harness_teardown_ms` by subtracting a generation-window bound from a bracket timestamp, so the two must share a basis; all three clocked harnesses derived their bounds from the `TurnClock` and let the bracket fall back to `StreamEvent.timestamp`'s `default_factory=datetime.now`, putting a monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact split `TurnClock` exists to remove, reintroduced at the one seam the clock did not own. Measured on a live antigravity turn: an `AgentEndEvent` stamped **17 us BEFORE its own last message finished**, which cannot happen (the event is constructed strictly after the final flush), and `decompose_turn` clamped that negative and published `0.0` — "measured, and instant", the CE058 confusion arrived at from the other direction — for a harness whose real tail is ~0.1 ms; it now records 0.035 ms. It surfaced on one harness only because the drift is tens of microseconds and antigravity is the only one that holds its process across turns, so nothing happens between its last flush and its end event; every other harness books a tail of 7-543 ms, where the drift is invisible rather than absent — which is why the fix is at every clocked site rather than at that one. SCOPE IS DERIVED, never a harness list: codex and opencode take their spans from the CLI's own epoch stamps, deliberately have no `TurnClock`, and are correctly invisible to the rule — a raw bracket is CONSISTENT with their bounds — and the day either adopts a clock the rule starts applying with no edit. BLIND SPOT, in the rule's docstring: presence, not correctness. It cannot tell `self.clock.now()` from a `datetime.now()` spelled out at the call site, because the three harnesses legitimately reach their clock three ways; the guard for the SOURCE is behavioural (`tests/_bracket_clock.py` injects a stand-in anchored a year from real time, so a reverted argument fails by a year rather than by the microseconds that separate the two clocks), which is the division of labour CE060 states — a rule removes the SILENT case, a default nobody chose), **CE063** (no module in `src/coder_eval/agents/` may import `busy_ms` — tool execution comes out of a generation window in exactly ONE place, `timing.py::subtract_tool_time`. Five reducers used to do it themselves while the head and tail were already computed centrally at the same seam, and that asymmetry is where every timing defect on this branch lived — none of them in the arithmetic, all of them in the bookkeeping AROUND it: when to reset a per-step span list (clearing it at `step_start` wiped a span before the flush could subtract it, a 100% overstatement of that window), when to clear a spent start stamp (a second flush with no intervening start republished the previous span — 3000 ms of generation for a 2000 ms turn), when to advance the mark. A sixth harness reaching for `busy_ms` rebuilds that, and its tool time is then subtracted TWICE — by the reducer and again by the collector — under-reporting generation on one harness only, which takes a corpus comparison to notice. A separate id from CE061 rather than a rebody: CE061 asks where a window's ARITHMETIC came from and four reducers still call `close_window`, so its property is live and unsuperseded; this asks whether a reducer subtracts at all. It deliberately does NOT reuse CE061's `_imports_the_helper`, whose bare-module-import branch exists so `timing.close_window(...)` counts as reaching the helper — inverted into a ban that branch flags four of the five reducers. CE061 is now **exemption-free**: claude-code was its one permanent `# noqa` and, with the subtraction moved, calls the shrunken `close_window` like the other four), **CE060** (in `src/coder_eval/agents/`, every `AssistantMessage(...)` must pass `message_id` explicitly — an identity invariant, which is why it is its own id rather than a second arm of CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field defaulted to `None` on every message it ever recorded, and the evalboard — which groups assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock gap when either side lacks one — collapsed a whole turn's generations into ONE timeline row as soon as the harness's generation windows became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on the day they were written — a snapshot is regenerated from whatever the code currently does, so it catches a later change and never an initial omission. The damage was not confined to the timeline, which is why "only granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the 10 s slow-generation bar were per-turn too. Unlike its two siblings it **derives its constructor set from each module's own `coder_eval.models` imports** instead of hardcoding the spelling, which closes exactly the blind spot CE058's clause below concedes: `claude_code_agent.py` binds only `AssistantMessage as AssistantMessageTelemetry`, so a name list guards that file's two construction sites purely by coincidence, and an arbitrary `as Msg` is missed outright. Widening CE058/CE059 the same way is recorded in `.claude/harness-candidates.md`. BLIND SPOT, in the rule's docstring: the runtime `None` — the kwarg must be PRESENT, not statically non-`None`, because OpenCode's `messageID` and Pi's `responseId` legitimately evaluate to `None` when the CLI omits them, and passing a fallback expression *is* deciding), **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md new file mode 100644 index 000000000..580cd5aa1 --- /dev/null +++ b/.claude/notes/agents.md @@ -0,0 +1,13 @@ +# Agents + +> Conventions and authority order: see [README.md](README.md). + +## Token accounting and the reconciliation message + +- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. + +- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. + +## Harness run-limit parity + +- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. **Pi** is the same shape — the CLI (`pi -p --mode json`) streams a real multi-step loop per `communicate()` (`turn_start`/`turn_end`), so `max_turns: N` counts native `turn_start` steps; Pi retries transient/provider errors INTERNALLY (`agent_end.willRetry`), and the reducer finalizes once at `agent_settled`/EOF (not the first `agent_end`), folding the retry cycles into one turn. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex, Antigravity, and Pi (all run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), `allowed_tools`/`disallowed_tools` on Pi (its built-in tool names are lowercase — `bash`/`read`/… — and cannot map to the Claude-namespaced config default, so forwarding them would strip the agent of ALL tools; warned+ignored like the three agents above) — Pi DOES enforce `system_prompt` (`--append-system-prompt`, a small win over OpenCode) and DOES honor `plugins` for skills (each resolved skills dir → a `--skill ` arg via the shared `_plugin_skill_dirs` resolver, recorded as `pi_skill_paths`, so it CAN run activation suites) but does NOT read `system_prompt_file`, and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md. diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md new file mode 100644 index 000000000..5a3a55a8f --- /dev/null +++ b/.claude/notes/contracts.md @@ -0,0 +1,9 @@ +# Contracts: criteria, datasets, judging + +> Conventions and authority order: see [README.md](README.md). + +## Datasets and aggregation + +- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.) + +- **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure. diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md new file mode 100644 index 000000000..d5ab9eb73 --- /dev/null +++ b/.claude/notes/isolation.md @@ -0,0 +1,7 @@ +# Isolation: docker, sandbox, detached grading + +> Conventions and authority order: see [README.md](README.md). + +## Detached grading and `Sandbox.adopt` + +- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses is ONE parameter (`grade_in_place`, resolved by `_gate_scope_for_grade`), not two: it shipped beside an `include_setup_phase` every caller passed as its exact complement, and a future caller setting one and forgetting the other would silently drop half of a SECURITY gate. Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and `_assert_regrade_honored` refuses a returned row whose `started_at` moved, because an image predating this change ignores the unknown `regrade` key and RUNS THE AGENT, which the host would otherwise fold back as the recorded row's verdict (the exact sibling of `_assert_grade_honored`, one release later). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. `grant_container_access` now RETURNS what it widened and `run()` restores it in the same `finally`: the two staging dirs are disposable, but the graded workspace is the caller's tree — an operator-supplied `--workspace` was left world-writable permanently. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: every container is launched `TELEMETRY_ENABLED=false` under the invariant "container silent, host emits once", and the grading path had inherited only the silent half. The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `git clone` now passes `--` before the URL (argv position 2, so a value beginning with `-` was parsed as an option). `Sandbox.resolve_files` is containment-checked for the same reason: criterion paths were the one task-authored path skipping `_resolve_within_sandbox`, and `Path(root) / '/etc/passwd'` is `/etc/passwd`. An escaping LITERAL now raises `CheckerMisuseError` rather than resolving to `[]`: returning no match books an eval-CONFIG error as an agent failure — a gating 0.0 reading "file does not exist" for a file that plainly does exist and that no agent behaviour could place inside the sandbox (CE039's exact distinction). `tasks/byod_smoke_test.yaml` was broken that way for several commits, checking `/opt/byod_marker` baked into the BYOD image, with only a task-log warning to show for it; it now asserts on the container with `run_command: test -f …`, which is what a claim about the IMAGE rather than about the agent's workspace should look like. The guard keys on the escaping path EXISTING, so a merely-absent absolute path stays an ordinary failing verdict, and the GLOB branch still warns-and-drops, since filtering some matches out of a search is its normal behaviour. A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (the one legitimate rewrite — we are already inside the container the driver asked for), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md new file mode 100644 index 000000000..c5c88b3e6 --- /dev/null +++ b/.claude/notes/orchestration.md @@ -0,0 +1,21 @@ +# Orchestration + +> Conventions and authority order: see [README.md](README.md). + +## Config merging and CLI overrides + +- **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly. + +- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only. + +## Execute vs. run: the grading switch + +- **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` with grading removed — the agent runs and the full trajectory is captured, but no criterion is checked, `weighted_score` is `None` (never `0.0`, which would be indistinguishable from "graded and scored zero"), and the row finalizes as **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are facts about the *run*, not about grading, and still apply (so `execute` still exits non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog check, and post-failure diagnostics); it crosses the docker boundary in `context.json` (defaulting to `True` in-container, so a host predating `execute` keeps grading). It is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because a task YAML must never declare itself ungraded; only the invoking command decides. `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code path. Three things are refused rather than degraded: `--junit-xml` (a report of verdicts, and there are none — though `reports_junit` still emits `` for an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades nothing), and simulation tasks (their turn-continuation logic reads criteria results, so an ungraded dialog would silently change its own stopping behavior). `stop_early:` blocks are inert under `execute` for the same reason the kill switch exists: the full trajectory is the deliverable. Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and grades with its own tests. + +## `--resume` is command-relative + +- **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate ` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`. + +## Early stop on criterion + +- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md new file mode 100644 index 000000000..e75298eb0 --- /dev/null +++ b/.claude/notes/permissions.md @@ -0,0 +1,7 @@ +# Permissions and the reference anti-cheat + +> Conventions and authority order: see [README.md](README.md). + +## Reference solutions and the anti-cheat window + +- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half. diff --git a/.claude/notes/persistence.md b/.claude/notes/persistence.md new file mode 100644 index 000000000..c8182b546 --- /dev/null +++ b/.claude/notes/persistence.md @@ -0,0 +1,4 @@ +# Persistence + +> Conventions and authority order: see [README.md](README.md). + diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md new file mode 100644 index 000000000..3e1ea1895 --- /dev/null +++ b/.claude/notes/reporting.md @@ -0,0 +1,15 @@ +# Reporting, pricing, harbor, telemetry + +> Conventions and authority order: see [README.md](README.md). + +## Published rates and run-time caps + +- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)`. The guard originally shipped on `RunSummary` alone, so the same 10-task `execute` run with one crash rendered "Pass Rate: n/a" in `run.md` and "Pass Rate: 0.0%" in `experiment.md`. `measured` is **counted evidence** (`tasks_measured` / `rows_measured` — rows carrying a `weighted_score`), never a bucket count: the first version tested `tasks_succeeded + tasks_failed == 0`, but `TIMEOUT` and the two budget stops are category `failed` and reachable under `execute` (`_check_run_limits` still runs on the ungraded branch), so ONE timed-out row in a 100-task ungraded night read as "measured" and published `pass_rate: 0.0` — a real 0% point on the evalboard trend for a run that graded nothing. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender. + +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. + +## Plugin and GitHub Action layout + +plugins/coder-eval/ # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills//SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `-` (`lint-tasks`, `check-skill`). Never `-`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling. + +action.yml # Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the `v` tag + cuts the Release, so nothing consumer-visible moves before the wheel is on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing. diff --git a/.claude/notes/timing.md b/.claude/notes/timing.md new file mode 100644 index 000000000..c4c486fdb --- /dev/null +++ b/.claude/notes/timing.md @@ -0,0 +1,4 @@ +# Timing + +> Conventions and authority order: see [README.md](README.md). + diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index f08442a23..16d4385d1 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -94,6 +94,9 @@ jobs: - name: Custom architectural lint (CE001+) run: .venv/bin/pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings + - name: Prose budget (docstring/comment ratchet) + run: .venv/bin/python -m tests.lint.prose_budget + # PHASE 2: Type checking - name: Type check with pyright run: .venv/bin/pyright diff --git a/CLAUDE.md b/CLAUDE.md index 611cd07ac..7f139d69e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,9 +3,9 @@ Working reference for AI assistants on the `coder_eval` codebase. Design *rationale* — why a subsystem is shaped the way it is, and which shipped defect -shaped it — lives in **`.claude/architecture-notes.md`**, which is not auto-loaded. Read -it before changing grading, resume, early stop, timing, the reference anti-cheat, or an -agent adapter. +shaped it — lives under **`.claude/notes/`**, which is not auto-loaded; start at +[`.claude/notes/README.md`](.claude/notes/README.md). Read it before changing grading, +resume, early stop, timing, the reference anti-cheat, or an agent adapter. User-facing documentation lives in [`docs/`](docs/index.md): start with the [User Guide](docs/USER_GUIDE.md) for CLI behaviour and the @@ -89,7 +89,7 @@ action.yml # Published composite GitHub Action ## Key Architectural Patterns -Each entry is a pointer. Full rationale: `.claude/architecture-notes.md`. +Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes/README.md`). - **Discriminated unions** for criteria types and template sources. - **Plugin registry**: `criteria/` auto-discovers via `pkgutil` + `@register_criterion`. @@ -207,6 +207,8 @@ make evalboard-verify # the JS half: tsc --noEmit + vitest + next build # Regenerate a generated surface — never hand-edit the output make docs-indexes # README/docs index tables from the mkdocs nav (CE028) make plugin-reference # the plugin's criteria reference from the models (CE033) + +make docs-budget # prose budget report; fails `make verify` if the total grows ``` Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is @@ -357,10 +359,14 @@ bandit, pre-commit, mcp - **Comments are a last resort** — default to ZERO comments. Names, types and small functions carry the meaning. A comment is allowed ONLY when it records something the code cannot say +- **A docstring states the contract, not the history** — what a caller must know to call + it correctly. Why the design is this shape belongs in `.claude/notes/`; what it used to + be belongs in git. `make docs-budget` reports the standing total and fails + `make verify` if it grows ## Notes for AI Assistants - Communication style: use ASD-STE-100 when you speak to the user. - Temporary files go in `tmp/`, not `/tmp`. -- Read `.claude/architecture-notes.md` before changing grading, resume, early stop, - timing, the reference anti-cheat, or any significant parts of this code's architecture. +- Read `.claude/notes/` before changing grading, resume, early stop, timing, the + reference anti-cheat, or any significant parts of this code's architecture. diff --git a/Makefile b/Makefile index ba200fcb0..62b03d52a 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra evalboard-verify clean run lint docs-indexes plugin-reference docker-image docker-image-full coder-eval-runtime docker-images +.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra evalboard-verify clean run lint docs-indexes plugin-reference docs-budget docker-image docker-image-full coder-eval-runtime docker-images # Single source of the installed coder-eval version (used to tag the docker # images). Referenced lazily inside the docker recipes, so it doesn't run on @@ -36,6 +36,9 @@ docs-indexes: ## Regenerate README/docs indexes from the mkdocs nav (SSOT) plugin-reference: ## Regenerate the plugin's bundled criteria reference from the models (SSOT) uv run python -m tests.lint.plugin_reference +docs-budget: ## Report the docstring/comment prose budget and check it against the baseline + uv run python -m tests.lint.prose_budget + typecheck: ## Run type checking with pyright uv run pyright # The CE036 contract engine executes checker code and feeds the early-stop @@ -64,6 +67,7 @@ verify: ## Run all verification steps (CI equivalent) uv run ruff check $(LINT_PATHS) uv run pyright uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings + uv run python -m tests.lint.prose_budget # uv run pip-audit --desc --skip-editable # uv run bandit -r src/ -ll --format json -o bandit-report.json uv run pytest tests/ -n auto -m "not live and not lint" --cov=coder_eval --cov-report=term-missing --cov-report=xml --cov-fail-under=80 diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py new file mode 100644 index 000000000..cee05a5bb --- /dev/null +++ b/tests/lint/prose_budget.py @@ -0,0 +1,348 @@ +"""Measure and ratchet the essay-shaped prose in ``src/coder_eval``. + +One gated number: ``essay_words`` — words in docstrings over 150 words (Typer command +docstrings exempt, they render as ``--help``) plus words in comment runs of three or +more consecutive lines. It may not exceed ``_ESSAY_BASELINE_WORDS``, so the house style +is *no new essays*, not *no new documentation*. + +Also resolves every ``Rationale: § `` pointer, and — under +``--assert-code-unchanged `` — proves a commit moved prose only, by comparing the +docstring-stripped AST and the multiset of functional directive comments against ``ref``. + +Stdlib only, and it never imports, execs or evals a file it measures. +""" + +from __future__ import annotations + +import ast +import io +import re +import subprocess +import sys +import tokenize +from collections import Counter +from pathlib import Path +from typing import NamedTuple + + +_DOCSTRING_ESSAY_WORDS = 150 +_COMMENT_BLOCK_LINES = 3 +_ESSAY_BASELINE_WORDS = 79_754 + +_SRC = Path("src/coder_eval") + +# Exempt by (path relative to src/coder_eval, function name) pair, and only for a +# function at module level: `Sandbox.run_command` is a method and a bare-name exemption +# would silently excuse it. Registered in src/coder_eval/cli/__init__.py. +_TYPER_COMMANDS = frozenset( + { + ("cli/run_command.py", "run_command"), + ("cli/execute_command.py", "execute_command"), + ("cli/plan_command.py", "plan_command"), + ("cli/evaluate_command.py", "evaluate_command"), + ("cli/report_command.py", "report_command"), + ("cli/aggregate_command.py", "aggregate_command"), + ("cli/export_command.py", "export_command"), + ("cli/harbor_command.py", "reward_command"), + ("cli/run_task_internal_command.py", "run_task_internal_command"), + } +) + +_DOCSTRING_OWNERS = (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef) + +_POINTER = re.compile(r"Rationale:\s*(\S+\.md)\s*§\s*(.+?)\s*$") + +_HEADING = re.compile(r"^##\s+(.+?)\s*$") + +# Executable directives that live in comments and therefore never enter the AST. +_DIRECTIVE = re.compile(r"^#\s*(?:noqa\b|type:\s*ignore\b|pyright:|nosec\b|pragma:|fmt:)") + + +class FileProse(NamedTuple): + """Per-file prose measurement. ``essays`` is ``(qualname, words)``, descending.""" + + docstring_words: int + comment_words: int + comment_blocks: int + essays: list[tuple[str, int]] + + @property + def total(self) -> int: + return self.docstring_words + self.comment_words + + +class Measurement(NamedTuple): + """Everything one tree scan produced: measured files, plus the ones that would not parse.""" + + files: dict[Path, FileProse] + skipped: list[Path] + + +def docstring_words(text: str) -> int: + return len(text.split()) + + +def comment_words(comment: str) -> int: + """Words in one comment token. The ``#`` itself is not a word.""" + return len(comment.lstrip().removeprefix("#").split()) + + +def _comment_runs(source: str) -> list[list[tokenize.TokenInfo]]: + """Comment tokens grouped into runs of consecutive lines.""" + runs: list[list[tokenize.TokenInfo]] = [] + previous_line = -2 + for token in tokenize.generate_tokens(io.StringIO(source).readline): + if token.type != tokenize.COMMENT: + continue + if token.start[0] == previous_line + 1 and runs: + runs[-1].append(token) + else: + runs.append([token]) + previous_line = token.start[0] + return runs + + +def measure_source(source: str, rel: str) -> FileProse | None: + """Measure one module's essay prose. ``None`` when it does not parse.""" + try: + tree = ast.parse(source) + runs = _comment_runs(source) + except (SyntaxError, tokenize.TokenError, ValueError): + return None + + top_level = {id(node) for node in tree.body} + essays: list[tuple[str, int]] = [] + for node in ast.walk(tree): + if not isinstance(node, _DOCSTRING_OWNERS): + continue + text = ast.get_docstring(node) + if text is None: + continue + name = "" if isinstance(node, ast.Module) else node.name + if (rel, name) in _TYPER_COMMANDS and id(node) in top_level: + continue + words = docstring_words(text) + if words > _DOCSTRING_ESSAY_WORDS: + essays.append((name, words)) + essays.sort(key=lambda essay: (-essay[1], essay[0])) + + blocks = [run for run in runs if len(run) >= _COMMENT_BLOCK_LINES] + return FileProse( + docstring_words=sum(words for _, words in essays), + comment_words=sum(comment_words(token.string) for run in blocks for token in run), + comment_blocks=len(blocks), + essays=essays, + ) + + +def measure(repo_root: Path) -> Measurement: + """Scan ``src/coder_eval``. Files that do not parse are skipped, never fatal.""" + files: dict[Path, FileProse] = {} + skipped: list[Path] = [] + for path in sorted((repo_root / _SRC).rglob("*.py")): + rel = path.relative_to(repo_root / _SRC) + prose = measure_source(path.read_text(encoding="utf-8"), rel.as_posix()) + if prose is None: + skipped.append(rel) + elif prose.total: + files[rel] = prose + return Measurement(files=files, skipped=skipped) + + +def total_words(files: dict[Path, FileProse]) -> int: + """The one summing seam the report and the gate both use.""" + return sum(prose.total for prose in files.values()) + + +def _subsystem(rel: Path) -> str: + return rel.parts[0] if len(rel.parts) > 1 else "top-level" + + +def render_report(measurement: Measurement) -> str: + grouped: dict[str, list[tuple[Path, FileProse]]] = {} + for rel, prose in measurement.files.items(): + grouped.setdefault(_subsystem(rel), []).append((rel, prose)) + + lines = ["PROSE BUDGET (docstrings over 150 words + comment runs of 3+ lines)", ""] + for subsystem in sorted(grouped, key=lambda name: (-sum(p.total for _, p in grouped[name]), name)): + rows = sorted(grouped[subsystem], key=lambda row: (-row[1].total, row[0].as_posix())) + lines.append(subsystem) + for rel, prose in rows: + lines.append( + f" {rel.as_posix():<48}{prose.docstring_words:>7} doc{prose.comment_words:>7} cmt{prose.total:>8}" + ) + lines.append(f" {'subtotal':<48}{'':>7} {'':>7} {sum(p.total for _, p in rows):>8}") + lines.append("") + + files = measurement.files + lines.append( + f"TOTAL {total_words(files)}" + f" = {sum(p.docstring_words for p in files.values())} docstring" + f" + {sum(p.comment_words for p in files.values())} comment" + ) + lines.append( + f" files with prose {len(files)}" + f" essays {sum(len(p.essays) for p in files.values())}" + f" blocks {sum(p.comment_blocks for p in files.values())}" + ) + if measurement.skipped: + lines.append(f"skipped: {', '.join(p.as_posix() for p in measurement.skipped)}") + + roster = sorted( + ((rel, name, words) for rel, prose in files.items() for name, words in prose.essays), + key=lambda row: (-row[2], row[0].as_posix(), row[1]), + ) + lines += ["", f"ESSAYS ({len(roster)} docstrings over {_DOCSTRING_ESSAY_WORDS} words)"] + lines += [f" {f'{rel.as_posix()}::{name}':<68}{words:>6}" for rel, name, words in roster] + return "\n".join(lines) + "\n" + + +def _prose_lines(source: str) -> list[str]: + """Every docstring and comment line in a module, for pointer scanning.""" + out: list[str] = [] + try: + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, _DOCSTRING_OWNERS): + out += (ast.get_docstring(node) or "").splitlines() + for token in tokenize.generate_tokens(io.StringIO(source).readline): + if token.type == tokenize.COMMENT: + out.append(token.string) + except (SyntaxError, tokenize.TokenError, ValueError): + return out + return out + + +def _headings(path: Path) -> set[str]: + if not path.is_file(): + return set() + return {match.group(1) for line in path.read_text(encoding="utf-8").splitlines() if (match := _HEADING.match(line))} + + +def check_pointers(repo_root: Path) -> list[str]: + """Every ``Rationale: § `` must resolve. Returns the failures.""" + failures: list[str] = [] + headings: dict[Path, set[str]] = {} + for path in sorted((repo_root / _SRC).rglob("*.py")): + rel = path.relative_to(repo_root / _SRC) + for line in _prose_lines(path.read_text(encoding="utf-8")): + match = _POINTER.search(line.strip()) + if not match: + continue + target, heading = Path(match.group(1)), match.group(2) + if target not in headings: + headings[target] = _headings(repo_root / target) + if not (repo_root / target).is_file(): + failures.append(f"{rel.as_posix()}: no such file {target.as_posix()}") + elif heading not in headings[target]: + failures.append(f"{rel.as_posix()}: {target.as_posix()} has no heading '## {heading}'") + return failures + + +def check(repo_root: Path) -> str | None: + """``None`` when the tree is at or under the baseline, else the failure message.""" + total = total_words(measure(repo_root).files) + if total <= _ESSAY_BASELINE_WORDS: + return None + return ( + f"prose budget exceeded: {total} essay words against a baseline of " + f"{_ESSAY_BASELINE_WORDS} (+{total - _ESSAY_BASELINE_WORDS}). " + "Move rationale to .claude/notes/, or lower the baseline if you removed prose." + ) + + +def code_shape(source: str) -> str: + """``ast.dump`` of the module with every docstring filtered out of its body. + + Filtered, not replaced by ``Pass``: a docstring-only body must not compare equal to + a ``pass``-bodied one. ``ast.unparse``/``compile`` reject the resulting empty body; + ``ast.dump`` does not. + """ + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, _DOCSTRING_OWNERS) and ast.get_docstring(node) is not None: + node.body = node.body[1:] + return ast.dump(tree) + + +def directive_comments(source: str) -> Counter[str]: + """Multiset of functional directive comments (``# noqa``, ``# nosec``, ...). + + These never enter the AST, so deleting one is a behaviour change ``code_shape`` is + structurally blind to. + """ + found: Counter[str] = Counter() + try: + tokens = list(tokenize.generate_tokens(io.StringIO(source).readline)) + except (tokenize.TokenError, IndentationError, SyntaxError, ValueError): + return found + for token in tokens: + if token.type == tokenize.COMMENT and _DIRECTIVE.match(token.string.strip()): + found[token.string.strip()] += 1 + return found + + +def _git(repo_root: Path, *args: str) -> tuple[int, str]: + result = subprocess.run( + ["git", *args], + cwd=repo_root, + capture_output=True, + text=True, + check=False, + ) + return result.returncode, result.stdout + + +def assert_code_unchanged(repo_root: Path, ref: str) -> list[str]: + """Report every ``src/coder_eval`` file whose code — not prose — differs from ``ref``.""" + code, listing = _git(repo_root, "diff", "--name-only", ref, "--", _SRC.as_posix()) + if code != 0: + return [f"git diff against {ref!r} failed"] + + findings: list[str] = [] + for name in sorted(filter(None, listing.splitlines())): + if not name.endswith(".py"): + continue + shown, before = _git(repo_root, "show", f"{ref}:{name}") + before = before if shown == 0 else "" + path = repo_root / name + after = path.read_text(encoding="utf-8") if path.is_file() else "" + try: + if code_shape(before) != code_shape(after): + findings.append(f"{name}: code changed (AST differs after stripping docstrings)") + except SyntaxError: + findings.append(f"{name}: could not parse both revisions") + continue + dropped = directive_comments(before) - directive_comments(after) + findings += [ + f"{name}: dropped directive comment {comment!r} x{count}" for comment, count in sorted(dropped.items()) + ] + return findings + + +def main(argv: list[str]) -> int: + repo_root = Path(__file__).resolve().parents[2] + + if argv[:1] == ["--assert-code-unchanged"]: + if len(argv) != 2: + print("usage: --assert-code-unchanged ", file=sys.stderr) + return 2 + findings = assert_code_unchanged(repo_root, argv[1]) + for finding in findings: + print(finding, file=sys.stderr) + return 1 if findings else 0 + + print(render_report(measure(repo_root)), end="") + + failed = False + for failure in check_pointers(repo_root): + print(f"unresolved pointer: {failure}", file=sys.stderr) + failed = True + if (message := check(repo_root)) is not None: + print(message, file=sys.stderr) + failed = True + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py new file mode 100644 index 000000000..e527ffe98 --- /dev/null +++ b/tests/test_prose_budget.py @@ -0,0 +1,255 @@ +"""Unit tests for the prose budget measurement (``tests/lint/prose_budget.py``). + +Every case runs against a synthetic tree in ``tmp_path`` or a plain string, never +against the real ``src/coder_eval`` — its word counts change with every prose commit, +so asserting on them here would make this file a second, drifting baseline. +""" + +from __future__ import annotations + +import textwrap +from pathlib import Path + +import pytest + +from tests.lint import prose_budget + + +def _words(count: int) -> str: + return " ".join(f"w{index}" for index in range(count)) + + +def _tree(tmp_path: Path, files: dict[str, str]) -> Path: + for rel, text in files.items(): + path = tmp_path / "src" / "coder_eval" / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(textwrap.dedent(text), encoding="utf-8") + return tmp_path + + +class TestDocstringMeasurement: + def test_long_function_docstring_is_counted(self) -> None: + prose = prose_budget.measure_source(f'def f():\n """{_words(200)}"""\n', "m.py") + assert prose is not None + assert prose.docstring_words == 200 + assert prose.essays == [("f", 200)] + + def test_exactly_the_threshold_is_not_an_essay(self) -> None: + prose = prose_budget.measure_source(f'def f():\n """{_words(150)}"""\n', "m.py") + assert prose is not None + assert prose.docstring_words == 0 + assert prose.essays == [] + + def test_one_word_over_the_threshold_is_counted(self) -> None: + prose = prose_budget.measure_source(f'def f():\n """{_words(151)}"""\n', "m.py") + assert prose is not None + assert prose.docstring_words == 151 + + def test_module_docstring_is_named_module(self) -> None: + prose = prose_budget.measure_source(f'"""{_words(200)}"""\n', "m.py") + assert prose is not None + assert prose.essays == [("", 200)] + + +class TestTyperExemption: + def test_module_level_typer_command_is_exempt(self) -> None: + source = f'def run_command():\n """{_words(400)}"""\n' + prose = prose_budget.measure_source(source, "cli/run_command.py") + assert prose is not None + assert prose.docstring_words == 0 + + def test_same_name_as_a_method_elsewhere_is_counted(self) -> None: + """``Sandbox.run_command`` — the case a bare-name exemption would silently excuse.""" + source = f'class Sandbox:\n def run_command(self):\n """{_words(400)}"""\n' + prose = prose_budget.measure_source(source, "sandbox.py") + assert prose is not None + assert prose.docstring_words == 400 + + def test_same_name_nested_inside_the_exempt_module_is_counted(self) -> None: + source = f'class Helper:\n def run_command(self):\n """{_words(400)}"""\n' + prose = prose_budget.measure_source(source, "cli/run_command.py") + assert prose is not None + assert prose.docstring_words == 400 + + +class TestCommentMeasurement: + def test_three_consecutive_lines_are_a_block(self) -> None: + prose = prose_budget.measure_source("# one two\n# three four\n# five six\nx = 1\n", "m.py") + assert prose is not None + assert prose.comment_words == 6 + assert prose.comment_blocks == 1 + + def test_a_bare_hash_adds_no_words_but_keeps_the_run_length(self) -> None: + prose = prose_budget.measure_source("# one two\n#\n# five six\nx = 1\n", "m.py") + assert prose is not None + assert prose.comment_words == 4 + assert prose.comment_blocks == 1 + + def test_two_consecutive_lines_are_not_a_block(self) -> None: + prose = prose_budget.measure_source("# one two\n# three four\nx = 1\n", "m.py") + assert prose is not None + assert prose.comment_words == 0 + + def test_a_blank_line_splits_the_run(self) -> None: + prose = prose_budget.measure_source("# one two\n# three four\n\n# five six\nx = 1\n", "m.py") + assert prose is not None + assert prose.comment_words == 0 + + def test_two_separate_runs_are_both_counted(self) -> None: + source = "# a b\n# c d\n# e f\nx = 1\n# g h\n# i j\n# k l\n# m n\ny = 2\n" + prose = prose_budget.measure_source(source, "m.py") + assert prose is not None + assert prose.comment_words == 14 + assert prose.comment_blocks == 2 + + def test_the_hash_is_not_a_word(self) -> None: + assert prose_budget.comment_words("# foo bar") == 2 + + +class TestMeasureTree: + def test_a_syntax_error_is_skipped_not_fatal(self, tmp_path: Path) -> None: + root = _tree( + tmp_path, + { + "broken.py": "def f(:\n", + "good.py": f'def f():\n """{_words(200)}"""\n', + }, + ) + measurement = prose_budget.measure(root) + assert measurement.skipped == [Path("broken.py")] + assert measurement.files[Path("good.py")].docstring_words == 200 + + def test_total_words_sums_docstrings_and_comments(self, tmp_path: Path) -> None: + root = _tree( + tmp_path, + { + "a.py": f'def f():\n """{_words(200)}"""\n', + "b.py": "# one two\n# three four\n# five six\nx = 1\n", + }, + ) + assert prose_budget.total_words(prose_budget.measure(root).files) == 206 + + +class TestCheck: + def test_at_the_baseline_it_passes(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = _tree(tmp_path, {"a.py": f'def f():\n """{_words(200)}"""\n'}) + monkeypatch.setattr(prose_budget, "_ESSAY_BASELINE_WORDS", 200) + assert prose_budget.check(root) is None + + def test_one_word_above_the_baseline_fails(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = _tree(tmp_path, {"a.py": f'def f():\n """{_words(200)}"""\n'}) + monkeypatch.setattr(prose_budget, "_ESSAY_BASELINE_WORDS", 199) + message = prose_budget.check(root) + assert message is not None + assert "200" in message and "199" in message + + +class TestPointers: + def _root(self, tmp_path: Path, source: str, notes: str) -> Path: + root = _tree(tmp_path, {"a.py": source}) + target = root / ".claude" / "notes" / "timing.md" + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(notes, encoding="utf-8") + return root + + def test_a_resolving_pointer_passes(self, tmp_path: Path) -> None: + source = 'def f():\n """Do it.\n\n Rationale: .claude/notes/timing.md § close_window\n """\n' + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n\nWhy.\n") + assert prose_budget.check_pointers(root) == [] + + def test_a_pointer_in_a_comment_resolves(self, tmp_path: Path) -> None: + source = "# Rationale: .claude/notes/timing.md § close_window\nx = 1\n" + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n\nWhy.\n") + assert prose_budget.check_pointers(root) == [] + + def test_a_missing_file_fails(self, tmp_path: Path) -> None: + source = '"""Rationale: .claude/notes/absent.md § close_window"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n") + failures = prose_budget.check_pointers(root) + assert len(failures) == 1 + assert "no such file" in failures[0] + + def test_a_missing_heading_fails(self, tmp_path: Path) -> None: + source = '"""Rationale: .claude/notes/timing.md § open_window"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n") + failures = prose_budget.check_pointers(root) + assert len(failures) == 1 + assert "no heading" in failures[0] + + def test_a_heading_with_backticks_resolves(self, tmp_path: Path) -> None: + source = '"""Rationale: .claude/notes/timing.md § `--resume` is command-relative"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## `--resume` is command-relative\n") + assert prose_budget.check_pointers(root) == [] + + def test_a_heading_with_a_colon_resolves(self, tmp_path: Path) -> None: + source = '"""Rationale: .claude/notes/timing.md § Execute vs. run: the grading switch"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## Execute vs. run: the grading switch\n") + assert prose_budget.check_pointers(root) == [] + + +class TestCodeShape: + def test_prose_only_differences_compare_equal(self) -> None: + before = 'def f(a):\n """One.\n\n Long rationale.\n """\n # why\n # more why\n return a + 1\n' + after = 'def f(a):\n """Two."""\n return a + 1\n' + assert prose_budget.code_shape(before) == prose_budget.code_shape(after) + + def test_a_one_statement_edit_differs(self) -> None: + before = 'def f(a):\n """One."""\n return a + 1\n' + after = 'def f(a):\n """One."""\n return a + 2\n' + assert prose_budget.code_shape(before) != prose_budget.code_shape(after) + + def test_a_docstring_only_body_does_not_raise(self) -> None: + assert prose_budget.code_shape('def f():\n """Only this."""\n') + + def test_a_docstring_only_body_is_not_a_pass_body(self) -> None: + docstring_only = prose_budget.code_shape('def f():\n """Only this."""\n') + pass_body = prose_budget.code_shape("def f():\n pass\n") + assert docstring_only != pass_body + + +class TestDirectiveComments: + def test_a_directive_inside_a_block_is_found(self) -> None: + source = "# context\n# more context\nx = 1 # noqa: CE051\n" + assert prose_budget.directive_comments(source) == {"# noqa: CE051": 1} + + def test_dropping_a_directive_shows_up_as_a_difference(self) -> None: + before = "# context\n# noqa: CE051\n# more\nx = 1\n" + after = "# context\n# more\nx = 1\n" + dropped = prose_budget.directive_comments(before) - prose_budget.directive_comments(after) + assert dropped == {"# noqa: CE051": 1} + + def test_ordinary_prose_is_not_a_directive(self) -> None: + assert prose_budget.directive_comments("# just a comment\nx = 1\n") == {} + + @pytest.mark.parametrize( + "comment", + [ + "# type: ignore[return-value]", + "# pyright: ignore[reportMissingImports]", + "# pyright: reportIncompatibleVariableOverride=false", + "# nosec B310", + "# pragma: no cover", + "# fmt: off", + ], + ) + def test_every_tracked_directive_form_is_recognised(self, comment: str) -> None: + assert prose_budget.directive_comments(f"x = 1 {comment}\n") == {comment: 1} + + +class TestRenderReport: + def test_the_report_names_files_subsystems_the_total_and_the_essays(self, tmp_path: Path) -> None: + root = _tree( + tmp_path, + { + "a.py": f'def essay_fn():\n """{_words(200)}"""\n', + "agents/b.py": "# one two\n# three four\n# five six\nx = 1\n", + }, + ) + report = prose_budget.render_report(prose_budget.measure(root)) + assert "a.py" in report + assert "top-level" in report and "agents" in report + assert "subtotal" in report + assert "TOTAL 206" in report + assert "ESSAYS" in report + assert "a.py::essay_fn" in report + assert "200" in report From 5801474302025d31fd28e7e3b9b7db57bb8dc0dc Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 17:53:22 -0700 Subject: [PATCH 02/19] =?UTF-8?q?docs:=202/7=20=E2=80=94=20move=20timing?= =?UTF-8?q?=20and=20permissions=20rationale=20into=20.claude/notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cuts the four densest files in the tree from 6,874 essay words to 533, leaving the caller-facing contract and the couplings a future editor would break, and moving the design rationale to `.claude/notes/{timing,permissions,persistence}.md`. Kept in the source: the `timing.ts` parity claim and its shared fixture, `TurnClock`'s not-for-deadlines hazard, the raw-total ordering hazard and its division guard, the stacked-window contract, and every CE reference whose text stayed. Deleted: history git already holds, and `TurnClock`'s per-harness roster, which `docs/agents/HARNESS_PARITY.md` owns. Also corrects a stale relocated claim: the notes said the task directory was not shielded, but `Orchestrator._communicate_with_retry` chmods it alongside the reference. No executable statement changed — proved per file by `prose_budget --assert-code-unchanged`, which compares the docstring-stripped AST and the functional directive comments. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2 --- .claude/notes/agents.md | 2 +- .claude/notes/permissions.md | 120 ++++++- .claude/notes/persistence.md | 66 ++++ .claude/notes/timing.md | 306 +++++++++++++++++ src/coder_eval/fs_permissions.py | 274 +++++---------- src/coder_eval/path_utils.py | 114 ++----- src/coder_eval/streaming/collector.py | 152 +++------ src/coder_eval/timing.py | 465 +++++++------------------- tests/lint/prose_budget.py | 2 +- 9 files changed, 779 insertions(+), 722 deletions(-) diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index 580cd5aa1..d2088c335 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -6,7 +6,7 @@ - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. -- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. +- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. The residual is almost always positive; a NEGATIVE one means the captured generations over-report some bucket, which is why the note's wording is branched — a `-512` entry must not read as "billed but not surfaced". ## Harness run-limit parity diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md index e75298eb0..49591654e 100644 --- a/.claude/notes/permissions.md +++ b/.claude/notes/permissions.md @@ -4,4 +4,122 @@ ## Reference solutions and the anti-cheat window -- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half. +- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory **is** shielded alongside it: it previously was not, because under docker it was bind-mounted `:ro` and the chmod returned EROFS, but it is now a read-write throwaway copy (`docker_runner._prepare_task_dir_mount`) so the window applies — which matters because the task dir holds grading material beyond the reference (`run_command` fixtures, expected outputs, and, for a task laid out flat, every SIBLING task's reference). What the window does NOT hide is the task DEFINITION: `task.yaml` is also staged at `/work/input` for the in-container orchestrator, and that mount is untouched. Hiding the criteria from the agent is a separate, unsolved problem. Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half. + +## The stacked chmod window + +The agent under evaluation runs with the same filesystem view as the harness: in +`driver: tempdir` it is an ordinary process on the host, and in `driver: docker` the +orchestrator and the agent share one container. Any directory the harness can read, the +agent can read too — including the task directory and the reference solution. An agent +that greps for the reference does not solve the task, it copies the answer. + +`set_permissions` closes that window for the duration of an `async with` block. The +orchestrator wraps every `agent.communicate` call in it, so the staged reference directory +is unreadable exactly while the agent is executing, and readable again by the time +criteria and judges run. The task directory is shielded by the same window: under docker +it is mounted as a throwaway COPY at a fixed container path, which is what makes it +chmod-able without touching the user's checked-out `tasks/` tree. + +It shields grading MATERIAL that happens to live in the task directory (a `reference/` +subdirectory, fixtures), not the task DEFINITION: `task.yaml` is separately staged at +`/work/input`, which the agent can still read. + +### Why a stack and not a refcount + +Windows nest with *different* modes, so what an exit has to restore is the mode of the +enclosing window — not "the original", and not "is anyone still holding it". A refcount +cannot express that: it would see a nested re-grant as just another holder and silently +leave the outer mode in place. The stack also subsumes what a refcount did, for free — +two windows applying the same mode push two identical entries, and the inner pop +re-applies the outer's identical mode instead of restoring the pre-window one. + +The inner, more permissive form exists for one intended consumer: **live success +criteria**. Early-stop verdicts are computed while the agent turn is still running — i.e. +inside the 000 window — so a live criterion that needs to consult the reference solution +has to be able to read it exactly then, while the agent still cannot. That is why +`READ_ONLY_MODE` is public. It is a designed seam, not speculative generality. + +Only the REFERENCE is shielded, never the sandbox. A live criterion reading the agent's +own output files needs no window change at all — and should not get one. Reading the +static reference mid-turn cannot break the `LiveVerdict` monotonicity contract; reading +the half-written sandbox can, and is the "end-state peeking" `live_verdict` rules out. + +### Not wired up yet + +The remaining work, for whoever picks it up: + +- `EarlyStopWatcher._evaluate_impl` wraps its verdict loop in a `READ_ONLY_MODE` window + over the reference. One window per round, around the loop rather than per criterion — + that is the tightest placement, which matters because a chmod is global filesystem state + and the agent is running CONCURRENTLY: the re-grant is visible to it too, for as long as + it is open. +- That loop is a `StreamCallback` (plain `def`), so it needs a synchronous twin of + `set_permissions` pushing onto the same registry — the stack is already thread-safe, so + the twin is small. +- `live_verdict` gains NO parameter. It reads the reference from a per-task accessor + instead. That accessor must be a `ContextVar`, NOT `os.environ`: `run_batch -j 8` runs + many orchestrators in one process, so a process-global would leak one task's reference + into a sibling's verdict, silently and only under parallelism. (`REFERENCE_DIR` today is + set only in the `env=` dict handed to `run_command` subprocesses, so it is not readable + in-process.) + +## Locking and crash safety + +The registry is keyed by the *resolved* path so a directory reached by two different +relative routes is one entry, and guarded by a plain `threading` lock rather than an +`asyncio` one because the crash-safety handlers run outside the event loop and must be +able to take it. It is an `RLock`, not a `Lock`: `restore_all()` runs from a signal +handler, which can be delivered on the main thread while atexit's `restore_all()` is +already mid-flight, and a non-reentrant lock deadlocks the interpreter at exit — exactly +when restoring matters most. + +`pop` chmods INSIDE the lock. Releasing first would let a concurrent `push()` observe the +path still at the restricted mode and record THAT as its `original` — so its own pop would +then leave the path at 000 permanently, the exact failure this module exists to prevent. +A failed restore keeps the entry rather than dropping it: `restore_all()` is the last +chance to put the path back, and it can only do that while it still holds the pre-window +mode. + +Crash handlers are installed from the event-loop (main) thread, before the chmods are +offloaded. `signal.signal` raises `ValueError` anywhere else, so installing from inside +the `to_thread` worker — as an earlier revision did — silently failed and left SIGTERM +with no restore at all. The flag latches only when the signal handlers really went in, so +a call from a worker thread is retried later rather than latching a no-op as done. + +Installation is deliberately not done at import time: `sandbox.py` imports this module, so +an import-time install would rewrite SIGINT/SIGTERM disposition for every process that +merely imports `coder_eval` — including library embedders and host runs, where no window +is ever opened. + +The signal handler chains to the previous disposition. An operator's Ctrl-C must not be +swallowed, and SIGTERM must still terminate. `SIG_IGN` is the one disposition that is +neither callable nor `SIG_DFL` — it means "the process chose to ignore this", so restoring +and returning is the correct chain. `None` means a handler installed from C and not +retrievable from Python; treating it as `SIG_DFL` restores default termination. + +The push sits INSIDE the `try`, so the `finally` always runs, and it is shielded: a +cancellation landing on the await (task-timeout watchdog, sibling batch failure) still +raises `CancelledError` while the worker thread goes on to complete every chmod. With the +push above the `try` — as an earlier revision had it — that left the paths at mode 000 +with no matching pop: unreadable for the rest of the run, and a stale registry entry that +poisoned the next window on the same path. + +## strict=True and the hard-fail path + +Under `strict`, a chmod refusal on an existing path raises `PermissionWindowError` instead +of warning. An unprotected run that reports a normal pass/fail is worse than no run: +nothing downstream can tell it apart from a protected one. `Sandbox.set_permissions` sets +it whenever the window is actually enforced (in-container). + +A missing path is the common, benign case (the task has no reference) and only debug-logs. +A genuine refusal — read-only mount, foreign owner — warns even when not strict, because +the operator should know the run is not protected. + +`_push_all` under `strict` leaves the paths pushed before the failure applied: the context +manager's `finally` cannot see a return value that never came. That is deliberate — +`restore_all` (atexit / signal) still holds their pre-window modes, and unwinding there +would swallow the failure that must abort the run. + +Pre-window modes are captured rather than hardcoded, so a repo that ships `0o750` task +dirs stays `0o750` on restore. diff --git a/.claude/notes/persistence.md b/.claude/notes/persistence.md index c8182b546..37aeea65d 100644 --- a/.claude/notes/persistence.md +++ b/.claude/notes/persistence.md @@ -2,3 +2,69 @@ > Conventions and authority order: see [README.md](README.md). +## write_text_atomic + +A plain `write_text` truncates first, so a SIGKILL or a full disk mid-write leaves a +half-file. For `task.json` that is worse than no file: a truncated record parses as +*malformed*, which the recovery paths treat as "not complete" — so a later `--resume` +re-executes the task and pays for the agent again, and the row vanishes from `run.json`. +One writer, so the orchestrator and the detached grade's write-back cannot have different +crash semantics for the same file. + +`O_NOFOLLOW` is not tidiness: without it, a pre-planted `task.json.tmp` *symlink* in a +shared run directory makes this an arbitrary-file-overwrite primitive — and one that +bypasses the destination symlink refusal in `evaluate`'s write-back, since that guard +checks the destination while the truncation happens through the temp name. + +### Why the temp name must be unique rather than fixed + +`os.replace` is the only step that can be interrupted without trace, and this function +exists precisely because the process may be SIGKILLed (the docker host-heartbeat watchdog +does exactly that) — so a crash between `open` and `replace` WILL sometimes leave the temp +file behind. Under a fixed name, `O_EXCL` then turned that leftover into a permanent +refusal to write the record at all: the row reported ERROR, `--resume` saw no `task.json`, +re-ran the task into the same run dir, and hit the same stale file — an unbounded loop +that re-pays for the agent every time. A unique name (pid + random) keeps `O_EXCL`'s +guarantee while making a leftover inert. It can litter a dead `.tmp` beside the record +after a hard kill; that is strictly better than wedging finalization, and the litter is +recognisable by its embedded pid. + +### Why the mode is 0o644 + +The same mode a plain `write_text` produced, and the widest one that is never group- or +world-*writable* whatever the umask. Creating it 0600 broke the docker driver on Linux: +the in-container orchestrator writes `task.json` as root straight into the bind-mounted +host run dir, and the host then reads it back as the invoking uid — an unguarded read that +raises `PermissionError` for every task. A result record is not a secret, and the symlink +hazard is closed by `O_NOFOLLOW` and the unpredictable name rather than by the mode. + +## rmtree_restrictive + +Plain `rmtree(..., ignore_errors=True)` silently declines on a tree left at mode 000 by a +killed run: `scandir` on a 000 directory raises `PermissionError`, the `rmdir`s then fail +with ENOTEMPTY, and every one of those is swallowed — leaving an orphaned tempdir holding +the reference solution, with no log line. + +An `onexc` handler cannot fix it either: the failing call is the directory +`open`/`scandir` that drives the walk, which the handler has no way to resume. So +traversal is restored on the way DOWN first, then the tree is deleted. + +## Run-directory filename constants + +The per-task filenames are module-level constants rather than literals because ~12 sites +name them — including three that `rglob` for the first — and two half-copies of the same +string in different packages is how a rename becomes a silent no-op on the sites it +missed. `REFERENCE_COPY_IGNORE` is shared for the same reason: the host-side docker mount +and the per-run staged copy are the SAME operation on two mutually exclusive driver paths, +so a literal at each site would make `$REFERENCE_DIR` contents driver-dependent the moment +one of them grew an entry. + +`prior.json` is never written by a run: it is only ever an input to +`coder-eval evaluate` / `run --resume` over a `driver: docker` row, staged into the +grading container's read-only input mount. + +A regrade writes to `grade.log`, not `task.log`, because the log handler opens its file +`mode="w"` — pointing a detached or resumed grade at `task.log` truncated the agent +trajectory log the run had already paid for. `grade.docker.log` exists for the same reason +one layer down: on the `run --resume` path `docker.log` is already the executed +container's log. diff --git a/.claude/notes/timing.md b/.claude/notes/timing.md index c4c486fdb..37992850c 100644 --- a/.claude/notes/timing.md +++ b/.claude/notes/timing.md @@ -2,3 +2,309 @@ > Conventions and authority order: see [README.md](README.md). +## TurnClock + +A turn's bounds and its durations have to share a basis or they can disagree, and the +disagreement lands in a field measured in milliseconds. Two concrete failures the class +removes: + +- Antigravity computed its window span on the MONOTONIC clock while unioning WALL-clock + tool intervals and subtracting one from the other. That is the only reason its window + could go negative at all, and the clamp that hid it was indistinguishable from a real + instant generation. +- Pi stamped with naive-LOCAL `datetime.now()`, and claude-code did the same. A DST + transition or an NTP step inside a turn lands directly in a generation window — an + hour-long jump in a millisecond field. Nightly runs start at 04:18 and run for hours, + so it is reachable rather than theoretical. A monotonic-derived stamp cannot express it. + +It is an EXTRACTION, not an invention: antigravity already captured this exact pair at +the top of `communicate` and simply did not use it for later stamps. + +Stamps stay naive local, matching what the rest of the telemetry and the persisted +`execution_started_at` already are, so no consumer changes. + +Within a turn the derived stamp is monotonic-accurate and may drift from real wall time; +each turn re-anchors. That is intended — do not "fix" it by re-reading the wall clock, +which is the property being removed. + +One per turn, never module-level and never reused: a long run would accumulate drift +between the pair and real wall time. The turn-state constructors take it as an argument +so the lifetime is visible in the signature, and so a unit test can pass a fake straight +in. An end-to-end test driving `communicate()` cannot — the state is built inside it, out +of the caller's reach — so those replace the class through the agent module instead +(`tests/_bracket_clock.py`). Both reach the same object. + +Which harnesses use it — for their window bounds and, since **CE064**, for their turn +bracket — is stated in `docs/agents/HARNESS_PARITY.md` (the `clock basis for recorded +stamps` and `turn bracket` rows), the designated SSOT for per-harness composition. +Asserting it anywhere else is the drift that put a wrong OpenCode row in that table for +months. + +## _require_same_awareness + +Subtracting a naive and an aware stamp raises `TypeError: can't subtract offset-naive and +offset-aware datetimes` deep inside the arithmetic, surfaces out of +`EventCollector.build_turn_record`, and kills the turn with a message naming neither the +field nor the harness. The guard turns that into a statement of which pair disagreed and +which side is aware. + +Unreachable from this repo today, and that is the point: every stamp in `agents/` and +`streaming/` is a naive `datetime.now()`, so this guards the SEAM rather than a live +defect. The exposure it is for is a third-party agent registered through the +`coder_eval.plugins` SPI, which lives outside `src/coder_eval/agents/` and which no lint +rule scoped to that directory could ever see. That is why it is a runtime guard and not +a rule. + +Only the MIX raises. An agent internally consistent in UTC is not this function's +problem, and neither is one that is consistently naive. + +## busy_ms + +The union, not the sum. Tool intervals overlap in practice — Antigravity resolves several +calls from one `Step` and backgrounds anything over ten seconds; Codex spawns collab +agents that run concurrently — so adding their durations over-counts busy time by exactly +the overlap. Subtracting such a sum from a generation window understates generation and, +with enough concurrency, drives it negative: four concurrent 400 ms calls inside a +1000 ms window sum to 1600 ms, clamping the result to the `0.0` that "unknown timing says +unknown" exists to eliminate. + +Clipping to `[lo, hi]` is the other half: a tool that opened before this window only spent +part of its life inside it, and only that part is not generation time here. + +The spans are awareness-checked as well as the bounds, not instead of them: the clipping +compares each span against BOTH `lo` and `hi`, so a guard on the bounds alone would leave +the function uncovered by it. + +An EMPTY span list is checked not at all, bounds included. The comprehension never runs, +nothing is compared and nothing is subtracted, so there is no pair for the guard to be +about — and raising there would reject a call that has always returned `0.0`. + +## union_ms + +`busy_ms` with the window set to the spans' own bounds. It exists because two callers had +copy-pasted that same `min`/`max`/`busy_ms` tail — `tests/_fixtures/golden_streams/_scrub.py` +(the golden sensor) and `scripts/timing/decompose_run.py` (the live residual gate) — and +they answer the same question about the same recorded commands, so a divergence would let +one pass while the other failed. Each keeps its own stamp parsing and span building, +because their input shapes genuinely differ; only this tail is shared. + +That it does not filter `end < start` holds only while every caller does. A new caller +that skips the check gets whatever `busy_ms` does with an inverted pair, which is to +discard it — silently, rather than by this function's stated contract. + +## close_window + +The shape all five reducers share, returning the RAW window; tool execution comes back +out centrally in `subtract_tool_time`. + +`mark` is keyword-only and has NO default so that no reducer can open a window without +stating what it tiles from — the defect Pi shipped with, measuring from its own turn start +so that every inter-turn gap fell into no bucket at all. Note what the signature does and +does not buy: it constrains the call SHAPE, not the VALUE. A reducer can still pass the +wrong mark; what it cannot do is fail to have one. + +`item_start` is this emission's own first stamp, when the harness has one. The `min()` +against `mark` is the tiling defense and nothing else: a stamp that went backwards must +never push the window start past the first item and invert the span. claude-code passes +none — its stream carries no per-emission item start — so its window opens exactly at the +mark. + +It deliberately does not return `completed`. The window always ends at `now`, which the +caller passed in, so handing it back would be an argument returned unchanged — +redundancy dressed as symmetry. + +## decompose_turn + +`tool_spans` is what keeps the four buckets disjoint, and omitting it is a double-count +rather than a lost refinement. A tool is not confined to a generation window: Antigravity +force-closes an orphan at finalization, which stamps its completion inside the tail, and +it backgrounds anything over ten seconds, which can straddle either end. Such a span is +subtracted out of the windows AND counted in the tool bucket, so leaving it in the head or +tail books it twice — measured on the committed `antigravity_d_orphaned_tool` fixture as a +residual of -86% of wall clock. + +The union and not the sum, because concurrent tool calls otherwise book their overlap +twice — measured: one live Pi turn overlapped a `Write` and a `Bash` by 18.4 ms. + +`EventCollector` is the sole caller, and deliberately so: this is the one place the two +values are computed, after which they are persisted on `TurnRecord` and every later +consumer READS them rather than recomputing. The golden-stream sensor asserts on the +dumped record, and `scripts/timing/decompose_run.py` reads the stored fields — neither can +call this, because `task.json` carries no `AgentStartEvent` stamp to recompute a head from. + +What the head CONTAINS differs per harness and is deliberately NOT split: a harness that +spawns its process per turn fuses boot, provider resolution, dispatch and TTFT — measured +on OpenCode, the process spawns in 3 ms and the first event lands at 3921 ms — while one +that spawns it once at startup has no boot inside the turn to fuse in. No stream carries a +marker between those parts. Naming these for the interval they MEASURE rather than for +what they contain is the whole point; `docs/agents/HARNESS_PARITY.md` holds the +per-harness composition. + +## main_thread_tool_spans + +The span set the generation subtraction, the head and the tail are all measured against, +so they cannot disagree about which calls exist. Shared with +`reports_stats.turn_time_buckets`, which answers the same question about a finished +`TurnRecord` — a second typed copy of this rule is how two report surfaces come to publish +two different tool totals for one run. (`scripts/timing/decompose_run.py` keeps its own, +over raw `task.json` dicts rather than models; that is the sanctioned third reader, and +`tests/test_timing_close_window.py::TestTheThreeToolUnionsAgree` pins all three together.) + +Sub-agent tools are excluded. `_overhead_ms` once filtered its GENERATIONS to the main +thread and then passed EVERY command, so its claim to keep all four buckets measuring one +thread was true only by luck: a child nests inside the parent Agent call, whose own +interval the union already covers — but Codex's recovered child tools carry the CHILD's +clock, so nothing made it true by construction. The evalboard's twin (`toolExecutionMs`) +does filter, so the two agreed by accident. + +A sub-agent's tool ids are reachable only through the messages that own them: a child +generation carries `parent_tool_use_id`, and its `tool_use_ids` are the calls it made. + +## _WINDOW_TOLERANCE_MS + +One millisecond, the coarsest unit a field named `_ms` can honestly be published in: a +producer that records microsecond-precision bounds and rounds its duration to whole +milliseconds is within its rights, and crashing its turns over 0.001 ms would relocate a +defect rather than remove one. The exposure the check is for is a third-party agent +registered through the `coder_eval.plugins` SPI, which is exactly the producer most likely +to round — so the tolerance has to admit it. + +It still catches everything it is for. The defect class is a reducer that NARROWED or +WIDENED a window without moving its bounds — subtracting its own tool time, most plausibly +— which is tens to thousands of milliseconds, three to six orders of magnitude above this. + +## subtract_tool_time + +Five reducers used to do the subtraction themselves — four through `close_window` as they +flushed, claude-code once at finalization — while the head and tail were already computed +centrally. That asymmetry was the complexity, and every timing defect this branch fixed +lived in the per-reducer bookkeeping around the subtraction rather than in the subtraction +itself: when to reset a span list, when to clear a start stamp, when to advance a mark. A +reducer now publishes the RAW window and keeps only the genuinely harness-shaped decision, +which is where its window opens. + +Non-mutating for aliasing reasons rather than repeated calls. Every agent builds its +terminal event as `AgentEndEvent(messages=list(...))` — that copies the LIST, not the +message objects — so writing in place would reach back into the agent's own live state +from the collector, which is exactly the layering "the collector is the sole capture seam" +exists to prevent. It is also unconditionally safe for a caller that builds a record +twice: `EarlyStopWatcher` holds one collector across a turn's tool-call rounds and calls +`build_turn_record` on every one. + +Grouping by identical bounds rather than `message_id`: Codex splits one window across two +sub-messages (thinking and action) that share `started_at` and `completed_at` and divide +the window by output-token share, so subtracting the group's overlap from each part +separately would subtract it twice and stop the parts summing to the window. Bounds +identity also covers OpenCode and Pi, which can legitimately carry `message_id is None` — +keying on the id would silently collapse every id-less message of a turn into one group. + +### Why the raw total must equal the bounds + +That equality is what lets `generation_duration_ms` stay a PUBLISHED field rather than one +the collector derives from the bounds. Deriving it instead was considered and cut — it +would cost five reducers, a regeneration of every golden and a rewrite of CE059, whose +exemption keys on the kwarg being present at the call site — and the assertion is the +sensor that makes deferring that safe. A mismatch means a reducer narrowed or widened a +window without moving its bounds, which is the drift +`tests/_fixtures/golden_streams/_scrub.py::assert_timing_captured`'s "bounds that span it" +check catches one replay at a time. + +It OVERLAPS with CE061 and is kept anyway. All five reducers build the window with +`close_window(mark=…, now=…)` and write `started_at=started, completed_at=now`, and CE061 +— now exemption-free — forces that shape statically, so the equality is largely true by +construction. What the runtime check adds is the half an import-level check cannot see: a +reducer that bypasses `close_window`, and a third-party agent registered through the +`coder_eval.plugins` SPI, which lives outside `src/coder_eval/agents/` where no lint rule +reaches it. It is not load-bearing on its own. + +Raising kills the turn, and that is accepted — the same trade `_require_same_awareness` +makes at this seam. The condition is unreachable without a reducer bug; all five are +exercised by the golden corpus and by the ms-exact identity contract. + +### Why the zero-total skip runs before the equality check + +`close_window` clamps an inverted window — `now` before `mark`, two clocks disagreeing — +to `0.0` while the bounds it writes still say `completed_at < started_at`, so the bounds +span is NEGATIVE and the equality fails. That is a measured inversion, the case +`decompose_turn` deliberately clamps because both ends were observed; raising on it would +kill turns on exactly the shape the clamp exists to tolerate. The cost is that a `0.0` +published beside a POSITIVE window slips through — a shape no in-tree reducer produces, +and one that reads downstream as "measured, and instant" rather than as a crashed turn. + +### Why the apportioned shares are not rounded + +The last member takes the remainder, so the parts reconstruct the group's net exactly +without rounding — while rounding each earlier share UP could push `assigned` past `net` +and hand the last member a NEGATIVE duration. That needs a net of well under a microsecond +(a window almost entirely covered by tool execution) and so had never been seen, but a +negative generation is an invariant break, not a rounding artifact. + +## _overhead_ms + +Measured against `AssistantMessage` entries only: a simulation turn interleaves +`UserMessage` entries, and a reconciled turn ends with a `ReconciliationMessage` that +carries no timestamps at all, so indexing the raw list would measure the wrong thing or +raise. + +A message whose `generation_duration_ms` is `None` is skipped. That field is the +codebase's own marker for "no window was measurable here", and every producer of one +stamps `started_at == completed_at == datetime.now()` at *append* time as an admitted +placeholder — Codex's rollout rebuild (`_messages_from_items`), both Codex sub-agent +recovery builders, and Claude's `_synthesize_subagent_terminal_message`. Reading those +stamps as window bounds turns a placeholder into a measurement: a Codex turn rebuilt from +its rollout stamps every message at turn END, which would book the entire turn as harness +startup. It is the same exemption CE059 makes for the same reason. + +`min` / `max` rather than the first and last list entries, because the list is not ordered +by time — Codex appends recovered sub-agent messages after the parent's last flush. +Positional access made the result depend on append order, which nothing enforces. + +Main thread only, the same rule its two sibling call sites already apply +(`codex_agent._token_usage_from_messages` and `scripts/timing/decompose_run.py`). A +sub-agent's generations carry the spawning Agent call's `parent_tool_use_id`, and the +identity these two values complete sums generation over the main thread ONLY — the parent +tool call's own interval already spans the sub-agent's whole run. Bracketing the span with +a sub-agent message therefore shrinks the head or the tail by time no other bucket claims, +and Codex's recovered child messages carry the CHILD's clock, so the bracket can move +either way. + +## Why the subtraction and the head/tail may run in either order + +`EventCollector.build_turn_record` calls `subtract_tool_time` before `_overhead_ms`, and +that order is NOT load-bearing: `_overhead_ms` reads only each message's bounds, its +main-thread flag, and whether its duration is `None` — none of which `subtract_tool_time` +changes. What IS load-bearing is that both are handed the SAME span set. + +## The TypeScript twin + +`evalboard/lib/timing.ts::busyMs` subtracts tool time from a task's WALL CLOCK to produce +the Unaccounted residual. It answers the same question about the same `task.json`, so the +two must agree — and neither owns the numbers: `tests/_fixtures/timing_union_cases.json` +does, and both suites replay it. + +The four-bucket identity has a second implementation there too: the evalboard's Unaccounted +cell (`_sections.tsx`) subtracts the same buckets from the same wall clock, as `pricing.ts` +mirrors `pricing.py`. It does not recompute a head or a tail (it reads the stored fields), +so a change in `timing.py` needs a TS change only when it alters what the buckets mean; +adding a fifth bucket means touching that cell and `sumHarnessOverhead`. + +## Why timing.py is a cycle-free leaf + +It sits outside `agents/` because `EventCollector` consumes it, and importing anything +under `agents/` pulls in every agent, which imports `streaming/`. The same reasoning as +`models/cli_match.py`. + +## Where a reducer's window opens + +NO harness subtracts tool execution from its own generation windows. Each publishes the +RAW window it measured, and `subtract_tool_time` takes the UNION of the tool intervals +back out of them once, for all five, at the single capture seam — the same place the head +and the tail are already computed. + +A reducer's only remaining timing decision is where its window opens, which is the one +genuinely harness-shaped part: two interleave a tool into a single window outright +(Antigravity, whose Step for the tool arrives and only a later `usage_metadata` Step cuts +the message, and Codex, whose `_flush_message` window extends to the last item's +`completed_at_ms`) while the other three tile the turn contiguously, so a call open at a +boundary runs inside two windows. Central subtraction handles both without either reducer +knowing which it is. diff --git a/src/coder_eval/fs_permissions.py b/src/coder_eval/fs_permissions.py index 499c8320a..8509129df 100644 --- a/src/coder_eval/fs_permissions.py +++ b/src/coder_eval/fs_permissions.py @@ -1,111 +1,30 @@ """Temporary filesystem-permission windows for anti-cheat. -The agent under evaluation runs with the same filesystem view as the harness: -in ``driver: tempdir`` it is an ordinary process on the host, and in -``driver: docker`` the orchestrator and the agent share one container. Any -directory the harness can read, the agent can read too -- including the task -directory and the reference solution. An agent that greps for the reference -does not solve the task, it copies the answer. - -:func:`set_permissions` closes that window: it chmods the target paths to a -mode (0o000 by default) for the duration of an ``async with`` block and falls -back on exit. The orchestrator wraps every ``agent.communicate`` call in it, so -the staged reference directory is unreadable exactly while the agent is -executing, and readable again by the time criteria and judges run. The task -directory is shielded by the same window: under docker it is mounted as a -throwaway COPY at a fixed container path, which is what makes it chmod-able -without touching the user's checked-out ``tasks/`` tree. See the call site in -``Orchestrator._communicate_with_retry``. - -This shields grading MATERIAL that happens to live in the task directory (a -``reference/`` subdirectory, fixtures), not the task DEFINITION: ``task.yaml`` -is separately staged at ``/work/input``, which the agent can still read. - -Windows **stack**, which is what makes a mid-turn re-grant expressible: code -that runs inside the turn but is not the agent can open a narrower window to -read a shielded path, and the enclosing 000 is restored when it closes:: +The agent shares the harness's filesystem view, so it can read anything the +harness can — the staged reference solution included. +:func:`set_permissions` chmods them (0o000 by default) for the body of an +``async with`` block and falls back on exit; the orchestrator wraps every +``agent.communicate`` call in it. + +Windows **stack**; an inner one may be MORE permissive:: async with set_permissions([reference], mode=RESTRICTED_MODE): ... # agent turn: 000 async with set_permissions([reference], mode=READ_ONLY_MODE): - ... # this code can read: 555 + ... # can read: 555 ... # back to 000 -The inner form exists for ONE intended consumer: **live success criteria**. -Early-stop verdicts are computed while the agent turn is still running -- i.e. -inside the 000 window -- so a live criterion that needs to consult the reference -solution has to be able to read it exactly then, while the agent still cannot. -A flat set/restore cannot express that, and a refcount actively breaks it (it -treats the inner re-grant as just another holder and leaves 000 in place). That -is why this is a stack, and why :data:`READ_ONLY_MODE` is public. It is a -designed seam, not speculative generality. - -NOT WIRED UP YET. The remaining work, for whoever picks it up: - -* ``EarlyStopWatcher._evaluate_impl`` wraps its verdict loop in a - ``READ_ONLY_MODE`` window over the reference. One window per round, around - the loop rather than per criterion -- that is the tightest placement, which - matters because a chmod is global filesystem state and the agent is running - CONCURRENTLY: the re-grant is visible to it too, for as long as it is open. -* That loop is a ``StreamCallback`` (plain ``def``), so it needs a synchronous - twin of :func:`set_permissions` pushing onto this same ``_registry`` -- the - stack is already thread-safe, so the twin is small. -* ``live_verdict`` gains NO parameter. It reads the reference from a per-task - accessor instead. That accessor must be a ``ContextVar``, NOT ``os.environ``: - ``run_batch -j 8`` runs many orchestrators in one process, so a process-global - would leak one task's reference into a sibling's verdict, silently and only - under parallelism. (``REFERENCE_DIR`` today is set only in the ``env=`` dict - handed to ``run_command`` subprocesses, so it is not readable in-process.) - -Scope note: only the REFERENCE is shielded, never the sandbox. A live criterion -reading the agent's own output files therefore needs no window change at all -- -and should not get one. Reading the static reference mid-turn cannot break the -``LiveVerdict`` monotonicity contract; reading the half-written sandbox can, and -is the "end-state peeking" ``live_verdict``'s own docstring rules out. - -Two further properties: - -* **Pre-window mode capture, not a hardcoded restore.** The outermost exit - restores the mode actually observed, so a repo that ships ``0o750`` task dirs - stays ``0o750``. -* **Crash-safe.** Unwinds are also registered with :mod:`atexit` and on - ``SIGINT``/``SIGTERM``, so a killed run does not leave a checked-out - ``tasks/`` tree at mode 000. +The inner form is for live success criteria; not wired up yet. + +Pre-window modes are captured, not hardcoded; unwinds also run from +:mod:`atexit` and on ``SIGINT``/``SIGTERM``. .. warning:: - **This is defense-in-depth, not a boundary.** ``chmod`` is a DAC control. - Two separate facts limit it against an agent running as root in the same - container: - - * *Reading* a mode-000 path is bypassed via ``CAP_DAC_OVERRIDE`` / - ``CAP_DAC_READ_SEARCH``; the docker driver drops both, which does close - that half (verified: a ``chmod 000`` dir is readable by container root by - default, and denied once the caps are gone). - * *Restoring* the mode is gated on **owner-or-CAP_FOWNER**, and container - root owns the bind-mounted copy, so ``chmod 755 /work/references`` puts it - back and the agent reads the solution. ``FOWNER``/``CHOWN`` are NOT dropped - to stop that, because the in-container orchestrator that applies the window - is the same root process with the same caps: dropping them breaks the - harness's own chmod wherever the mount preserves a non-root owner (native - Linux), i.e. on exactly the hosts where the drop would otherwise bite. - Closing this half needs a different uid, not a smaller capability set. - - A third limit is about *time*, not permissions: the window spans - ``agent.communicate``, so between turns (and after the last one, while - criteria run) the path is back at its pre-window mode. Nothing reaps the - agent's child processes at turn end, so a backgrounded ``while ! cat ...`` - loop started during a turn succeeds the moment the window closes. Reading is - only half of that — the docker mount is read-WRITE by necessity, so the same - loop could *overwrite* the reference and drive ``reference_comparison`` to - 1.0. The overwrite half IS closed, by a content hash taken at staging time - and re-verified before grading - (``Orchestrator._verify_reference_integrity``); the read half is not. - - So this stops passive reads and accidental leakage; it does not stop an - agent that deliberately re-opens the path or waits the window out. Full - containment needs the agent to run as a non-root uid that does not own the - reference, and the reference to be unreadable for the agent's whole lifetime - rather than per-turn (follow-up). + **Defense-in-depth, not a boundary.** ``chmod`` is a DAC control; container + root can re-open the path or wait the window out. + Rationale: .claude/notes/permissions.md § Reference solutions and the anti-cheat window + +Rationale: .claude/notes/permissions.md § The stacked chmod window """ from __future__ import annotations @@ -152,32 +71,20 @@ class PermissionWindowError(RuntimeError): class _PermissionStack: """Process-wide stack of applied modes, per resolved path. - A plain stack, not a refcount: the whole point is that windows nest with - *different* modes, so what an exit has to restore is the mode of the - enclosing window -- not "the original", and not "is anyone still holding - it". A refcount cannot express that; it would see a nested re-grant as just - another holder and silently leave the outer mode in place. + A plain stack, not a refcount: windows nest with *different* modes, so what + an exit restores is the enclosing window's mode. - The stack also subsumes what a refcount did, for free: two windows applying - the same mode push two identical entries, and the inner pop re-applies the - outer's (identical) mode instead of restoring the pre-window one. + Keyed by the *resolved* path, so a directory reached by two routes is one + entry. Guarded by a ``threading`` lock, not an ``asyncio`` one, because the + crash-safety handlers run outside the event loop and must be able to take it. - Keyed by the *resolved* path so a directory reached by two different - relative routes is one entry. Guarded by a plain ``threading.Lock`` rather - than an ``asyncio.Lock`` because the crash-safety handlers (:mod:`atexit`, - signal handlers) run outside the event loop and must be able to take it. + Rationale: .claude/notes/permissions.md § Locking and crash safety """ def __init__(self) -> None: - # Whether the crash handlers are installed. An instance attribute rather - # than a module-level global: the state belongs to the registry whose - # entries the handlers restore, and a mutable module global read only by - # its own writer reads as dead to static analysis. self._handlers_installed = False - # RLock, not Lock: restore_all() runs from a signal handler, which can be - # delivered on the main thread while atexit's restore_all() is already - # mid-flight. A non-reentrant lock deadlocks the interpreter at exit — - # exactly when restoring matters most. + # RLock, not Lock: a signal handler's restore_all() can land while + # atexit's is mid-flight, and a non-reentrant lock deadlocks at exit. self._lock = threading.RLock() # resolved path -> (mode before the outermost window, applied-mode stack) self._entries: dict[Path, tuple[int, list[int]]] = {} @@ -202,10 +109,8 @@ def push(self, path: Path, mode: int, *, strict: bool = False) -> bool: original = path.stat().st_mode & 0o7777 os.chmod(path, mode) except OSError as e: - # A missing path is the common, benign case (task has no - # reference). A genuine chmod refusal (read-only mount, foreign - # owner) is worth a warning: the window is not in place and the - # operator should know this run is not protected. + # A missing path is the common, benign case. A genuine refusal + # means this run is not protected, so the operator hears about it. if isinstance(e, FileNotFoundError): logger.debug("set_permissions: %s does not exist; nothing to do", path) return False @@ -214,9 +119,8 @@ def push(self, path: Path, mode: int, *, strict: bool = False) -> bool: + "the agent would be able to read it during this turn" ) if strict: - # Fail closed. An unprotected run that reports a normal - # pass/fail is worse than no run: nothing downstream can tell - # it apart from a protected one. + # Fail closed: an unprotected run that reports a normal + # pass/fail is indistinguishable from a protected one. raise PermissionWindowError(message) from e logger.warning("%s", message) return False @@ -239,10 +143,9 @@ def pop(self, path: Path) -> None: original, applied = entry applied.pop() target = applied[-1] if applied else original - # chmod INSIDE the lock. Releasing first would let a concurrent push() - # observe the path still at the restricted mode and record THAT as its - # `original` — so its own pop would then leave the path at 000 - # permanently, the exact failure this module exists to prevent. + # chmod INSIDE the lock: releasing first lets a concurrent push() + # record the restricted mode as its `original` and strand the path + # at 000 permanently. try: os.chmod(path, target) logger.debug("set_permissions: %s <- %#o", path, target) @@ -253,10 +156,8 @@ def pop(self, path: Path) -> None: target, e, ) - # Keep the entry: restore_all() (atexit / signal) is the last - # chance to put this path back, and it can only do that while it - # still holds the pre-window mode. Dropping the entry here would - # strip the crash path of the only record of `original`. + # Keep the entry: restore_all() is the last chance to put this + # path back, and it needs the pre-window mode this entry holds. return if not applied: del self._entries[path] @@ -265,20 +166,15 @@ def ensure_crash_handlers(self) -> None: """Install atexit + signal restores once, before the first window opens. MUST be called from the main thread: ``signal.signal`` raises - ``ValueError`` anywhere else. :func:`set_permissions` calls it on the - event-loop thread before offloading the chmods, which is what makes the - signal half actually take effect — installing from inside the - ``to_thread`` worker (as an earlier revision did) silently failed and - left SIGTERM with no restore at all. - - Deliberately NOT done at import time: ``sandbox.py`` imports this module, - so an import-time install would rewrite SIGINT/SIGTERM disposition for - every process that merely imports coder_eval — including library - embedders and host runs, where no window is ever opened and this registry - stays empty. The whole install runs under the lock so a concurrent caller - cannot observe a half-installed state, and the flag latches only when the - signal handlers really went in, so a call from a worker thread is retried - from the main thread later rather than latching a no-op as done. + ``ValueError`` anywhere else. Deliberately NOT done at import time — + ``sandbox.py`` imports this module, and an import-time install would + rewrite SIGINT/SIGTERM disposition for every process that merely imports + ``coder_eval``. + + The flag latches only when the signal handlers really went in, so a call + from a worker thread is retried from the main thread later. + + Rationale: .claude/notes/permissions.md § Locking and crash safety """ with self._lock: if self._handlers_installed: @@ -308,9 +204,9 @@ def _make_signal_handler( """Build a handler that restores ``registry``, then chains to ``previous``. Chaining matters twice over: an operator's Ctrl-C must not be swallowed, and - SIGTERM must still terminate. ``SIG_IGN`` is the one disposition that is - neither callable nor ``SIG_DFL`` — it means "the process chose to ignore - this", so restoring and returning is the correct chain. + SIGTERM must still terminate. + + Rationale: .claude/notes/permissions.md § Locking and crash safety """ def _handler(sig: int, frame: FrameType | None) -> None: @@ -320,9 +216,8 @@ def _handler(sig: int, frame: FrameType | None) -> None: elif previous == signal.SIG_IGN: return else: - # SIG_DFL, or None == handler installed from C and not retrievable - # from Python. Treating both as SIG_DFL restores default - # termination; swallowing it would make SIGTERM stop killing us. + # SIG_DFL, or None == a handler installed from C. Treating both as + # SIG_DFL keeps SIGTERM terminating. signal.signal(sig, signal.SIG_DFL) os.kill(os.getpid(), sig) @@ -344,10 +239,8 @@ def _install_crash_handlers(registry: _PermissionStack) -> bool: previous = signal.getsignal(signum) signal.signal(signum, _make_signal_handler(registry, previous)) except (ValueError, OSError) as e: - # Not on the main thread, or the platform lacks the signal. atexit - # still covers the ordinary-exit case, but SIGTERM does NOT run - # atexit — so a killed run can strand the tree at mode 000. That is - # worth more than a debug line. + # atexit covers ordinary exit, but SIGTERM does NOT run atexit — + # a killed run can strand the tree at mode 000. Warn, don't debug. installed_all = False logger.warning( "set_permissions: could not install a restore handler for signal %s (%s); " @@ -368,34 +261,25 @@ async def set_permissions( """Chmod ``paths`` to ``mode`` for the body, then fall back on exit. Windows NEST, and an inner window may be *more* permissive than the one - around it -- that is the point. Exiting restores the enclosing window's - mode, and only the outermost exit restores the pre-window mode:: - - async with set_permissions([reference], mode=RESTRICTED_MODE): - ... # agent turn: 000 - async with set_permissions([reference], mode=READ_ONLY_MODE): - ... # something mid-turn reads: 555 - ... # back to 000, not to 755 + around it. Exiting restores the enclosing window's mode; only the outermost + exit restores the pre-window mode. ``None`` entries and duplicates are dropped, so callers can pass optional - paths (``[task_dir, reference_dir]``) without pre-filtering. Paths are - resolved before use so the stack keys are canonical. - - The unwind runs in a ``finally``, so it happens on the exception path too - -- an agent crash or turn timeout must not leave the tree unreadable. + paths without pre-filtering. Paths are resolved before use. The unwind runs + in a ``finally``, so an agent crash or turn timeout cannot leave the tree + unreadable. Args: paths: Directories (or files) to chmod. ``None`` entries are skipped. mode: Permission bits to apply. Defaults to :data:`RESTRICTED_MODE`. strict: Raise :class:`PermissionWindowError` when an existing path cannot be chmod'd, instead of warning and continuing unprotected. - Set by ``Sandbox.set_permissions`` whenever the window is actually - enforced (in-container), so a broken anti-cheat control fails the - run rather than producing a normal-looking score. Raises: PermissionWindowError: under ``strict``, when a path exists but the chmod was refused. + + Rationale: .claude/notes/permissions.md § strict=True and the hard-fail path """ resolved: list[Path] = [] seen: set[Path] = set() @@ -405,8 +289,7 @@ async def set_permissions( try: candidate = Path(raw).resolve() except OSError as e: - # Same fail-open outcome as a chmod refusal, so it gets the same - # visibility — this path is NOT shielded during the turn. + # Same fail-open outcome as a chmod refusal, same visibility. logger.warning("set_permissions: could not resolve %s (%s); it will not be shielded", raw, e) continue if candidate in seen: @@ -414,39 +297,33 @@ async def set_permissions( seen.add(candidate) resolved.append(candidate) - # Install the crash restores HERE, on the event-loop (main) thread, and not - # inside _push_all: signal.signal() raises ValueError off the main thread, so - # installing from the to_thread worker below silently installed nothing. + # HERE, on the event-loop (main) thread, not inside _push_all: + # signal.signal() raises ValueError off the main thread. if resolved: _registry.ensure_crash_handlers() - # chmod is a syscall per path; offload so a slow network filesystem doesn't - # stall the event loop that is about to drive the agent's streaming turn. + # chmod is a syscall per path; offload so a slow filesystem doesn't stall + # the event loop about to drive the agent's streaming turn. held: list[Path] = [] push_task: asyncio.Future[list[Path]] | None = None try: if resolved: - # The push sits INSIDE the try, so the finally below ALWAYS runs. - # asyncio.shield protects the inner task, not this await: a - # cancellation landing here (task_timeout watchdog, sibling batch - # failure) still raises CancelledError out of the await while the - # worker thread goes on to complete every chmod. With the push above - # the try -- as an earlier revision had it -- that left the paths at - # mode 000 with no matching pop: unreadable for the rest of the run, - # and a stale registry entry that poisoned the next window on the - # same path. + # INSIDE the try, so the finally ALWAYS runs. shield protects the + # inner task, not this await: a cancellation still raises here while + # the worker completes every chmod — which is why the pop below + # joins the task rather than assuming nothing landed. + # Rationale: .claude/notes/permissions.md § Locking and crash safety push_task = asyncio.ensure_future(asyncio.to_thread(_push_all, resolved, mode, strict)) held = await asyncio.shield(push_task) yield finally: if push_task is not None and not held: - # We were cancelled mid-push. The shielded worker is still running - # and still chmod'ing; join it so `held` names exactly what landed - # and the unwind below cannot race it. + # Cancelled mid-push: join the shielded worker so `held` names + # exactly what landed and the unwind cannot race it. held = await asyncio.shield(push_task) if held: - # Shielded: the unwind MUST run even when the surrounding task is - # being cancelled (task_timeout watchdog), or the tree stays at 000. + # Shielded: the unwind MUST run under cancellation too, or the + # tree stays at 000. await asyncio.shield(asyncio.to_thread(_pop_all, held)) @@ -454,10 +331,11 @@ def _push_all(paths: list[Path], mode: int, strict: bool) -> list[Path]: """Push every path, returning only those that must later be popped. Under ``strict`` a refused chmod raises, and the paths pushed before it are - left applied: the context manager's ``finally`` cannot see a return value - that never came. That is deliberate -- ``_registry.restore_all`` (atexit / - signal) still holds their pre-window modes, and the alternative (unwinding - here) would swallow the failure that must abort the run. + left applied — the context manager's ``finally`` cannot see a return value + that never came. ``_registry.restore_all`` still holds their pre-window + modes. + + Rationale: .claude/notes/permissions.md § strict=True and the hard-fail path """ return [path for path in paths if _registry.push(path, mode, strict=strict)] diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index 46c97f54e..c8536cfdf 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -16,94 +16,58 @@ TASK_LOG_FILENAME = "task.log" -# Where a RE-GRADE's log goes. `task_log_handler` opens its file `mode="w"`, so -# pointing a detached/resumed grade at task.log truncated the agent trajectory -# log the run had already paid for — thousands of lines replaced by the grading -# pass's handful. That directly contradicts `_apply_resume`'s own contract -# ("to_grade is deliberately NOT cleared: its artifacts are the run's output and -# the very thing being graded"), and task.log is a documented run artifact. +# A RE-GRADE's log goes here, NOT to task.log: the log handler opens its file +# `mode="w"`, so a detached grade pointed at task.log truncates the agent +# trajectory the run already paid for. +# Rationale: .claude/notes/persistence.md § Run-directory filename constants GRADE_LOG_FILENAME = "grade.log" # The per-task result record, and the pre-grade snapshot a detached grade keeps -# beside it. Module-level because ~12 sites name them — including three that -# `rglob` for the first — and two half-copies of the same string in different -# packages is how a rename becomes a silent no-op on the sites it missed. +# beside it. TASK_JSON_FILENAME = "task.json" PRE_GRADE_JSON_FILENAME = "task.execute.json" # The already-executed row a DETACHED GRADE seeds from, staged into the grading -# container's read-only input mount. Never written by a run; only ever an input -# to `coder-eval evaluate` / `run --resume` over a `driver: docker` row. +# container's read-only input mount. Never written by a run. PRIOR_RESULT_FILENAME = "prior.json" # The container's own stdout+stderr transcript, and the name it is folded back -# under after a GRADING container. Constants rather than literals for the reason -# CE053 states: the producer lives in ``isolation/`` and the consumer in -# ``orchestration/``, the fold-back is guarded by ``is_file()``, so a rename on -# the producing side would degrade the copy to a silent no-op and discard the -# only record of why a grading container failed. +# under after a GRADING container. Constants, not literals (CE053): the +# fold-back is guarded by ``is_file()``, so a rename on the producing side +# would degrade the copy to a silent no-op. DOCKER_LOG_FILENAME = "docker.log" -# Named for the PHASE. On the ``run --resume`` path ``docker.log`` is already -# taken by the executed container's log, and overwriting it would repeat the -# task.log/grade.log truncation bug one layer down. GRADE_DOCKER_LOG_FILENAME = "grade.docker.log" # The virtualenv directory `setup` creates and `adopt` discovers. Named because # whether it is on PATH decides which binaries a criterion resolves. VENV_DIRNAME = ".venv" -# Ignore list for every copy of a reference solution tree. A module-level -# constant, not an inline literal at each call site: the host-side docker mount -# (`DockerRunner._prepare_reference_mount`) and the per-run staged copy -# (`orchestration.evaluation.stage_reference_dir`) are the SAME operation on two -# mutually exclusive driver paths, so a literal at each site would make -# ``$REFERENCE_DIR`` contents driver-dependent the moment one of them grew an -# entry. +# Ignore list for every copy of a reference solution tree. Shared, because the +# host-side docker mount and the per-run staged copy are the SAME operation on +# two mutually exclusive driver paths. +# Rationale: .claude/notes/persistence.md § Run-directory filename constants REFERENCE_COPY_IGNORE = [".git"] def write_text_atomic(path: Path, text: str) -> None: """Write ``text`` to ``path`` via a temp file + ``os.replace``. - A plain ``write_text`` truncates first, so a SIGKILL or a full disk mid-write - leaves a half-file. For ``task.json`` that is worse than no file: a truncated - record parses as *malformed*, which the recovery paths treat as "not - complete" — so a later ``--resume`` re-executes the task and pays for the - agent again, and the row vanishes from ``run.json``. One writer, so the - orchestrator and the detached grade's write-back cannot have different crash - semantics for the same file. + A plain ``write_text`` truncates first, so a crash mid-write leaves a + half-file — and a truncated ``task.json`` parses as *malformed*, which the + recovery paths read as "not complete", so ``--resume`` pays for the agent + again. One writer, so every producer of that file has the same crash + semantics. The temp file is opened ``O_CREAT | O_EXCL | O_NOFOLLOW`` under a name that - is UNIQUE per call. Without ``O_NOFOLLOW`` a pre-planted ``task.json.tmp`` - *symlink* in a shared run directory makes this an arbitrary-file-overwrite - primitive — and one that bypasses the destination symlink refusal in - ``evaluate``'s write-back, since the guard checks the destination while the - truncation happens through the temp name. - - The name must be unique, not fixed, and that is a correctness requirement - rather than tidiness. ``os.replace`` is the only step that can be interrupted - without trace, and this function exists precisely because the process may be - SIGKILLed (the docker host-heartbeat watchdog does exactly that) — so a - crash between ``open`` and ``replace`` WILL sometimes leave the temp file - behind. Under a fixed name, ``O_EXCL`` then turned that leftover into a - permanent refusal to write the record at all: the row reported ERROR, and - ``--resume`` saw no ``task.json``, re-ran the task into the same run dir, and - hit the same stale file — an unbounded loop that re-pays for the agent every - time. A unique name keeps ``O_EXCL``'s guarantee while making a leftover - inert. It can litter a dead ``.tmp`` beside the record after a hard kill; - that is strictly better than wedging finalization, and the litter is - recognisable by its embedded pid. - - Mode is ``0o644`` — the same mode a plain ``write_text`` produced, and the - widest one that is never group- or world-*writable* whatever the umask. - Creating it 0600 broke the docker driver on Linux: - the in-container orchestrator writes ``task.json`` as root straight into the - bind-mounted host run dir, and the host then reads it back as the invoking - uid — an unguarded read that raises ``PermissionError`` for every task. A - result record is not a secret, and the symlink hazard is closed by - ``O_NOFOLLOW`` and the unpredictable name rather than by the mode. + is UNIQUE per call. ``O_NOFOLLOW`` closes a symlink-plant overwrite + primitive; the unique name keeps ``O_EXCL``'s guarantee while making a + leftover from a SIGKILLed predecessor inert instead of a permanent refusal + to write the record. Mode is ``0o644`` — do not narrow it; the docker driver + reads this file back as a different uid. + + Rationale: .claude/notes/persistence.md § write_text_atomic """ # pid + random: unique across concurrent writers AND across a crashed - # predecessor, so O_EXCL can never collide with our own leftovers. + # predecessor, so O_EXCL cannot collide with our own leftovers. tmp = path.with_name(f"{path.name}.{os.getpid()}.{secrets.token_hex(4)}.tmp") flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0) fd = os.open(tmp, flags, 0o644) @@ -140,14 +104,12 @@ def digest_tree(root: Path) -> str: def rmtree_restrictive(root: Path) -> None: """``rmtree`` a tree that may have been left at mode 000 by a killed run. - Plain ``rmtree(..., ignore_errors=True)`` silently declines here: ``scandir`` - on a 000 directory raises ``PermissionError``, the ``rmdir``s then fail with - ENOTEMPTY, and every one of those is swallowed — leaving an orphaned tempdir - holding the reference solution, with no log line. + Plain ``rmtree(..., ignore_errors=True)`` silently declines on such a tree, + and an ``onexc`` handler cannot fix it either — the failing call is the + ``scandir`` that drives the walk. So traversal is restored on the way DOWN + first, then the tree is deleted. - An ``onexc`` handler cannot fix it either: the failing call is the directory - ``open``/``scandir`` that drives the walk, which the handler has no way to - resume. So restore traversal on the way DOWN first, then delete. + Rationale: .claude/notes/persistence.md § rmtree_restrictive """ for dirpath, dirnames, _filenames in os.walk(root, topdown=True, onerror=lambda _e: None): for name in (dirpath, *(os.path.join(dirpath, d) for d in dirnames)): @@ -226,11 +188,8 @@ def format_task_log_id(variant_id: str, task_id: str, replicate_index: int = 0) - Batch ``stream_label`` - CLI tqdm progress-bar postfix - Shape mirrors ``build_task_run_dir`` (same three segments, same NN padding - via ``replicate_subdir_name``) so log tags and on-disk paths stay in - lockstep. Callers MUST use this helper rather than hand-rolling the - f-string so future format changes (e.g., NN → NNN) touch exactly one - place. + Shape mirrors ``build_task_run_dir``, so log tags and on-disk paths stay in + lockstep. Callers MUST use this helper rather than hand-rolling the f-string. """ return f"{variant_id}/{task_id}/{replicate_subdir_name(replicate_index)}" @@ -245,18 +204,13 @@ def create_latest_symlink(runs_base: Path, run_id: str) -> None: run_id: ID of the current run (e.g., "2025-10-09_15-30-45") """ latest_link = runs_base / "latest" - # Use relative path for symlink target (just the run_id directory name) - # This ensures the symlink works correctly when both are in the same directory + # Relative target, so the symlink resolves from the runs base itself. target = Path(run_id) try: - # Remove existing symlink/file if latest_link.exists() or latest_link.is_symlink(): latest_link.unlink() - - # Create symlink with relative path latest_link.symlink_to(target, target_is_directory=True) except (OSError, NotImplementedError): - # Windows may not support symlinks, skip gracefully if platform.system() != "Windows": raise diff --git a/src/coder_eval/streaming/collector.py b/src/coder_eval/streaming/collector.py index c4949d1d8..3e516d293 100644 --- a/src/coder_eval/streaming/collector.py +++ b/src/coder_eval/streaming/collector.py @@ -1,23 +1,16 @@ """EventCollector: reduce a standardized event stream into a TurnRecord. -This is the single, agent-agnostic place where the persisted ``TurnRecord`` -(and therefore ``task.json``) is assembled from the event stream — so adding a -new agent means emitting the standard events, with capture coming for free -(no per-agent telemetry-assembly code). +The single, agent-agnostic place where the persisted ``TurnRecord`` (and +therefore ``task.json``) is assembled — so a new agent emits the standard events +and gets capture for free. -Reduction split: +An agent attaches its own collector alongside the caller's ``stream_callback`` +and returns ``build_turn_record()`` from ``communicate()``. -- ``commands`` are derived from the ``ToolEndEvent`` stream (every tool call, - including crash-orphaned ones force-closed as ``unresolved``), ordered by the - tool's ``sequence_number``. This is the genuine "events are the source of - truth" path for tool telemetry. -- The per-message telemetry / token payload (the intricate, SDK-specific token - machinery the plan defers) rides on the terminal ``AgentEndEvent`` and is read - back verbatim — no re-derivation, so token correctness is untouched. - -An agent attaches its own ``EventCollector`` alongside the caller's callback and -returns ``build_turn_record()`` from ``communicate()``; the orchestrator keeps -reading the return value (and ``pending_turn`` on crash), now event-derived. +``commands`` are derived from the ``ToolEndEvent`` stream (crash-orphaned calls +included, force-closed as ``unresolved``), ordered by ``sequence_number``. The +per-message telemetry / token payload rides on the terminal ``AgentEndEvent`` and +is read back verbatim, never re-derived. """ from __future__ import annotations @@ -64,12 +57,8 @@ def __init__(self) -> None: self._agent_end: AgentEndEvent | None = None def on_event(self, event: StreamEvent) -> None: - # Only the main agent's own events shape its TurnRecord. This guard is - # forward-looking: nested sub-agent events (parent_thread_id set) are NOT - # emitted by any agent yet (sub-agent nesting is deferred), so this branch - # is currently never taken. It's here so that when nesting lands, child - # events are skipped here and attributed via the finalization payload - # rather than corrupting the main-agent record. + # Only the main agent's own events shape its TurnRecord. Forward-looking: + # no agent emits nested sub-agent events yet, so this never fires today. if event.parent_thread_id is not None: return @@ -77,11 +66,9 @@ def on_event(self, event: StreamEvent) -> None: self._iteration = event.iteration self._user_input = event.prompt self._agent_start_at = event.timestamp - # A new turn has begun, so the previous turn's terminal event is no - # longer this turn's. Every agent builds a fresh collector per - # communicate(), but EarlyStopWatcher keeps ONE across retries: left - # stale, it would pair this attempt's start with the last attempt's - # end and publish the clamped inversion as a measured 0.0. + # EarlyStopWatcher keeps ONE collector across retries: left stale, + # this would pair the new start with the last attempt's end and + # publish the clamped inversion as a measured 0.0. self._agent_end = None if event.model: self._model = event.model @@ -103,12 +90,10 @@ def visible_turn_count(self) -> int: ``TurnRecord`` (minus its trailing final-reply entry, which cannot exist while the turn is still running). - Agents whose SDK has no meaningful native turn counter — Codex and - Antigravity each deliver a single SDK turn per ``communicate()`` — enforce - ``run_limits.max_turns`` against this. Reading it from the collector rather - than from each agent's own scratch list is what makes the cap mean the same - thing on both: the collector is the single agent-agnostic capture path, and - keying on ``tool_id`` means a re-emitted end event cannot double-count. + Agents whose SDK has no meaningful native turn counter (Codex, + Antigravity) enforce ``run_limits.max_turns`` against this, so the cap + means the same thing on both. Keying on ``tool_id`` means a re-emitted + end event cannot double-count. """ return len(self._commands) @@ -120,47 +105,20 @@ def _overhead_ms( ) -> tuple[float | None, float | None]: """The turn's head and tail — the wall clock the generations do not cover. - Measured against ``AssistantMessage`` entries only: a simulation turn - interleaves ``UserMessage`` entries, and a reconciled turn ends with a - ``ReconciliationMessage`` that carries no timestamps at all, so indexing - the raw list would measure the wrong thing or raise. - - Two further restrictions, both of which are the difference between a - measurement and an invention: + Measured against ``AssistantMessage`` entries only, skipping any whose + ``generation_duration_ms`` is ``None`` — that field is the codebase's + marker for "no window was measurable here", and its producers stamp a + placeholder ``started_at == completed_at`` that would otherwise read as a + measurement (the same exemption CE059 makes). - A message whose ``generation_duration_ms`` is ``None`` is SKIPPED. That - field is the codebase's own marker for "no window was measurable here", - and every producer of one stamps ``started_at == completed_at == - datetime.now()`` at *append* time as an admitted placeholder — Codex's - rollout rebuild (``_messages_from_items``), both Codex sub-agent - recovery builders, and Claude's ``_synthesize_subagent_terminal_message``. - Reading those stamps as window bounds turns a placeholder into a - measurement: a Codex turn rebuilt from its rollout stamps every message - at turn END, which would book the entire turn as harness startup. It is - the same exemption CE059 makes for exactly the same reason. + ``min``/``max``, not the first and last entries: the list is not ordered + by time. MAIN THREAD ONLY, so all four buckets measure one thread. - ``min`` / ``max`` rather than the first and last list entries, because - the list is not ordered by time — Codex appends recovered sub-agent - messages after the parent's last flush. Positional access made the - result depend on append order, which nothing enforces. + ``tool_spans`` is REQUIRED, never defaulted: its one caller computes the + set once and hands the same object to both consumers, and a fallback here + would build a SECOND set. - ``tool_spans`` is REQUIRED, never defaulted. Its one caller computes the - set once and hands the same object to both consumers; a fallback branch - here would build a SECOND set, which is precisely what the comment at - that call site says must never happen — the subtraction and the - head/tail have to agree about which calls exist or the buckets stop - being disjoint. - - MAIN THREAD ONLY, the third restriction and the same rule its two - sibling call sites already apply (``codex_agent._token_usage_from_messages`` - and ``scripts/timing/decompose_run.py``). A sub-agent's generations - carry the spawning Agent call's ``parent_tool_use_id``, and the identity - these two values complete sums generation over the main thread ONLY — - the parent tool call's own interval already spans the sub-agent's whole - run. Bracketing the span with a sub-agent message therefore shrinks the - head or the tail by time no other bucket claims, and Codex's recovered - child messages carry the CHILD's clock, so the bracket can move either - way. Excluding them keeps all four buckets measuring one thread. + Rationale: .claude/notes/timing.md § _overhead_ms """ generations = [ m @@ -182,16 +140,13 @@ def _reconciled_messages(messages: list[TranscriptMessage], usage: TokenUsage) - """Append a ``ReconciliationMessage`` so the transcript's token buckets sum to ``usage`` (the authoritative turn total). - The per-``AssistantMessage`` stream consistently under-reports the bill — - a fixed prompt slice (~512 input tokens on Claude) is billed on no - SDK-emitted message, and sub-agent input/cache only partially bubbles up. - We book that residual once, explicitly, as a synthetic entry rather than - smearing fabricated tokens across real generations. After this, any - consumer that sums the four token buckets across the transcript reproduces - ``usage`` exactly — no separate aggregate needed. Only assistant - generations carry agent-billed tokens, so the residual is measured against - them (simulator ``UserMessage`` tokens are a separate bill). Emitted only + The per-message stream under-reports the bill, so the residual is booked + once, explicitly, rather than smeared across real generations. After + this, summing the four buckets across the transcript reproduces ``usage`` + exactly. Measured against assistant generations only, and emitted only when some bucket actually diverges. + + Rationale: .claude/notes/agents.md § Token accounting and the reconciliation message """ in_sum = out_sum = cw_sum = cr_sum = 0 for m in messages: @@ -206,10 +161,9 @@ def _reconciled_messages(messages: list[TranscriptMessage], usage: TokenUsage) - d_cr = usage.cache_read_input_tokens - cr_sum if d_in == 0 and d_out == 0 and d_cw == 0 and d_cr == 0: return messages - # The residual is almost always positive (tokens billed but not streamed). - # A negative residual means the captured generations OVER-report the turn - # total for some bucket; word the note for that case so a "-512" entry - # doesn't read as "billed but not surfaced". + # A negative residual means the captured generations OVER-report some + # bucket; word the note for that case so "-512" doesn't read as + # "billed but not surfaced". positive = d_in >= 0 and d_out >= 0 and d_cw >= 0 and d_cr >= 0 note = ( "Tokens the agent billed but never surfaced as a generation " @@ -239,8 +193,7 @@ def build_turn_record(self) -> TurnRecord: commands = self._ordered_commands() if end is None: - # No terminal event yet (e.g. mid-stream snapshot). Return a minimal - # record from the granular events we have. + # No terminal event yet (mid-stream snapshot): minimal record. return TurnRecord( iteration=self._iteration, user_input=self._user_input, @@ -258,33 +211,20 @@ def build_turn_record(self) -> TurnRecord: tokens if (not tokens.is_empty() or tokens.total_cost_usd is not None) else None ) - # The authoritative turn total (token_usage) is the source of truth, but - # the per-message stream under-reports it. Book the residual as a single - # synthetic ReconciliationMessage so the transcript's token buckets sum - # to the total — making the stream self-reconciling for any downstream - # consumer (e.g. the evalboard) without a competing aggregate. messages: list[TranscriptMessage] = list(end.messages) - # Tool execution comes out of the generation windows HERE, once, for - # every harness — the reducers publish raw windows. - # - # The span set is computed ONCE and handed to both consumers. That is - # the invariant worth protecting, and it is the one that is easy to - # break: the subtraction and the head/tail must agree about which calls - # exist, or the buckets stop being disjoint. (The ORDER of the two is - # not load-bearing — `_overhead_ms` reads only the bounds, the - # main-thread flag and whether the duration is `None`, none of which - # `subtract_tool_time` changes. Do not add a comment claiming it is.) + # ONE span set, handed to BOTH consumers: the subtraction and the + # head/tail must agree about which calls exist, or the buckets stop being + # disjoint. (Their ORDER is not load-bearing. Do not claim it is.) + # Rationale: .claude/notes/timing.md § Why the subtraction and the head/tail may run in either order tool_spans = main_thread_tool_spans(messages, self._commands.values()) messages = subtract_tool_time(messages, tool_spans) if token_usage is not None: messages = self._reconciled_messages(messages, token_usage) startup_ms, teardown_ms = self._overhead_ms(messages, tool_spans) - # The turn's tool bucket, stored rather than left to be re-derived. It - # is the UNION (never the sum) of the SAME span set above, so all four - # buckets are measured against one selection. `None` when no bounded - # span was recorded — a turn that ran tools and timed none is not a - # turn whose tools took no time (CE058). + # The UNION (never the sum) of the SAME span set above. `None` when no + # bounded span was recorded: a turn that ran tools and timed none is not + # one whose tools took no time (CE058). tool_union = union_ms(tool_spans) if tool_spans else None return TurnRecord( diff --git a/src/coder_eval/timing.py b/src/coder_eval/timing.py index 4ce7d219a..71eded408 100644 --- a/src/coder_eval/timing.py +++ b/src/coder_eval/timing.py @@ -1,27 +1,21 @@ """Wall-clock arithmetic for a turn, defined once and shared. -A cycle-free leaf (the ``models/cli_match.py`` rationale): it sits outside -``agents/`` because ``EventCollector`` consumes it, and importing anything -under ``agents/`` pulls in every agent, which imports ``streaming/``. +A cycle-free leaf: it sits outside ``agents/`` because ``EventCollector`` +consumes it, and importing anything under ``agents/`` pulls in every agent, +which imports ``streaming/``. NO harness subtracts tool execution from its own generation windows. Each -publishes the RAW window it measured, and ``subtract_tool_time`` below -takes the UNION of the tool intervals back out of them once, for all five, at -the single capture seam — the same place the head and the tail are already -computed. A reducer's only remaining timing decision is where its window -opens, which is the one genuinely harness-shaped part: two interleave a tool -into a single window outright (Antigravity, whose Step for the tool arrives and -only a later ``usage_metadata`` Step cuts the message, and Codex, whose -``_flush_message`` window extends to the last item's ``completed_at_ms``) while -the other three tile the turn contiguously, so a call open at a boundary runs -inside two windows. Central subtraction handles both without either reducer -knowing which it is. - -There is a TypeScript twin, ``evalboard/lib/timing.ts::busyMs``, which -subtracts tool time from a task's WALL CLOCK to produce the Unaccounted -residual. It answers the same question about the same ``task.json``, so the -two must agree — neither owns the numbers: ``tests/_fixtures/timing_union_cases.json`` -does, and both suites replay it. +publishes the RAW window it measured, and ``subtract_tool_time`` takes the +UNION of the tool intervals back out of them once, for all five, at the single +capture seam. A reducer's only remaining timing decision is where its window +opens. + +There is a TypeScript twin, ``evalboard/lib/timing.ts::busyMs``, answering the +same question about the same ``task.json``, so the two must agree — and neither +owns the numbers: ``tests/_fixtures/timing_union_cases.json`` does, and both +suites replay it. + +Rationale: .claude/notes/timing.md § Where a reducer's window opens """ import math @@ -35,50 +29,23 @@ class TurnClock: """One (wall, monotonic) pair per turn; every later stamp derives from it. - A turn's bounds and its durations have to share a basis or they can - disagree, and the disagreement lands in a field measured in milliseconds. - Two concrete failures this removes: - - * Antigravity computed its window span on the MONOTONIC clock while - unioning WALL-clock tool intervals and subtracting one from the other. - That is the only reason its window could go negative at all, and the - clamp that hid it was indistinguishable from a real instant generation. - * Pi stamped with naive-LOCAL ``datetime.now()``, and claude-code did the - same. A DST transition or an NTP step inside a turn lands directly in a - generation window — an hour-long jump in a millisecond field. Nightly - runs start at 04:18 and run for hours, so it is reachable rather than - theoretical. A monotonic-derived stamp cannot express it. - - It is an EXTRACTION, not an invention: antigravity already captured this - exact pair at the top of ``communicate`` and simply did not use it for - later stamps. - - Stamps stay NAIVE LOCAL, matching what the rest of the telemetry and the - persisted ``execution_started_at`` already are, so no consumer changes. - - Within a turn the derived stamp is monotonic-accurate and may drift from - real wall time; each turn re-anchors. That is intended — do not "fix" it by - re-reading the wall clock, which is the property being removed. + A turn's bounds and its durations must share a basis, or they disagree in a + field measured in milliseconds. Stamps stay NAIVE LOCAL, matching the rest of + the telemetry and the persisted ``execution_started_at``. ONE PER TURN, never module-level and never reused across turns: a long run - would accumulate drift between the pair and real wall time. The turn-state - constructors take it as an argument so the lifetime is visible in the - signature, and so a unit test can pass a fake straight in. An end-to-end - test driving ``communicate()`` cannot: the state is built inside it, out of - the caller's reach, so those replace this class through the agent module - instead (``tests/_bracket_clock.py``). Both reach the same object. + accumulates drift between the pair and real wall time. Within a turn the + derived stamp is monotonic-accurate and may drift from real wall time; each + turn re-anchors. That is intended — do not "fix" it by re-reading the wall + clock, which is the property being removed. NOT for deadlines. Those stay on ``time.monotonic()`` directly: a deadline must not move when the wall clock steps. - Antigravity, Pi and claude-code use it — for their window bounds and, since - CE064, for their turn bracket. Codex and OpenCode do not. This docstring - deliberately says no more than that: asserting a current property of two - other modules from here is the drift that put a wrong OpenCode row in the - parity table for months, and that table is the designated SSOT for - per-harness composition. See the `clock basis for recorded stamps` row in - docs/agents/HARNESS_PARITY.md, and the paragraph below it for why each - unconverted harness stays that way. + Which harnesses use it is the `clock basis for recorded stamps` row in + docs/agents/HARNESS_PARITY.md, the SSOT for per-harness composition. + + Rationale: .claude/notes/timing.md § TurnClock """ def __init__(self) -> None: @@ -92,23 +59,11 @@ def now(self) -> datetime: def _require_same_awareness(a: datetime, b: datetime, *, field: str) -> None: """Raise if one stamp is timezone-aware and the other is naive. - Subtracting the two raises ``TypeError: can't subtract offset-naive and - offset-aware datetimes`` deep inside the arithmetic below, which surfaces - out of ``EventCollector.build_turn_record`` and kills the turn with a - message naming neither the field nor the harness. This turns that into a - statement of which pair disagreed and which side is aware. - - Unreachable from this repo today, and that is the point: every stamp in - ``agents/`` and ``streaming/`` is a naive ``datetime.now()`` (verified by - grep — zero ``timezone.utc`` / ``astimezone`` / ``tzinfo`` hits), so this - guards the SEAM rather than a live defect. The exposure it is actually for - is a third-party agent registered through the ``coder_eval.plugins`` SPI, - which lives outside ``src/coder_eval/agents/`` and which no lint rule - scoped to that directory could ever see. That is why this is a runtime - guard and not a rule. - - Only the MIX raises. An agent that is internally consistent in UTC is not - this function's problem, and neither is one that is consistently naive. + Only the MIX raises: an agent consistently aware, or consistently naive, is + not this function's problem. It turns a bare ``TypeError`` raised deep in the + arithmetic into a statement of which pair disagreed and which side is aware. + + Rationale: .claude/notes/timing.md § _require_same_awareness """ if (a.tzinfo is None) == (b.tzinfo is None): return @@ -125,32 +80,16 @@ def _require_same_awareness(a: datetime, b: datetime, *, field: str) -> None: def busy_ms(spans: list[tuple[datetime, datetime]], lo: datetime, hi: datetime) -> float: """Wall milliseconds inside ``[lo, hi]`` where at least ONE span was running. - The union, not the sum. Tool intervals overlap in practice — Antigravity - resolves several calls from one ``Step`` and backgrounds anything over ten - seconds; Codex spawns collab agents that run concurrently — so adding - their durations over-counts the busy time by exactly the overlap. - Subtracting such a sum from a generation window understates generation - and, with enough concurrency, drives it negative: four concurrent 400 ms - calls inside a 1000 ms window sum to 1600 ms, clamping the result to the - ``0.0`` that "unknown timing says unknown" exists to eliminate. - - Clipping to ``[lo, hi]`` is the other half: a tool that opened before this - window only spent part of its life inside it, and only that part is not - generation time here. - - Every stamp reaching this function is a naive ``datetime.now()`` today — - that is true of all of ``agents/`` and ``streaming/`` — so a mixed pair - means an agent has started recording aware stamps, and - ``_require_same_awareness`` names which pair rather than letting a bare - ``TypeError`` escape from the arithmetic. The spans are checked as well as - the bounds, not instead of them: the clipping below compares each span - against BOTH ``lo`` and ``hi``, so a guard on the bounds alone would leave - this function uncovered by it. - - An EMPTY span list is checked NOT AT ALL, bounds included. The comprehension - never runs, nothing is compared and nothing is subtracted, so there is no - pair for the guard to be about — and raising there would reject a call that - has always returned ``0.0``. + The UNION, not the sum: overlapping tool intervals would otherwise over-count + the busy time by exactly the overlap and drive a generation window negative. + Spans are clipped to ``[lo, hi]``, so a tool that opened earlier contributes + only the part that ran inside this window. + + Raises ``TypeError`` (via ``_require_same_awareness``) if the bounds and the + spans do not share a timezone awareness. An EMPTY span list returns ``0.0`` + and is not checked at all. + + Rationale: .claude/notes/timing.md § busy_ms """ if not spans: return 0.0 @@ -175,23 +114,13 @@ def busy_ms(spans: list[tuple[datetime, datetime]], lo: datetime, hi: datetime) def union_ms(spans: list[tuple[datetime, datetime]]) -> float: """Wall milliseconds at least ONE span was running, over their full extent. - ``busy_ms`` with the window set to the spans' own bounds. It exists because - two callers had copy-pasted that same ``min``/``max``/``busy_ms`` tail — - ``tests/_fixtures/golden_streams/_scrub.py`` (the golden sensor) and - ``scripts/timing/decompose_run.py`` (the live residual gate) — and they - answer the same question about the same recorded commands, so a divergence - would let one pass while the other failed. Each keeps its OWN stamp parsing - and span building, because their input shapes genuinely differ; only this - tail is shared. + ``busy_ms`` with the window set to the spans' own bounds. It does NOT filter ``end < start``. EVERY caller drops those while building - its span list — ``main_thread_tool_spans`` below (shared by - the collector and the report layer), ``_scrub.py`` and - ``decompose_run.py`` — so guarding again here would be a second rule about - the same input in a second place. That reasoning holds only while it stays - true of every caller: a new one that skips the check gets whatever - ``busy_ms`` does with an inverted pair, which is to discard it, but - silently rather than by this function's stated contract. + its span list, so a new caller that skips the check gets ``busy_ms``'s + silent discard rather than this function's stated contract. + + Rationale: .claude/notes/timing.md § union_ms """ if not spans: return 0.0 @@ -201,38 +130,22 @@ def union_ms(spans: list[tuple[datetime, datetime]]) -> float: def close_window(*, mark: datetime, now: datetime, item_start: datetime | None = None) -> tuple[datetime, float]: """Open one generation window at ``mark`` and close it at ``now``: its ``(started, span_ms)``. - The shape all five reducers share. What it returns is the RAW window — - tool execution is taken back out of it once, centrally, in - ``subtract_tool_time`` below, which is the only place - that arithmetic lives. It used to happen here too, per flush, and in - claude-code at finalization; the per-reducer bookkeeping that required - (a span list, its reset rule, the set of still-open calls) is where every - timing defect on this branch actually lived. - - ``mark`` is where the window opens: the previous flush's close, which is - what makes the windows TILE the turn contiguously instead of leaving the - model time that PRODUCED an item attributed to nothing. It is keyword-only - and has NO default so that no reducer can open a window without stating - what it tiles from — which is the defect pi shipped with, measuring from - its own turn start so that every inter-turn gap fell into no bucket at all. - Note what the signature does and does not buy: it constrains the call - SHAPE, not the VALUE. A reducer can still pass the wrong mark; what it - cannot do is fail to have one. - - ``item_start`` is this emission's own first stamp, when the harness has - one. The ``min()`` against ``mark`` is the tiling defense and nothing else: - a stamp that went backwards must never push the window start PAST the first - item and invert the span. claude-code passes none — its stream carries no - per-emission item start — so its window opens exactly at the mark. - - The result is clamped at ``0.0``: an inverted window (``now`` before - ``mark``, two clocks disagreeing) is a measured zero, not a negative - generation. - - It deliberately does NOT return ``completed``. The window always ends at - ``now``, which the caller passed in, so handing it back would be an - argument returned unchanged — redundancy dressed as symmetry. Call sites - write ``completed_at=now`` directly. + The shape all five reducers share, and what it returns is the RAW window — + tool execution comes back out centrally, in ``subtract_tool_time``. + + ``mark`` is where the window opens: the previous flush's close, which is what + makes the windows TILE the turn contiguously. It is keyword-only with NO + default so that no reducer can open a window without stating what it tiles + from. + + ``item_start`` is this emission's own first stamp, when the harness has one; + the ``min()`` against ``mark`` stops a backwards stamp inverting the span. + + The span is clamped at ``0.0``: an inverted window is a measured zero, not a + negative generation. ``completed`` is deliberately not returned — it is + always ``now``, which the caller already has. + + Rationale: .claude/notes/timing.md § close_window """ started = min(mark, item_start) if item_start is not None else mark return started, max(0.0, (now - started).total_seconds() * 1000.0) @@ -247,61 +160,26 @@ def decompose_turn( ) -> tuple[float | None, float | None]: """Wall ms before the first generation window opens, and after the last closes. - The turn's two unexplained ends. Between them the windows tile (each - harness's generation mark runs to the next) and tool execution is already - subtracted inside them, so head + generation + UNION(tool) + tail is the - whole turn — the union and not the sum, because concurrent tool calls - otherwise book their overlap twice (``busy_ms`` above, and measured: one - live Pi turn overlapped a ``Write`` and a ``Bash`` by 18.4 ms). - - ``tool_spans`` is what keeps those four buckets DISJOINT, and omitting it - is a double-count rather than a lost refinement. A tool is not confined to - a generation window: Antigravity force-closes an orphan at finalization - (``antigravity_agent.py``), which stamps its completion inside the tail, - and it backgrounds anything over ten seconds, which can straddle either - end. Such a span is subtracted out of the windows AND counted in the tool - bucket, so leaving it in the head or tail books it twice — measured on the - committed ``antigravity_d_orphaned_tool`` fixture as a residual of -86% of - wall clock. So the head and tail exclude tool time by the same rule and - the same helper the windows use. - - ``EventCollector`` is the SOLE caller, and deliberately so: this is the one - place the two values are computed, after which they are persisted on - ``TurnRecord`` and every later consumer READS them rather than recomputing. - The golden-stream sensor asserts on the dumped record, and - ``scripts/timing/decompose_run.py`` reads the stored fields — neither can - call this, because ``task.json`` carries no ``AgentStartEvent`` stamp to - recompute a head from. - - The head means ONE thing on all five: wall clock from the turn starting - until the harness first observed model output. Every reducer opens its first - generation window at that same instant, which is what keeps the two buckets - disjoint. What the head CONTAINS still differs and is deliberately NOT - split: a harness that spawns its process PER TURN fuses that boot, provider - resolution, dispatch and TTFT — measured on OpenCode, the process spawns in - 3 ms and the first event lands at 3921 ms — while one that spawns it once at - startup and holds it across turns has no boot inside the turn to fuse in. No - stream carries a marker between those parts. Naming these for the - interval they MEASURE rather than for what they contain is the whole point; - see docs/agents/HARNESS_PARITY.md for the per-harness composition. - - Every stamp reaching this function is a naive ``datetime.now()`` — that is - true of all of ``agents/`` and ``streaming/`` today — so a mixed pair means - an agent has started recording aware stamps, and ``_require_same_awareness`` - says so rather than letting a bare ``TypeError`` escape and kill the turn. - - ``None`` means never measured — a turn that produced no generation, or a - snapshot taken before the terminal event. Never 0.0, which would claim a - measurement was taken and came back instant (CE058). A measured inversion - (the two clocks disagreeing) IS a real zero and clamps, because both ends - were observed. - - NOTE the four-bucket identity has a second implementation in TypeScript — - the evalboard's Unaccounted cell (``_sections.tsx``) subtracts the same - buckets from the same wall clock, as ``pricing.ts`` mirrors ``pricing.py``. - It does not recompute a head or a tail (it reads the stored fields), so a - change HERE needs a TS change only when it alters what the buckets mean; - adding a fifth bucket means touching that cell and ``sumHarnessOverhead``. + The turn's two unexplained ends. Between them the windows tile and tool + execution is already subtracted, so head + generation + UNION(tool) + tail + is the whole turn. + + ``tool_spans`` keeps the four buckets DISJOINT: a tool is not confined to a + generation window, so a span left in the head or tail is booked twice. + + ``EventCollector`` is the SOLE caller: computed once, persisted on + ``TurnRecord``, READ by everyone later. + + The head means ONE thing on all five harnesses: wall clock from the turn + start until the harness first observed model output. What it CONTAINS + differs per harness (docs/agents/HARNESS_PARITY.md). + + ``None`` means never measured — never ``0.0`` (CE058). A measured inversion + IS a real zero and clamps, because both ends were observed. + + Raises ``TypeError`` (via ``_require_same_awareness``) on a mixed pair. + + Rationale: .claude/notes/timing.md § decompose_turn """ spans = tool_spans or [] head = tail = None @@ -323,30 +201,19 @@ def main_thread_tool_spans( The span set the generation subtraction, the head and the tail are all measured against, so they cannot disagree about which calls exist. Shared - with ``reports_stats.turn_time_buckets``, which answers the same question - about a finished ``TurnRecord`` — a second typed copy of this rule is how - two report surfaces come to publish two different tool totals for one run. - (``scripts/timing/decompose_run.py`` keeps its own, over raw ``task.json`` - dicts rather than models; that is the sanctioned third reader, and - ``tests/test_timing_close_window.py::TestTheThreeToolUnionsAgree`` pins all - three together.) - - Sub-agent tools are excluded, and that used to be the gap: ``_overhead_ms`` - filtered its GENERATIONS to the main thread and then passed EVERY command, - so its claim to keep all four buckets measuring one thread was true only by - luck. It held because a child nests inside the parent Agent call, whose own - interval the union already covers — but Codex's recovered child tools carry - the CHILD's clock, so nothing made it true by construction. The evalboard's - twin (``toolExecutionMs``) does filter, so the two agreed by accident. - - A sub-agent's tool ids are reachable only through the messages that own - them: a child generation carries ``parent_tool_use_id``, and its - ``tool_use_ids`` are the calls it made. - - An inverted pair (``end`` before ``start``) is dropped here rather than - passed on. ``busy_ms`` would discard it anyway, but ``timing.union_ms`` - documents that it does NOT filter them because its callers do — so this is - the caller keeping that true. + with ``reports_stats.turn_time_buckets``; + ``tests/test_timing_close_window.py::TestTheThreeToolUnionsAgree`` pins this, + that, and ``scripts/timing/decompose_run.py`` together. + + Sub-agent tools are excluded — a child's calls are already covered by the + parent Agent call's own interval. A sub-agent's tool ids are reachable only + through the messages that own them: a child generation carries + ``parent_tool_use_id``, and its ``tool_use_ids`` are the calls it made. + + An inverted pair (``end`` before ``start``) is dropped here, which is what + keeps ``union_ms``'s "every caller filters" contract true. + + Rationale: .claude/notes/timing.md § main_thread_tool_spans """ sub_agent_tool_ids = { tool_id @@ -365,19 +232,11 @@ def main_thread_tool_spans( #: How far a published window may sit from the span its own bounds describe. -#: -#: ONE MILLISECOND, which is the coarsest unit a field named ``_ms`` can -#: honestly be published in: a producer that records microsecond-precision -#: bounds and rounds its duration to whole milliseconds is within its rights, -#: and crashing its turns over 0.001 ms would be the guard relocating a defect -#: rather than removing one. The exposure this check is actually for is a -#: third-party agent registered through the ``coder_eval.plugins`` SPI, which is -#: exactly the producer most likely to round — so the tolerance has to admit it. -#: -#: It still catches everything it is for. The defect class is a reducer that -#: NARROWED or WIDENED a window without moving its bounds — subtracting its own -#: tool time, most plausibly — which is tens to thousands of milliseconds, three -#: to six orders of magnitude above this. +#: ONE MILLISECOND — the coarsest unit a field named ``_ms`` can honestly be +#: published in, so a producer that rounds is admitted while the defect class +#: (a reducer narrowing or widening a window without moving its bounds) is tens +#: to thousands of ms, orders of magnitude above it. +#: Rationale: .claude/notes/timing.md § _WINDOW_TOLERANCE_MS _WINDOW_TOLERANCE_MS = 1.0 @@ -387,81 +246,28 @@ def subtract_tool_time( ) -> list[TranscriptMessage]: """Take tool execution back out of the generation windows it overlapped. - THE one place this happens. Five reducers used to do it themselves — four - through ``close_window`` as they flushed, claude-code once at finalization — - while the head and tail were already computed centrally, right here. That - asymmetry was the complexity, and every timing defect this branch fixed - lived in the per-reducer bookkeeping around the subtraction rather than in - the subtraction itself: when to reset a span list, when to clear a start - stamp, when to advance a mark. A reducer now publishes the RAW window and - keeps only the genuinely harness-shaped decision, which is where its window - opens. - - NON-MUTATING, and the reason is aliasing rather than repeated calls. Every - agent builds its terminal event as ``AgentEndEvent(messages=list(...))`` — - that copies the LIST, not the message objects — so writing in place would - reach back into the agent's own live state from the collector, which is - exactly the layering "the collector is the sole capture seam" exists to - prevent. ``model_copy`` keeps it one-directional. It is also unconditionally - safe for any caller that builds a record twice: ``EarlyStopWatcher`` holds - one collector across a turn's tool-call rounds and calls - ``build_turn_record`` on every one. - - GROUPED BY IDENTICAL BOUNDS, not by ``message_id``. Codex splits one window - across two sub-messages (thinking and action) that share ``started_at`` and - ``completed_at`` and divide the window by output-token share; subtracting - the group's overlap from each part separately would subtract it twice and - stop the parts summing to the window. Bounds identity covers that, and it - also covers OpenCode and Pi, which can legitimately carry - ``message_id is None`` — so keying on the id would silently collapse every - id-less message of a turn into one group. - - MAIN THREAD ONLY. A sub-agent generation (``parent_tool_use_id`` set) is - skipped: its own tools are not in this span set, and the Agent call that - spawned it already covers its whole run. - - A ``generation_duration_ms`` of ``None`` means no window was ever measured - (codex's rollout rebuild, claude's synthesized sub-agent terminal), so there - is nothing to subtract from and it passes through untouched — never - coerced to ``0.0`` (CE058). Every non-``AssistantMessage`` entry — a - simulation ``UserMessage``, the appended ``ReconciliationMessage`` — passes - through by identity. - - A window entirely covered by tool execution reaches ``0.0``, and that is a - measurement rather than an absence. - - THE GROUP'S RAW TOTAL MUST EQUAL THE SPAN ITS BOUNDS DESCRIBE, and this - function raises if it does not. That equality is the contract that lets - ``generation_duration_ms`` stay a PUBLISHED field rather than one the - collector derives from the bounds: a reducer publishes the raw window it - measured, so the duration is ``completed_at - started_at`` (or, for a group - Codex split across two sub-messages, sums to it). Deriving it here instead - was considered and cut — it would cost five reducers, a regeneration of - every golden and a rewrite of CE059, whose exemption keys on the kwarg being - present at the call site — and this assertion is the sensor that makes - deferring that safe. A mismatch means a reducer narrowed or widened a window - without moving its bounds, which is the drift - ``tests/_fixtures/golden_streams/_scrub.py::assert_timing_captured``'s - "bounds that span it" check catches one replay at a time. - - It OVERLAPS with CE061 and is deliberately kept anyway. All five reducers - build the window with ``timing.close_window(mark=…, now=…)`` and write - ``started_at=started, completed_at=now``, and CE061 — now exemption-free — - forces that shape statically, so the equality is largely true by - construction. What this adds is the runtime half: a reducer that bypasses - ``close_window`` in a way an import-level check cannot see, and a - third-party agent registered through the ``coder_eval.plugins`` SPI, which - lives outside ``src/coder_eval/agents/`` where no lint rule reaches it. It - is not load-bearing on its own. - - RAISING KILLS THE TURN, and that is accepted — the same trade - ``timing._require_same_awareness`` makes at this seam. The condition is - unreachable without a reducer bug; all five are exercised by the golden - corpus and by the ms-exact identity contract. + THE one place this happens. Reducers publish RAW windows and never subtract. + + Returns a NEW list; messages are copied, never mutated — agents alias their + own live message objects into ``AgentEndEvent``, so writing in place would + reach back into agent state. + + Windows group by identical ``(started_at, completed_at)``, so the two + sub-messages Codex splits one window into share a single subtraction. + Sub-agent generations (``parent_tool_use_id`` set) are skipped. A + ``generation_duration_ms`` of ``None`` passes through untouched — never + coerced to ``0.0`` (CE058) — as does every non-``AssistantMessage`` entry. + A window fully covered by tool execution reaches ``0.0``: a measurement. + + Raises ``ValueError`` if a group's published total does not equal the span + its bounds describe. That equality is what lets ``generation_duration_ms`` + stay a PUBLISHED field; CE061 forces the same shape statically. Raising + kills the turn, which is accepted. + + Rationale: .claude/notes/timing.md § subtract_tool_time """ - # (index, raw window ms) per group. The raw value is captured HERE, where - # the message is already narrowed to AssistantMessage, so the apportioning - # loop below needs no second narrowing. + # (index, raw window ms) per group, captured while the message is already + # narrowed to AssistantMessage. groups: dict[tuple[datetime, datetime], list[tuple[int, float]]] = {} for index, message in enumerate(messages): if not isinstance(message, AssistantMessage): @@ -474,20 +280,14 @@ def subtract_tool_time( out = list(messages) for (started, completed), members in groups.items(): raw_total = sum(raw for _, raw in members) - # Nothing to apportion, and dividing by it is a ZeroDivisionError. A - # group already at zero stays at zero. + # Nothing to apportion, and the loop below divides by it — a group already + # at zero stays at zero. # - # THE SKIP RUNS BEFORE THE CHECK BELOW, and that order is load-bearing - # rather than incidental. `close_window` clamps an inverted window — - # `now` before `mark`, two clocks disagreeing — to `0.0` while the - # bounds it writes still say `completed_at < started_at`, so `bounds_ms` - # is NEGATIVE and the equality fails. That is a measured inversion, the - # case `decompose_turn` deliberately clamps because both ends were - # observed; raising on it would kill turns on exactly the shape the - # clamp exists to tolerate. The cost is that a `0.0` published beside a - # POSITIVE window slips through — a shape no in-tree reducer produces, - # and one that reads downstream as "measured, and instant" rather than - # as a crashed turn. + # ORDER IS LOAD-BEARING: this skip runs BEFORE the equality check below, + # because `close_window` clamps an inverted window to 0.0 while its bounds + # still say `completed_at < started_at` — a measured inversion the check + # would otherwise raise on. + # Rationale: .claude/notes/timing.md § subtract_tool_time if raw_total <= 0: continue bounds_ms = (completed - started).total_seconds() * 1000.0 @@ -507,14 +307,9 @@ def subtract_tool_time( assigned = 0.0 for n, (index, raw) in enumerate(members): # The last member takes the remainder so the parts reconstruct the - # group's net exactly, rather than drifting by the rounding. - # NOT rounded. The last member already takes the remainder, so the - # parts reconstruct the group's net exactly without it — while - # rounding each earlier share UP could push `assigned` past `net` - # and hand the last member a NEGATIVE duration. That needs a net of - # well under a microsecond (a window almost entirely covered by - # tool execution) and so had never been seen, but a negative - # generation is an invariant break, not a rounding artifact. + # group's net exactly. NOT rounded: rounding an earlier share up could + # push `assigned` past `net` and hand the last member a NEGATIVE + # duration. share = net - assigned if n == len(members) - 1 else net * (raw / raw_total) out[index] = out[index].model_copy(update={"generation_duration_ms": share}) assigned += share diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index cee05a5bb..b810657b4 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 79_754 +_ESSAY_BASELINE_WORDS = 73_413 _SRC = Path("src/coder_eval") From 4943b72e2545d5383c392bfa2519b3b1dd1a0132 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 18:32:03 -0700 Subject: [PATCH 03/19] =?UTF-8?q?docs:=203/7=20=E2=80=94=20move=20agent-ad?= =?UTF-8?q?apter=20rationale=20into=20.claude/notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cuts the five harness adapters and their eight helpers from 17,227 essay words to 4,456, and gives `.claude/notes/agents.md` the shared story the adapters kept telling five times over: the turn lifecycle, first-window seeding, per-harness generation marks, token accounting and cost resolution, why a clean exit can still be a crash, and how each CLI is reaped. Every per-harness parity claim either stays in the adapter or is already a row in docs/agents/HARNESS_PARITY.md, which the taxonomy makes the SSOT for cross-harness facts. Both SDK#24168 FIXMEs stay FIXMEs. No executable statement changed — proved per file by `prose_budget --assert-code-unchanged`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2 --- .claude/notes/agents.md | 616 +++++++++++++- src/coder_eval/agents/__init__.py | 7 +- src/coder_eval/agents/_logging.py | 29 +- src/coder_eval/agents/_skills.py | 35 +- src/coder_eval/agents/antigravity_agent.py | 625 +++++--------- src/coder_eval/agents/claude_code_agent.py | 931 +++++++------------- src/coder_eval/agents/codex_agent.py | 939 ++++++++------------- src/coder_eval/agents/opencode_agent.py | 576 +++++-------- src/coder_eval/agents/pi_agent.py | 453 ++++------ src/coder_eval/agents/registry.py | 32 +- src/coder_eval/agents/watchdog.py | 47 +- src/coder_eval/streaming/events.py | 12 +- src/coder_eval/streaming/renderers.py | 11 +- src/coder_eval/streaming/wire.py | 19 +- tests/lint/prose_budget.py | 2 +- 15 files changed, 1896 insertions(+), 2438 deletions(-) diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index d2088c335..656ccce74 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -4,10 +4,622 @@ ## Token accounting and the reconciliation message -- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. +- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout; both harnesses' turn totals already include sub-agent cost. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. The residual is almost always positive; a NEGATIVE one means the captured generations over-report some bucket, which is why the note's wording is branched — a `-512` entry must not read as "billed but not surfaced". ## Harness run-limit parity -- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. **Pi** is the same shape — the CLI (`pi -p --mode json`) streams a real multi-step loop per `communicate()` (`turn_start`/`turn_end`), so `max_turns: N` counts native `turn_start` steps; Pi retries transient/provider errors INTERNALLY (`agent_end.willRetry`), and the reducer finalizes once at `agent_settled`/EOF (not the first `agent_end`), folding the retry cycles into one turn. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex, Antigravity, and Pi (all run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), `allowed_tools`/`disallowed_tools` on Pi (its built-in tool names are lowercase — `bash`/`read`/… — and cannot map to the Claude-namespaced config default, so forwarding them would strip the agent of ALL tools; warned+ignored like the three agents above) — Pi DOES enforce `system_prompt` (`--append-system-prompt`, a small win over OpenCode) and DOES honor `plugins` for skills (each resolved skills dir → a `--skill ` arg via the shared `_plugin_skill_dirs` resolver, recorded as `pi_skill_paths`, so it CAN run activation suites) but does NOT read `system_prompt_file`, and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md. +- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode and Pi each keep a native unit too, because their CLIs stream a real multi-step loop per `communicate()` (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. + + The **known unfixed divergences** — which config fields each harness does and does not enforce, and the per-harness `agent.plugins[].path` depth (claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, which is the costly direction: no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, reading exactly like a skill that never triggers; held to the plugin-root shape for `SKILL_SOURCE_PATH` by CE045) — are the table's to state, not this file's. Full table + rationale: docs/agents/HARNESS_PARITY.md. + +## Shared turn lifecycle + +Every adapter drives the same skeleton, on the base class: `_begin_turn()` resets the +pending slot and bumps the iteration counter, `_end_turn_ok()` marks the turn clean, and +`_mark_stopped()` closes the agent. Before raising on a mid-turn failure an adapter sets +`pending_turn` to a `crashed=True` `TurnRecord` and raises bare, which is what lets the +orchestrator drain the partial record and un-bump the iteration. + +The record is BUILT before `_end_turn_ok()` on every harness: a failure inside the +reduction is a failed turn, and `_end_turn_ok` would already have cleared the rollback +flag `discard_pending_turn` needs. + +Three exit paths converge on `finalize`, and it is idempotent on all of them, because the +protocol allows EXACTLY ONE `AgentEndEvent` per `communicate()`: the clean return, the +crash/timeout kernel, and the outer `except` that can fire *after* a normal finalize (a +failure while building the record). The first call wins, so a late crash cannot emit a +second terminal event into the caller's `stream_callback` — it still raises, so the +failure is not swallowed. + +`TurnEndStatus` mirrors `AgentEndStatus` value-for-value precisely so the conversion in +`finalize` is total: an unmapped future member raises loudly instead of silently +bucketing to COMPLETED. + +Status precedence is the same everywhere: timeout > stopped_early > max_turns_exhausted > +completed. `stopped_early` outranks the cap because an armed criterion deciding the +outcome is the more specific reason to have cut the run, and every loop checks it first. + +## Why a post-stop exception is not a crash + +Once the loop has broken on purpose — a cooperative stop or the turn cap — an exception +raised while tearing the stream down must NOT be escalated. Escalating triggers the +orchestrator's retry with the watcher's decision still latched, so the retry stops at turn +0 having spent nothing useful; a cap-break is the same shape, where the retry burns the +budget again and re-hits the cap. `ended_cleanly` is the guard. + +## Why the constructors declare every kwarg + +`create_agent` calls `agent_class(config, route=route, **kwargs)` through a +`cast(Any, ...)`, so pyright checks nothing at the call site; a `**_` sink would mean +nothing checks it at runtime either. The orchestrator depends on that `TypeError` as a +signal — it gates `cost_log_tags` on `supports_cost_log_tags` precisely because the +agent-agnostic factory would otherwise forward it into constructors that do not declare +it. A mis-gated kwarg must be loud, not silently dropped. + +`route` is accepted for factory parity and deliberately unused by the CLI-driven +harnesses: those CLIs own their own provider configuration. + +## First-generation window seeding + +`harness_startup_ms` is defined as the wall clock from the turn starting until the harness +first observed model output, and that instant is also where the first generation window +opens — which is what keeps the head and the generation disjoint so the four-bucket +identity closes. + +Without the re-seed the mark is stamped when the turn state is built, BEFORE +`AgentStartEvent` is emitted, so the head is a small negative that `decompose_turn` clamps +to `0.0` — a clamped inversion published as "measured, and instant", the exact confusion +CE058 exists to prevent. The fault is not the clamp: the head was measured against the +WRONG INSTANT. Everything the harness spent booting, resolving a provider and reaching its +first token was booked as msg0's generation instead — ~3.6 s per turn on claude-code, +~4.7 s on Antigravity against a later-window median of 3.3 s. + +**ONCE PER TURN, and that is the whole contract.** The seeding hook runs for every stream +event; re-seeding on each would stop the windows tiling and drop the gap before the next +emission — a tool result landing, then the next request going out — into no bucket at all, +which is the defect Pi shipped with. The flag needs no reset: a fresh turn state (and a +fresh `TurnClock`) is built per `communicate()`, so it is per-attempt by construction. If +a future harness reuses a turn state, the reset belongs there. + +A turn that never observes model output keeps the turn-entry mark and clamps to `0.0` +exactly as before. That is the correct degradation, not a gap. + +**Antigravity gates on the Step SOURCE.** `StepSource` carries `SYSTEM` and `USER` besides +`MODEL`, and `StepType` carries `SYSTEM_MESSAGE` / `COMPACTION` / `FINISH`; the SDK's event +processor queues every `step_update` verbatim, so a turn can legitimately open with one. +Seeding on such a Step would put the mark BEFORE the model spoke and hand the remainder +back to msg0's generation — the defect the seeding exists to remove. The same gate guards +text streaming. + +What differs between claude-code and Antigravity is not in-process versus subprocess — +both spawn a binary. Antigravity spawns its `localharness` ONCE, in `start()`, and holds +it across every `communicate()`, so there is no boot inside a turn for the head to +contain: it is dispatch plus time to first token. claude-code spawns a fresh CLI per turn +and fuses that boot in. The head means the same thing on both; only its COMPOSITION +differs, which is a real property of the harness rather than a measurement artifact. + +**One route is operator-reachable and worth knowing.** claude-code sets +`include_partial_messages=True` BEFORE spreading `**self.config.sdk_options`, so +`-D agent.sdk_options.include_partial_messages=false` turns the raw stream off, and with +it the re-seed — the head silently returns to the clamped `0.0` it used to publish. +Nothing warns; the degradation is safe but the number changes meaning. + +## Per-harness generation marks + +Where each reducer OPENS its window — the one genuinely harness-shaped timing decision +left. The central rule this sits under (raw windows, one subtraction seam, and why +interleaving versus tiling does not matter to it) is +[timing.md § Where a reducer's window opens](timing.md); do not restate it here. + +- **claude-code** tiles from the previous emission's arrival. The mark is deliberately + NOT advanced when a tool result arrives: resetting it there opened the next window at + the instant the RESULT landed rather than at the previous window's close, so everything + between the tool finishing and its result reaching the handler (SDK transport, CLI + processing, next-request dispatch) fell into no bucket. Measured on + `tasks/dataset_example.yaml`: a 21.5 ms `Write` followed by a 2511.7 ms round trip — + 21% of an 11.7 s turn accounted to nothing. A tool-heavy shape hides this because the + tool union absorbs the interval; a fast tool leaves it exposed. +- **OpenCode** tiles from the previous `step_finish`. The CLI announces a step only once + it is already producing one, so a window bounded by `step_start` drops the model time + that PRODUCED the step. Measured on `tasks/hello_date` with a live claude-haiku-4.5: two + gaps of 857 ms and 851 ms carrying no tool (the `Write` inside them took 7 ms), 24% of + the turn's wall clock — enough on its own to hold OpenCode above the evalboard's 25% + "Unaccounted" red threshold. +- **Pi** tiles from the previous `turn_end`. It was the only harness measuring from its + own `turn_start`, so the wall clock between one `turn_end` and the next `turn_start` — + the model time that PRODUCED that turn — fell into no bucket at all. +- **Codex** tiles from the previous flush's end. The SDK stamps an item with the moment it + began EXECUTING, so seeding from it discarded the gap between the last item's completion + and this one's start. Measured on `tasks/hello_date`: a `Write` emission spanning 2 ms + reported 98 output tokens while 2694 ms of real generation sat in the preceding gap; + across that turn only 15.8% of the 17 s wall clock was accounted for. +- **Antigravity** does not tile at all — it interleaves the tool INTO the window. Do NOT + "simplify" it to resetting the mark when a tool ends — that loses + real model time: measured on run `2026-09-09_04-18-50`, task + `skill-rpa-uia-google-search`, a harness-local `Read` closed 8 ms after it opened while + 6.4 s of model time separated the two flushes around it. Publishing the RAW window + handles that case AND its opposite (a 43 s `Bash`, where the model time really is the + flush-to-DONE remainder). + +On every harness the mark advances ONLY after a flush that actually appended a message: a +step that never finished published nothing, so tiling past it would attribute its time to +whichever step finishes next. + +Pi and OpenCode also clear the step's own start stamp at flush, because it has been SPENT +into the message. It is passed to `close_window` as `item_start`, whose `min()` pulls the +window open to cover it; left in place, a second `turn_end`/`step_finish` with no +intervening start — a duplicate or replayed line, which these reducers promise to survive +— reopens the next window back at the previous step's start and publishes that whole span +a second time (reproduced on Pi: 3000 ms of generation for a 2000 ms turn). Pi clears its +text and tool-id lists for the same reason: otherwise the previous turn's text is +re-emitted as its own assistant message and the same `tool_use_ids` are re-listed, so one +tool call appears to belong to two generations. + +There is no per-step span list to reset any more, and that whole class of defect went with +it: the central subtraction sees every span at once and clips each to the window it +overlaps, so a call closing in the gap before a step start needs nobody to remember it. +The rule that used to live there was wrong once on OpenCode — clearing at `step_start` +wiped the span before `step_finish` could subtract it, a 100% overstatement of that window. + +## Why a clean exit can still be a crash + +An exit code of 0 with no telemetry is indistinguishable from a real pass in every +aggregate, and file-based criteria can still score it SUCCESS. Worse, a turn with no +tokens is one whose `max_total_tokens` / `max_usd` gates could never have tripped no +matter how much the run actually billed. So the CLI harnesses crash rather than score: + +- **Vocabulary drift** — a clean exit that recognized NO event from the harness's known + set. This has happened: OpenCode once parsed the `session.next.*` server vocabulary + instead of the CLI's own and scored SUCCESS 1.0 with zero turns and zero tokens. +- **Finished steps with no tokens** (OpenCode) — the same outcome one layer down. Keying + on "recognized nothing" alone left it reachable: a `step_finish` carrying no `tokens` + key recognizes three events, books an all-zero `TokenUsage`, and `EventCollector` maps + that to `token_usage=None` — a COMPLETED turn with no tokens, no cost, no warning. The + arm keys on a step the CLI reported as FINISHED, its own claim that a generation + completed, rather than on `usage.is_empty()` alone, which would also condemn a stream + cut before any step could finish. +- **A terminal provider error** (Pi, OpenCode) — `pi -p` exits 0 after exhausting its + internal retries, so without the crash the turn books as a clean COMPLETED + (`FinalStatus.FAILURE`, category "failed"), silently depressing the measured pass rate. + Crashing routes it to `FinalStatus.ERROR`, which is excluded from outcomes. +- **A CLI that closed its stream but would not exit** within the grace period. + +Every arm is gated on `stopped_early` / `max_turns_exhausted`, because an intentional cut +can land before the clearing event arrives. Pi's error case shows why: `error_message` is +set at an error `turn_end` and cleared only by a LATER non-error `turn_end`, but a +`max_turns` / `should_stop` cut can fire at the next `turn_start`, leaving a stale error +from a turn Pi was still retrying. Without the guard that clean, budget-exhausted cut +would crash and burn retries, contradicting the documented "finalizes cleanly as +`max_turns_exhausted`, no crash" contract. + +OpenCode has one escape hatch, `require_token_telemetry`, for a provider or auth mode that +reports no usage at all — where crashing every turn makes the harness unusable rather than +merely imprecise. It deliberately does NOT cover vocabulary drift: that arm has silently +zeroed a whole run before, and no provider quirk explains it. + +## Token accounting, per harness + +Keep the buckets straight: `uncached_input_tokens` is the FRESH prompt slice only, because +cost bills it at the input rate and the cache buckets separately. + +- **claude-code** prefers `ResultMessage.model_usage`, the SDK's cumulative per-model + billing; summed and priced at list rates it equals `total_cost_usd` exactly, and it + captures sub-agent consumption the assistant-message stream does not. The per-call + telemetry sum is the fallback (exact only when every token-bearing emission carries an + id), and the `usage` snapshot the last resort. Cost is backfilled from the rate card + when a turn was timed out or killed, so there is no terminal `ResultMessage` — the + tokens are already captured, so this is pure pricing. +- **Codex/OpenAI** report `input_tokens` INCLUSIVE of the cached prefix, and bill no + separate cache-write fee. So the fresh slice is `input - cached`, `cache_creation` is 0, + and `cache_read` is `cached`. +- **OpenCode** is the one stream where the convention must be decided per step: `input` + is either already the fresh slice (flat, `total = input + output + reasoning + cache`) + or inclusive of the cache (nested, the OpenAI `prompt_tokens` convention). The stream's + own `total` arbitrates, so a CLI upgrade that flips the convention re-classifies itself + instead of silently mis-booking a bucket. With no cache traffic the conventions agree. +- **Pi** reports the fresh slice directly, so it maps straight across. +- **Gemini/Antigravity** reports `prompt` (with `cached` a subset), `candidates` and + `thoughts`. Fresh input is `prompt - cached`, cache_read is `cached`, cache_creation is + 0 (no cache-write fee), and output is `candidates + thoughts` — Gemini bills thinking as + output. + +`reasoning` bills at the output rate everywhere but is reported apart from `output`, so it +is folded into the turn total while the per-message record keeps it separately. + +## Why token-shape drift warns instead of raising + +A bare `int()` raises on anything non-numeric, which `communicate`'s `except Exception` +turns into an `AgentCrashError` (`max_retries=2`) — so ONE mistyped bucket burns three +full attempts and lands the task as ERROR. That is the opposite of the policy every +neighbouring field follows. A changed type in the `tokens` dict is drift, so it is +reported once per turn and the turn survives on the buckets it could read. It is also what +makes the "never raises on bad input" line-handler contract true. + +The event-vocabulary check cannot see inside `usage`, which is why the CLI harnesses carry +extra guards: a renamed or absent `usage` object, an all-zero bucket set, and a +`totalTokens` that no longer reconciles with the summed buckets each warn once. Without +them a CLI upgrade silently zeroes the run's tokens and cost and blinds the budget gates. + +## Cost: the stream versus the rate card + +A non-zero cost the CLI reported always wins — it is the provider's own accounting, and on +OpenRouter per-request routing makes it strictly better than a static headline rate. The +rate card fills two gaps that would otherwise book tokens with no money: + +- the stream reported no cost at all (a provider or auth mode that omits it, or a turn + that died before its first finished step), or +- it reported `0` for tokens the rate card prices above zero. OpenCode reports 0 when its + own model registry has no price for the model, or under subscription-style auth; a true + $0 and a present-but-always-zero cost field are indistinguishable from the stream alone, + and understating cost silently defeats `max_usd`, which is the worse failure. A + genuinely free model has an all-zero rate entry (or none), so it still resolves to 0. + +The Claude SDK's own `costUSD` is a client-side estimate assuming Anthropic pricing, so it +is wrong for an open-weight model behind LiteLLM and is repriced from the token buckets at +the model's real rate. The buckets are untouched, so the reconciliation invariant holds — +only the cost scalar changes. An unpriced model sets the cost to `None` (an honest N/A) +**and warns**, because a silent `None` makes the orchestrator skip the `max_usd` gate with +no diagnostic. + +## Codex rollout rebuild + +Codex runs every sub-agent on its own child thread whose events never reach the parent +stream, and that thread persists with *Limited* rollout policy, which drops +`commandExecution` events. So neither the live stream nor `thread.read` surfaces the +sub-agent's shell commands, and `thread/tokenUsage/updated` only ever reports the PARENT +thread. + +But the child rollout ALWAYS persists the raw `function_call` / `local_shell_call` / +`custom_tool_call` ResponseItems and a `token_count` event with the child's cumulative +usage. Recovery mines that file, emitting one `CommandTelemetry` per inner call plus one +nested `parent_tool_use_id`-tagged `AssistantMessage` carrying that generation's real +tokens — which `_fold_subagent_tokens` then sums into the turn total, reaching the same +end state Claude gets naturally. + +Recovery runs on a turn-cap stop, and that is deliberate: it is the only writer of those +tagged messages, so skipping it drops the child threads' spend from the run's cost +entirely. A cap is a routine ending, so paying ~2 s of rollout polling beats under- +reporting spend on every capped run that spawned a sub-agent. The recovered calls land in +the trajectory beyond the cap's count, the same way a force-closed orphan does — the cap +bounds what the model was allowed to DO, not what the record may explain. It is still +skipped on a cooperative stop: an armed gate has already decided the run, and children may +have no rollout yet. + +The rollout file can lag the parent's `turn/completed` by a beat (the recorder flushes on +a background task), so the lookup polls ~2 s — but bails immediately when +`/sessions` does not exist at all, since no flush can ever land there. + +The thread-cumulative baseline is the other half. `ThreadTokenUsage.total` counts the whole +THREAD, and the Codex thread is created once per task and reused, so by turn N it still +carries turns 1..N-1. The orchestrator sums per-turn usages, so handing it the cumulative +figure books turn 1 again on turn 2 and turns 1-2 again on turn 3 — a sum of prefix sums, +inflating an N-turn task by roughly (N+1)/2. Subtracting the previous turn's snapshot +leaves this turn. A total that moved BACKWARDS means the thread restarted, so the snapshot +is already turn-local and is returned whole rather than clamped to zero. + +On a crash the SDK total never arrives and the per-generation tokens on the flushed +messages are used instead — but the baseline must still advance past them, or the next +turn's delta re-books everything the crashed turn already reported. + +## Why the generation is split into sub-messages + +Codex flushes one generation as up to two `AssistantMessage`s — thinking and action — +sharing one `message_id` and one pair of bounds. The FIRST carries the generation's +input/cache, because those are per-CALL billing figures that must not be split; the rest +carry 0 so per-`message_id` sums do not double-count. Generation TIME is different: it is +a property of the content, so it is apportioned by output-token share, with the last part +taking the remainder. Concentrating it on the first reported the thinking row as the +entire generation and the action row as instant. + +Sharing the bounds is what makes the split safe: the central subtraction groups the two +parts and takes the tool overlap out ONCE rather than once per part. + +This weighs by output tokens while the evalboard's own mixed-emission split weighs by +CONTENT SIZE. Deliberate, not an oversight to unify — here the SDK hands over a real +per-spec token count, so there is no need to approximate one from content length. + +## Antigravity Step interleaving and the background poll + +`receive_steps()` exhausts with a tool call still open when the model kicks off a +`run_command` as a background task and goes idle without waiting for it. +`Conversation.wait_for_wakeup()` is an unimplemented stub on the Local harness connection +(always returns False regardless of pending state), so the agent polls for progress +itself — gated on that orphaned-tool signal, so a normal turn takes the branch zero times. + +The poll loop is bounded by a FRACTION of the turn's `timeout`, not by `timeout` itself: a +check against the identical value races the `ThreadedWatchdog` non-deterministically for +who fires first, while a smaller fraction is a strictly earlier, non-racing deadline whose +whole purpose is to win that race. 0.8 leaves the watchdog a fifth of the budget as margin +for this loop's own exit bookkeeping. Without the bound, a tool call spuriously left +ACTIVE with no real background job behind it finalized immediately and graded whatever the +agent had produced; bounding it only by a fixed cycle count disconnected from `timeout` +(120 × 5 s = 600 s, double the framework's default `turn_timeout: 300`) makes the graceful +path unreachable — the watchdog always wins, and the same spurious-orphan turn burns the +full turn timeout before crashing with zero criteria evaluated. + +The cycle cap is the SOLE bound when a task sets no timeout at all. It is deliberately not +"break after N consecutive empty polls": `receive_steps()` returns identically empty +whether a backgrounded job is still running or will never resolve, and there is no signal +that tells the two apart except waiting. A count small enough to matter would abort real +slow jobs (confirmed cases needed up to ~60 consecutive 5 s empty polls before +succeeding); one large enough to be safe barely improves on a flat cap. + +`has_orphaned_tool_call` is an ALLOWLIST on ACTIVE, not a denylist on "not yet closed". +`StepStatus` also has WAITING_FOR_USER (the harness blocked on a question no one will +answer in a headless eval), CANCELED and UNKNOWN — none of which the closed set ever marks +done, and none of which the poll loop should wait out, since they will never become DONE +on their own. + +## The receive_steps re-entrancy window + +`receive_steps()` is two nested async generators: the public one delegates to the +connection layer, which guards re-entrancy with a flag cleared only in its OWN `finally`. +`aclosing` closes the outer generator deterministically, but a `GeneratorExit` thrown into +a delegating generator does not synchronously propagate into the inner one it was +mid-iterating — confirmed live: the inner `finally` ran only after the outer's frame +unwound AND the loop processed the abandoned generator's finalizer, i.e. on a LATER event +loop turn. So a cooperative-stop `break` can leave the connection "receiving" for a short +bounded window, and the next `receive_steps()` call raises `RuntimeError` inside it. +Retrying with `asyncio.sleep(0)` gives the already-scheduled finalizer a turn, mirroring +the SDK's own handling of this exact error in `Conversation.send()`. Retrying is preferred +over the SDK's `wait_for_idle()` fallback, which discards steps already queued and would +silently drop real content. + +## Why the tool-call id falls back the way it does + +`call.id` is typed optional, and the fallback must be BOTH stable across a step's own +ACTIVE → DONE re-emissions — so an id-less call's DONE step closes the SAME id its ACTIVE +step opened, rather than minting a fresh one from an already-advanced counter and +stranding the ACTIVE entry as a permanent orphan that stalls the poll loop for its full +budget — AND unique across trajectories, since a sub-agent trajectory can reuse the same +low `step_index` values as the main one. It mirrors the SDK's own +`trajectory_id:step_index` scheme rather than inventing a separate one; the call index +further disambiguates multiple id-less calls within one step, which the SDK's scheme does +not. + +## Why only a RESOLVED tool is timed + +On Codex, when both SDK stamps are present ``timestamp`` becomes the tool's own START +rather than the completion instant, which would place the call after its own execution. +Ordering is unaffected either way: ``TurnRecord.commands`` is sorted on +``sequence_number``, not on ``timestamp``. + +An orphan force-closed by the end-of-turn sweep was never observed finishing, so the +instant the sweep runs is not a completion. Stamping it manufactures both an +`execution_completed_at` and the `duration_ms` derived from it, and the pair then reads as +a measured span that the central subtraction takes back out of a generation window it +never occupied. `execution_started_at` IS kept: the harness really did emit that start, +and one bound alone forms no span. Unknown status and unknown duration are one fact +(CE058) — claude-code's `_finalize_commands` leaves the same field `None` for the same +reason, rather than coercing it to `0.0`, which put an invented measurement on both sides +of `avg_command_time_ms`. + +## Tool-name and argument normalization + +Every criterion is written against the canonical (Claude) vocabulary, so each harness maps +its native tool names and per-tool argument keys onto it. Without the map a +`command_executed` with `tool_name: Bash` matches NOTHING on that harness, and the +shell-aware `parameters["command"]` extraction in `criteria/command_executed.py` degrades +to raw-JSON matching — so the same task scores differently per harness. Unknown names pass +through unchanged. + +Three cases are worth knowing: + +- **OpenCode's tool set varies by MODEL within the one harness.** A live 174-task run + showed DeepSeek using `write`/`edit` 199 times and `apply_patch` 0, while GPT-5.6 used + `apply_patch` 120 times and `write`/`edit` 0. Unmapped, every `tool_name: Write` / `Edit` + criterion scores 0 on a GPT-family model that edited the file correctly. +- **OpenCode has MOVED its file-path key.** A live capture emitted `filePath` while the + tool schemas registered by the installed CLI read `path`. Both spellings are mapped, so + telemetry stays canonical across the CLI versions a run might use; neither collides with + a legitimate parameter of those tools. +- **Pi's search tool is `find`** (glob-by-pattern), not `glob`; there is no `glob` tool in + its built-in set, so mapping `find` to the canonical `Glob` is what keeps + `command_executed` and `commands_efficiency` comparable. + +Antigravity additionally strips the result payload out of a tool call's arguments: the +harness folds result fields into the same `args` dict at DONE. Beyond a static key list, +any key that FIRST appears at DONE is treated as a result — which matters because +`skill_triggered` substring-searches every parameter value, so a leaked result could +false-positive. + +## Codex runs full-access on every permission mode + +`coder_eval` owns the isolation boundary either way — a docker container or an ephemeral +per-task tempdir — so Codex's own in-process OS sandbox is redundant. Worse, it actively +breaks on the paths the harness relies on: inside the container Landlock is unavailable, +on constrained CI agents the bwrap re-exec is denied, and on Windows there is no OS +sandbox at all. In each case a read-only or workspace-write run fails its writes silently +and scores 0 with no loud error. Dropping to full-access matches claude-code and +Antigravity, which run with no in-agent OS sandbox; it also keeps network on, so tool +installs work without extra sandbox config. + +The consequence is stated loudly at `start()` for EVERY mode, not just +`bypassPermissions`, so operators are not misled that plan/acceptEdits/default confine +Codex — none of them do. Adversarial or untrusted evals belong on the docker driver; the +tempdir/host driver is a working directory, not a confinement boundary. + +Approval mode is `deny_all` on every permission mode too. The SDK offers only two: +`auto_review`, which puts a SERVER-SIDE reviewer in the loop that can spuriously return +"declined" under gateway load — files silently not written, a failure mode Claude has no +analog for, since its Write/Edit permissions are decided client-side — and `deny_all`, +which despite the name means "run autonomously, never prompt, no reviewer": in-sandbox +operations execute directly and only escalations BEYOND the sandbox are refused. An eval +harness never wants a reviewer that can flake. + +## Codex login-shell PATH restoration + +Codex issues every shell command through a LOGIN shell — `bash -lc` on Linux, `zsh -lc` on +macOS. A login shell re-sources the system profile chain (`/etc/profile`, +`/etc/zprofile`'s path_helper), which unconditionally RESETS PATH and silently drops the +mock-CLI prepend passed through the app-server environment, so bare commands resolve to +the REAL CLIs — real-tenant contamination. + +The per-user dotfiles are sourced AFTER that chain, so a generated per-task HOME gets the +last word and re-prepends the mock dirs. Per-task rather than the user's real dotfiles so +parallel tasks with different mocks cannot collide. zsh selects its dotfiles by `ZDOTDIR` +rather than `HOME`, which is why both are pointed at the generated dir, and why all three +zsh files re-prepend: `/etc/zprofile` resets PATH BETWEEN `.zshenv` and `.zprofile`, and a +sourced user file may reset it again — a duplicate PATH entry is harmless, a lost prepend +is contamination. + +The env `HOME` exists ONLY so bash selects the generated file; the profile's first act is +to export the ORIGINAL home back, so git, npm and every `$HOME`-relative reference keep +working. Codex state (auth, rollout sessions) is pinned separately via `CODEX_HOME`, which +is created first because the binary hard-errors on an explicitly set path that does not +exist — hosts that auth via `CODEX_API_KEY` never ran `codex login`. + +`.bash_profile` mimics bash's first-found chain; the `.profile` twin sources only +`.profile`, since the bash-specific files may contain bashisms a POSIX shell would choke +on. **Known residual gap:** a NESTED bash/sh login shell inside a command re-reads the real +profiles and loses the prepend again. Nested zsh keeps it, because `ZDOTDIR` stays +exported. No-op on Windows, where Codex shells through PowerShell (`-NoProfile`) or +`cmd /c`, neither of which re-sources a profile chain. + +## Reaping the CLI harnesses + +`opencode run` leaves a local server child alive after the CLI exits, and it INHERITS the +stdout pipe — so EOF never arrives on its own, `readline()` would block to the turn +deadline, and signalling only the CLI pid orphans the child. Each invocation therefore runs +in its own session, so its pgid is the CLI's pid and the group holds only what that +invocation spawned; each read races against process exit, and a bounded drain collects the +tail. Sessions are persisted on disk, so killing a turn's server does not lose `--session` +continuity. + +stderr is drained CONCURRENTLY from the moment the CLI starts. Reading it only after exit +deadlocks the pair: a child that fills the ~64 KiB stderr pipe blocks on write, stops +emitting stdout, and never exits, so the turn hangs to its deadline. `docker_runner` dodges +this by merging stderr into stdout; here that would corrupt the nd-JSON, so stderr gets its +own reader. + +`_reap_orphaned_cli` is not merely a leak guard. Two exits reach `communicate`'s `finally` +with the child ALIVE — the `except Exception` crash and an external cancellation — and +neither passes through the graceful `kill()`. `AgentCrashError` is RETRIED, so attempt 2 +would spawn a SECOND CLI against the same sandbox and session while attempt 1 is still +editing the files the criteria are about to score, and whichever writer won would decide +the task's result. It is deliberately synchronous: it runs while a `CancelledError` is +propagating, where any await can itself be cut short. `docker_runner` kills its container +from `finally` for the same reason. + +The single-line read limit is raised because one nd-JSON event can carry a whole tool +result, which blows past `StreamReader`'s default 64 KiB cap and raises `ValueError` +mid-stream, killing the read loop. + +Pi keeps its session dir across `kill()` and removes it only in `stop()`: the +orchestrator's mid-turn backstop calls `kill()`, and dropping the dir there would break +resume across a retried turn. `_cleanup` always calls `stop()` after any `kill()`, so the +tempdir is still reclaimed. + +`_TERM_GRACE_SECONDS` is re-declared at the same value in both nd-JSON harnesses rather +than shared: the CLI-driver hoist that would unify their teardown constants and reducers is +a tracked follow-up. The shared plugin→skills resolver already lives in `agents/_skills.py`, +and `STDOUT_LINE_LIMIT_BYTES`, which IS canonical, is imported. + +## The system_prompt_semantics marker + +Each adapter declares how it treats `agent.system_prompt`: `append` (claude-code's +`claude_code` preset, Codex's `developer_instructions`, Antigravity's +`TemplatedSystemInstructions`, Pi's `--append-system-prompt`), `replace` (a claude-code +judge sub-agent, where the configured prompt IS the entire system prompt), or `unknown` +(OpenCode, which has no CLI knob at all). + +It is recorded per run because runs from BEFORE the marker existed did not share one +regime: claude-code used replace-on-set / empty-on-unset, and Codex silently DROPPED the +field. **A trend dashboard must not pool scores across that boundary**, and an absent +marker reads as a pre-marker run — which is why every adapter spreads the base +`get_environment_info()` first rather than emitting the marker conditionally (CE046). + +claude-code's is the only one derived per config rather than fixed, so it is computed +from the resolved prompt value and never recomputed independently — the persisted regime +cannot disagree with what was sent. + +## Skills, per harness + +A `plugins:` entry is a Claude-plugin root, and only the SKILLS half of it is honored +anywhere — a plugin's agents, hooks, commands and MCP servers have no equivalent outside +claude-code and are dropped. The manifest's `skills` field is read rather than `skills/` +being hardcoded, so a plugin that relocates its skills keeps working. + +- **OpenCode** maps each root to `skills.paths` via `OPENCODE_CONFIG_CONTENT`, which the + CLI merges as a final local-scope layer. That was chosen over writing + `/.opencode/skills/` because it writes nothing into the sandbox that is later + preserved as a run artifact and inspected by file criteria, and does not depend on how + the CLI resolves a project root from `--dir`. Verified orthogonal to `--pure`, which + skips external *plugins*, not configured skill paths. An inherited value is appended to + rather than clobbered, since the host may legitimately configure OpenCode the same way. +- **Pi** passes each as `--skill `. +- **Codex** symlinks (or copies, on Windows) each skill dir into `.agents/skills/`, which + the CLI auto-discovers from the working directory upward. +- **Antigravity** takes search paths natively via `skills_paths` — but those only drive + DISCOVERY. The file-tool allowlist is `workspaces` alone, so the skill roots must appear + there too, or the agent discovers a skill and every read of its `SKILL.md` is denied as + out-of-workspace. + +A bare skills directory is used as-is only when the root declares no `skills/` subdir. +That is deliberately not a fallback for a root that HAS one: `skills.paths` is scanned +recursively and a repo root can contain self-referential symlinks (`UiPath/skills` has +`plugins/uipath -> ..`), which resolves skills through an arbitrary path and silently drops +duplicate names. + +Every way this can come up empty is logged loudly — an unresolved env var, a missing dir, +a root with no `/SKILL.md` under it. A plugin whose skills never reach the agent +still *looks* like a normal run, which is precisely the failure the logging closes: the +run measures the model WITHOUT the skill under test. + +## Why the registry rejects a re-registration + +Re-registering the SAME classes is legitimate (an idempotent built-in reload). Re- +registering a kind with a DIFFERENT implementation is a silent shadow: which agent runs +would depend on entry-point discovery order, which is not stable across environments — a +reproducibility hole. Two plugins must not claim the same `agent.type`. + +The registry is keyed by the kind STRING so a built-in `AgentKind` member and a +plugin-supplied raw string collide on one key (`AgentKind` is a `StrEnum`), which is what +lets a plugin register a brand-new kind that is not an enum member. Its imports are +`TYPE_CHECKING`-only so it imports nothing from `coder_eval` at runtime, keeping the edge +one-way — the plugin loader and the models layer import the registry, never the reverse. +`create_agent` deliberately does not import `coder_eval.plugins` itself for the same +reason; callers reach a config through `parse_agent_config`, which loads them. + +## The threaded watchdog + +`asyncio.wait_for` is not enough for these harnesses: the Claude SDK wraps its subprocess +in anyio cancel scopes that suppress `asyncio.CancelledError`, so cooperative cancellation +does not reliably stop a stuck CLI, and a blocking SDK call lands a cancel only at an await +point. A `threading.Timer` on a daemon OS thread fires even when the event loop is starved +or stuck on subprocess I/O. + +SIGKILL is what actually releases stdout/stdin and unblocks the anyio readers so the async +generator unwinds. claude-code therefore pre-constructs its transport when a timeout is +set — the SDK's default path creates it internally and never exposes the subprocess +handle. That handle is captured in the watchdog CLOSURE rather than read from the agent, +so a stale watchdog from an earlier turn cannot kill a later turn's subprocess. + +Timeout detection checks both the watchdog flag AND the wall clock: the flag-only check +races the watchdog, misreporting a timeout as a generic error when the handler is entered +just before the flag flips. On the happy path only the flag is trusted, so a wall-clock +drift during post-loop cleanup cannot reclassify a successful turn as a timeout. + +`kill_sync` runs on the watchdog thread and must not await. Antigravity's cancel and +disconnect are async-only, so its hook only records intent — the genuine teardown happens +via the asyncio-task cancel the watchdog also delivers and the subsequent exit-stack close. + +## Pi + +`agent_start` can appear MORE THAN ONCE per invocation, because Pi auto-retries a +transient provider error internally — so `agent_end` (which carries `willRetry`) is NOT +terminal. `agent_settled`, or stdout EOF, is; emitting the single `AgentEndEvent` on the +first `agent_end` would cut the turn off mid-retry. + +That retry loop is also why `on_turn_start` closes a dangling `TurnStartEvent` before +opening the next one. A generation aborted mid-turn — a provider error before the +assistant message completed, the defining `willRetry` case — otherwise leaves the stream +carrying N starts and N-1 ends, breaking the one-pair-per-inner-turn contract renderers +depend on. `finalize` closes only the LAST open turn, so it cannot cover this. + +The session id is sanitized because dataset-row tasks have path-shaped ids +(`suite/row_3`, set in `task_loader`) and Pi derives its session file from the id under +`--session-dir` — so a raw `/` resolves to a non-existent subdir and fails the row before +any work is done. diff --git a/src/coder_eval/agents/__init__.py b/src/coder_eval/agents/__init__.py index c56ecf392..010f2a9c4 100644 --- a/src/coder_eval/agents/__init__.py +++ b/src/coder_eval/agents/__init__.py @@ -23,10 +23,9 @@ def register_builtins(registry: type[AgentRegistry]) -> None: # Reference the imported classes so the registration side effect is explicit # and a future refactor that drops the top-level imports fails loudly here. _ = (ClaudeCodeAgent, CodexAgent, AntigravityAgent, OpenCodeAgent, PiAgent, NoOpAgent) - # Rot-protection: the decorators fire on import, but assert the built-ins are - # actually registered so a future lazy-import refactor (which would leave the - # import-cached modules' decorators un-run) fails loudly instead of silently - # registering nothing. + # The decorators fire on import, but assert anyway: a lazy-import refactor + # would leave the import-cached modules' decorators un-run, and this fails + # loudly instead of registering nothing. for kind in ( AgentKind.CLAUDE_CODE, AgentKind.CODEX, diff --git a/src/coder_eval/agents/_logging.py b/src/coder_eval/agents/_logging.py index 6ffbb5744..1feeb9686 100644 --- a/src/coder_eval/agents/_logging.py +++ b/src/coder_eval/agents/_logging.py @@ -36,27 +36,22 @@ def log_raw_sdk_event( ) -> None: """Dump an SDK event verbatim, the instant it arrives, when opted in. - Gated behind ``CODER_EVAL_RAW_SDK_LOG`` so normal runs stay quiet. Emits at - INFO so it shows up in task.log without flipping the whole logger to DEBUG. - Shared by every agent so the dump format is identical across backends. - - For each event we log, in order: - * the caller-supplied ``header_fields`` (e.g. ``type=`` for Claude, - ``method=``/``root_type=`` for Codex), - * the full ``repr(repr_target)`` (untruncated), and - * a sorted ``key=value`` dump of every public attribute of - ``attr_target`` (defaulting to ``repr_target``), so token fields like - ``usage`` / ``model_usage`` are visible exactly as the SDK delivered - them even when ``repr`` is terse. + Gated behind ``CODER_EVAL_RAW_SDK_LOG`` so normal runs stay quiet, and emitted + at INFO so it reaches task.log without flipping the whole logger to DEBUG. + Shared by every agent, so the dump format is identical across backends. + + Logs the caller's ``header_fields``, then the UNTRUNCATED + ``repr(repr_target)``, then a sorted dump of every public attribute of + ``attr_target`` (default ``repr_target``) — so token fields are visible exactly + as the SDK delivered them even when ``repr`` is terse. Args: log: The agent's prefixed logger adapter. repr_target: The object to ``repr()`` in full. - attr_target: The object to introspect for the attribute dump. Defaults - to ``repr_target`` (Codex passes the notification's item root here, - falling back to the notification when the root is absent). - header_fields: Arbitrary ``key=value`` pairs rendered into the header - line, in insertion order. + attr_target: The object to introspect. Codex passes the notification's + item root, falling back to the notification when the root is absent. + header_fields: ``key=value`` pairs rendered into the header line, in + insertion order. """ if not raw_sdk_logging_enabled(): return diff --git a/src/coder_eval/agents/_skills.py b/src/coder_eval/agents/_skills.py index 60bdb11b1..7ca52d6f2 100644 --- a/src/coder_eval/agents/_skills.py +++ b/src/coder_eval/agents/_skills.py @@ -1,14 +1,9 @@ """Shared ``agent.plugins`` -> skills-directory resolver for CLI harnesses. -Both OpenCode (maps each skills dir into ``skills.paths`` in -``OPENCODE_CONFIG_CONTENT``) and Pi (passes each as a ``--skill `` argument) -honor only the *skills* half of a Claude plugin. This module holds that one -resolver so neither agent has to reach into the other's private module for it -(the alternative — ``pi_agent`` importing ``opencode_agent._plugin_skill_dirs`` — -coupled the two harnesses through an implementation-private symbol). +One resolver, so neither OpenCode nor Pi has to reach into the other's private +module for it. -Only the skills half of a plugin is honored. A plugin's agents, hooks, commands -and MCP servers have no CLI equivalent and are dropped by both harnesses. +Rationale: .claude/notes/agents.md § Skills, per harness """ from __future__ import annotations @@ -33,10 +28,8 @@ def _manifest_skill_dirs(root: Path) -> list[Path]: """Skill directories a Claude-plugin root declares, in manifest order. Reads the ``skills`` field of ``/.claude-plugin/plugin.json`` (a string - or a list of strings, each relative to the root) and falls back to the - convention default ``/skills`` when the manifest is absent, unreadable, - or declares none. Honoring the manifest rather than hardcoding ``skills/`` - keeps a plugin that relocates its skills working on both harnesses. + or a list, each relative to the root), falling back to ``/skills`` when + the manifest is absent, unreadable, or declares none. """ manifest = root.joinpath(*_PLUGIN_MANIFEST_RELPATH) declared: list[str] = [] @@ -64,11 +57,10 @@ def _plugin_skill_dirs( """Resolve ``plugins:`` entries to skill-directory paths for a CLI harness. Returns the skills-parent directories (each holding ``/SKILL.md``) that - a ``type: local`` plugin root declares. Shared by OpenCode (``skills.paths`` - in ``OPENCODE_CONFIG_CONTENT``) and Pi (a ``--skill `` argument each); - ``harness`` only labels the diagnostics. Every way this can come up empty is - logged rather than passed over: a plugin whose skills never reach the agent - still *looks* like a normal run, which is precisely the failure this closes. + a ``type: local`` plugin root declares. ``harness`` only labels the + diagnostics. EVERY way this can come up empty is logged rather than passed + over, because a plugin whose skills never reach the agent still *looks* like a + normal run. """ resolved: list[str] = [] for plugin in plugins or []: @@ -90,12 +82,9 @@ def _plugin_skill_dirs( ) continue candidates = [directory for directory in _manifest_skill_dirs(root) if directory.is_dir()] - # A path that is ALREADY a bare skills directory (//SKILL.md) - # has no `skills/` subdir, so use it as-is. Deliberately not a fallback for - # a root that HAS one: `skills.paths` is scanned recursively and a repo - # root can contain self-referential symlinks (UiPath/skills has - # `plugins/uipath -> ..`), which resolves skills through an arbitrary path - # and silently drops duplicate names. + # A path that is ALREADY a bare skills directory has no `skills/` subdir, + # so use it as-is. Deliberately NOT a fallback for a root that HAS one. + # Rationale: .claude/notes/agents.md § Skills, per harness if not candidates: candidates = [root] for directory in candidates: diff --git a/src/coder_eval/agents/antigravity_agent.py b/src/coder_eval/agents/antigravity_agent.py index 209a7e6de..1923c4742 100644 --- a/src/coder_eval/agents/antigravity_agent.py +++ b/src/coder_eval/agents/antigravity_agent.py @@ -1,17 +1,15 @@ """Antigravity agent implementation using the official google-antigravity SDK. -The backend drives Google's Antigravity agent *local harness* (the bundled -``localharness`` binary shipped in the ``google-antigravity`` wheel) via the -SDK's :class:`LocalAgentConfig`. It authenticates against the Gemini Developer -API with ``GEMINI_API_KEY`` and runs entirely on the local machine, editing -files inside the sandbox working directory — so coder_eval's on-disk success -criteria see the agent's writes exactly as they do for Claude / Codex. - -Why this surface (and not the branded ``agy`` CLI or the remote SDK): the -standalone Antigravity CLI cannot authenticate headlessly with a Gemini API key -(only interactive OAuth), and the Interactions API runs in a *remote* cloud -sandbox whose edits never land in our local dir. The local harness is the only -non-deprecated path that satisfies headless + GEMINI_API_KEY + local execution. +Drives Google's Antigravity agent *local harness* (the bundled ``localharness`` +binary) via the SDK's :class:`LocalAgentConfig`, authenticating against the +Gemini Developer API with ``GEMINI_API_KEY`` and running entirely on the local +machine — so coder_eval's on-disk success criteria see the agent's writes exactly +as they do for Claude and Codex. + +It is the only non-deprecated surface that satisfies headless + GEMINI_API_KEY + +local execution: the branded ``agy`` CLI cannot authenticate headlessly with an +API key, and the Interactions API runs in a REMOTE sandbox whose edits never land +in our dir. All SDK imports are lazy (inside ``start`` / helpers), mirroring CodexAgent, so this module imports cleanly when the optional ``[antigravity]`` extra is absent; @@ -73,79 +71,40 @@ logger = logging.getLogger(__name__) -# Recommended Gemini coding model when a task pins no ``agent.model`` and neither -# ``--model`` nor ``ANTIGRAVITY_MODEL`` is set. Gemini 3.5 Flash is Antigravity 2.0's -# default coding model (2026-05) — it outperforms the older Gemini 3.1 Pro on coding / -# agentic benchmarks while running faster; ``medium`` thinking is its daily-driver default. +# Fallback when a task pins no ``agent.model`` and neither ``--model`` nor +# ``ANTIGRAVITY_MODEL`` is set: Antigravity 2.0's own default coding model. _DEFAULT_MODEL = "gemini-3.5-flash" # How often to re-check for progress once an orphaned (backgrounded) tool call is -# detected. receive_steps() returns instantly empty ONLY when the connection is -# already idle with nothing queued -- which is exactly the state right after a -# background job leaves the model idle, so the common re-check is cheap. (It CAN -# still await indefinitely if called while genuinely non-idle work is in flight; -# see the poll loop's own comment in communicate() for that case.) -# Conversation.wait_for_wakeup() is an unimplemented stub on the Local harness -# connection this agent uses (always returns False, regardless of pending state, -# confirmed against the installed SDK's source) — so this file drives its own -# sleep-and-retry poll instead. Not user-configurable: a tuning constant, not a -# feature. +# detected. `Conversation.wait_for_wakeup()` is an unimplemented stub on this +# SDK's Local harness, so the agent polls itself. A tuning constant, not a knob. +# Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll _BACKGROUND_POLL_INTERVAL_SECONDS = 5.0 -# Bound on retrying a receive_steps() call that hits the SDK's re-entrancy guard -# (see _drain()'s docstring) -- each retry yields one event-loop turn via -# asyncio.sleep(0) for the prior drain's already-scheduled generator cleanup to -# land. Confirmed live against the real SDK's generator-delegation shape that -# this clears within 2 turns; this constant carries a 2.5x margin, not a -# separately-tuned budget. +# Retries for a receive_steps() call that hits the SDK's re-entrancy guard; each +# yields one event-loop turn for the prior drain's cleanup to land. Confirmed live +# to clear within 2 turns, so this is a 2.5x margin rather than a tuned budget. +# Rationale: .claude/notes/agents.md § The receive_steps re-entrancy window _RECEIVE_STEPS_REENTRY_RETRIES = 5 -# Fraction of the turn's configured `timeout` the poll loop is allowed to spend -# waiting on a backgrounded tool call, before giving up and finalizing through -# its OWN graceful path (force-close the orphan as unresolved, grade normally) -# instead of running into the ThreadedWatchdog's harder cutoff at `timeout` -# itself. Deliberately a FRACTION of `timeout`, not `timeout` itself: a check -# against the identical value the watchdog uses races it non-deterministically -# for who fires first (the bug an earlier review round removed); a check -# against a smaller fraction is a strictly earlier, non-racing internal -# deadline whose whole purpose is to reliably win that race. 0.8 leaves the -# watchdog a fifth of the turn's budget as margin for this loop's own exit -# bookkeeping (the warning log, finalize()'s force-close/grade pass) to -# complete before the harder cancellation would land anyway. -# -# This bound is what actually matters: without it, a tool call spuriously left -# ACTIVE with no real background job behind it (observed live -- see the final -# validation run) used to finalize immediately pre-fix and grade whatever the -# agent had already produced. Bounding this loop only by a fixed cycle count -# disconnected from `timeout` (as an earlier revision did: 120 * 5s = 600s, -# double the framework's own default `turn_timeout: 300` in -# experiments/default.yaml) makes the graceful path unreachable in practice -- -# the watchdog always wins first, and the SAME spurious-orphan turn now burns -# the full turn timeout before crashing as TurnTimeoutError with zero criteria -# evaluated, a strict regression for that input class. +# Fraction of the turn's `timeout` the poll loop may spend waiting on a +# backgrounded tool call before finalizing through its OWN graceful path +# (force-close the orphan, grade normally). A FRACTION, never `timeout` itself: a +# check against the identical value races the ThreadedWatchdog for who fires +# first, while an earlier deadline reliably wins. +# Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll _POLL_DEADLINE_TIMEOUT_FRACTION = 0.8 -# Cap on poll *cycles* per turn -- the SOLE bound when a task sets no -# run_limits.turn_timeout/task_timeout at all (timeout=None), since -# _POLL_DEADLINE_TIMEOUT_FRACTION has nothing to multiply in that case. Also a -# backstop against a very large configured timeout turning this loop into an -# effectively unbounded wait: 120 * 5s = 10 minutes, ~2x the worst real -# backgrounded-job duration observed in confirmed-broken tasks (60-300s). -# -# Deliberately NOT "break after N consecutive empty polls" instead: the real -# SDK's receive_steps() returns identically empty whether a backgrounded job is -# still genuinely running OR will never resolve at all (confirmed live against -# the installed SDK) -- there is no signal that tells these two cases apart -# except waiting. A consecutive-empty-count small enough to matter would also -# abort real slow jobs (the confirmed cases needed up to ~60 consecutive 5s- -# empty polls before succeeding); one large enough to be safe barely improves -# over this flat cap. A flat, data-grounded cap is the honest option. +# Cap on poll *cycles* -- the SOLE bound when a task sets no timeout at all, and +# a backstop against a very large one. 120 * 5s = 10 minutes, ~2x the worst real +# backgrounded-job duration observed (60-300s). Deliberately NOT "break after N +# consecutive empty polls". +# Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll _MAX_BACKGROUND_POLLS = 120 -# Antigravity builtin tool name -> canonical Claude-ish tool name, so cross-agent -# success criteria (command_executed / commands_efficiency / skill_triggered) and -# reports key on the SAME tool names the Claude / Codex backends emit. Unmapped -# tool names pass through unchanged. +# Antigravity builtin tool name -> the canonical (Claude) vocabulary every +# criterion is written against. Unmapped names pass through unchanged. +# Rationale: .claude/notes/agents.md § Tool-name and argument normalization _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP: dict[str, str] = { "run_command": "Bash", "create_file": "Write", @@ -162,31 +121,23 @@ } # Tool-call arg keys the harness ADDS at completion (the result payload), not -# model-supplied inputs — stripped from CommandTelemetry.parameters and mined for -# the tool result instead. This is the STATIC backstop; the live mapping ALSO -# strips any key that first appears at tool-DONE (see ``_params``), so tool- -# specific result fields (LS ``results``, WebSearch ``summary``) never leak into -# parameters — important because ``skill_triggered`` substring-searches every -# parameter value and a leaked result could otherwise false-positive. +# model-supplied inputs. The STATIC backstop; ``_params`` also strips any key +# that first appears at DONE. A leaked result would false-positive +# ``skill_triggered``, which substring-searches every parameter value. _RESULT_ARG_KEYS: frozenset[str] = frozenset( {"exit_code", "combined_output", "diff_block", "output", "stdout", "stderr", "result", "results", "summary"} ) -# Antigravity per-tool INPUT-arg key -> canonical (Claude-ish) key, so cross-agent -# success criteria (command_executed keys on Bash ``parameters["command"]``; LS on -# ``path``) and reports read the SAME parameter names the Claude/Codex backends -# emit. Keyed by the canonical tool name (post tool-name mapping). Unlisted keys -# pass through unchanged. +# Antigravity per-tool INPUT-arg key -> canonical (Claude) key, keyed by the +# canonical tool name (post tool-name mapping). Unlisted keys pass through. _ANTIGRAVITY_ARG_RENAME: dict[str, dict[str, str]] = { "Bash": {"command_line": "command"}, "LS": {"directory_path": "path"}, } -# google.antigravity.types.Step{Status,Type,Source,Target} VALUES we branch on, -# mirrored as plain strings so this module needs no SDK import (the SDK is an -# optional extra; only ``start()`` touches it). Named constants — not bare string -# literals — so an antigravity StepStatus.ERROR comparison is not mistaken for a -# coder_eval FinalStatus member-name denylist (lint rule CE018). +# Step{Status,Type,Source,Target} VALUES we branch on, mirrored as plain strings +# so this module needs no SDK import. Named constants, not bare literals, so a +# StepStatus.ERROR compare is not mistaken for a FinalStatus denylist (CE018). _STATUS_ACTIVE = "ACTIVE" _STATUS_DONE = "DONE" _STATUS_ERROR = "ERROR" @@ -204,11 +155,11 @@ def _enum_value(x: Any) -> Any: def _to_token_usage(usage: Any, model: str | None) -> TokenUsage: """Map a ``google.antigravity.types.UsageMetadata`` to coder_eval ``TokenUsage``. - Gemini reports ``prompt`` (with ``cached`` as a subset), ``candidates`` (output - excluding thinking) and ``thoughts`` (reasoning). coder_eval's buckets: - uncached input = prompt - cached; cache_read = cached; cache_creation = 0 - (Gemini bills no separate cache-write fee); output = candidates + thoughts - (Gemini bills thinking as output). Cost is rate-carded from the bare model id. + Gemini reports ``prompt`` (with ``cached`` a subset), ``candidates`` and + ``thoughts``; cache_creation is 0 because Gemini bills no cache-write fee, and + output folds in thinking because Gemini bills it as output. + + Rationale: .claude/notes/agents.md § Token accounting, per harness """ prompt = getattr(usage, "prompt_token_count", 0) or 0 cached = getattr(usage, "cached_content_token_count", 0) or 0 @@ -230,13 +181,12 @@ def _to_token_usage(usage: Any, model: str | None) -> TokenUsage: class AntigravityAgent(Agent[AntigravityAgentConfig]): """Implementation of the Agent interface for Google Antigravity (Gemini).""" - # The step loop has a between-steps guard where the cooperative - # ``should_stop`` check runs, so this agent supports early-stop-on-criterion. + # The step loop has a between-steps guard where `should_stop` runs. supports_cooperative_stop: ClassVar[bool] = True - # Antigravity has always appended (TemplatedSystemInstructions wraps - # system_instructions around its own harness prompt), so its runs are - # comparable across the marker boundary. + # TemplatedSystemInstructions wraps system_instructions around the harness's + # own prompt, and always has — so runs ARE comparable across the marker. + # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker system_prompt_semantics: ClassVar[SystemPromptSemantics] = "append" def __init__( @@ -257,17 +207,15 @@ def __init__( self.config = config self.route = route or DirectRoute() self.working_directory: Path | None = None - # The live SDK Agent session + its AsyncExitStack (entered in start(), - # closed in stop()). The exit-stack teardown terminates the localharness - # subprocess, so reaping it is what stop()/kill() rely on. + # The live SDK Agent session + its AsyncExitStack. The exit-stack teardown + # terminates the localharness subprocess, which is what stop()/kill() rely + # on. self._sdk_agent: Any = None self._exit_stack: AsyncExitStack | None = None - # Absolute dirs to prepend to PATH so sandbox mock CLIs shadow real ones - # for the harness's run_command tool — handed to the SDK's per-agent env - # seam at start() (see _harness_env). + # Dirs prepended to PATH so sandbox mock CLIs shadow real ones for the + # harness's run_command tool (see _harness_env). self._env_path_prepend: list[str] = [] - # _state / _iteration / _iteration_was_incremented / pending_turn lifecycle - # bookkeeping lives on the Agent base class (shared defaults + helpers). + # Turn-lifecycle bookkeeping lives on the Agent base class. self._log = PrefixedAdapter(logger, {"prefix": instance_name}) def _effective_model(self) -> str: @@ -277,13 +225,13 @@ def _effective_model(self) -> str: def _resolve_skills_paths(self, plugin_tools_dir: str | None) -> list[str]: """Resolve skill search-path roots for the harness's native ``skills_paths``. - Mirrors the source resolution the Codex backend uses: collect ``type: local`` - plugin paths from ``config.plugins`` (env-expanded) plus the runtime - ``plugin_tools_dir``. For each source, hand the harness the directory that - *directly* parents skill dirs — either ``/skills`` (a plugin-marketplace - / repo root) or ```` itself (already a skills dir) — whichever actually - contains a ``/SKILL.md``. The harness auto-discovers skills under those - roots; no symlinking is needed (unlike Codex, Antigravity takes search paths). + For each ``type: local`` plugin path (env-expanded) plus the runtime + ``plugin_tools_dir``, hands the harness the directory that DIRECTLY parents + skill dirs — ``/skills`` or ```` itself, whichever holds a + ``/SKILL.md``. Unlike Codex, Antigravity takes search paths, so no + symlinking is needed. + + Rationale: .claude/notes/agents.md § Skills, per harness """ sources: list[Path] = [] for plugin in self.config.plugins or []: @@ -297,8 +245,8 @@ def _resolve_skills_paths(self, plugin_tools_dir: str | None) -> list[str]: if path.is_dir(): sources.append(path) else: - # Loud: an unresolved env var (e.g. unset $SKILLS_REPO_PATH) or a - # missing dir silently drops the skills, so the agent runs blind. + # Loud: an unresolved env var or a missing dir drops the skills + # silently, so the agent runs blind. hint = "env var likely unset" if "$" in expanded else "path does not exist" self._log.warning("Plugin skills path did not resolve: %r → %r (%s)", raw, expanded, hint) if plugin_tools_dir and Path(plugin_tools_dir).is_dir(): @@ -330,12 +278,9 @@ def _resolve_workspaces(self, skills_paths: list[str]) -> list[str]: """Workspace roots for the harness's ``workspace_only`` file-tool policy. The sandbox working directory (the write target) plus the resolved skill - roots. ``skills_paths`` only drives skill *discovery*; the file-tool - allowlist is governed solely by ``workspaces``, so the skill roots must - appear here too — otherwise the agent discovers a skill but every read of - its ``SKILL.md`` is denied as out-of-workspace. The roots are bind-mounted - into the sandbox at the same path by the shared docker plugin auto-mount, - mirroring how Claude reads skills from the mounted plugin path. + roots. ``skills_paths`` drives DISCOVERY only; the file-tool allowlist is + ``workspaces`` alone, so a skill root missing here is discovered and then + denied on every read of its ``SKILL.md``. """ return [str(self.working_directory), *skills_paths] @@ -343,16 +288,14 @@ def _harness_env(self) -> dict[str, str] | None: """Per-agent environment for the localharness subprocess (``LocalAgentConfig.env``). Returns the mock-CLI PATH prepend as a one-key overlay, or ``None`` when no - mock dirs are configured (so the SDK spawns with a plain inherited env). The - SDK merges this over ``os.environ`` at spawn (``{**os.environ, **env}``), so - naming only ``PATH`` leaves every other inherited variable untouched. The - same overlay is handed to the harness as its ``run_command`` environment, so - mock CLIs shadow the real ones inside the agent's shell too. + mock dirs are configured. The SDK merges it over ``os.environ`` at spawn, + so naming only ``PATH`` leaves every other inherited variable untouched. + The same overlay becomes the harness's ``run_command`` environment. """ if not self._env_path_prepend: return None - # Match the parent process's own casing (Windows exports ``Path``) so the - # merge overrides the inherited entry instead of adding a sibling key. + # Match the parent's own casing (Windows exports ``Path``) so the merge + # overrides the inherited entry instead of adding a sibling key. path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH") merged = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key) or ""]) return {path_key: merged} @@ -367,20 +310,15 @@ async def start( """Initialize and start the Antigravity agent's local harness session. Args: - working_directory: Path to the sandbox working directory. The primary - ``workspace`` so file writes (and run_command) operate there — - process-cwd-independent, so concurrent host-mode tasks don't race. - Resolved skill roots are added alongside it so the agent can read - skill files — see ``_resolve_workspaces``. - env_path_prepend: Absolute directories to prepend to PATH (typically the - resolved ``SandboxConfig.mock_path_dirs``) so mock CLIs shadow the real - ones for the harness's ``run_command`` tool — same mock-shadowing - contract as the Claude/Codex backends. Delivered through the SDK's - per-agent ``env`` seam (see ``_harness_env``), so concurrent tasks get - genuinely separate environments rather than a time-sliced global one. - plugin_tools_dir: A skills/plugin source root. Resolved (together with - ``config.plugins``) into the harness's native ``skills_paths`` so the - agent can discover and engage UiPath skills — see ``_resolve_skills_paths``. + working_directory: The sandbox dir, and the primary ``workspace`` so + writes and run_command operate there — process-cwd-independent, so + concurrent host-mode tasks don't race. + env_path_prepend: Dirs prepended to PATH so mock CLIs shadow the real + ones (the shared mock-shadowing contract), delivered through the + SDK's per-agent ``env`` seam so concurrent tasks get genuinely + separate environments rather than a time-sliced global one. + plugin_tools_dir: A skills/plugin source root, resolved together with + ``config.plugins`` into the harness's native ``skills_paths``. """ self.working_directory = Path(working_directory) self._env_path_prepend = list(env_path_prepend or []) @@ -396,55 +334,41 @@ async def start( ) from e try: - # GEMINI_API_KEY authenticates the harness. None lets the SDK read it - # from the environment itself (and raise a clear error if truly unset). + # None lets the SDK read GEMINI_API_KEY itself, and raise a clear + # error if truly unset. api_key = os.getenv("GEMINI_API_KEY") or None skills_paths = self._resolve_skills_paths(plugin_tools_dir) cfg = LocalAgentConfig( model=self._effective_model(), api_key=api_key, # File tools are confined to ``workspaces`` by the auto-prepended - # workspace_only policy. Scope to the sandbox workdir (the write - # target — process-cwd-independent so concurrent host-mode tasks - # don't race) PLUS the resolved skill roots, so the agent can READ - # each SKILL.md. ``skills_paths`` only feeds discovery; the file-tool - # allowlist is ``workspaces`` alone, so without the roots here every - # skill read is denied as out-of-workspace. The roots are already - # bind-mounted into the sandbox at the same path by the shared - # docker plugin auto-mount (the path Claude reads skills from too). + # workspace_only policy — see _resolve_workspaces for why the skill + # roots must be in here and not only in ``skills_paths``. workspaces=self._resolve_workspaces(skills_paths), - # Autonomous execution: approve every tool call (incl. run_command), - # which the default LocalAgentConfig policy would otherwise deny. - # ``permission_mode`` is deliberately NOT mapped onto these policies — - # it does not confine this agent, exactly as on Codex. coder_eval's - # isolation boundary is the driver (a docker container or an ephemeral - # per-task tempdir), so an in-agent approval policy is redundant, and - # the modes below bypassPermissions differ only in what they'd ask a - # human about — there is no human on a headless eval path. Declared as - # such in the parity table so it is visible rather than silent. + # Autonomous execution: approve every tool call, which the default + # policy would deny. ``permission_mode`` is deliberately NOT mapped + # here — it does not confine this agent, exactly as on Codex, and + # docs/agents/HARNESS_PARITY.md says so rather than leaving it + # silent. The isolation boundary is the driver. policies=[policy.allow_all()], system_instructions=self.config.system_prompt or None, - # Skill discovery: hand the harness the search-path roots that parent - # the UiPath skill dirs. Unlike Codex (which symlinks into - # .agents/skills/), Antigravity takes skill search paths natively. + # Skill discovery: the search-path roots that parent the skill dirs. skills_paths=skills_paths, - # Mock-CLI PATH shadowing, per agent. The SDK merges this over the - # inherited os.environ when it spawns the localharness, so two - # concurrent tasks never see each other's mock dirs. + # Mock-CLI PATH shadowing, per agent: two concurrent tasks never + # see each other's mock dirs. env=self._harness_env(), ) - # Attach the configured thinking level (reasoning effort) onto every - # resolved model's Gemini endpoint. The SDK validates the model list in - # a model_validator; we set options on the resolved targets after build. + # Thinking level onto every resolved model's endpoint. The SDK + # validates the model list in a model_validator, so options are set on + # the resolved targets after build. level = types.ThinkingLevel(self.config.thinking_level) for target in cfg.models or []: endpoint = getattr(target, "endpoint", None) if isinstance(endpoint, types.GeminiAPIEndpoint): endpoint.options = types.GeminiModelOptions(thinking_level=level) - # Enter the SDK Agent context (boots the localharness subprocess + - # opens the conversation). Held open across communicate() calls and - # closed in stop(). + # Boots the localharness subprocess + opens the conversation. Held + # open across communicate() calls, closed in stop(). self._exit_stack = AsyncExitStack() self._sdk_agent = await self._exit_stack.enter_async_context(SdkAgent(cfg)) self._log.debug("Antigravity local harness started (model=%s)", self._effective_model()) @@ -460,31 +384,14 @@ async def _drain( ) -> None: """Consume one ``receive_steps()`` cycle onto ``state``, honoring a cooperative stop mid-stream. Shared by the initial drain and each poll - cycle's re-drain in ``communicate`` so this shape lives in one place. - - ``receive_steps()`` is actually TWO nested async generators: the public - ``Conversation.receive_steps()`` we call here delegates internally - (``async for step in self._connection.receive_steps(): yield step``) to - the connection layer, which guards re-entrancy with an ``_is_receiving`` - flag cleared only in its OWN ``finally``. ``aclosing`` on the outer - generator closes IT deterministically, but a ``GeneratorExit`` thrown - into a delegating generator does not synchronously propagate into the - inner one it was mid-iterating -- confirmed live: the inner ``finally`` - only ran after the outer's frame was unwound AND the event loop had - processed the abandoned inner generator's async-gen finalizer, i.e. on a - LATER event-loop turn, not within the ``aclosing`` block itself. So a - cooperative-stop ``break`` here can still leave the connection - "receiving" for a short, bounded window afterward, and the NEXT - ``receive_steps()`` call (the next poll cycle, or the next turn in a - multi-turn dialog) can raise ``RuntimeError`` during that window. The - retry below -- yielding via ``asyncio.sleep(0)`` and trying again -- - gives that already-scheduled finalizer a turn to run, mirroring the - SDK's OWN handling of this exact ``RuntimeError`` in - ``Conversation.send()`` (falls back to ``wait_for_idle()``); retrying - the drain itself is preferred here over that fallback since - ``wait_for_idle()`` discards any steps already queued, which would - silently drop real content instead of just retrying past a transient - window. + cycle's re-drain, so this shape lives in one place. + + A cooperative-stop ``break`` can leave the SDK connection "receiving" for + a short bounded window, and the NEXT ``receive_steps()`` call raises + ``RuntimeError`` inside it. The retry below yields an event-loop turn for + the already-scheduled generator finalizer to run. + + Rationale: .claude/notes/agents.md § The receive_steps re-entrancy window """ for attempt in range(_RECEIVE_STEPS_REENTRY_RETRIES): try: @@ -495,10 +402,9 @@ async def _drain( state.stopped_early_hit = True self._log.debug("Cooperative stop requested; ending step loop at this boundary") break - # The turn cap shares this boundary: the step that reached the - # cap is kept whole, the next is never pulled. Checked after - # the cooperative stop so an armed early-stop still reports as - # STOPPED_EARLY when both would fire on the same step. + # The turn cap shares this boundary: the step that reached + # the cap is kept whole, the next is never pulled. After the + # cooperative stop, so an armed early-stop wins a tie. if state.max_turns_reached(): state.max_turns_hit = True self._log.debug( @@ -532,16 +438,13 @@ async def communicate( conversation is cancelled (best-effort) and the turn finalizes cleanly as ``STOPPED_EARLY`` (``crashed=False``). - ``max_turns`` caps VISIBLE turns — tool calls, the unit - ``reports_stats.visible_turn_count`` counts — enforced in-stream on the same - step-loop boundary as the cooperative stop. Claude Code's native SDK cap - counts assistant messages instead; one ``communicate()`` here is a single SDK - turn, so a native counter would cap at 1 and mean nothing. See - docs/agents/HARNESS_PARITY.md. + ``max_turns`` caps VISIBLE turns — resolved tool calls — enforced in-stream + on the same boundary as the cooperative stop: one ``communicate()`` here is + a single SDK turn, so a native counter would cap at 1 and mean nothing. + See docs/agents/HARNESS_PARITY.md. Drives one logical turn: ``conversation.send(prompt)`` then iterate - ``receive_steps()`` until the turn goes idle, mapping the Gemini step - stream onto the standardized event protocol. + ``receive_steps()`` until the turn goes idle. Raises: RuntimeError: If the agent is not started. @@ -554,14 +457,12 @@ async def communicate( assert self.config.type is not None, "AntigravityAgent requires AgentConfig.type before communicate()" self._begin_turn() - # Raw monotonic, and deliberately not the turn clock: this seeds the - # poll deadline below and `duration_seconds`, neither of which may move - # when the wall clock steps. `TurnClock` is for the RECORDED stamps. + # Raw monotonic, deliberately NOT the turn clock: this seeds the poll + # deadline and `duration_seconds`, neither of which may move when the wall + # clock steps. `TurnClock` is for the RECORDED stamps. turn_start_time = time.monotonic() - # ONE clock per turn. This is the (monotonic, wall) pair the reducer - # already captured here and then failed to use for its later stamps — - # which is why its window span was monotonic while its tool intervals - # were wall, and why the two could disagree. + # ONE clock per turn, so the window bounds and the tool intervals + # subtracted from them share a basis. clock = TurnClock() task_id = str(self.config.type) model = self._effective_model() @@ -584,15 +485,10 @@ async def communicate( ) try: - # `timestamp` from the TURN CLOCK, not the event model's raw - # `datetime.now()` default: this bound is subtracted against window - # bounds the same clock produced (`decompose_turn`), and two bases - # in one subtraction is what `TurnClock` exists to remove. Measured - # HERE: this harness's tail came out at -0.017 ms — an end stamped - # 17 us before its own last message finished — which clamped to the - # `0.0` that means "measured, and instant" (CE058). It holds its - # process across turns, so its true tail is ~0.1 ms, which is the - # only scale at which the drift between two clocks can flip a sign. + # From the TURN CLOCK, not the model's raw `datetime.now()` default: + # this bound is subtracted against window bounds the same clock + # produced, and two bases in one subtraction clamped this harness's + # -0.017 ms tail to a measured 0.0 (CE058). emit.on_event( AgentStartEvent( task_id=task_id, @@ -615,34 +511,21 @@ def _on_turn_timeout() -> None: emit.on_event(TurnStartEvent(task_id=task_id, turn_id=turn_id, model=model)) conversation = self._sdk_agent.conversation poll_count = 0 - # Bound the poll loop's OWN exit by a fraction of `timeout` so its - # graceful path (force-close the orphan, grade normally) reliably - # wins the race against the ThreadedWatchdog's harder cutoff at - # `timeout` itself, instead of the watchdog always firing first — - # see _POLL_DEADLINE_TIMEOUT_FRACTION's comment for why a FRACTION - # of `timeout` doesn't race it the way an identical value would. - # `timeout=None` has nothing to derive a fraction from, so the - # cycle-based _MAX_BACKGROUND_POLLS is the sole bound in that case. + # Bound the poll loop's OWN exit earlier than the watchdog's, so + # its graceful path reliably wins that race. `timeout=None` has + # nothing to take a fraction of, so the cycle cap is the sole bound. poll_deadline = turn_start_time + timeout * _POLL_DEADLINE_TIMEOUT_FRACTION if timeout else None try: await conversation.send(user_input) - # The cooperative should_stop poll runs AFTER process_step (the - # emission that lets the watcher latch on the deciding tool - # call) and BEFORE the next step is pulled — the deciding step - # is kept, the next is not. No-op when should_stop is None. + # should_stop runs AFTER process_step (the emission the watcher + # latches on) and BEFORE the next step is pulled. await self._drain(conversation, state, should_stop) - # The model may have kicked off a run_command as a background - # task and gone idle without waiting for it — receive_steps() - # then exhausts with that tool call still open (never reached - # DONE/ERROR). Conversation.wait_for_wakeup() is an unimplemented - # stub on this SDK's Local harness (always returns False, - # regardless of pending state — confirmed against the installed - # source and live-tested), so poll for progress ourselves - # instead, gated on that orphaned-tool signal so a normal turn - # (which always closes its tool calls before the stream - # exhausts) takes this branch zero times and finalizes exactly - # as fast as today. + # The model may background a run_command and go idle, so + # receive_steps() exhausts with that call still open. Gated on + # the orphaned-tool signal, so a normal turn takes this branch + # zero times. + # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll while ( not state.stopped_early_hit and not state.max_turns_hit @@ -658,19 +541,14 @@ def _on_turn_timeout() -> None: self._log.debug("Polling for backgrounded work (orphaned tool call); attempt %d", poll_count) await asyncio.sleep(_BACKGROUND_POLL_INTERVAL_SECONDS) if state.timeout_hit or (poll_deadline is not None and time.monotonic() >= poll_deadline): - # The watchdog decided to fire during the sleep above, or - # this loop's own (earlier) deadline just passed: skip the - # re-drain (which could itself await indefinitely on - # genuinely non-idle work) rather than waiting for the - # loop's own head check to catch it next cycle. + # Skip the re-drain, which could itself await + # indefinitely on genuinely non-idle work. break if should_stop is not None and should_stop(): state.stopped_early_hit = True break - # A re-drain honors the turn cap the same way the initial one - # does (the check lives in _drain), so a poll cycle can also - # be the cycle that reaches it; the loop head above then - # stops polling instead of waiting out the background work. + # A re-drain honors the turn cap too (the check lives in + # _drain), so a poll cycle can be the one that reaches it. await self._drain(conversation, state, should_stop) if ( @@ -679,10 +557,9 @@ def _on_turn_timeout() -> None: and not state.max_turns_hit and not state.timeout_hit ): - # Exited via this loop's own bound (poll_deadline or the - # cycle cap), not an external stop/timeout -- the tool call - # is force-closed as unresolved in finalize() below and the - # turn is still graded normally on everything else. + # Exited via this loop's OWN bound, not an external + # stop/timeout: the call is force-closed as unresolved and + # the turn is still graded normally on everything else. bound = ( f"poll_deadline ({_POLL_DEADLINE_TIMEOUT_FRACTION:.0%} of {timeout:g}s turn timeout)" if poll_deadline is not None @@ -692,10 +569,8 @@ def _on_turn_timeout() -> None: self._log.warning(msg, bound, poll_count) if state.stopped_early_hit or state.max_turns_hit: - # Best-effort server-side cancel, mirrors kill(); a raising - # cancel() lands in the guarded handler below. Single check - # point covers a stop from either the initial drain or any - # poll cycle, so cancel() fires exactly once either way. + # Best-effort server-side cancel. One check point, so it + # fires exactly once whichever drain stopped. with contextlib.suppress(Exception): await conversation.cancel() except asyncio.CancelledError: @@ -706,12 +581,8 @@ def _on_turn_timeout() -> None: if state.timeout_hit: self._finalize_and_raise_timeout(state.finalize, timeout or 0, cause=e) if state.ended_cleanly: - # The turn already stopped cleanly (e.g. the generator's - # aclose() raised on the break); escalating to a crash - # would trigger the orchestrator's retry with the watcher's - # decision still latched → immediate stop-at-turn-0 on the - # retry (wasted spend). A cap-break is the same shape: the - # retry would burn the budget again and re-hit the cap. + # Already stopped on purpose — do not escalate. + # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) else: self._finalize_and_raise_crash( @@ -730,9 +601,7 @@ def _on_turn_timeout() -> None: raise except Exception as e: if state.ended_cleanly and not state.timeout_hit: - # Same retry-poisoning guard as the inner handler: the turn already - # ended cleanly (cooperative stop or turn cap), so finalize instead - # of crashing. + # Same retry-poisoning guard as the inner handler. self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) else: self._finalize_and_raise_crash( @@ -741,10 +610,8 @@ def _on_turn_timeout() -> None: self._state = AgentState.WORKING self._end_turn_ok() - # Precedence matches Claude: timeout (raised above) > stopped_early > - # max_turns_exhausted > completed. stopped_early outranks the cap because an - # armed criterion deciding the outcome is the more specific reason to have - # cut the run, and the step loop checks it first. + # Precedence: timeout (raised above) > stopped_early > max_turns > done. + # Rationale: .claude/notes/agents.md § Shared turn lifecycle if state.stopped_early_hit: status = AgentEndStatus.STOPPED_EARLY elif state.max_turns_hit: @@ -771,9 +638,8 @@ def kill_sync(self) -> None: """Best-effort synchronous abort for the watchdog thread (cannot await). Antigravity's cancel/disconnect are async-only, so the genuine teardown - happens via the asyncio-task cancel the watchdog also delivers (which - unwinds ``receive_steps``) and the subsequent ``stop()`` exit-stack close. - This hook only records intent. + happens via the watchdog's asyncio-task cancel and the subsequent + ``stop()`` exit-stack close. This hook only records intent. """ self._state = AgentState.ERROR @@ -815,7 +681,7 @@ class _AntigravityTurnState: repeatedly through ACTIVE -> DONE transitions; ``usage_metadata`` lands once per generation on a DONE/terminal step (summing them == the turn total); a tool call carries a stable ``id`` and its result is folded into expanded - ``args`` (``exit_code`` / ``combined_output`` / ``diff_block``) at DONE. + ``args`` at DONE. """ def __init__( @@ -842,10 +708,8 @@ def __init__( self.iteration = iteration self.model = model self.turn_start_time = turn_start_time - # Every wall stamp below derives from this, so the tool spans and the - # window bounds they are subtracted from share one basis. Injected, not - # read from a module global, so a test supplies a fake instead of - # monkeypatching `datetime` out from under the reducer. + # Injected, not read from a module global, so a test supplies a fake + # instead of monkeypatching `datetime` out from under the reducer. self.clock = clock self.max_turns = max_turns @@ -860,28 +724,21 @@ def __init__( self._output_parts: list[str] = [] self._assistant_turns = 0 - # Tool tracking: emit ToolStart on first sight of an id; ToolEnd at DONE. + # ToolStart on first sight of an id; ToolEnd at DONE. self._next_seq = 0 self._seen_tools: set[str] = set() self._closed_tools: set[str] = set() self._open_tools: dict[str, CommandTelemetry] = {} - # Raw arg keys present when a tool was first seen (its model-supplied - # inputs). Used at DONE to distinguish inputs from harness-appended - # result fields, whatever they're named for that tool. + # Arg keys present when a tool was first seen (its model-supplied + # inputs), used at DONE to tell them from harness-appended result fields. self._tool_input_keys: dict[str, set[str]] = {} - # Most recently seen StepStatus per tool id, for has_orphaned_tool_call - # below — deliberately separate from _closed_tools, which only tracks - # the DONE/ERROR terminal states relevant to result reporting. + # Most recently seen StepStatus per tool id, for has_orphaned_tool_call. + # Separate from _closed_tools, which tracks only DONE/ERROR. self._tool_last_status: dict[str, Any] = {} # Content blocks accumulated since the last per-generation flush. self._blocks: list[ContentBlock] = [] - # Generation-window mark: where the CURRENT generation started. Set to - # the turn's own start so the first window includes prompt submission - # and connection setup — real time the model call cost, and the same - # choice Claude makes (its mark is also the turn start). Both stamps - # come from the SAME instant, captured by communicate(), so the - # recorded bounds and the measured duration describe one span. - # Advanced only by a flush that actually emitted a message. + # Where the CURRENT generation started, advanced only by a flush that + # actually emitted a message. self._gen_mark_wall: datetime = clock.now() # Re-seeded ONCE, at the first observed Step. See # `_seed_first_generation_window`. @@ -891,69 +748,35 @@ def __init__( def ended_cleanly(self) -> bool: """True once the loop broke on purpose (cooperative stop or the turn cap). - Both are non-crash terminations, so a stray exception raised while unwinding - the step generator afterwards must not be escalated into a retry. + Both are non-crash terminations, so a stray exception raised while + unwinding the step generator afterwards must not be escalated. """ return self.stopped_early_hit or self.max_turns_hit def max_turns_reached(self) -> bool: """True once this turn has produced ``max_turns`` visible turns. - Delegates the count to the collector (``EventCollector.visible_turn_count``) - — the single agent-agnostic capture path, so one ``max_turns`` value means - the same thing here and on Codex. It counts RESOLVED tool calls (the end - event), which also means the call that reaches the cap keeps its result - instead of being force-closed as unresolved. + Delegates to ``EventCollector.visible_turn_count``, the single + agent-agnostic capture path, so one ``max_turns`` means the same thing here + and on Codex. It counts RESOLVED tool calls, so the call that reaches the + cap keeps its result instead of being force-closed as unresolved. """ return self.max_turns is not None and self.collector.visible_turn_count >= self.max_turns def _seed_first_generation_window(self, source: Any) -> None: """Move the first window's mark to the first observed MODEL output. - ``harness_startup_ms`` is defined as the wall clock from the turn - starting until the harness first observed model output, and that instant - is also where the first generation window opens — which is what keeps - the head and the generation disjoint so the four-bucket identity still - closes. - - Without this ``_gen_mark_wall`` is stamped when the turn state is built, - BEFORE ``AgentStartEvent`` is emitted, so the head is a small negative - that ``decompose_turn`` clamps to ``0.0`` — a clamped inversion - published as "measured, and instant", which is the exact confusion CE058 - exists to prevent everywhere else. Everything before the first ``Step`` - — dispatch and time to first token — was booked as the first - generation instead: ~4.7 s per turn on this harness, measured against a - later-window median of 3.3 s. - - What differs from claude-code is not in-process versus subprocess — - this harness spawns a ``localharness`` binary too. It is spawned ONCE, - in ``start()``, and held across every ``communicate()``, so there is no - boot inside a turn for the head to contain: it is dispatch plus time to - first token. claude-code spawns a fresh CLI per turn and so fuses that - boot in. The head means the same thing on both; only its COMPOSITION - differs, which is a real property of the harness rather than a - measurement artifact. - - GATED ON ``source``, because the field is defined as model output and - the SDK streams Steps that are not. ``StepSource`` carries ``SYSTEM`` - and ``USER`` besides ``MODEL``, and ``StepType`` carries - ``SYSTEM_MESSAGE`` / ``COMPACTION`` / ``FINISH``; the SDK's event - processor queues every ``step_update`` verbatim, so a turn can - legitimately open with one. Seeding on such a Step would put the mark - BEFORE the model spoke and hand the remainder back to msg0's - generation, which is the defect this method exists to remove. The same - gate guards text streaming a few lines below, for the same reason. - - ONCE PER TURN, and that is the whole contract. ``process_step`` runs for - every Step in the turn; re-seeding on each would stop the windows tiling - and drop the gap before the next emission into no bucket at all, which - is the defect pi shipped with. The flag needs no reset: a fresh turn - state (and a fresh ``TurnClock``) is built per ``communicate()``, so it - is per-attempt by construction. - - A turn that streams no MODEL Step at all never latches, keeps the - turn-entry mark and clamps to ``0.0`` exactly as before — the same - fail-safe degradation as an unrecognized source. + GATED ON ``source``, because ``harness_startup_ms`` is defined as model + output and the SDK streams Steps that are not: a turn can legitimately open + with a SYSTEM or USER Step, and seeding on one would put the mark BEFORE + the model spoke. The same gate guards text streaming below. + + ONCE PER TURN, and that is the whole contract: re-seeding would stop the + windows tiling. The flag needs no reset — a fresh turn state is built per + ``communicate()``. A turn that streams no MODEL Step keeps the turn-entry + mark and clamps to ``0.0``, which is the correct degradation. + + Rationale: .claude/notes/agents.md § First-generation window seeding """ if self._first_output_seen or _enum_value(source) != _SOURCE_MODEL: return @@ -993,20 +816,10 @@ def process_step(self, step: Any) -> None: def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call_index: int) -> None: raw_name = _enum_value(call.name) - # call.id is usually present ("a tool call carries a stable id" per this - # class's docstring), but the SDK types it as optional. The fallback must - # be BOTH stable across a step's own ACTIVE -> DONE re-emissions (same - # step_index) -- so an id-less call's DONE step closes the SAME cid its - # ACTIVE step opened, rather than minting a fresh id from a counter that - # already advanced, which would strand the ACTIVE entry as a permanent, - # never-closing "orphan" and stall the poll loop for its full budget -- - # AND unique across trajectories: the SDK keys its own step tracking on - # (trajectory_id, step_index), since a sub-agent trajectory can reuse the - # same low step_index values as the main one. Mirrors the SDK's own - # `trajectory_id:step_index` id scheme (falling back to bare step_index - # when trajectory_id is empty, e.g. no sub-agent involved) rather than - # inventing a separate one; call_index further disambiguates multiple - # id-less tool calls within the same step, which the SDK's scheme does not. + # call.id is usually present but the SDK types it optional. The fallback + # mirrors the SDK's own `trajectory_id:step_index` scheme; call_index + # further disambiguates multiple id-less calls within one step. + # Rationale: .claude/notes/agents.md § Why the tool-call id falls back the way it does trajectory_id = getattr(step, "trajectory_id", "") or "" step_key = f"{trajectory_id}:{step.step_index}" if trajectory_id else str(step.step_index) cid = call.id or f"{raw_name}_{step_key}_{call_index}" @@ -1071,12 +884,10 @@ def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call def _params(tool_name: str, args: dict[str, Any], input_keys: set[str] | None) -> dict[str, Any]: """Model-supplied inputs only, renamed to canonical cross-agent keys. - A key is treated as a (dropped) result field when it is in the static - ``_RESULT_ARG_KEYS`` backstop OR — given the input-key snapshot taken at - tool start — it first appeared at DONE (harness-appended output, whatever - the tool names it). Surviving input keys are renamed via - ``_ANTIGRAVITY_ARG_RENAME`` so ``command_executed`` / reports key on the - same names (``command`` / ``path``) the Claude/Codex backends emit. + A key is a (dropped) result field when it is in the static + ``_RESULT_ARG_KEYS`` backstop OR first appeared at DONE, given the + input-key snapshot taken at tool start. Survivors are renamed to the + canonical vocabulary. """ rename = _ANTIGRAVITY_ARG_RENAME.get(tool_name, {}) out: dict[str, Any] = {} @@ -1091,35 +902,18 @@ def _params(tool_name: str, args: dict[str, Any], input_keys: set[str] | None) - def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: """Cut accumulated blocks into one AssistantMessage carrying this gen's tokens. - Keeping per-message token buckets summing to the turn total means the - EventCollector's reconciliation step books a zero residual. + Keeping the per-message buckets summing to the turn total means the + collector's reconciliation books a zero residual. """ if not self._blocks and gen.is_empty(): return now_wall = self.clock.now() - # Do NOT "simplify" this to resetting the mark when a tool ends. That - # loses real model time: measured on run 2026-09-09_04-18-50, task - # skill-rpa-uia-google-search, a harness-local Read closed 8 ms after - # it opened while 6.4 s of model time separated the two flushes around - # it — a reset would have reported 8 ms and dropped the 6.4 s. - # Publishing the RAW window and letting the collector subtract the tool - # union handles that case AND its opposite (a 43 s Bash, where the - # model time really is the flush-to-DONE remainder). - # - # This harness interleaves a tool INTO a window rather than tiling - # around it, so the window legitimately contains time that is not model - # time. `timing.subtract_tool_time` clips the union to these - # bounds and takes it out. Measured here before any of that existed: a - # Bash opening 1.7 ms before the flush drove Sum(generation) + - # Sum(command) 0.26 ms PAST the turn wall, on a turn whose whole - # headroom was 1.4 ms. - # - # The span used to be read off `time.monotonic()` while these intervals - # were wall, and subtracting one from the other is the only reason this - # window could go negative — a clamp that was indistinguishable from a - # real instant generation. Both bounds now derive from `self.clock`, so - # the disagreement is unrepresentable and the branch that hid it is - # gone. + # Do NOT "simplify" this to resetting the mark when a tool ends: this + # harness interleaves a tool INTO a window rather than tiling around it, + # so the RAW window legitimately contains time that is not model time and + # the collector clips the tool union out of it. Resetting instead drops + # the model time around a fast tool. + # Rationale: .claude/notes/agents.md § Per-harness generation marks _, generation_ms = close_window(mark=self._gen_mark_wall, now=now_wall) for i, block in enumerate(self._blocks): block.sequence = i @@ -1137,16 +931,14 @@ def _flush_generation(self, gen: TokenUsage, reasoning_tokens: int) -> None: reasoning_tokens=reasoning_tokens, model=self.model, # The Step stream carries no message id, and the evalboard's - # SAME_EMISSION_GAP_MS fallback cannot split this harness's - # contiguous windows — see docs/agents/HARNESS_PARITY.md. + # gap fallback cannot split contiguous windows. message_id=f"{self.turn_id}-msg-{self._assistant_turns}", ) ) self._assistant_turns += 1 self._blocks = [] - # Advance the mark ONLY after a message was actually appended. The - # early return above means a no-op flush leaves the window open, so a - # later real generation still measures from where it began. + # Advance ONLY after a message was appended: a no-op flush leaves the + # window open, so a later real generation still measures from its start. self._gen_mark_wall = now_wall def _agent_output(self) -> str: @@ -1161,17 +953,12 @@ def has_orphaned_tool_call(self) -> bool: ACTIVE — the structural signature of a backgrounded task the model went idle on without waiting for. See ``communicate``'s poll loop. - Deliberately an ALLOWLIST on ACTIVE, not a denylist on "not yet closed - via _closed_tools" alone: the SDK's StepStatus also has WAITING_FOR_USER - (the harness is blocked on a question that will never be answered in - this headless eval), CANCELED, and UNKNOWN — none of which _closed_tools - ever marks done (that set only tracks DONE/ERROR, the states relevant to - result reporting), but none of which the poll loop should ever wait out - either, since they will never become DONE on their own. Checking the - allowlisted ACTIVE status is what tells these apart from a genuine - in-flight background job. The `not in _closed_tools` guard is layered on - top (not a substitute) purely as a monotonicity backstop, in case a - closed id's last-seen entry were ever left at ACTIVE by a re-emission. + An ALLOWLIST on ACTIVE, never a denylist on "not yet closed": the SDK also + has WAITING_FOR_USER, CANCELED and UNKNOWN, none of which the poll loop + should wait out. The `not in _closed_tools` guard is layered on top as a + monotonicity backstop, not a substitute. + + Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll """ return any(cid not in self._closed_tools and s == _STATUS_ACTIVE for cid, s in self._tool_last_status.items()) @@ -1203,9 +990,8 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso if self._blocks: self._flush_generation(TokenUsage(), 0) - # AgentEndStatus and TurnEndStatus are parallel by value; convert directly - # (mirrors the Codex sibling) so an unmapped future member raises loudly - # instead of silently bucketing to COMPLETED. + # Parallel by value, so an unmapped future member raises loudly instead + # of silently bucketing to COMPLETED. turn_status = TurnEndStatus(status.value) self.emit.on_event( @@ -1232,8 +1018,7 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso crash_reason=crash_reason, max_turns_exhausted=status is AgentEndStatus.MAX_TURNS_EXHAUSTED, duration_seconds=time.monotonic() - self.turn_start_time, - # One basis with the window bounds — see the AgentStartEvent - # site in `communicate`. + # One basis with the window bounds — see the AgentStartEvent site. timestamp=self.clock.now(), ) ) diff --git a/src/coder_eval/agents/claude_code_agent.py b/src/coder_eval/agents/claude_code_agent.py index cfcd5843d..c773e981e 100644 --- a/src/coder_eval/agents/claude_code_agent.py +++ b/src/coder_eval/agents/claude_code_agent.py @@ -21,11 +21,11 @@ query, ) -# Private SDK import — the public `query()` API doesn't expose the subprocess -# handle, but we need it to SIGKILL on timeout (the SDK's anyio task groups -# swallow asyncio cancellation, so cooperative cancel doesn't preempt a stuck -# CLI). If this import breaks on an SDK upgrade, the threaded watchdog loses -# its kill target and timeouts will no longer be enforced at the agent layer. +# Private SDK import: the public `query()` API does not expose the subprocess +# handle, which the watchdog needs to SIGKILL on timeout. If this breaks on an +# SDK upgrade the watchdog loses its kill target and timeouts stop being +# enforced at the agent layer. +# Rationale: .claude/notes/agents.md § The threaded watchdog from claude_agent_sdk._internal.transport.subprocess_cli import SubprocessCLITransport # SystemPromptPreset is not re-exported from the SDK root, so claude_agent_sdk.types @@ -107,16 +107,13 @@ def _is_text_block(block: Any) -> bool: def _distribute_output_tokens(total: int, weights: list[int]) -> list[int]: """Split a call's output_tokens across its block-emissions by content weight. - The Anthropic API reports output_tokens per API *call*, not per content - block, but the CLI surfaces one call as several per-block emissions. To make - each emission's recorded output sensible (rather than dumping the whole - call on the first block and zeroing the rest), we apportion the call total - across emissions by a content-length proxy (thinking/text length, or tool - name + serialized args length). + The API reports output_tokens per API *call*, but the CLI surfaces one call + as several per-block emissions, so the total is apportioned by a + content-length proxy rather than dumped on the first block. - Uses the largest-remainder (Hamilton) method so the returned integers sum - EXACTLY to ``total`` — per-message output stays reconcilable with the - iteration aggregate. Falls back to an even split when all weights are zero. + Largest-remainder (Hamilton), so the returned integers sum EXACTLY to + ``total`` and per-message output stays reconcilable with the aggregate. Falls + back to an even split when all weights are zero. """ n = len(weights) if n == 0: @@ -162,13 +159,11 @@ def _tool_result_text(content: Any) -> str: def _is_task_notification(message: Any) -> bool: """Check if message is a TaskNotificationMessage (sub-agent terminal event). - It is a SystemMessage carrying per-sub-agent ``usage`` (a TaskUsage), keyed - by the spawning ``tool_use_id``. It also has ``session_id`` + ``usage``, so - it would otherwise be misread as the final ResultMessage — hence the - explicit guard, checked before ``_is_sdk_result_message``. Identified by the - SDK type or ``subtype`` (both reliably present on the real message); we avoid - attribute-presence sniffing so it can't misfire on test mocks. The ``subtype`` - fallback also lets duck-typed mocks (not real SDK instances) be recognized. + It carries ``session_id`` + ``usage``, so it would otherwise be misread as + the final ResultMessage — hence this guard, checked FIRST. Identified by the + SDK type or ``subtype`` rather than attribute-presence sniffing, so it cannot + misfire on a mock; the ``subtype`` fallback is what lets a duck-typed mock be + recognized. """ return isinstance(message, TaskNotificationMessage) or getattr(message, "subtype", None) == "task_notification" @@ -188,16 +183,13 @@ def _is_sdk_result_message(message: Any) -> bool: class _ClaudeTurnState: """Per-turn mutable scratch state for one ``ClaudeCodeAgent.communicate`` call. - Holds every cross-branch local the SDK-stream pump mutates and exposes one - method per message kind (``on_*``) plus ``dispatch`` (the type-dispatch - ladder, order-preserving) and ``finalize`` (terminal ``AgentEndEvent`` + - partial-record build). A plain module-private class (NOT Pydantic, NOT - exported) — it carries a back-reference to the agent so it can reuse the - agent's existing helpers (``_finalize_commands`` / ``_build_token_usage`` / - ``_format_messages`` / …). The two raw lists are kept DISTINCT and typed - separately: ``messages`` (raw SDK ``Message`` objects, fed to - ``_update_state_from_messages`` / ``_format_messages``) and ``sdk_messages`` - (telemetry ``TranscriptMessage`` objects, carried on ``AgentEndEvent``). + Holds every cross-branch local the SDK-stream pump mutates, with one method + per message kind plus ``dispatch`` and ``finalize``. A back-reference to the + agent lets it reuse the agent's helpers. + + The two raw lists are DISTINCT and must stay so: ``messages`` holds raw SDK + ``Message`` objects and ``sdk_messages`` holds telemetry + ``TranscriptMessage`` objects, carried on ``AgentEndEvent``. """ def __init__( @@ -245,21 +237,12 @@ def __init__( self.sequence_number = 0 self.last_assistant_message_index: int | None = None - # ONE clock per turn, and every wall stamp this turn records derives - # from it — the window bounds below, the tool spans - # `_resolve_pending_command` stamps, the fallback tool timestamp. The - # central subtraction clips those WALL tool spans to these WALL window - # bounds, so the two sharing one basis is what keeps the arithmetic - # meaningful; before `TurnClock` they shared only naive-LOCAL - # `datetime.now()`, which a DST transition or an NTP step inside a turn - # lands directly in a generation window — an hour-long jump in a - # millisecond field, on runs that start at 04:18 and last hours. - # Injectable so a test supplies a fake rather than monkeypatching this - # module's `datetime` global, which a derived stamp silently escapes — - # leaving the test passing against the real clock. - # `turn_start_time` stays raw monotonic and is untouched: - # `duration_seconds` and the turn deadline read it, and a deadline must - # not move when the wall clock steps. + # ONE clock per turn, and every wall stamp this turn records derives from + # it, so the central subtraction clips WALL tool spans to WALL window + # bounds. Injectable so a test supplies a fake rather than monkeypatching + # this module's `datetime`, which a derived stamp silently escapes. + # `turn_start_time` stays raw monotonic: a deadline must not move when the + # wall clock steps. self.clock = clock or TurnClock() self.last_event_wall: datetime = self.clock.now() # Re-seeded ONCE, at the first observed model output. See @@ -311,8 +294,8 @@ def turn_tokens(self, turn_id: str) -> TokenUsage | None: def dispatch(self, message: Message) -> None: """Record the raw message and route it to its per-kind handler. - Order is load-bearing: ``_is_sdk_result_message`` must be checked before - ``_is_user_message``; the TaskNotification guard before the result guard. + ORDER IS LOAD-BEARING: ``_is_sdk_result_message`` before + ``_is_user_message``, and the TaskNotification guard before both. """ self.messages.append(message) msg_type = type(message).__name__ @@ -442,12 +425,9 @@ def on_assistant_message(self, message: Message) -> None: out_tok = int(msg_usage.get("output_tokens", 0) or 0) self.pending_delta_output_tokens = None - # The RAW window. Tool execution comes out of it once, centrally, in - # `EventCollector.build_turn_record` — so this harness now asks the same - # helper as the other four and CE061 no longer needs its one permanent - # exception. The mark is the only harness-shaped decision left, and it - # stays here: `started` is the mark, since this stream carries no + # The RAW window. `started` is the mark, since this stream carries no # per-emission item start to pull the window open to. + # Rationale: .claude/notes/agents.md § Per-harness generation marks started, raw_generation_ms = close_window(mark=generation_started_wall, now=message_arrival_wall) assistant_telemetry = AssistantMessageTelemetry( started_at=started, @@ -478,9 +458,9 @@ def on_assistant_message(self, message: Message) -> None: self.last_event_wall = message_arrival_wall def on_task_notification(self, message: Message) -> None: - """TaskNotification carries lossy per-sub-agent usage; we capture it from - the Agent tool-result instead. This guard exists only to keep - ``_is_sdk_result_message`` from misreading it (session_id + usage).""" + """TaskNotification carries LOSSY per-sub-agent usage, captured from the + Agent tool-result instead. This exists only to keep + ``_is_sdk_result_message`` from misreading it.""" pass def on_result_message(self, message: Message) -> None: @@ -516,52 +496,18 @@ def on_result_message(self, message: Message) -> None: def _seed_first_generation_window(self) -> None: """Move the first window's mark to the first observed model output. - ``harness_startup_ms`` is defined as the wall clock from the turn - starting until the harness first observed model output, and that instant - is also where the first generation window opens — which is what keeps - the head and the generation disjoint so the four-bucket identity still - closes. - - Without this the mark is stamped in ``__init__``, BEFORE - ``AgentStartEvent`` is emitted, so the head comes out negative and - ``decompose_turn`` clamps it to ``0.0``. The fault is NOT the clamp — - that function's own docstring is right that a measured inversion is a - real zero, because both ends were observed. The fault is that the head - was measured against the WRONG INSTANT: the mark sat before the turn - bracket rather than at the first observed model output, so the interval - being measured was not the one the field is defined as. Everything the - CLI spent booting, resolving a provider and reaching its first token was - booked as msg0's generation instead: ~3.6 s per turn on this harness, - inflating every generation figure, the Generation split and the 10 s - slow-generation bar. - - The old rejection rested on this harness running the model in-process. - It does not: ``claude-agent-sdk`` spawns the ``claude`` CLI over - ``anyio.open_process`` and ``_pump_messages`` calls ``query()`` once - per ``communicate()`` — a fresh CLI per turn, the same shape as codex, - opencode and pi. - - ONCE PER TURN, and that is the whole contract. ``message_start`` arrives - for every API call in the turn; re-seeding on each would stop the - windows tiling and drop the gap before the next emission — a tool result - landing, then the next request going out — into no bucket at all, which - is the defect pi shipped with. The flag needs no reset: a fresh - ``_ClaudeTurnState`` is built per ``communicate()``, so it is - per-attempt by construction. If a future harness reuses a turn state, - the reset belongs there and not here. - - A turn with no ``message_start`` — a mocked ``query()``, a crash before - the first event — never calls this, keeps the turn-entry mark and clamps - to ``0.0`` exactly as before. That is the correct degradation rather - than a gap. - - One route to it is OPERATOR-REACHABLE and worth knowing: this harness - sets ``include_partial_messages=True`` BEFORE spreading - ``**self.config.sdk_options``, so + ONCE PER TURN, and that is the whole contract: re-seeding on every + ``message_start`` would stop the windows tiling and drop the gap before + the next emission into no bucket. The flag needs no reset — a fresh + ``_ClaudeTurnState`` is built per ``communicate()``. A turn with no + ``message_start`` keeps the turn-entry mark and clamps to ``0.0``, which is + the correct degradation. + + One route to that degradation is OPERATOR-REACHABLE: ``-D agent.sdk_options.include_partial_messages=false`` turns the raw - stream off, and with it this re-seed — the head silently returns to the - clamped ``0.0`` it used to publish. Nothing warns; the degradation is - safe but the number changes meaning. + stream off and with it this re-seed. Nothing warns. + + Rationale: .claude/notes/agents.md § First-generation window seeding """ if self.first_output_seen: return @@ -570,7 +516,7 @@ def _seed_first_generation_window(self) -> None: def on_stream_event(self, message: Message) -> None: """Recover cumulative output_tokens from raw ``message_start`` / - ``message_delta`` stream events (handles both sub-cases internally).""" + ``message_delta`` stream events.""" evt: dict[str, Any] = getattr(message, "event", None) or {} evt_type = evt.get("type") if evt_type == "message_start": @@ -596,30 +542,13 @@ def on_stream_event(self, message: Message) -> None: def on_user_message(self, message: Message) -> None: """Process tool results (and a sub-agent's terminal generation) from a tool-result UserMessage. The sub-agent message is appended BEFORE the - tool-result loop — its position in ``sdk_messages`` is observable.""" - # The generation mark is DELIBERATELY NOT advanced here. It used to be - # reset to `self.clock.now()`, which opened the next window at the - # instant the tool RESULT arrived rather than tiling it from the - # previous window's close — so everything between the tool finishing - # and its result reaching this handler (SDK transport, CLI processing, - # next-request dispatch) fell into no bucket at all. Measured on - # `tasks/dataset_example.yaml`: a 21.5 ms `Write` followed by a 2511.7 ms - # round trip, which is 21% of an 11.7 s turn accounted to nothing and - # the reason CI's residual gate failed on that task while a - # `sleep`-heavy probe read 0.05%. A tool-heavy shape cannot see this: - # the tool union absorbs the interval. A fast tool leaves it exposed. - # - # Leaving the mark where `on_assistant_message` put it makes the next - # window run from the previous emission's arrival, so the windows tile - # the turn contiguously — the same rule pi follows with `gen_mark`, and - # the one pi was explicitly fixed for. - # - # The tool's OWN interval is not double-counted by this: it is a - # separate bucket, and `streaming/collector.py::subtract_tool_time` - # clips the tool union out of every window it overlaps, once, for all - # five harnesses. That is exactly why the mark can be left alone here — - # the reducer no longer has to carve the tool out of its own windows. + tool-result loop — its position in ``sdk_messages`` is observable. + + The generation mark is DELIBERATELY NOT advanced here: leaving it where + ``on_assistant_message`` put it is what makes the windows tile. + Rationale: .claude/notes/agents.md § Per-harness generation marks + """ sub_msg = self._agent._synthesize_subagent_terminal_message(message, self.sdk_model_used) if sub_msg is not None: self.sdk_messages.append(sub_msg) @@ -662,12 +591,10 @@ def on_user_message(self, message: Message) -> None: def _finalize_token_usage(self) -> TokenUsage: """Build the turn's cumulative TokenUsage, repricing for LiteLLM. - Extracted from ``finalize`` so the LiteLLM repricing *wiring* (not just the - static ``_reprice_for_litellm`` helper) is directly testable. The SDK's - cost estimate assumes Claude pricing and is wrong for an open-weight model - behind LiteLLM, so reprice the top-line from the token buckets at the - model's real rate — buckets untouched, so the reconciliation invariant - holds. + Extracted from ``finalize`` so the LiteLLM repricing *wiring*, not just + the helper, is directly testable. + + Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ usage = ( self._agent._build_token_usage( @@ -762,17 +689,15 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): """Implementation of the Agent interface for Claude Code using the SDK.""" - # The Claude message loop has a between-messages guard where the cooperative - # ``should_stop`` check runs, so this agent supports early-stop-on-criterion. + # The message loop has a between-messages guard where `should_stop` runs. supports_cooperative_stop: ClassVar[bool] = True - # This agent's __init__ accepts cost_log_tags and stamps them into - # ANTHROPIC_CUSTOM_HEADERS for the proxy-side actual-cost join (LiteLLM backend). + # __init__ accepts cost_log_tags and stamps them into ANTHROPIC_CUSTOM_HEADERS + # for the proxy-side actual-cost join (LiteLLM backend). supports_cost_log_tags: ClassVar[bool] = True - # One warning per agent for a replace-mode config with no prompt to replace - # with: _resolve_system_prompt() runs on every query and once per env-info - # snapshot, and a per-turn repeat would bury the rest of task.log. + # One warning per agent for a replace-mode config with no prompt: the resolver + # runs on every query, and a per-turn repeat would bury the rest of task.log. _warned_prompt_mode_downgrade: bool = False def __init__( @@ -789,48 +714,37 @@ def __init__( Args: config: Agent configuration route: API routing configuration. If None, uses DirectRoute. - instance_name: Short label used to prefix this instance's log - records (e.g. ``"coder"`` for the coding agent, - ``"simulator"`` for the tools-disabled user-simulator agent). - Lets you tell them apart in ``task.log`` when both run in - the same process. - extra_mcp_servers: Runtime-only in-process MCP servers (e.g. the - judge ``submit_verdict`` tool) merged into ``ClaudeAgentOptions.mcp_servers``. - NOT sourced from YAML — ``mcp_servers`` is in - ``_FRAMEWORK_OWNED_SDK_FIELDS`` and explicitly denied via - ``sdk_options`` for security. The judge criterion is the only - caller today. - cost_log_tags: LiteLLM-only correlation headers (``x-ce-run-id`` / - ``x-ce-task-id`` / ``x-ce-attempt``, with ``x-ce-iteration`` appended - per turn) stamped into ``ANTHROPIC_CUSTOM_HEADERS`` so a proxy-side - cost callback can attribute each call's real cost back to this run. - None on Direct/Bedrock. + instance_name: Short label prefixing this instance's log records, so + the coding agent and the user-simulator are distinguishable in + ``task.log`` when both run in the same process. + extra_mcp_servers: Runtime-only in-process MCP servers merged into + ``ClaudeAgentOptions.mcp_servers``. NOT sourced from YAML — + ``mcp_servers`` is explicitly denied via ``sdk_options`` for + security. The judge criterion is the only caller today. + cost_log_tags: LiteLLM-only correlation headers stamped into + ``ANTHROPIC_CUSTOM_HEADERS`` so a proxy-side cost callback can + attribute each call's real cost back to this run. None on + Direct/Bedrock. """ self.config = config self.route = route or DirectRoute() self._extra_mcp_servers = extra_mcp_servers or {} - # Correlation headers stamped on every SDK->proxy request (LiteLLM route - # only), so a proxy-side cost-logging callback can join each call's real - # usage.cost + cache buckets back to this run/task. None => no header - # (Direct/Bedrock, or when the orchestrator supplies none). This turn's - # iteration is appended per-communicate() in _build_claude_query. + # Correlation headers stamped on every SDK->proxy request (LiteLLM only). + # This turn's iteration is appended per-communicate(). self._cost_log_tags = cost_log_tags self.client: ClaudeSDKClient | None = None self.working_directory: Path | None = None - # _state / _iteration / _iteration_was_incremented / pending_turn lifecycle - # bookkeeping lives on the Agent base class (shared defaults + helpers). + # Turn-lifecycle bookkeeping lives on the Agent base class. self._sdk_options_dump: dict[str, Any] | None = None self._session_id: str | None = None - # Transport reference held only while a communicate() call is in flight, - # so kill() can reach into the CLI subprocess when the SDK swallows - # asyncio cancellation. + # Held only while a communicate() call is in flight, so kill() can reach + # the CLI subprocess when the SDK swallows asyncio cancellation. self._active_transport: SubprocessCLITransport | None = None self._env_path_prepend: list[str] = [] self._plugin_tools_dir: str | None = None self._log = PrefixedAdapter(logger, {"prefix": instance_name}) - # Deduplicate "unhandled SDK message type" warnings per agent - # instance — _format_messages runs many times per task and these - # types are stable for the lifetime of a session. + # Dedupe "unhandled SDK message type" warnings: _format_messages runs + # many times per task and the types are stable for a session. self._warned_unknown_types: set[str] = set() async def start( @@ -866,16 +780,14 @@ def _build_sdk_env( Args: route: API routing configuration. - path_prepend: Absolute directories to prepend (in order) to PATH so their - contents shadow same-named binaries in the parent PATH. Resolved by the - sandbox manager from ``SandboxConfig.mock_path_dirs``; the agent does no - filesystem inspection of its own. - plugin_tools_dir: Fallback canonical ``node_modules/@uipath`` to export as - ``PLUGIN_TOOLS_DIR`` when the process environment doesn't already - provide one. An external ``PLUGIN_TOOLS_DIR`` always wins. - cost_log_tags: LiteLLM-only correlation headers stamped (newline-separated - ``Name: Value``) into ``ANTHROPIC_CUSTOM_HEADERS``. Values must be - single-line ASCII (validated here) — the header block is CR/LF-delimited. + path_prepend: Directories prepended (in order) to PATH so they shadow + same-named binaries. Resolved by the sandbox manager; the agent + does no filesystem inspection of its own. + plugin_tools_dir: Fallback ``PLUGIN_TOOLS_DIR``; an external one wins. + cost_log_tags: LiteLLM-only correlation headers stamped + (newline-separated ``Name: Value``) into + ``ANTHROPIC_CUSTOM_HEADERS``. Values MUST be single-line ASCII, + validated here — the header block is CR/LF-delimited. Returns: Tuple of (env_vars_dict, model_override_or_None). @@ -888,8 +800,8 @@ def _build_sdk_env( prefix = os.pathsep.join(path_prepend) base_env["PATH"] = f"{prefix}{os.pathsep}{base_env.get('PATH', '')}" - # Pin UiPath CLI plugin discovery for the agent SDK subprocess. External - # env wins over sandbox-derived fallback so operators can override. + # Pin plugin discovery for the SDK subprocess; external env wins so + # operators can override. if tools_dir := os.environ.get("PLUGIN_TOOLS_DIR"): base_env["PLUGIN_TOOLS_DIR"] = tools_dir elif plugin_tools_dir: @@ -897,13 +809,10 @@ def _build_sdk_env( match route: case BedrockRoute() as br: - # `or ""` rather than asserting non-None here (unlike judge_bedrock.py's - # invoke_bedrock_judge_async): reaching a BedrockRoute at all already implies - # validate_api_keys()/resolve_route() confirmed the token upstream, and this - # is a pure env-dict builder with no error-reporting seam of its own — an - # empty token still produces a clear downstream SDK auth failure rather than - # a crash here. Kept deliberately lenient; do not "fix" to assert without - # also deciding how the resulting AssertionError should surface to the caller. + # `or ""` rather than asserting: reaching a BedrockRoute implies the + # token was confirmed upstream, and this is a pure env-dict builder + # with no error-reporting seam. Do not "fix" to an assert without + # deciding how the AssertionError should surface. env: dict[str, str] = { "CLAUDE_CODE_USE_BEDROCK": "1", "AWS_BEARER_TOKEN_BEDROCK": settings.aws_bearer_token_bedrock or "", @@ -920,12 +829,9 @@ def _build_sdk_env( case DirectRoute() as dr: # Neutralize inherited Bedrock creds: the CLI auto-selects Bedrock - # DIRECT when AWS_BEARER_TOKEN_BEDROCK is present in the inherited - # environment (same auto-selection the LiteLLM arm above guards - # against), so an explicit `route: direct` (e.g. via - # checker_context.api_route.route on a run whose agent is on - # Bedrock) would otherwise silently spend the operator's Bedrock - # bearer token instead of ANTHROPIC_API_KEY (PR #137 review). + # DIRECT whenever AWS_BEARER_TOKEN_BEDROCK is inherited, so an + # explicit `route: direct` would otherwise silently spend the + # operator's Bedrock token instead of ANTHROPIC_API_KEY. env = { "AWS_BEARER_TOKEN_BEDROCK": "", "CLAUDE_CODE_USE_BEDROCK": "", @@ -935,28 +841,23 @@ def _build_sdk_env( return {**base_env, **env}, dr.model case LiteLLMRoute() as cr: - # Point the SDK at the custom Anthropic-compatible endpoint (e.g. - # a LiteLLM gateway). These override any inherited value: the SDK - # merges {**os.environ, ..., **options.env} at spawn, so setting - # them here wins over the parent environment. + # Point the SDK at the custom Anthropic-compatible endpoint. These + # override any inherited value: the SDK merges + # {**os.environ, ..., **options.env} at spawn. env = { "ANTHROPIC_BASE_URL": settings.litellm_base_url or "", "ANTHROPIC_AUTH_TOKEN": settings.litellm_auth_token or "", - # Neutralize any inherited ANTHROPIC_API_KEY: auth on this - # route is the bearer ANTHROPIC_AUTH_TOKEN, and a stray - # x-api-key (e.g. a real Anthropic key exported from .env) - # would conflict with the gateway's key auth. + # Auth here is the bearer ANTHROPIC_AUTH_TOKEN; a stray + # x-api-key would conflict with the gateway's key auth. "ANTHROPIC_API_KEY": "", - # Claude Code attaches usage-attribution metadata (metadata.user_id) - # that Bedrock's requestMetadata regex rejects (HTTP 400) once LiteLLM - # forwards it to Bedrock. Disable it, mirroring the BedrockRoute case above. + # The attribution metadata Claude Code attaches is rejected by + # Bedrock's requestMetadata regex (HTTP 400) once LiteLLM + # forwards it. "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", - # Neutralize inherited Bedrock creds. The CLI auto-selects Bedrock DIRECT - # when AWS_BEARER_TOKEN_BEDROCK is present (`if(process.env.AWS_BEARER_TOKEN_BEDROCK)`), - # and that token is forwarded into docker task containers via the default - # env-passthrough allowlist — so without blanking it the CLI bypasses - # ANTHROPIC_BASE_URL (the LiteLLM proxy) and calls Bedrock directly. Empty string - # is falsy in the CLI's check, so this forces it back onto the gateway. + # Same auto-selection as the DirectRoute arm, and that token IS + # forwarded into docker task containers by the default + # env-passthrough allowlist — so without blanking it the CLI + # bypasses the LiteLLM proxy entirely. Empty is falsy there. "AWS_BEARER_TOKEN_BEDROCK": "", "CLAUDE_CODE_USE_BEDROCK": "", } @@ -965,17 +866,12 @@ def _build_sdk_env( if cr.small_model: env["ANTHROPIC_SMALL_FAST_MODEL"] = cr.small_model if cost_log_tags: - # Stamp every SDK->proxy request with correlation headers so a - # LiteLLM logging callback can attribute each call's real - # usage.cost + cache buckets back to this run/task/turn. Claude - # Code forwards ANTHROPIC_CUSTOM_HEADERS (newline-separated - # `Name: Value`) verbatim, incl. to a non-anthropic base URL. - # - # Sanitize at the seam: x-ce-task-id carries the author-defined - # task_id/variant_id, so a value with a CR/LF would inject extra - # headers into every SDK->proxy request (forged cost attribution, - # or an auth/routing header override). Non-ASCII also breaks the - # latin-1 header encoding. Reject both loudly rather than emit them. + # SANITIZE AT THE SEAM. x-ce-task-id carries the author-defined + # task_id/variant_id, and Claude Code forwards this block + # verbatim — so a CR/LF would inject extra headers into every + # SDK->proxy request (forged cost attribution, or an auth / + # routing override). Non-ASCII breaks the latin-1 encoding. + # Reject both loudly rather than emit them. for name, value in cost_log_tags.items(): joined = f"{name}{value}" if "\r" in joined or "\n" in joined or not joined.isascii(): @@ -990,12 +886,10 @@ def _resolve_effective_model( ) -> str | None: """Resolve the effective model and sync subprocess env on Bedrock. - Precedence: config_model (task YAML / --model / -D agent.model) wins - over the route default (BEDROCK_MODEL). On a Bedrock route, a bare alias - is auto-qualified with ``anthropic.`` and the region's inference-profile - prefix (``eu.``/``us.``/``apac.``) so the same value works across regions. - On Bedrock, the resolved value is always written to ``ANTHROPIC_MODEL`` - so the subprocess sees the same model as ``ClaudeAgentOptions.model``. + Precedence: ``config_model`` wins over the route default. On Bedrock a + bare alias is auto-qualified with the region's inference-profile prefix so + one value works across regions, and the resolved value is written to + ``ANTHROPIC_MODEL`` so the subprocess sees the same model as the options. """ if isinstance(self.route, BedrockRoute): if config_model is not None: @@ -1005,8 +899,7 @@ def _resolve_effective_model( env["ANTHROPIC_MODEL"] = effective return effective if isinstance(self.route, LiteLLMRoute): - # Same env-sync as Bedrock, but pass the id verbatim (no - # inference-profile qualification — the gateway maps it). + # Same env-sync, but verbatim: the gateway maps the id itself. effective = config_model or route_model if effective: env["ANTHROPIC_MODEL"] = effective @@ -1027,18 +920,14 @@ async def communicate( Args: user_input: The message/prompt to send stream_callback: Optional callback for real-time event streaming - timeout: Hard wall-clock deadline in seconds. When exceeded, a - watchdog task force-kills the CLI subprocess (the SDK's anyio - task groups suppress cooperative cancellation, so a graceful - asyncio.wait_for is not sufficient). - max_turns: Hard cap on inner-loop turns for this call. None defers - to the SDK default. - should_stop: Cooperative early-stop poll (early-stop-on-criterion). - When provided, it is checked after each dispatched SDK message; - the first True finalizes the turn cleanly as STOPPED_EARLY - (``crashed=False``, no raise) at the next message boundary. - ``None`` (default) leaves the message loop behaviorally - identical to before. + timeout: Hard wall-clock deadline in seconds. A watchdog force-kills + the CLI subprocess when it elapses — the SDK's anyio task groups + suppress cooperative cancellation, so `asyncio.wait_for` is not + sufficient. + max_turns: Hard cap on inner-loop turns. None defers to the SDK. + should_stop: Cooperative early-stop poll, checked after each dispatched + message; the first True finalizes cleanly as STOPPED_EARLY + (``crashed=False``, no raise) at the next boundary. Returns: TurnRecord containing the complete interaction @@ -1051,9 +940,8 @@ async def communicate( if not self.working_directory: raise RuntimeError("Agent not started. Call start() first.") - # AgentConfig.type is `str | None`, but the orchestrator, SubAgentRunner, - # and UserSimulator all set it before construction. Assert the invariant so - # streaming-event sites below can safely use `str(self.config.type)`. + # Every constructor path sets it; assert so the streaming-event sites + # below can use `str(self.config.type)`. assert self.config.type is not None, "ClaudeCodeAgent requires AgentConfig.type to be set before communicate()" # Reset the pending slot + bump the iteration counter (shared lifecycle). @@ -1062,18 +950,15 @@ async def communicate( turn_start_time = time.monotonic() deadline = turn_start_time + timeout if timeout is not None else None - # Event emission: the agent is the SOLE emitter. Events fan out to an - # internal EventCollector (which assembles the TurnRecord — the single, - # agent-agnostic capture path) and the caller's stream_callback. + # The agent is the SOLE emitter: events fan out to an internal + # EventCollector and the caller's stream_callback. task_id = str(self.config.type) # str() so a plugin subclass with a non-enum kind also works collector = EventCollector() emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) - # All per-turn scratch state lives on the state object so each stream - # branch is a method. Built BEFORE the try so the except/finally can - # finalize even when setup (_build_claude_query) crashes — `timeout_hit` - # is set by both the in-loop deadline break and the watchdog callback; - # Python bool assignment is atomic under the GIL, so no lock is needed. + # Built BEFORE the try so except/finally can finalize even when setup + # crashes. `timeout_hit` is written by both the in-loop deadline break and + # the watchdog callback; bool assignment is atomic under the GIL. state = _ClaudeTurnState( self, emit=emit, @@ -1087,9 +972,8 @@ async def communicate( deadline=deadline, ) - # stderr capture STAYS a communicate local: it is wired into the SDK - # options during setup (a construction-order hazard if it lived on the - # state, which is built first), and only the error ladder reads its lines. + # STAYS a communicate local: it is wired into the SDK options during + # setup, which the state (built first) would order-invert. stderr_lines: list[str] = [] def capture_stderr(line: str) -> None: @@ -1112,41 +996,30 @@ def capture_stderr(line: str) -> None: prompt=user_input, iteration=self._iteration, model=effective_model, - # Stamped from the TURN CLOCK, not the event model's raw - # `datetime.now()` default. This bound is subtracted against - # window bounds the same clock produced (`decompose_turn`), and - # two bases inside one subtraction is what `TurnClock` exists to - # remove. Measured: antigravity's tail came out at -0.017 ms — - # an `AgentEndEvent` stamped 17 us BEFORE its own last message - # finished, which cannot happen — and `decompose_turn` clamped - # it to the `0.0` that means "measured, and instant" (CE058). - # It only bites where the true interval is smaller than the - # drift between the two clocks, which is the one harness that - # holds its process across turns; the fix belongs at every - # clocked site regardless, since that is what makes the - # subtraction single-basis rather than usually-close. + # From the TURN CLOCK, not the model's raw `datetime.now()` + # default: this bound is subtracted against window bounds the + # same clock produced, and two bases in one subtraction publish + # a clamped inversion as a measured 0.0 (CE058). timestamp=state.clock.now(), ) ) - # IMPORTANT: the transport is captured in the closure (not read from - # self._active_transport) so a stale watchdog from an earlier turn - # cannot kill a subsequent turn's subprocess. + # Captured in the CLOSURE, not read from self._active_transport, so a + # stale watchdog from an earlier turn cannot kill this turn's process. watchdog_target = transport def _on_turn_timeout() -> None: state.timeout_hit = True self._kill_transport(watchdog_target) - # Only forward the transport kwarg when we actually built one — - # otherwise keep the call shape identical to the no-timeout path so - # mocks with strict (prompt, options) signatures keep working. + # Only when one was built, so mocks with strict (prompt, options) + # signatures keep working on the no-timeout path. query_kwargs: dict[str, Any] = {"prompt": user_input, "options": options} if transport is not None: query_kwargs["transport"] = transport self._log.debug("Starting agent query stream...") - # OS-thread watchdog: fires at `timeout` seconds regardless of - # event-loop liveness. Immune to anyio cancel-scope suppression. + # OS-thread watchdog: fires regardless of event-loop liveness, and is + # immune to anyio cancel-scope suppression. with ThreadedWatchdog( timeout_seconds=timeout, on_timeout=_on_turn_timeout, @@ -1158,24 +1031,20 @@ def _on_turn_timeout() -> None: self._log.debug("Agent query stream ended") except asyncio.CancelledError: - # The threaded watchdog cancels the running task via - # loop.call_soon_threadsafe(task.cancel) when it fires. If that - # cancel landed *because* of the timeout, re-raise as - # TurnTimeoutError so the retry system sees a terminal timeout (not a - # transient cancel). External cancels propagate unchanged. + # A cancel that landed BECAUSE of the timeout re-raises as + # TurnTimeoutError, so the retry system sees a terminal timeout rather + # than a transient cancel. External cancels propagate unchanged. if self._timed_out(state.timeout_hit, deadline): assert timeout is not None self._finalize_and_raise_timeout(state.finalize, timeout) - # Cancelled from outside this turn: park the telemetry on `pending_turn` - # for the caller to drain. Otherwise the `finally` below finalizes as - # COMPLETED, which keeps no record. + # Cancelled from outside: park the telemetry on `pending_turn`, or the + # `finally` finalizes as COMPLETED and keeps no record. if not state.finalized: self._finalize_external_cancel(state.finalize) raise except ProcessError as e: - # When the watchdog SIGKILLs the subprocess, the SDK surfaces it as a - # ProcessError (exit code -9). Classify as a timeout so the retry - # system doesn't treat it as a transient AGENT_CRASH. + # A watchdog SIGKILL surfaces as ProcessError (exit -9); classify it as + # a timeout so the retry system does not treat it as AGENT_CRASH. if self._timed_out(state.timeout_hit, deadline): assert timeout is not None self._finalize_and_raise_timeout(state.finalize, timeout, cause=e) @@ -1186,15 +1055,14 @@ def _on_turn_timeout() -> None: message = f"CLI process failed (exit code {e.exit_code}): {detail}" self._finalize_and_raise_crash(state.finalize, message, cause=e) except Exception as e: - # Same race as above: the watchdog may have killed the subprocess and - # the SDK may have re-raised as a generic Exception. Check both the - # flag AND the wall-clock in case the flag flip races with our catch. + # The SDK may re-raise a watchdog kill as a generic Exception. Check + # both the flag AND the wall clock, in case the flip races this catch. if self._timed_out(state.timeout_hit, deadline): assert timeout is not None self._finalize_and_raise_timeout(state.finalize, timeout, cause=e) if not self._max_turns_short_circuit(state.sdk_result_summary, "Generic Exception"): - # The SDK wraps ProcessError as a generic Exception via the message stream. - # Read the captured ResultMessage summary (if any) for diagnostic context. + # The SDK wraps ProcessError as a generic Exception via the + # message stream; read the ResultMessage summary for context. error_info = self._format_error_summary(state.sdk_result_summary) cause_stderr = self._extract_cause_stderr(e) stderr = self._build_stderr_message(cause_stderr, stderr_lines) @@ -1206,30 +1074,22 @@ def _on_turn_timeout() -> None: message = f"Communication with agent failed: {error_details}" self._finalize_and_raise_crash(state.finalize, message, cause=e) finally: - # Auto-finalize any path the except blocks didn't (happy path, - # max_turns short-circuit, in-loop timeout break). Idempotent - # (guarded by state.finalized) so the crash/timeout branches that - # already finalized are a no-op here. Exactly one AgentEndEvent + one - # EventCollector-built TurnRecord are produced on every exit path. + # Auto-finalize any path the except blocks did not. Idempotent, so + # exactly one AgentEndEvent is produced on every exit path. if not state.finalized: if state.timeout_hit: assert timeout is not None state.finalize(AgentEndStatus.TIMEOUT, crashed=True, crash_reason=format_timeout_reason(timeout)) elif state.stopped_early_hit: - # Clean cooperative stop: NOT a crash, NOT a timeout. The - # max_turns_exhausted promotion in finalize() only fires for - # COMPLETED, so STOPPED_EARLY survives; the post-finally - # timeout raise is gated on timeout_hit, which is False here. + # NOT a crash, NOT a timeout. The max_turns promotion in + # finalize() only fires for COMPLETED, so this survives. state.finalize(AgentEndStatus.STOPPED_EARLY, crashed=False, crash_reason=None) else: state.finalize(AgentEndStatus.COMPLETED, crashed=False, crash_reason=None) self._active_transport = None - # Only trust `timeout_hit` in the happy path: if the loop completed - # cleanly, a wall-clock drift during post-loop cleanup would falsely - # classify a successful turn as a timeout. The watchdog and in-loop guard - # are the authoritative signals. (pending_turn already set by the - # finalize(TIMEOUT) call in the finally above.) + # Only the flag here, never the wall clock: a drift during post-loop + # cleanup would misclassify a successful turn as a timeout. if state.timeout_hit: assert timeout is not None raise TurnTimeoutError(timeout, iteration=self._iteration) @@ -1239,8 +1099,7 @@ def _on_turn_timeout() -> None: # This turn completed successfully — the iteration increment stands. self._end_turn_ok() - # The TurnRecord is the EventCollector's reduction of the events emitted - # above — single, agent-agnostic capture path (no parallel record build). + # The collector's reduction of the events emitted above. return collector.build_turn_record() async def _pump_messages( @@ -1252,21 +1111,16 @@ async def _pump_messages( ) -> None: """Drive the SDK message stream for one turn (extracted from ``communicate``). - Kept separate so the added cooperative-stop check keeps ``communicate`` - under ruff's statement cap. ``query`` is still resolved as a module global - at call time, so ``patch("...claude_code_agent.query", ...)`` test mocks - keep working. - - Two break conditions, and the order matters: - - - The wall-clock guard runs at the TOP of the loop — it breaks BEFORE the - message is dispatched, so the over-deadline message is DISCARDED (no - append, no events). Do NOT relocate this to a post-loop check. - - The cooperative stop runs AFTER ``state.dispatch(message)`` so a watcher - observing events during dispatch can flip its flag on THIS message and - the next message is never pulled — the deciding message is kept, the - next is not. No-op when ``should_stop is None`` (behaviorally identical - to before). + Kept separate so the cooperative-stop check keeps ``communicate`` under + ruff's statement cap. ``query`` is still resolved as a module global at + call time, so ``patch("...claude_code_agent.query", ...)`` mocks work. + + Two break conditions, and the ORDER MATTERS: + + - The wall-clock guard runs at the TOP, so an over-deadline message is + DISCARDED — no append, no events. Do NOT move it to a post-loop check. + - The cooperative stop runs AFTER ``state.dispatch(message)``, so a watcher + can flip its flag on THIS message and the next is never pulled. """ async for message in query(**query_kwargs): if deadline is not None and time.monotonic() > deadline: @@ -1288,24 +1142,18 @@ def _build_claude_query( ) -> tuple[ClaudeAgentOptions, SubprocessCLITransport | None, str | None]: """Build the SDK options (+ a timeout-only transport) for one turn. - Returns ``(options, transport, effective_model)``. ``transport`` is None - unless a ``timeout`` is set — it is pre-constructed only so the watchdog - can hard-kill the subprocess (the SDK's default path creates it internally - and never exposes it). ``effective_model`` is the resolved model id (may be - None on a DirectRoute with no configured model). ``stderr_callback`` is - wired into the options here but owned by ``communicate`` — it must exist - before the options are built, and only the error ladder reads its lines. + ``transport`` is None unless a ``timeout`` is set: it is pre-constructed + only so the watchdog can hard-kill the subprocess. ``effective_model`` may + be None on a DirectRoute with no configured model. ``stderr_callback`` is + wired in here but owned by ``communicate``. """ assert self.working_directory is not None # guaranteed by communicate's guard above # Process plugins: copy from config and replace env vars in paths. plugins = process_plugins(self.config.plugins or [], log=self._log) # type: ignore[arg-type] - # Build env overrides and resolve model for the configured API route. - # Precedence: task/CLI agent.model > route default (e.g. BEDROCK_MODEL). - # Per-turn cost-correlation headers (LiteLLM route only): the run/task tag - # from the orchestrator plus this turn's iteration, so the proxy-side cost - # log can be joined back to the exact turn. + # Per-turn cost-correlation headers (LiteLLM only): the run/task tag plus + # this turn's iteration, so the proxy-side cost log joins to the turn. cost_log_tags: dict[str, str] | None = None if self._cost_log_tags is not None: cost_log_tags = {**self._cost_log_tags, "x-ce-iteration": str(self._iteration)} @@ -1322,21 +1170,17 @@ def _build_claude_query( if "ToolSearch" not in disallowed_tools: disallowed_tools.append("ToolSearch") - # The SDK maps system_prompt=None to `--system-prompt ""` (an explicit - # EMPTY custom prompt) and a plain string to a full replacement — either - # way Claude Code's default behavioral guidance (parallel tool-call - # batching, conciseness) is lost. So ALWAYS send the claude_code preset: - # without `append` the CLI runs its default prompt; with it the configured - # prompt is appended. exclude_dynamic_sections keeps the prompt static - # across runs (the per-run tempdir path would otherwise be baked into the - # system prompt, breaking prompt caching and run comparability); the SDK - # re-injects the stripped sections into the first user message. - # system_prompt_mode="replace" (judge sub-agents) opts out of the preset: - # the configured prompt IS the entire system prompt. + # ALWAYS the claude_code preset: the SDK maps `system_prompt=None` to an + # explicit EMPTY prompt and a plain string to a full replacement, either + # of which loses Claude Code's default behavioral guidance. + # `exclude_dynamic_sections` keeps the prompt static across runs — the + # per-run tempdir path would otherwise be baked in, breaking prompt caching + # and comparability — and the SDK re-injects the stripped sections into the + # first user message. `system_prompt_mode="replace"` opts out. system_prompt = self._resolve_system_prompt() # as_posix(), not str(): bash on Windows strips backslashes from unquoted - # paths, so a redirect like `> D:\foo\bar` ends up writing to "Dfoobar". + # paths, so `> D:\foo\bar` writes to "Dfoobar". options = ClaudeAgentOptions( cwd=self.working_directory.as_posix(), permission_mode=self.config.permission_mode.value, @@ -1347,12 +1191,11 @@ def _build_claude_query( plugins=plugins, # type: ignore[arg-type] stderr=stderr_callback, # Capture stderr for better error messages env=env, - # Subscribe to raw stream events so we can recover the *cumulative* - # output_tokens for each emission from ``message_delta.usage`` events. - # Claude Code CLI ships AssistantMessage.usage.output_tokens with only - # a partial streaming snapshot (anthropics/claude-code#22686), so - # summing per-message values undercounts by 10x+. Without this flag - # StreamEvents are suppressed by the SDK. + # Recovers the CUMULATIVE output_tokens per emission from + # `message_delta.usage`: the CLI ships only a partial streaming + # snapshot (anthropics/claude-code#22686), so summing per-message + # values undercounts by 10x+. Without this the SDK suppresses + # StreamEvents. It also gates the first-window re-seed. include_partial_messages=True, system_prompt=system_prompt, setting_sources=self.config.setting_sources if self.config.setting_sources is not None else ["project"], @@ -1364,14 +1207,12 @@ def _build_claude_query( **self.config.sdk_options, ) - # Dump SDK options for later inspection (captures all 37+ fields including defaults). + # For later inspection: captures every field, defaults included. self._sdk_options_dump = dump_dataclass(options) - # When a timeout is set, pre-construct the transport so we retain a - # reference to the subprocess for hard-kill; the SDK's default path - # creates this internally and never exposes it. When no timeout is set we - # leave it None so the SDK uses its own default (keeps the door open for - # tests that mock query() without a real CLI). + # Pre-constructed only under a timeout, to retain the subprocess handle + # for hard-kill. None otherwise, so the SDK uses its own default and tests + # can mock query() without a real CLI. transport: SubprocessCLITransport | None = None if timeout is not None: transport = SubprocessCLITransport(prompt=user_input, options=options) @@ -1381,22 +1222,20 @@ def _build_claude_query( def _resolve_system_prompt(self) -> str | SystemPromptPreset: """The system-prompt VALUE that actually goes on the wire. - Single source of truth for both the options builder and the - ``system_prompt_semantics`` run-record marker (derived from this value, - never re-computed), so the persisted regime can never disagree with what - was sent. Returning the value rather than a mode string is what lets the - caller skip a type-narrowing re-check of the invariant resolved here. + Single source of truth for the options builder AND the + ``system_prompt_semantics`` run marker, which is derived from this value + and never recomputed — so the persisted regime cannot disagree with what + was sent. - ``replace`` requires a configured prompt (the config validator rejects - the pair at load, but a mutated or hand-built config falls back to the - preset here — fail open to append). + ``replace`` requires a configured prompt. The config validator rejects the + pair at load, but a hand-built config falls open to the preset here. """ if self.config.system_prompt_mode == "replace": if self.config.system_prompt is not None: return self.config.system_prompt if not self._warned_prompt_mode_downgrade: - # Warn once per agent so the downgrade is visible in task.log - # rather than only inferable from run.json's marker. + # Once per agent, so the downgrade is visible in task.log rather + # than only inferable from run.json's marker. self._warned_prompt_mode_downgrade = True logger.warning( "system_prompt_mode='replace' with no system_prompt — falling back to the claude_code " @@ -1410,13 +1249,12 @@ def _resolve_system_prompt(self) -> str | SystemPromptPreset: def get_environment_info(self) -> dict[str, Any]: """Record which system-prompt regime built this run's prompts. - ``append`` = the claude_code preset (dynamic sections excluded) with the - configured system_prompt, if any, appended; ``replace`` = the configured - prompt is the ENTIRE system prompt (judge sub-agents). Unlike the other - agents this is per-config, not fixed, so it overrides the base ClassVar - with the resolved value. Runs from before this marker existed used - replace-on-set / empty-on-unset semantics — trend dashboards must not - pool scores across that boundary. + ``append`` = the claude_code preset with the configured prompt appended; + ``replace`` = the configured prompt IS the entire system prompt (judge + sub-agents). Unlike the other agents this is per-config, not fixed, so it + overrides the base ClassVar with the resolved value. + + Rationale: .claude/notes/agents.md § The system_prompt_semantics marker """ semantics: SystemPromptSemantics = "replace" if isinstance(self._resolve_system_prompt(), str) else "append" return {**super().get_environment_info(), "system_prompt_semantics": semantics} @@ -1429,20 +1267,17 @@ async def stop(self) -> None: async def kill(self) -> None: """Force-terminate the in-flight Claude CLI subprocess, if any. - Async wrapper around ``kill_sync`` for callers that prefer async. - The threaded watchdog inside communicate() uses ``_kill_transport`` - directly on a captured transport (not via ``self._active_transport``) - to avoid a cross-turn race where a stale watchdog could kill a later - turn's subprocess. + Async wrapper around ``kill_sync``. The watchdog inside communicate() uses + ``_kill_transport`` on a CAPTURED transport instead, to avoid a stale + watchdog killing a later turn's subprocess. """ self.kill_sync() def kill_sync(self) -> None: """Synchronously SIGKILL the in-flight Claude CLI subprocess, if any. - Safe to call from a non-asyncio thread (e.g. a ``threading.Timer`` - callback). Reads ``self._active_transport`` once; if a later turn - has already cleared it, this is a no-op. + Safe from a non-asyncio thread. Reads ``self._active_transport`` once; a + no-op if a later turn already cleared it. """ self._kill_transport(self._active_transport) @@ -1450,10 +1285,8 @@ def kill_sync(self) -> None: def _timed_out(timeout_hit: bool, deadline: float | None) -> bool: """Return True if the turn has exceeded its deadline by either path. - Checks both the watchdog flag AND the wall clock. The flag-only check - races with the watchdog: if the handler was entered just before the - watchdog flipped the flag, we'd misreport a timeout as a generic - error. Checking wall-clock is the belt that catches that case. + BOTH the watchdog flag and the wall clock: a flag-only check races the + watchdog and misreports a timeout entered just before the flip. """ if timeout_hit: return True @@ -1463,10 +1296,10 @@ def _timed_out(timeout_hit: bool, deadline: float | None) -> bool: def _kill_transport(transport: SubprocessCLITransport | None) -> None: """SIGKILL the subprocess behind `transport`, if any. - The SDK wraps the subprocess in anyio cancel scopes that suppress - asyncio.CancelledError, so cooperative cancellation doesn't reliably - stop a stuck CLI. Sending SIGKILL releases stdout/stdin, which - unblocks the anyio readers so the async generator unwinds cleanly. + SIGKILL releases stdout/stdin, which unblocks the anyio readers so the + async generator unwinds cleanly. + + Rationale: .claude/notes/agents.md § The threaded watchdog """ if transport is None: return @@ -1491,11 +1324,10 @@ def _finalize_commands( for tool_id, cmd_data in pending_commands.items(): cmd = cmd_data["telemetry"] if cmd.result_status is None: - # Unknown status and unknown duration are the same fact: nothing - # resolved this command, so nothing timed it either. duration_ms - # is deliberately LEFT as None here — it used to be coerced to - # 0.0, which put an invented measurement on both sides of - # avg_command_time_ms and dragged the average toward zero. + # Unknown status and unknown duration are one fact: nothing + # resolved this command, so nothing timed it. `duration_ms` is + # deliberately left None (CE058). + # Rationale: .claude/notes/agents.md § Why only a RESOLVED tool is timed cmd.result_status = "unknown" unknown_status_count += 1 self._log.warning( @@ -1522,14 +1354,11 @@ def _finalize_commands( def _aggregate_model_usage(model_usage: dict[str, Any] | None) -> TokenUsage | None: """Sum the SDK ResultMessage ``model_usage`` into a cumulative TokenUsage. - ``model_usage`` maps each model id to its cumulative billing for the - session — ``{model: {inputTokens, outputTokens, cacheReadInputTokens, - cacheCreationInputTokens, costUSD, ...}}`` (camelCase, unlike ``usage``). - This is the SDK's authoritative cost breakdown: summed and priced it + Maps each model id to its cumulative session billing (camelCase, unlike + ``usage``). The SDK's authoritative cost breakdown: summed and priced it reconciles to ``total_cost_usd`` exactly, and it INCLUDES sub-agent - consumption (notably cache-creation/input) that the assistant-message - stream and the ``usage`` snapshot under-report. Returns None when absent - or empty so the caller can fall back. + consumption the stream under-reports. None when absent, so the caller can + fall back. """ if not isinstance(model_usage, dict) or not model_usage: return None @@ -1567,82 +1396,24 @@ def _build_token_usage( Source-of-truth order: - 1. ``ResultMessage.model_usage`` — the SDK's cumulative per-model billing. - Summed + priced at list rates it equals ``total_cost_usd`` exactly, - and it captures sub-agent token consumption (especially cache-creation - and input) that the assistant-message stream and the ``usage`` snapshot - do NOT — sub-agent emissions are only partially (sometimes never) - bubbled into the recorded stream. This is authoritative; prefer it. - - 2. Per-call telemetry stream (sum) — used when ``model_usage`` is absent - (e.g. legacy/mock SDKs). Recorded usage is deduped by - ``message_id``, so summing is exact when every token-bearing emission - carries an id; this still beats the ``usage`` snapshot, which - under-reports the cache-read cascade ~2-3x on multi-call runs. - - 3. ``ResultMessage.usage`` snapshot — last resort. - - ``total_cost_usd`` comes from ``model_usage.costUSD`` when present, else - the ResultMessage ``total_cost_usd`` — the real billed total. When BOTH - are absent (timeout / kill — there is no terminal ``ResultMessage``), it - is backfilled from the priced token buckets via ``calculate_cost``, - mirroring the Codex self-pricing path so killed turns still record a cost - instead of ``—``. The tokens are already captured; this is pure pricing. - - WHY THIS FIELD IS NECESSARY — AND WHY YOU CANNOT DERIVE IT FROM ``messages`` - --------------------------------------------------------------------------- - There are two distinct token views, and they are NOT meant to reconcile: - - * **Billing view** — ``model_usage`` (this method's output, carried as - ``AgentEndEvent.usage`` → ``TurnRecord.token_usage``). The complete, - cost-accurate per-model total for the turn. Cost/budget/reports read - this. - * **Attribution view** — the per-message ``messages`` transcript. Useful - for per-generation / per-sub-agent display (group by - ``parent_tool_use_id``), but NOT cost-complete. - - Summing the per-generation ``AssistantMessage`` entries does NOT, on its - own, equal ``model_usage`` — for three independent reasons (all confirmed - empirically against live ``claude_subagent_test`` runs — see the dumps - under ``tmp/agentusage-*`` and ``CODER_EVAL_RAW_SDK_LOG=1``): - - 1. **Per-step input/output are lossy snapshots.** The streamed - ``message_start``/``message_delta`` ``usage`` under-reports - ``input_tokens`` (and ``output_tokens``) vs. the billed total; the - docs say to "prefer the result message." Only ``output`` gets a - ``message_delta`` correction — ``input`` never does. - 2. **Sub-agent generations are not fully streamed.** A sub-agent's calls - never emit ``message_start``/``message_delta`` into the parent stream; - its terminal generation arrives only as the Agent tool result (we - synthesize it — see ``_synthesize_subagent_terminal_message``), and - its input is billed to ``model_usage`` without a corresponding parent - message. - 3. **A fixed ~512-token input is billed but never streamed.** Across runs - ``model_usage.inputTokens`` exceeds the sum of ALL captured generations - by a constant ~512 — and this gap is IDENTICAL with prompt caching - disabled (``DISABLE_PROMPT_CACHING=1`` → cw=cr=0), so it is NOT a - cache-bucketing artifact. ``message_start`` count == captured - ``AssistantMessage`` count (we drop nothing); the 512 simply belongs to - no SDK-emitted message. It is an upstream billing-vs-stream property, - not a capture bug. - - Cache (``cache_creation`` + ``cache_read``) reconciles from the generation - messages to the token; only ``input``/``output`` carry the residual above. - ``costUSD``/``total_cost_usd`` are themselves client-side estimates (the - SDK computes them locally) — the authoritative figure is Anthropic's - Usage/Cost API. - - HOW THE STREAM RECONCILES (the residual is booked, not smeared). The full - ``TurnRecord.messages`` stream DOES sum to ``token_usage`` exactly — - because ``EventCollector`` appends one synthetic ``ReconciliationMessage`` - (``role="reconciliation"``) carrying the per-bucket residual (this method's - ``model_usage`` total minus the sum of the real generations). The residual - is the three sources above. Do NOT instead smear by-difference tokens onto - the real ``AssistantMessage`` generations: that would fabricate - per-generation numbers matching no real call and break per-sub-agent - attribution. ``token_usage`` (billing) stays authoritative for - cost/budget/reports; ``messages`` (one reconciliation entry included) is - the attribution view that now sums to the same total. + 1. ``ResultMessage.model_usage`` — the SDK's cumulative per-model billing, + authoritative and inclusive of sub-agent consumption. Prefer it. + 2. Per-call telemetry stream (sum) — used when ``model_usage`` is absent. + Exact only when EVERY token-bearing emission carries a ``message_id``, + since that is what the dedup keys on. + 3. ``ResultMessage.usage`` snapshot — last resort; it under-reports the + cache-read cascade ~2-3x on multi-call runs. + + ``total_cost_usd`` comes from ``model_usage.costUSD``, else the + ResultMessage total, else the priced buckets (a killed turn has no + terminal ``ResultMessage``). + + **The BILLING view: summing ``messages`` does not reproduce it on its + own.** ``EventCollector`` books the residual as one synthetic + ``ReconciliationMessage``, so the stream still sums to this total. Do NOT + smear by-difference tokens onto real generations. + + Rationale: .claude/notes/agents.md § Token accounting, per harness """ from_models = ClaudeCodeAgent._aggregate_model_usage(sdk_result_model_usage) if from_models is not None: @@ -1656,9 +1427,8 @@ def _build_token_usage( for m in assistant_msgs if m.input_tokens or m.output_tokens or m.cache_creation_tokens or m.cache_read_tokens ] - # Summing is exact only when every token-bearing emission has an id (so - # the dedup in communicate() applied and no ResultMessage backfill was - # mixed in). Otherwise defer to the ResultMessage summary. + # Exact only when every token-bearing emission has an id, so the dedup + # applied and no ResultMessage backfill was mixed in. if token_bearing and all(m.message_id for m in token_bearing): return ClaudeCodeAgent._backfill_cost( TokenUsage( @@ -1687,9 +1457,7 @@ def _build_token_usage( def _price_from_buckets(usage: TokenUsage, model: str | None) -> float | None: """Price the four token buckets at ``model``'s list rate. - Shared by ``_backfill_cost`` (price-if-absent) and ``_reprice_for_litellm`` - (always-reprice). Returns ``None`` when ``model`` is unset or absent from - the rate card. + ``None`` when ``model`` is unset or absent from the rate card. """ if not model: return None @@ -1705,12 +1473,9 @@ def _price_from_buckets(usage: TokenUsage, model: str | None) -> float | None: def _backfill_cost(usage: TokenUsage, model: str | None) -> TokenUsage: """Price the token buckets when the SDK gave no cost (timeout / kill). - On a clean turn the SDK supplies ``costUSD`` / ``total_cost_usd``. When a - turn is timed out or killed there is no terminal ``ResultMessage``, so the - cost is absent even though the tokens are fully captured. Backfill it from - the rate card — the same self-pricing the Codex agent always does — so the - recorded top-line cost matches the evalboard simulator instead of ``—``. - A no-op when the cost is already set or the model is unknown/unpriced. + A timed-out or killed turn has no terminal ``ResultMessage``, so the cost + is absent even though the tokens are fully captured. A no-op when the cost + is already set or the model is unpriced. """ if usage.total_cost_usd is not None or not model: return usage @@ -1718,9 +1483,8 @@ def _backfill_cost(usage: TokenUsage, model: str | None) -> TokenUsage: if cost is not None: usage.total_cost_usd = cost else: - # Model not in the rate card — the turn reverts to a null cost (the - # pre-#386 symptom). Surface it so a stale pricing table is visible - # rather than silently reproducing "Cost = —" for new models. + # Not in the rate card, so the turn reverts to a null cost. Surface + # it, or a stale pricing table silently reads as "Cost = —". logger.warning("No pricing for model %r; timeout/kill turn cost left unset", model) return usage @@ -1728,18 +1492,15 @@ def _backfill_cost(usage: TokenUsage, model: str | None) -> TokenUsage: def _reprice_for_litellm(usage: TokenUsage, model: str | None) -> None: """Recompute the top-line cost for the LiteLLM backend, in place. - The Claude Agent SDK's ``costUSD``/``total_cost_usd`` is a client-side - estimate that assumes Claude/Anthropic pricing, so it is wrong for an - open-weight model driven through LiteLLM. Reprice from the (already - authoritative) token buckets at the model's real rate. The token buckets - are left untouched, so the per-message stream / reconciliation invariant - is unaffected — only the cost scalar changes. - - An unknown/unpriced model sets the cost to ``None`` (an honest "N/A") - **and logs a warning** — mirroring ``_backfill_cost`` — because a proxy - model missing from the rate card otherwise silently yields - ``total_cost_usd = None``, which makes the orchestrator skip the - ``max_usd`` gate with no diagnostic. + The SDK's cost estimate assumes Anthropic pricing, so it is wrong behind + LiteLLM. The token buckets are left UNTOUCHED, so the reconciliation + invariant is unaffected — only the cost scalar changes. + + An unpriced model sets the cost to ``None`` **and warns**: a silent + ``None`` makes the orchestrator skip the ``max_usd`` gate with no + diagnostic. + + Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ cost = ClaudeCodeAgent._price_from_buckets(usage, model) usage.total_cost_usd = cost @@ -1760,29 +1521,21 @@ def get_sdk_options(self) -> dict[str, Any] | None: def _try_parse_json_value(content: Any) -> dict[str, Any] | list[Any] | None: """Return the parsed JSON object or array from content, else None. - Strict telemetry-capture variant. ``coder_eval.formatting._extract_json`` - is the lenient display-path variant — keep behaviour aligned when you - change one, but they are intentionally separate: the telemetry path - feeds ``CommandTelemetry.result_data`` where false positives persist - into ``task.json`` and downstream dashboards. - - Accepts the two SDK-delivered shapes for ToolResultBlock.content: a plain - string, or a list of content blocks (MCP tools use this, e.g. - [{"type": "text", "text": "..."}]). Within the first 200 characters, - looks for the first line whose first non-whitespace character is `{` or - `[` and parses from there using raw_decode, so prefix noise (e.g. warning - lines the `uip` CLI prints before the JSON body) and trailing garbage are - tolerated. Requiring the brace to start a line avoids false positives - from incidental `{` or `[` embedded inside text (e.g. the Read tool's - line-numbered source where `items: list = []` would otherwise parse as - an empty list). The 200-char cap further rules out braces buried deep in - long text output. If the candidate fails to parse, returns None — no - fragment fallback, which would surface misleading partial captures from - truncated payloads. Bare empty containers (`{}` / `[]`) are rejected: - a non-empty dict or list is evidence of real structured content. - Primitives (strings, numbers, booleans, null) are rejected for the same - reason — a bare primitive adds no information beyond result_summary. - Non-JSON tool output is normal; parse failures are swallowed silently. + The STRICT telemetry-capture variant. ``formatting._extract_json`` is the + lenient display-path twin — keep behaviour aligned, but they are + intentionally separate: this one feeds ``CommandTelemetry.result_data``, + where a false positive persists into ``task.json`` and downstream + dashboards. + + Accepts a plain string or a list of content blocks (MCP tools use the + latter), and parses with ``raw_decode`` from the first line whose first + non-whitespace character is ``{`` or ``[``, so prefix noise and trailing + garbage are tolerated. Requiring the brace to START A LINE is what stops + an incidental ``[`` inside text matching. + + Rejected on purpose: a failed parse (no fragment fallback), bare ``{}`` / + ``[]``, and bare primitives. Non-JSON tool output is normal, so failures + are silent. """ if isinstance(content, list): text_parts = [ @@ -1795,9 +1548,8 @@ def _try_parse_json_value(content: Any) -> dict[str, Any] | list[Any] | None: content = "".join(text_parts) if not isinstance(content, str): return None - # Only look for the JSON start within the first 200 chars — enough to skip - # a few prefix warning lines but not so lax that a brace buried in a long - # text body gets mistaken for a structured payload. + # First 200 chars only: enough for a few prefix warning lines, not so lax + # that a brace buried in long text reads as a structured payload. match = re.search(r"(?:^|\n)[^\S\n]*[{[]", content[:_JSON_START_SEARCH_LIMIT]) if not match: return None @@ -1805,9 +1557,8 @@ def _try_parse_json_value(content: Any) -> dict[str, Any] | list[Any] | None: parsed, _ = json.JSONDecoder().raw_decode(content, match.end() - 1) except ValueError: return None - # Reject bare empty containers ({} / []): a non-empty dict or list is - # evidence of real structured content, an empty one is indistinguishable - # from an accidental match and adds nothing over result_summary. + # A non-empty dict or list is evidence of real structured content; an + # empty one is indistinguishable from an accidental match. if isinstance(parsed, (dict, list)) and parsed: return parsed return None @@ -1828,24 +1579,18 @@ def _tool_end_status(cls, is_error: bool, content: Any) -> ToolEndStatus: def _synthesize_subagent_terminal_message(message: Any, model: str | None) -> AssistantMessageTelemetry | None: """Materialize a sub-agent's TERMINAL generation as an AssistantMessage. - A sub-agent that calls tools runs several generations. Its intermediate - ones bubble into the parent stream as ``parent_tool_use_id``-tagged - assistant messages, but its terminal generation is delivered as the Agent - tool RESULT (``UserMessage.tool_use_result``), never as a streamed - message. We synthesize it as one so the sub-agent's full lifecycle lives - in the transcript and per-sub-agent usage is recoverable by grouping - messages on ``parent_tool_use_id`` — no separate sidecar field needed. - - ``tool_use_result.usage`` is the terminal call's usage breakdown (input / - output / cache-creation / cache-read — complete, incl. the cache-read that - ``TaskNotification.usage`` drops). It is terminal-only, not cumulative, so - it does NOT overlap the bubbled intermediate generations (its output is - the final reply's alone). Returns None for non-sub-agent tool results - (regular Bash/Write/etc. carry ``tool_use_result`` but no ``agentId``). - - The token total is unaffected: the normal path derives it from - ``ResultMessage.model_usage`` (which ignores this transcript), so the - synthetic message is purely additive for attribution/display. + A sub-agent's intermediate generations bubble into the parent stream as + ``parent_tool_use_id``-tagged messages, but its terminal one is delivered + as the Agent tool RESULT and never streamed. Synthesizing it puts the + sub-agent's full lifecycle in the transcript, so per-sub-agent usage is + recoverable by grouping on that id. + + ``tool_use_result.usage`` is the terminal call's own breakdown — complete, + and terminal-only, so it does NOT overlap the bubbled intermediates. + Returns None for a non-sub-agent tool result (no ``agentId``). + + The token total is unaffected: it derives from ``model_usage``, which + ignores this transcript, so the synthetic message is purely additive. """ tur = getattr(message, "tool_use_result", None) if not isinstance(tur, dict) or "agentId" not in tur: @@ -1874,18 +1619,13 @@ def _int(value: Any) -> int: except (TypeError, ValueError): return 0 - # This generation arrives as a tool result and is never streamed, so no - # window exists to measure — None (unknown), not 0.0 (instant). - # - # Deliberately NOT on the turn's `TurnClock`, and the only wall stamp in - # this harness that is not. These two bounds are an admitted - # PLACEHOLDER, not a measurement: `generation_duration_ms is None` and - # `parent_tool_use_id` is set, which is exactly what excludes this - # message from `subtract_tool_time` and from `_overhead_ms`'s - # head/tail bracket. A stamp no arithmetic reads has no basis to share, - # and threading a clock into a `@staticmethod` to produce one would - # claim otherwise. Codex's rollout rebuild stamps the same placeholder - # the same way, for the same reason. + # Never streamed, so no window exists to measure: None (unknown), not 0.0. + # These bounds are an admitted PLACEHOLDER, which is why they are + # deliberately NOT on the turn's `TurnClock` — the only wall stamp in this + # harness that is not. `generation_duration_ms is None` plus a set + # `parent_tool_use_id` is exactly what excludes this message from the + # subtraction and the head/tail bracket, so the stamp is read by no + # arithmetic and has no basis to share. now = datetime.now() return AssistantMessageTelemetry( started_at=now, @@ -1922,9 +1662,8 @@ def _resolve_pending_command( pending_commands: Map of tool_id -> {telemetry, command_start_time} processed_results: Set of already-processed tool IDs (for duplicate detection) now: This turn's ``TurnClock`` reading, passed in rather than read - here. The span stamped below is clipped against the window - bounds the same clock produced, so a second basis at this one - call site would put two clocks inside one subtraction. + here: the span stamped below is clipped against window bounds the + same clock produced. """ # Normalize content to string for storage content_str = str(content) if content is not None else "" @@ -1944,22 +1683,16 @@ def _resolve_pending_command( cmd.result_summary = content_str if content_str else None cmd.result_data = ClaudeCodeAgent._try_parse_json_value(content) - # Wall-clock execution bounds. `execution_completed_at` is the - # turn clock's reading; `execution_started_at` is reconstructed by - # subtracting the measured monotonic duration. This avoids storing - # a separate wall-clock start (we don't have one without - # restructuring pending_commands further) while still giving - # consumers two explicit timestamps with the right delta — and the - # reconstruction is now exact rather than approximate, since the - # turn clock is itself monotonic-derived. + # `execution_started_at` is RECONSTRUCTED by subtracting the measured + # monotonic duration from the turn clock's reading, which is exact + # because the turn clock is itself monotonic-derived. cmd.execution_completed_at = now cmd.execution_started_at = cmd.execution_completed_at - timedelta(milliseconds=duration_ms) if is_error: cmd.error_message = content_str - # Permission-blocked tool use is abnormal flow — warn so it - # surfaces in runs that don't have DEBUG enabled. + # Abnormal flow: warn so it surfaces without DEBUG enabled. content_lower = content_str.lower() if any( phrase in content_lower @@ -1982,16 +1715,8 @@ def _resolve_pending_command( def _build_stderr_message(sdk_stderr: str | None, stderr_lines: list[str]) -> str: """Combine SDK stderr with captured stderr lines, filtering out placeholder text. - The SDK often returns a hardcoded placeholder like "Check stderr output for details" - instead of actual error content. The real error details are in stderr_lines captured - via the stderr callback. - - Args: - sdk_stderr: The stderr string from ProcessError (may be a placeholder) - stderr_lines: Lines captured via the stderr callback during execution - - Returns: - Combined stderr message with real content, or "No stderr captured" + The SDK often returns a hardcoded placeholder instead of real error + content; the details are in the lines captured via the stderr callback. """ parts = [] @@ -2009,14 +1734,8 @@ def _build_stderr_message(sdk_stderr: str | None, stderr_lines: list[str]) -> st def _extract_cause_stderr(error: Exception) -> str | None: """Walk the exception __cause__ chain looking for a ProcessError with stderr. - The SDK re-raises ProcessError as a generic Exception via the Query message stream. - This method recovers the original stderr from the cause chain. - - Args: - error: The caught exception - - Returns: - stderr string from the original ProcessError, or None + The SDK re-raises ProcessError as a generic Exception via the Query + message stream, so the original stderr is only in the cause chain. """ cause = error.__cause__ depth = 0 @@ -2058,11 +1777,9 @@ def _max_turns_short_circuit(self, summary: ResultSummary | None, branch_label: def _summarize_result(msg: Message) -> ResultSummary | None: """Build a ``ResultSummary`` from an SDK ResultMessage, or None. - Returns None only when ``msg`` lacks the SDK ResultMessage shape - (``session_id`` + ``usage``). For real ResultMessages the SDK - always provides ``subtype`` (it's a required dataclass field), so - any missing/non-string value is treated as ``"unknown"`` rather - than silently disabling the summary downstream. + None only when ``msg`` lacks the ResultMessage shape. ``subtype`` is a + required dataclass field there, so a missing value becomes ``"unknown"`` + rather than silently disabling the summary downstream. """ if not _is_sdk_result_message(msg): return None @@ -2080,11 +1797,10 @@ def _summarize_result(msg: Message) -> ResultSummary | None: def _format_error_summary(summary: ResultSummary | None) -> str | None: """Format an errored ``ResultSummary`` for surfacing to the user. - Prefers free-form ``result`` text; falls back to the - ``subtype``/``stop_reason`` classification when ``result`` is - unset (which is the common shape on hard CLI crashes). Returns - None when there is nothing useful to surface, so callers can - decide whether to fall back to stderr. + Prefers free-form ``result`` text, falling back to the + ``subtype``/``stop_reason`` classification — the common shape on a hard + CLI crash. None when there is nothing useful, so the caller can fall back + to stderr. """ if summary is None or not summary.is_error: return None @@ -2104,8 +1820,7 @@ def _update_state_from_messages(self, messages: list[Message]) -> None: Args: messages: List of messages from the agent (SDK objects) """ - # Check for explicit error messages (use getattr for safe access). - # ResultMessage.is_error is intentionally NOT a state-change trigger — + # `ResultMessage.is_error` is intentionally NOT a state-change trigger: # the agent may recover from a tool error on a later turn. for msg in messages: if getattr(msg, "error", None): diff --git a/src/coder_eval/agents/codex_agent.py b/src/coder_eval/agents/codex_agent.py index 649818b1f..1264f5fbb 100644 --- a/src/coder_eval/agents/codex_agent.py +++ b/src/coder_eval/agents/codex_agent.py @@ -69,51 +69,28 @@ "Glob": "shell", } -# Approval mode — the SAME for every permission mode (no per-mode mapping). -# -# The Codex SDK exposes exactly two approval modes: -# - auto_review → AskForApproval.on_request + a SERVER-SIDE ApprovalsReviewer. -# The reviewer (an extra app-server/gateway decision) adjudicates each -# apply_patch/shell escalation. Under gateway load it can spuriously return -# "declined" — files silently not written. Claude has no analog: its -# Write/Edit permissions are decided CLIENT-SIDE with no model/reviewer in -# the loop, so it never hits this failure mode. -# - deny_all → AskForApproval.never + NO reviewer. Despite the name, this is -# the "run autonomously, never prompt, no reviewer" mode: in-sandbox -# operations (apply_patch within cwd, shell) execute directly; only -# escalations BEYOND the sandbox are refused. -# -# An eval harness never wants a reviewer that can flake, so EVERY permission mode -# uses deny_all. The trust boundary is coder_eval's per-run sandbox — a docker -# container or an ephemeral tempdir — NOT Codex's in-process OS sandbox, which -# _build_thread_options always drops to full-access (rationale there). +# Approval mode — the SAME for every permission mode. Despite the name this is +# the "run autonomously, never prompt, no reviewer" mode: in-sandbox operations +# execute directly, and only escalations BEYOND the sandbox are refused. The +# alternative puts a server-side reviewer in the loop that can flake. +# Rationale: .claude/notes/agents.md § Codex runs full-access on every permission mode _CODEX_APPROVAL_MODE = "deny_all" # Provider id registered in thread config when CODEX_BASE_URL routes to a # custom endpoint. _CUSTOM_PROVIDER_ID = "custom" -# Codex apply_patch (Write/Edit) statuses that mean the patch did not apply. -# PatchApplyStatus enum values are inProgress/completed/failed/declined — never -# the literal "error" the code used to compare against, so a failed patch was -# silently recorded as a successful Write. We use this only to classify the -# Write TELEMETRY honestly (failed → error). We do NOT fail or retry the turn on -# it: "declined" can only come from an approval reviewer, and every permission -# mode uses deny_all (no reviewer; see _CODEX_APPROVAL_MODE), so -# in-sandbox apply_patch is applied directly and "declined" should not occur; -# "failed" (diff context mismatch) is self-healed by the model within the turn, -# and grading checks the actual files regardless. +# apply_patch statuses meaning the patch did not apply. Used ONLY to classify the +# Write telemetry honestly, never to fail or retry the turn: "declined" needs an +# approval reviewer, which no permission mode configures, and "failed" (a diff +# context mismatch) is self-healed by the model within the turn. Grading checks +# the actual files regardless. _FILE_CHANGE_FAILURE_STATUSES = frozenset({"failed", "declined"}) -# Codex thread-item types that carry transcript CONTENT or session metadata -# rather than a tool call. Everything else streamed as item/started+item/completed -# is treated as a tool call (see _run_turn_with_streaming), so new Codex tool -# kinds are captured automatically instead of being silently dropped — the old -# code hard-coded only commandExecution/fileChange. -# - reasoning / agentMessage -> assistant transcript blocks (handled inline) -# - userMessage -> prompt echo (skipped) -# - contextCompaction / entered|exitedReviewMode / hookPrompt / plan -> -# session lifecycle + planning items, not agent tool calls. +# Thread-item types carrying transcript CONTENT or session metadata rather than a +# tool call. Everything ELSE streamed as item/started+item/completed is treated as +# a tool call, so a new Codex tool kind is captured automatically instead of being +# silently dropped. _CONTENT_ITEM_TYPES = frozenset( { "reasoning", @@ -127,8 +104,9 @@ } ) -# Friendly tool-name labels for known Codex tool item types. Unknown tool types -# fall back to the raw item type (so they still surface, just un-prettied). +# Codex item type -> the canonical (Claude) vocabulary every criterion is written +# against. Unknown types fall back to the raw item type, so they still surface. +# Rationale: .claude/notes/agents.md § Tool-name and argument normalization _TOOL_ITEM_NAMES: dict[str, str] = { "commandExecution": "Bash", "fileChange": "Write", @@ -140,17 +118,14 @@ "imageView": "ImageView", } -# collabAgentToolCall.tool value that spawns a NEW sub-agent (vs "wait"/messaging -# operations that act on an already-spawned agent). Only spawns register a new -# child thread to recover. Lowercased to match _status_value, which normalizes -# the SDK's "spawnAgent" enum value to lowercase. +# The collabAgentToolCall.tool value that spawns a NEW sub-agent; "wait" and +# messaging act on an already-spawned one, so only spawns register a child thread +# to recover. Lowercased to match _status_value's normalization. _COLLAB_SPAWN_TOOL = "spawnagent" -# Friendly tool-name labels for the raw ResponseItem function-call names found in -# a sub-agent's on-disk rollout (see _recover_subagent_tool_calls). The child -# thread persists `function_call`/`local_shell_call` ResponseItems unconditionally -# even though its `commandExecution` events are dropped under Limited persistence, -# so the rollout is the only place the sub-agent's inner tool calls survive. +# Tool names for the raw ResponseItem function calls in a sub-agent's on-disk +# rollout, which is the only place its inner tool calls survive. +# Rationale: .claude/notes/agents.md § Codex rollout rebuild _ROLLOUT_FN_NAMES: dict[str, str] = { "exec_command": "Bash", "shell": "Bash", @@ -165,11 +140,9 @@ _ROLLOUT_TOOL_CALL_TYPES = frozenset({"function_call", "local_shell_call", "custom_tool_call"}) _ROLLOUT_TOOL_OUTPUT_TYPES = frozenset({"function_call_output", "custom_tool_call_output"}) -# Sentinel returned by ``next(stream_iter, _STREAM_DONE)`` at stream end. We pass -# a default rather than catching StopIteration because a StopIteration raised -# inside ``asyncio.to_thread`` is converted by asyncio into a TypeError -# ("StopIteration interacts badly with generators…") that escapes ``except -# StopIteration`` — masking the real turn-failure reason on the stream-end path. +# A default rather than catching StopIteration: one raised inside +# ``asyncio.to_thread`` is converted by asyncio into a TypeError that escapes +# ``except StopIteration``, masking the real turn-failure reason. _STREAM_DONE = object() @@ -192,30 +165,21 @@ class _ItemTiming(NamedTuple): def _item_timing(started_ms: int | None, completed_ms: int | None, sdk_duration_ms: float | None) -> _ItemTiming: """Resolve a tool item's timing from the SDK's millisecond stamps. - One helper for all three telemetry builders, so a command, a file change - and an MCP call cannot disagree about what a missing stamp means. - - BOTH stamps or neither. Pairing a real stamp with ``_ms_to_dt(None)`` — - which is ``datetime.now()`` — would fabricate an interval out of one - reading and the current time, so the raw ``int | None`` values are checked - BEFORE conversion, never after. - - With both present, ``timestamp`` becomes the tool's own start. It used to - be ``datetime.now()`` at COMPLETION, which places the call after its own - execution. Ordering is unaffected either way: ``TurnRecord.commands`` is - sorted on ``sequence_number`` (``collector._ordered_commands``). - - Without them, the SDK item's own ``duration_ms`` is used only when it - reports something. A ``0`` (or, defensively, a negative) there is an - UNREPORTED duration, not an instant command — 70 of 211 commands in one - nightly reported ``0`` for calls the message gaps show took seconds — so - it becomes ``None`` and leaves both sides of every average instead of - dragging them toward zero. - - ``generation_completed_at`` is deliberately absent from this tuple, for - all three builders: it means "when the model finished emitting the - ``tool_use`` block", which Codex's stream does not carry per tool. - Deriving it from the flush time would be a guess. + One helper for all three telemetry builders, so a command, a file change and + an MCP call cannot disagree about what a missing stamp means. + + BOTH stamps or neither: pairing a real stamp with ``_ms_to_dt(None)`` — which + is ``datetime.now()`` — fabricates an interval out of one reading and the + current time, so the raw values are checked BEFORE conversion. + + Without them, the SDK item's own ``duration_ms`` is used only when it reports + something. A ``0`` there is an UNREPORTED duration, not an instant command (70 + of 211 commands in one nightly reported ``0`` for calls the message gaps show + took seconds), so it becomes ``None`` (CE058). + + ``generation_completed_at`` is deliberately absent for all three: it means + "when the model finished emitting the ``tool_use`` block", which Codex's + stream does not carry per tool, and the flush time would be a guess. """ if started_ms is not None and completed_ms is not None: started = _ms_to_dt(started_ms) @@ -227,8 +191,8 @@ def _item_timing(started_ms: int | None, completed_ms: int | None, sdk_duration_ # anomaly remains visible in the record. duration_ms=max(0.0, float(completed_ms - started_ms)), ) - # Explicit `is None or <= 0`, never `sdk_duration_ms or None` — that is - # the CE058 coalesce read backwards, and it hides the decision being made. + # Explicit `is None or <= 0`, never `sdk_duration_ms or None`: that is the + # CE058 coalesce read backwards, and hides the decision being made. duration = None if sdk_duration_ms is None or sdk_duration_ms <= 0 else float(sdk_duration_ms) return _ItemTiming( timestamp=datetime.now(), @@ -248,8 +212,7 @@ def _fresh_input_tokens(raw_input: int, cached: int) -> int: """The fresh (uncached) prompt slice = tokens written to cache this call. Single definition of the OpenAI cache-write convention, shared by the - per-message (`_flush_message`) and per-turn (`_token_usage_from_sdk`) paths so - they can't drift if the billing model ever changes. + per-message and per-turn paths so they cannot drift. """ return max(raw_input - cached, 0) @@ -257,9 +220,11 @@ def _fresh_input_tokens(raw_input: int, cached: int) -> int: class _ThreadTotals(NamedTuple): """A snapshot of the Codex SDK's thread-cumulative ``ThreadTokenUsage.total``. - Held on the agent across turns (the thread outlives the turn) so each turn can - report its own slice instead of the running total. ``input`` is the full prompt - count, cached prefix included — the SDK's convention, not ours. + Held across turns (the thread outlives the turn) so each turn reports its own + slice, not the running total. ``input`` is the full prompt count, cached + prefix INCLUDED — the SDK's convention, not ours. + + Rationale: .claude/notes/agents.md § Codex rollout rebuild """ input: int = 0 @@ -269,9 +234,9 @@ class _ThreadTotals(NamedTuple): def since(self, baseline: "_ThreadTotals") -> "_ThreadTotals": """This turn's tokens = the cumulative snapshot minus the previous one. - A total that moved BACKWARDS means the thread restarted under us (a fresh - thread counts from zero), so the snapshot is already turn-local: return it - whole rather than clamping every bucket to zero and losing the turn. + A total that moved BACKWARDS means the thread restarted, so the snapshot + is already turn-local: return it whole rather than clamping to zero and + losing the turn. """ if self.input < baseline.input or self.output < baseline.output or self.cached < baseline.cached: return self @@ -285,9 +250,8 @@ def since(self, baseline: "_ThreadTotals") -> "_ThreadTotals": def _message_uncached_input(m: AssistantMessage) -> int: """A captured generation's fresh (uncached) input. - Single definition of the per-message convention, shared by the cost and the - fold-up paths. Codex children carry 0 ``cache_creation`` (no cache-write fee), - but fold both defensively so nothing is dropped if that ever changes. + Codex children carry 0 ``cache_creation`` (no cache-write fee), but both are + folded defensively so nothing is dropped if that changes. """ return m.input_tokens + m.cache_creation_tokens @@ -297,14 +261,12 @@ def _message_uncached_input(m: AssistantMessage) -> int: # supported"), so this is a fixed constant, not an operator knob. _CODEX_WIRE_API = "responses" -# Login-shell profile files generated into the per-task HOME (see -# _setup_login_shell_home). ``.bash_profile`` is what ``bash -l`` reads; -# ``.profile`` covers ``sh``/``dash`` login shells. The zsh trio covers macOS, -# where zsh is the default shell: ``.zshenv`` runs in EVERY zsh, ``.zprofile`` -# in login shells (AFTER /etc/zprofile, whose path_helper resets PATH), and -# ``.zshrc`` in interactive shells - including codex's shell snapshot, which -# sources it explicitly. zsh selects its dotfiles via $ZDOTDIR (fallback -# $HOME), so _build_codex_env points ZDOTDIR at the generated home. +# Login-shell profiles generated into the per-task HOME. ``.bash_profile`` is +# what ``bash -l`` reads and ``.profile`` covers sh/dash; the zsh trio covers +# macOS, where ``.zshenv`` runs in EVERY zsh, ``.zprofile`` in login shells +# (AFTER /etc/zprofile's path_helper) and ``.zshrc`` in interactive ones, +# including codex's shell snapshot. +# Rationale: .claude/notes/agents.md § Codex login-shell PATH restoration _LOGIN_PROFILE_NAMES = (".bash_profile", ".profile", ".zshenv", ".zprofile", ".zshrc") _ZSH_PROFILE_NAMES = frozenset({".zshenv", ".zprofile", ".zshrc"}) @@ -326,21 +288,17 @@ def _get_item_root(notification: Any) -> Any: class _CodexTurnState: """Per-turn mutable scratch state for one ``CodexAgent.communicate`` call. - Holds the stream-pump locals and the assistant-transcript reconstruction - buffers, with one method per notification kind (``on_*``) plus ``dispatch`` - (returns True on ``turn/completed`` to break the pump), ``_record_block`` / - ``_flush_message`` (the per-generation message cutter) and ``finalize`` (the - terminal ``AgentEndEvent`` + partial-record build). A plain module-private - class (NOT Pydantic, NOT exported) with a back-reference to the agent for its - existing helpers (``_tool_name`` / ``_telemetry_for_item`` / … ). + Holds the stream-pump locals and the transcript reconstruction buffers, with + one method per notification kind plus ``dispatch`` (True on ``turn/completed`` + to break the pump), ``_flush_message`` and ``finalize``. ``commands`` and ``messages`` are the SAME list objects ``communicate`` owns, - held by identity (no copy) so a mid-turn crash keeps the partial transcript. - The finalize inputs (``sdk_token_usage`` / ``result_turn`` / ``result_text``) - are COMMITTED by ``communicate`` only after the pump returns cleanly — they - default to None/None/"" so a crashed turn finalizes from the captured messages - (the live pump scratch ``turn_result`` / ``latest_token_usage`` / - ``agent_message_chunks`` is intentionally NOT what finalize reads). + held by identity (no copy), so a mid-turn crash keeps the partial transcript. + + The finalize inputs are COMMITTED by ``communicate`` only after the pump + returns cleanly, defaulting to None/None/"" — so a crashed turn finalizes from + the captured messages, and the live pump scratch is intentionally NOT what + finalize reads. """ def __init__( @@ -393,14 +351,11 @@ def __init__( self.open_blocks: list[ContentBlock] = [] self.open_start_ms: int | None = None self.open_end_ms: int | None = None - # Where the NEXT generation window starts: the previous flush's end. - # Windows tile the turn contiguously, as they do on Antigravity and - # claude-code. None until the first flush, which falls back to its own - # first item — the SDK gives no "turn began" stamp, and inventing one - # from time.time() would mix our clock with the SDK's inside a single - # subtraction. Advanced ONLY by a flush that actually appended a - # message, so a no-op flush leaves the window open and a later real - # generation still measures from where it began. + # Where the NEXT generation window starts: the previous flush's end. None + # until the first flush, which falls back to its own first item — the SDK + # gives no "turn began" stamp, and inventing one from time.time() would + # mix our clock with the SDK's inside one subtraction. + # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark_ms: int | None = None self.start_ms_by_id: dict[str, int] = {} self.blocks_by_id: dict[str, ContentBlock] = {} @@ -428,11 +383,11 @@ def _record_block(self, block: ContentBlock, item_id: str, completed_ms: int | N def _flush_message(self, last: Any) -> None: """Cut the open buffer into AssistantMessage(s) for one generation. - ``last`` is the SDK ``TokenUsageBreakdown`` for the generation that - produced these blocks (or None for a safety flush with no usage). Emits - ONE sub-message per block kind (thinking vs tool/text), all sharing this - generation's ``message_id``; the first carries the gen's input/cache, the - rest carry 0 so per-message_id sums don't double-count. + ``last`` is the SDK breakdown for the generation that produced these + blocks (None for a safety flush). Emits ONE sub-message per block kind, + all sharing this generation's ``message_id``. + + Rationale: .claude/notes/agents.md § Why the generation is split into sub-messages """ if not self.open_blocks: self.reasoning_placeholders = [] @@ -441,13 +396,12 @@ def _flush_message(self, last: Any) -> None: cached = (getattr(last, "cached_input_tokens", 0) or 0) if last else 0 raw_input = (getattr(last, "input_tokens", 0) or 0) if last else 0 total_output = (getattr(last, "output_tokens", 0) or 0) if last else 0 - # The fresh (uncached) prompt slice is plain input — OpenAI bills no - # separate cache-write fee, so Codex carries 0 cache_creation. + # OpenAI bills no separate cache-write fee, so cache_creation is 0. gen_input = _fresh_input_tokens(raw_input, cached) gen_cache_write = 0 reasoning_tok = (getattr(last, "reasoning_output_tokens", 0) or 0) if last else 0 - # Resolve text-less reasoning blocks: show a policy placeholder when - # reasoning was billed, else drop the block. + # A text-less reasoning block becomes a placeholder when reasoning was + # billed, and is dropped otherwise. if self.reasoning_placeholders: if reasoning_tok > 0: for blk in self.reasoning_placeholders: @@ -463,27 +417,18 @@ def _flush_message(self, last: Any) -> None: thinking_blocks = [b for b in self.open_blocks if b.block_type == "thinking"] action_blocks = [b for b in self.open_blocks if b.block_type != "thinking"] - # The window runs from the PREVIOUS flush's end, not from this - # generation's first item. The SDK stamps an item with the moment it - # began EXECUTING, so seeding from it discarded the model time that - # produced the item — the gap between the last item's completion and - # this one's start. Measured on tasks/hello_date: a Write emission - # spanning 2 ms (start 20:50:39.063, end .065) reported 98 output - # tokens, and the 2694 ms of real generation sat in the preceding gap, - # attributed to nothing. Across that turn only 15.8% of the 17 s wall - # clock was accounted for. Tiling matches Antigravity and claude-code, - # and is what lets Sum(generation) + Sum(tool) reconcile to the turn. + # From the PREVIOUS flush's end, not this generation's first item: the + # SDK stamps an item with the moment it began EXECUTING, so seeding there + # discards the model time that produced it. mark_ms = self.gen_mark_ms if self.gen_mark_ms is not None else self.open_start_ms window_end_ms = self.open_end_ms if self.open_end_ms is not None else self.open_start_ms mark = _ms_to_dt(mark_ms) completed = _ms_to_dt(window_end_ms) - # The RAW window. It is extended to the LAST item's completion, so a - # generation containing a tool call already CONTAINS that tool's - # execution — but taking it back out is no longer this reducer's job. - # `timing.subtract_tool_time` does it for all five, which is - # also what makes the sub-message split below safe: the two specs share - # these bounds, so the collector groups them and subtracts the overlap - # ONCE rather than once per part. + # The RAW window, extended to the LAST item's completion — so a + # generation containing a tool call already CONTAINS its execution, and + # the collector takes it back out. That is also what makes the sub-message + # split safe: the two specs SHARE these bounds, so the overlap is + # subtracted once rather than once per part. started, gen_ms = close_window( mark=mark, now=completed, @@ -496,29 +441,20 @@ def _flush_message(self, last: Any) -> None: think_out = reasoning_tok if action_blocks else total_output action_out = max(total_output - reasoning_tok, 0) if thinking_blocks else total_output - # Sub-message specs in generation order (thinking first). The FIRST - # carries the gen's input/cache — those are per-CALL billing figures - # and must not be split. Generation TIME is different: it is a - # property of the content, so it is apportioned below. + # Thinking first. The FIRST carries the gen's input/cache: per-CALL + # billing figures that must not be split. Generation TIME is a property + # of the content, so it IS apportioned below. specs: list[tuple[list[ContentBlock], int, int]] = [] if thinking_blocks: specs.append((thinking_blocks, think_out, reasoning_tok)) if action_blocks: specs.append((action_blocks, action_out, 0)) - # Split gen_ms across the sub-messages by their own OUTPUT-TOKEN - # share, giving the last the remainder so the parts reconstruct - # gen_ms (to float precision — the shares are rounded to 1e-6 ms, so - # do not assert exact equality on an arbitrary measured window). - # Concentrating it all on the first reported the thinking row as the - # entire generation and the action row as instant. With no output - # recorded anywhere, split evenly — there is nothing to weigh by, and - # one row taking all of it would be a guess dressed as a measurement. - # - # NOTE this weighs by output tokens while the evalboard's own - # mixed-emission split weighs by CONTENT SIZE. Deliberate, not an - # oversight to unify: here the SDK hands us a real per-spec token - # count, so there is no need to approximate one from content length. + # By OUTPUT-TOKEN share, the last taking the remainder so the parts + # reconstruct gen_ms to float precision (shares round to 1e-6 ms, so do + # not assert exact equality on a measured window). With no output anywhere, + # split evenly. The evalboard's twin weighs by CONTENT SIZE instead, which + # is deliberate, not an oversight to unify. out_total = sum(out_tok for _, out_tok, _ in specs) gen_parts: list[float] = [] assigned = 0.0 @@ -571,12 +507,11 @@ def ended_cleanly(self) -> bool: def max_turns_reached(self) -> bool: """True once this turn has produced ``max_turns`` visible turns. - Delegates the count to the collector (``EventCollector.visible_turn_count``) - so Codex and Antigravity cap on one shared definition rather than each - agent's own scratch list — ``self.commands`` skips items whose telemetry the - SDK does not resolve, while the collector counts every emitted tool end, - which is exactly what lands in ``TurnRecord.commands``. Codex delivers one - SDK turn per ``communicate()``, so the SDK's own turn counter would cap at 1. + Delegates to ``EventCollector.visible_turn_count`` rather than + ``self.commands``, which SKIPS items whose telemetry the SDK does not + resolve; the collector counts every emitted tool end, which is what lands + in ``TurnRecord.commands``. Codex delivers one SDK turn per + ``communicate()``, so a native counter would cap at 1. """ return self.max_turns is not None and self.collector.visible_turn_count >= self.max_turns @@ -619,11 +554,9 @@ def on_item_started(self, notification: Any) -> None: if root_type is not None and root_type not in _CONTENT_ITEM_TYPES: tool_id = item_id or f"{root_type}_{self.next_sequence}" self.seq_by_id[tool_id] = self.next_sequence - # The start stamp is known HERE, so record it on the start - # telemetry too. close_open_tools publishes this object verbatim - # for an orphan, and without it Codex was the only harness whose - # unresolved tool calls could not be placed on a timeline at all - # (OpenCode, Pi and Antigravity all set it at tool start). + # Recorded on the START telemetry too: close_open_tools publishes + # this object verbatim for an orphan, and without it an unresolved + # call cannot be placed on a timeline at all. started_at = _ms_to_dt(started_at_ms) if started_at_ms is not None else None start_tel = CommandTelemetry( tool_name=self._agent._tool_name(root_type), @@ -636,8 +569,8 @@ def on_item_started(self, notification: Any) -> None: self.open_tools[tool_id] = start_tel self.emit.on_event(ToolStartEvent(task_id=self.task_id, turn_id=self.turn_id, tool=start_tel)) self.next_sequence += 1 - # Record the tool_use block now; is_error patched at item/completed, - # even after the message is flushed (held by reference in blocks_by_id). + # is_error is patched at item/completed even after the message is + # flushed, because the block is held by reference. block = ContentBlock(block_type="tool_use", sequence=0, tool_use_id=tool_id) self.blocks_by_id[tool_id] = block self._record_block(block, tool_id, None) @@ -655,10 +588,8 @@ def on_item_completed(self, notification: Any) -> None: # This tool is now resolved — drop it from the orphan set. self.open_tools.pop(tool_id, None) - # The SDK reports both ends of the execution; `on_item_started` - # banked the start. Passing them in is what lets the builders - # record real execution bounds instead of a duration the SDK often - # leaves at 0. + # `on_item_started` banked the start; passing both in is what lets + # the builders record real bounds instead of the SDK's frequent 0. telemetry, is_error = self._agent._telemetry_for_item( root, root_type, @@ -683,8 +614,8 @@ def on_item_completed(self, notification: Any) -> None: status=ToolEndStatus.ERROR if is_error else ToolEndStatus.OK, ) ) - # Patch the block recorded at item/started (held by reference, so this - # lands even post-flush) + extend the still-open message's end time. + # Patch the block recorded at item/started, and extend the still-open + # message's end time. if tool_id in self.blocks_by_id: self.blocks_by_id[tool_id].is_error = is_error if ( @@ -706,7 +637,7 @@ def on_item_completed(self, notification: Any) -> None: elif root_type == "reasoning": # OpenAI never returns raw CoT, so a text-less item becomes a - # placeholder block, resolved with its token count at flush. + # placeholder, resolved with its token count at flush. reasoning_id = getattr(root, "id", f"reasoning_{self.next_sequence}") parts = getattr(root, "content", None) or getattr(root, "summary", None) or [] text = "\n".join(p for p in parts if p) @@ -716,8 +647,8 @@ def on_item_completed(self, notification: Any) -> None: self.reasoning_placeholders.append(block) elif root_type == "agentMessage": - # Full assistant text — append a text block. The message is cut at the - # following tokenUsage event (the generation boundary), not here. + # The message is cut at the following tokenUsage event (the + # generation boundary), not here. message_item_id = getattr(root, "id", f"msg_{self.next_sequence}") text = getattr(root, "text", "") or "" if text: @@ -775,19 +706,15 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso return self.finalized = True - # Prefer the SDK total (deltas off the thread-cumulative figure); on - # crash/timeout it stays None, so fall back to the per-generation tokens - # already captured on the messages — those are per-turn to begin with, but - # the thread baseline still has to move past them or the NEXT turn's delta - # re-books this one. + # On crash/timeout the SDK total stays None, so fall back to the + # per-generation tokens on the messages — but the thread baseline still + # has to move past them, or the NEXT turn's delta re-books this one. token_usage = self._agent._token_usage_from_sdk(self.sdk_token_usage) if token_usage is None: token_usage = self._agent._token_usage_from_messages(self.messages) self._agent._advance_usage_baseline(token_usage) - # Codex bills sub-agents on separate threads, so fold the recovered child - # generations into the turn total — matching Claude's bubbled-up totals. - # Folded AFTER the baseline advance: the SDK total covers the parent thread - # only, so child tokens must not shift the parent's baseline. + # AFTER the baseline advance: the SDK total covers the parent thread only, + # so child tokens must not shift the parent's baseline. token_usage = self._agent._fold_subagent_tokens(token_usage, self.messages) self.emit.on_event( @@ -834,13 +761,11 @@ def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reaso class CodexAgent(Agent[CodexAgentConfig]): """Implementation of the Agent interface for OpenAI Codex using the Codex SDK.""" - # The notification pump has a between-items guard where the cooperative - # ``should_stop`` check runs, so this agent supports early-stop-on-criterion. + # The pump has a between-items guard where `should_stop` runs. supports_cooperative_stop: ClassVar[bool] = True - # Codex appends system_prompt as developer_instructions on top of its base - # prompt. Runs from before this marker existed silently DROPPED the field — - # dashboards must not pool system_prompt-setting tasks across that boundary. + # `system_prompt` maps to developer_instructions, ON TOP of the base prompt. + # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker system_prompt_semantics: ClassVar[SystemPromptSemantics] = "append" def __init__( @@ -861,9 +786,8 @@ def __init__( self.route = route or DirectRoute() self.codex_client: Any = None self.thread: Any = None - # Thread-cumulative token snapshot as of the END of the last finalized turn. - # The thread outlives the turn, so this is what makes each turn's usage its - # own delta rather than the running total (see _token_usage_from_sdk). + # Thread-cumulative snapshot as of the END of the last finalized turn: + # what makes each turn's usage its own delta, not the running total. self._thread_usage_baseline = _ThreadTotals() self.working_directory: Path | None = None self._env_path_prepend: list[str] = [] @@ -871,10 +795,9 @@ def __init__( # _state / _iteration / _iteration_was_incremented / pending_turn lifecycle # bookkeeping lives on the Agent base class (shared defaults + helpers). self._log = PrefixedAdapter(logger, {"prefix": instance_name}) - # Live handle to the in-flight turn, set by _run_turn_with_streaming and - # cleared in its finally. kill()/kill_sync() use it to interrupt a stuck - # turn — the watchdog's task.cancel() alone can't preempt a blocking SDK - # call (it lands only at an await point, which we now create via to_thread). + # Live handle to the in-flight turn, so kill()/kill_sync() can interrupt a + # stuck one: the watchdog's task.cancel() lands only at an await point, + # which the to_thread offload is what creates. self._active_turn_handle: Any = None async def start( @@ -906,17 +829,15 @@ async def start( env_override = self._build_codex_env() config = CodexConfig(env=env_override) if env_override else None - # Initialize the Codex client (context manager compatible). Close any - # prior client first: start() is driven through execute_with_retry, so - # a retried start would otherwise orphan the previous app-server - # subprocess + reader threads (reaped only at final cleanup). + # Close any prior client FIRST: start() runs through + # execute_with_retry, so a retried start would otherwise orphan the + # previous app-server subprocess and its reader threads. self._close_client() self.codex_client = Codex(config=config) self._log.debug("Codex client initialized") - # Authenticate with the API key when one is configured. Without this - # the app-server falls back to an existing ChatGPT login, so headless - # API-key runs (CI) would otherwise fail to authenticate. + # Without this the app-server falls back to an existing ChatGPT + # login, so headless API-key runs (CI) fail to authenticate. api_key = os.getenv("CODEX_API_KEY") if api_key: try: @@ -980,18 +901,16 @@ async def communicate( turn_start_time = time.monotonic() - # Event emission: the agent is the SOLE emitter; events fan out to an - # internal EventCollector (which assembles the TurnRecord — the single, - # agent-agnostic capture path) and the caller's stream_callback. + # The agent is the SOLE emitter: events fan out to an internal + # EventCollector and the caller's stream_callback. task_id = str(self.config.type) # str() so a plugin subclass with a non-enum kind also works collector = EventCollector() emit = CompositeStreamCallback([c for c in (collector, stream_callback) if c is not None]) # Codex has no per-API-call boundary: one thread.turn() == one turn_id. turn_id = f"codex-{self._iteration}" - # All per-turn scratch lives on the state so the stream-pump branches are - # methods; the same commands/messages lists flow through the pump and - # finalize. timeout_hit is set by the watchdog callback (atomic bool). + # The same commands/messages lists flow through the pump and finalize. + # `timeout_hit` is written by the watchdog callback (atomic bool). state = _CodexTurnState( self, emit=emit, @@ -1038,9 +957,8 @@ def _on_turn_timeout() -> None: emit.on_event(TurnStartEvent(task_id=task_id, turn_id=turn_id, model=self._effective_model())) try: - # Commit the pump's results onto the state only on a CLEAN - # return; a crash skips this, so finalize reads the crash - # defaults (None/"") and falls back to the captured messages. + # Committed only on a CLEAN return; a crash skips this, so + # finalize reads the defaults and falls back to the messages. state.result_turn, state.sdk_token_usage, state.result_text = await self._run_turn_with_streaming( state, should_stop ) @@ -1052,11 +970,8 @@ def _on_turn_timeout() -> None: if state.timeout_hit: self._finalize_and_raise_timeout(state.finalize, timeout or 0, cause=e) if state.ended_cleanly: - # The turn already stopped cleanly; escalating to a crash - # would trigger the orchestrator's retry with the watcher's - # decision still latched → immediate stop-at-turn-0 on the - # retry (wasted spend). A cap-break is the same shape: the - # retry would burn the budget again and re-hit the cap. + # Already stopped on purpose — do not escalate. + # Rationale: .claude/notes/agents.md § Why a post-stop exception is not a crash self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) else: self._finalize_and_raise_crash( @@ -1065,32 +980,26 @@ def _on_turn_timeout() -> None: if state.timeout_hit: # Watchdog fired but the pump finished before the cancel landed. - # Route through the shared kernel so this path sets _state=ERROR - # like every other timeout/crash path (it previously did not — a - # latent inconsistency now fixed). + # Routed through the shared kernel so this path sets _state=ERROR + # like every other timeout path. assert timeout is not None self._finalize_and_raise_timeout(state.finalize, timeout) except (AgentCrashError, TurnTimeoutError): # Already funneled through finalize by the inner handlers. raise except asyncio.CancelledError: - # Non-timeout cancel (external cancellation, or a cancel during - # thread_start before the watchdog block). The timeout path already - # finalized above; otherwise close the AgentStart so the event tree - # stays balanced and the pending-turn contract holds. finalize is + # External, or during thread_start before the watchdog block. Close + # the AgentStart so the event tree stays balanced; finalize is # idempotent, so the timeout case is a no-op here. if not state.finalized: self._finalize_external_cancel(state.finalize) raise except Exception as e: - # Catches failures OUTSIDE the inner turn block — notably thread_start - # and _format_turn_result. Without this, such errors escape as a bare - # exception: the orchestrator never drains pending_turn and _iteration - # stays incremented, violating the pending-turn contract. + # Failures OUTSIDE the inner turn block, notably thread_start. Without + # this they escape bare: the orchestrator never drains pending_turn + # and _iteration stays incremented. if state.ended_cleanly and not state.timeout_hit: - # Same retry-poisoning guard as the inner handler: the turn already - # ended cleanly (cooperative stop or turn cap), so finalize instead - # of crashing. + # Same retry-poisoning guard as the inner handler. self._log.warning("Ignoring post-stop exception; finalizing cleanly: %s", e) else: self._finalize_and_raise_crash( @@ -1100,11 +1009,8 @@ def _on_turn_timeout() -> None: self._state = AgentState.WORKING self._end_turn_ok() - # The TurnRecord is the EventCollector's reduction of the emitted events. - # Precedence matches Claude: timeout (raised above) > stopped_early > - # max_turns_exhausted > completed. stopped_early outranks the cap because an - # armed criterion deciding the outcome is the more specific reason to have - # cut the run, and the pump checks it first. + # Precedence: timeout (raised above) > stopped_early > max_turns > done. + # Rationale: .claude/notes/agents.md § Shared turn lifecycle if state.stopped_early_hit: status = AgentEndStatus.STOPPED_EARLY elif state.max_turns_hit: @@ -1118,8 +1024,8 @@ async def stop(self) -> None: """Stop the agent and tear down the Codex SDK session. ``Codex(config=...)`` eagerly spawns an app-server subprocess plus reader - threads; ``close()`` reaps them. Skipping it leaks a subprocess + threads - per task across a batch run, so close before nulling the reference. + threads, so close before nulling the reference or a batch run leaks one + set per task. """ self._close_client() self.thread = None @@ -1137,7 +1043,7 @@ def kill_sync(self) -> None: """Synchronous abort for the watchdog thread (cannot await coroutines). Best-effort: interrupt the in-flight turn so the blocked stream iteration - unblocks, then close the client. Safe to call at any time and idempotent. + unblocks, then close the client. Idempotent. """ self._interrupt_active_turn() self._close_client() @@ -1163,14 +1069,11 @@ def _close_client(self) -> None: def get_environment_info(self) -> dict[str, Any]: """Record the resolved Codex routing so runs are auditable/comparable. - Always emits ``system_prompt_semantics`` (from the base). The routing keys - (``codex_base_url_host`` / ``codex_wire_api`` / ``codex_api_version`` / - ``codex_model_is_deployment``) are added only when a custom endpoint is - configured (CODEX_BASE_URL): on a custom endpoint the model is an - operator-chosen alias (a deployment name on Azure), so two operators' + The routing keys are added only under a custom endpoint, where the model is + an operator-chosen alias (a deployment name on Azure) and two operators' ``gpt-5-codex`` deployments are otherwise indistinguishable in run - artifacts. The host (not the full URL) is recorded to avoid leaking any - embedded credentials; the API key is never recorded. + artifacts. The HOST, not the full URL, is recorded, so an embedded + credential cannot leak; the API key is never recorded. """ info: dict[str, Any] = dict(super().get_environment_info()) base_url = self._resolve_base_url() @@ -1187,15 +1090,11 @@ def get_environment_info(self) -> dict[str, Any]: def _setup_skills(self, plugin_tools_dir: str | None) -> None: """Set up .agents/skills directory from plugins or plugin_tools_dir. - Codex auto-discovers skills in .agents/skills/ directories scanned from - the working directory up through parent directories to repo root. - - Skills are collected from two sources: - 1. config.plugins - task-defined plugins with type='local' and path pointing to skills - 2. plugin_tools_dir parameter - runtime plugin directory + Codex auto-discovers skills in ``.agents/skills/``, scanned from the working + directory up to the repo root, so each source's skill dirs are symlinked + (or copied, on Windows) into it. - Supports SKILL.md files following the Agent Skills open standard. - Creates .agents/skills/ directory and symlinks/copies skill directories. + Rationale: .claude/notes/agents.md § Skills, per harness """ if not self.working_directory: return @@ -1215,9 +1114,8 @@ def _setup_skills(self, plugin_tools_dir: str | None) -> None: skills_sources.append(plugin_path) self._log.debug(f"Found skills from plugin: {plugin_path}") else: - # Loud: an unresolved env var (e.g. unset - # $SKILLS_REPO_PATH) or missing dir silently drops - # the plugin's skills, so the agent runs blind. + # Loud: an unresolved env var or missing dir drops the + # skills silently, so the agent runs blind. hint = "env var likely unset" if "$" in expanded_path else "path does not exist" self._log.warning( f"Plugin skills path did not resolve: {path_str!r} " @@ -1239,10 +1137,8 @@ def _setup_skills(self, plugin_tools_dir: str | None) -> None: try: agents_skills_dir.mkdir(parents=True, exist_ok=True) - # Symlink or copy skills from all sources. A source may either - # contain skill dirs directly (//SKILL.md) or be a - # Claude plugin-marketplace root whose skills live one level deeper - # (/skills//SKILL.md). Scan both layouts. + # A source may hold skill dirs directly or be a plugin root whose + # skills live one level deeper. Scan both layouts. for skills_source in skills_sources: scan_dirs = [skills_source] nested = skills_source / "skills" @@ -1271,8 +1167,8 @@ def _setup_skills(self, plugin_tools_dir: str | None) -> None: if linked: self._log.debug(f"Linked {len(linked)} skill(s) into {agents_skills_dir}") else: - # Sources existed but no SKILL.md was found under them or their - # skills/ subdir — codex will run without any skill context. + # Sources existed but held no SKILL.md, so codex runs with no + # skill context at all. self._log.warning( f"0 skills linked into {agents_skills_dir} despite " + f"{len(skills_sources)} plugin source(s): " @@ -1292,34 +1188,28 @@ def _resolve_base_url() -> str | None: def _resolve_api_version() -> str | None: """Azure OpenAI ``api-version`` from CODEX_API_VERSION, or None. - Azure's Responses endpoint requires an ``api-version`` query parameter on - every request; when set it is injected as the custom provider's - ``query_params``. Plain OpenAI / gateway endpoints leave this unset. + Azure's Responses endpoint requires it on every request, injected as the + provider's ``query_params``. Other endpoints leave it unset. """ return os.getenv("CODEX_API_VERSION") or None def _effective_model(self) -> str | None: """Resolve the model: task/CLI ``agent.model`` wins, else CODEX_MODEL. - Mirrors the Claude agent's ``config_model or route_model`` precedence - (where the route fallback is BEDROCK_MODEL); here the fallback is the - settings-backed CODEX_MODEL. + Mirrors the Claude agent's precedence, with CODEX_MODEL as the fallback. """ return self.config.model or settings.codex_model def _build_codex_env(self) -> dict[str, str] | None: """Build the environment passed to the Codex app-server. - Carries the API key (``CODEX_API_KEY``, read by the codex binary when a - model provider's ``env_key`` points at it) and, when the sandbox - resolved ``mock_path_dirs``, a PATH with those directories prepended so - mock CLIs shadow the real ones for the agent's shell commands. The base - URL is NOT an env var the binary honors — it is applied through the - model provider config in ``_build_thread_options`` instead. + Carries ``CODEX_API_KEY`` and, when the sandbox resolved mock dirs, a PATH + with those prepended. The base URL is NOT an env var the binary honors — + it goes through the model provider config instead. - The SDK merges this dict over ``os.environ`` for the app-server process - (and normalizes the PATH key case-insensitively), so a full PATH value - here safely replaces the inherited one. + The SDK merges this PARTIAL dict over ``os.environ`` (normalizing the PATH + key case-insensitively), so a full PATH value here safely replaces the + inherited one. """ env: dict[str, str] = {} api_key = os.getenv("CODEX_API_KEY") @@ -1331,18 +1221,14 @@ def _build_codex_env(self) -> dict[str, str] | None: env[path_key] = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key, "")]) self._log.debug(f"PATH prepend: {os.pathsep.join(self._env_path_prepend)}") if self._login_shell_home is not None: - # Point login shells at the generated profile dir (see - # _setup_login_shell_home) while pinning codex state (auth, rollout - # sessions) to its real location — _codex_home() reads the same - # resolution for sub-agent rollout recovery, so both sides agree. - # HOME steers bash/sh; ZDOTDIR steers zsh (the macOS default - # shell), which ignores HOME for dotfile selection when it is set. + # Login shells point at the generated profile dir while codex state + # stays pinned to its real location. HOME steers bash/sh; ZDOTDIR + # steers zsh, which ignores HOME for dotfile selection when set. env["HOME"] = str(self._login_shell_home) env["ZDOTDIR"] = str(self._login_shell_home) # The binary hard-errors on an explicitly set CODEX_HOME that does - # not exist (unset, it materializes the ~/.codex default itself) — - # hosts that auth via CODEX_API_KEY never ran `codex login`, so - # the dir may not exist yet. Create it before pinning. + # not exist, and a host that auths via CODEX_API_KEY never ran + # `codex login`. Create it before pinning. codex_home = self._codex_home() codex_home.mkdir(parents=True, exist_ok=True) env["CODEX_HOME"] = str(codex_home) @@ -1356,47 +1242,32 @@ def _login_shell_profiles_supported() -> bool: def _setup_login_shell_home(self) -> None: """Create a per-task HOME whose profiles restore the mock PATH prepend. - Codex issues every shell command through the user's default shell as a - login shell: ``bash -lc`` on Linux, ``zsh -lc`` on macOS (its default - shell; fish would need the same treatment here if codex ever picks - it). A login shell re-sources the system profile chain - - ``/etc/profile`` on Linux, ``/etc/zprofile``'s path_helper on macOS - - which unconditionally RESETS PATH, silently dropping the mock-CLI - prepend passed via the app-server environment, so bare commands - resolve to the REAL CLIs (real-tenant contamination). The per-user - dotfiles are sourced AFTER that chain, so a generated per-task HOME - (wired up in ``_build_codex_env``; codex state stays in CODEX_HOME) - gets the last word and re-prepends the mock dirs: - ``.bash_profile``/``.profile`` for bash/sh (selected via env HOME) and - ``.zshenv``/``.zprofile``/``.zshrc`` for zsh (selected via env - ZDOTDIR; ``.zshrc`` also feeds codex's shell snapshot, which sources - it explicitly). Per-task rather than the user's real dotfiles so - parallel tasks with different mocks cannot collide. No-op without mock - dirs or on non-POSIX hosts: Windows codex shells through PowerShell - (``-NoProfile``) or ``cmd /c``, neither of which re-sources a profile - chain that resets PATH, so the plain env prepend survives there as-is. - - Bash uses the env HOME only to PICK the profile file; the generated - profile's first act is to export the ORIGINAL home back, so the - sourced user profile and the command body see the real ``$HOME`` - (git config, tool caches, ``$HOME``-relative sourcing keep working). - Known residual gap: a NESTED bash/sh login shell inside a command - re-reads the real profiles and loses the prepend again (nested zsh - keeps it - ZDOTDIR stays exported). + Codex issues every shell command through a LOGIN shell, which re-sources + the system profile chain and unconditionally RESETS PATH — silently + dropping the mock-CLI prepend, so bare commands resolve to the REAL CLIs + (real-tenant contamination). The per-user dotfiles are sourced AFTER that + chain, so a generated per-task HOME gets the last word. + + Per-task rather than the user's real dotfiles, so parallel tasks with + different mocks cannot collide. No-op without mock dirs or on non-POSIX + hosts, where no profile chain resets PATH. + + **Known residual gap:** a NESTED bash/sh login shell inside a command + re-reads the real profiles and loses the prepend again. Nested zsh keeps + it, because ZDOTDIR stays exported. + + Rationale: .claude/notes/agents.md § Codex login-shell PATH restoration """ self._cleanup_login_shell_home() if not (self._env_path_prepend and self._login_shell_profiles_supported()): return original_home = os.environ.get("HOME", "") - # Where the user's REAL zsh dotfiles live: their own ZDOTDIR when set, - # else their home (zsh's fallback). + # The user's REAL zsh dotfile dir: their own ZDOTDIR, else their home. original_zdotdir = os.environ.get("ZDOTDIR", "") or original_home - # The profile only ever executes under a POSIX shell, so the PATH - # separator is ':' regardless of the host building it. + # Executed only under a POSIX shell, so ':' regardless of the host. quoted_prepend = shlex.quote(":".join(self._env_path_prepend)) export_line = f'export PATH={quoted_prepend}:"$PATH"' - # Track the dir BEFORE writing so a failed write can't orphan it — - # the except below (and any later cleanup) always sees it. + # Tracked BEFORE writing, so a failed write cannot orphan it. home = Path(tempfile.mkdtemp(prefix="coder-eval-codex-home-")) self._login_shell_home = home try: @@ -1408,8 +1279,8 @@ def _setup_login_shell_home(self) -> None: original_zdotdir=original_zdotdir, generated_home=str(home), ) - # newline="\n": the profile must stay LF-only no matter which host - # builds it, or bash sees literal \r at end of line. + # LF-only no matter which host builds it, or bash sees a literal + # carriage return at end of line. (home / name).write_text(content, encoding="utf-8", newline="\n") except Exception: self._cleanup_login_shell_home() @@ -1425,29 +1296,23 @@ def _login_profile_content( original_zdotdir: str = "", generated_home: str = "", ) -> str: - """One generated profile file: restore the ORIGINAL ``$HOME``, source - the user's own counterpart file (so image/user setup isn't lost), then - re-prepend the mock dirs. - - The env HOME pointing at the generated dir exists ONLY so bash selects - this file; exporting the original home back on the first line keeps - every ``$HOME`` consumer (git, npm, the sourced profile's own - ``$HOME/.bashrc`` references) on the real home. - - ``.bash_profile`` mimics bash's first-found chain over the original - home; the ``.profile`` twin (read by ``sh``/``dash`` login shells) - sources only ``.profile`` — the bash-specific files may contain - bashisms a POSIX shell would choke on. - - The zsh files each source their exact counterpart from the user's real - zsh dotfile dir (``original_zdotdir``) - zsh reads ALL of its startup - files, not a first-found chain. ``.zshenv`` additionally re-pins - ZDOTDIR to the generated home AFTER sourcing: the user's ``.zshenv`` - may redefine ZDOTDIR, which would steer the rest of the startup chain - away from the generated ``.zprofile``/``.zshrc``. Each zsh file - re-prepends because /etc/zprofile resets PATH BETWEEN ``.zshenv`` and - ``.zprofile``, and a sourced user file may reset it again; a duplicate - PATH entry is harmless, a lost prepend is contamination. + """One generated profile: restore the ORIGINAL ``$HOME``, source the + user's own counterpart, then re-prepend the mock dirs. + + The env HOME pointing at the generated dir exists ONLY so bash selects this + file; exporting the original back on the first line keeps every ``$HOME`` + consumer on the real home. + + ``.bash_profile`` mimics bash's first-found chain; the ``.profile`` twin + sources only ``.profile``, since the bash-specific files may contain + bashisms a POSIX shell would choke on. The zsh files each source their + EXACT counterpart, because zsh reads ALL of its startup files rather than a + first-found chain, and ``.zshenv`` re-pins ZDOTDIR after sourcing in case + the user's own redefined it. Each zsh file re-prepends: /etc/zprofile + resets PATH between ``.zshenv`` and ``.zprofile``, and a duplicate PATH + entry is harmless where a lost prepend is contamination. + + Rationale: .claude/notes/agents.md § Codex login-shell PATH restoration """ lines = [ "# Generated by coder_eval (CodexAgent): the system profile chain resets", @@ -1492,51 +1357,32 @@ def _cleanup_login_shell_home(self) -> None: def _build_thread_options(self) -> dict[str, Any]: """Build thread_start options from agent config. - Returns a dict with sandbox, approval_mode, and config parameters - for thread_start() based on permission_mode, allowed_tools, and disallowed_tools. + Rationale: .claude/notes/agents.md § Codex runs full-access on every permission mode """ from openai_codex.api import ApprovalMode, Sandbox # pyright: ignore[reportPrivateImportUsage] options: dict[str, Any] = {} - # Pin the model when one is resolved; otherwise Codex picks its default - # and two runs can silently differ. + # Pin the model, or Codex picks its default and two runs silently differ. effective_model = self._effective_model() if effective_model: options["model"] = effective_model self._log.debug(f"Codex model pinned to {effective_model}") - # system_prompt maps to developer_instructions: injected ON TOP of Codex's - # base prompt, matching the append-only contract of the shared config field - # (Claude Code appends via the claude_code preset; Antigravity via - # TemplatedSystemInstructions). base_instructions (full replacement of the - # base prompt) is deliberately not exposed. + # ON TOP of Codex's base prompt, matching the append-only contract of the + # shared config field. `base_instructions` (full replacement) is + # deliberately not exposed. if self.config.system_prompt is not None: options["developer_instructions"] = self.config.system_prompt permission_mode = self.config.permission_mode.value approval_mode_str = _CODEX_APPROVAL_MODE - # Codex always runs full-access. coder_eval owns this run's isolation - # boundary either way — a docker container (docker driver) or an ephemeral - # per-task tempdir it creates and discards (tempdir driver); those are the - # only two drivers — so Codex's own in-process OS sandbox (Landlock/seatbelt) - # is always redundant. Worse, it actively breaks on the paths we rely on: - # inside the container Landlock is unavailable, on constrained CI agents the - # bwrap re-exec is denied ("bwrap: execvp .../codex: Permission denied"), and - # on Windows there is no OS sandbox at all — in each case a read-only / - # workspace-write run fails its writes/execs silently and scores 0 with no - # loud error. Dropping to full-access matches Claude Code and Antigravity, - # which run with no in-agent OS sandbox; hard isolation of untrusted actions - # is the docker driver's job, and approval_mode stays deny_all regardless. - # - # Consequence: permission_mode does NOT confine Codex — every mode resolves - # to full-access. The docker driver is the only OS-level write boundary here; - # the tempdir/host driver is a working directory, not a confinement boundary - # (same as Claude Code / Antigravity already run there), so adversarial or - # untrusted evals belong on the docker driver. _log_config_enforcement - # surfaces this. full-access (danger-full-access) also keeps network on, so - # tool installs (the UiPath CLI, npm/pip) work without extra sandbox config. + # ALWAYS full-access: permission_mode does NOT confine Codex. The docker + # driver is the only OS-level write boundary; the tempdir/host driver is a + # working directory, not a confinement boundary, so adversarial or + # untrusted evals belong on docker. _log_config_enforcement says so. + # Rationale: .claude/notes/agents.md § Codex runs full-access on every permission mode options["sandbox"] = Sandbox.full_access options["approval_mode"] = ApprovalMode(approval_mode_str) @@ -1564,12 +1410,10 @@ def _build_thread_options(self) -> dict[str, Any]: + "do not rely on it as a security boundary." ) - # Route through a custom endpoint (e.g. an OpenAI-/responses-compatible - # gateway, or Azure OpenAI) when CODEX_BASE_URL is set. The codex binary - # has no base-URL env var — a model provider must be defined in config and - # selected, with env_key naming the env var that holds the key - # (CODEX_API_KEY). For Azure, CODEX_API_VERSION adds the required - # ``api-version`` query param and CODEX_MODEL is the deployment name. + # The codex binary has no base-URL env var: a model provider must be + # defined in config and selected, with env_key naming the key's variable. + # For Azure, CODEX_API_VERSION adds the required query param and + # CODEX_MODEL is the deployment name. base_url = self._resolve_base_url() if base_url: options["model_provider"] = _CUSTOM_PROVIDER_ID @@ -1582,15 +1426,13 @@ def _build_thread_options(self) -> dict[str, Any]: "name": "Custom", "base_url": base_url, "env_key": "CODEX_API_KEY", - # The pinned codex binary only supports the Responses wire API - # (it rejects `wire_api = "chat"` as "no longer supported"), so - # this is fixed rather than configurable. + # Fixed, not configurable: the pinned binary rejects + # `wire_api = "chat"` as "no longer supported". "wire_api": _CODEX_WIRE_API, } api_version = self._resolve_api_version() if api_version: - # Azure requires ?api-version=… on every request; the codex binary - # appends these to the provider's request URL. + # Azure requires ?api-version=... on every request. provider["query_params"] = {"api-version": api_version} tool_config["model_providers"] = {_CUSTOM_PROVIDER_ID: provider} self._log.debug( @@ -1612,11 +1454,8 @@ def _log_config_enforcement(self) -> None: self._log.debug(f"Disallowed tools: {', '.join(self.config.disallowed_tools)}") self._log.debug(f"Permission mode: {self.config.permission_mode.value}") - # Codex always runs full-access regardless of permission_mode (see - # _build_thread_options — its in-process OS sandbox is redundant given - # coder_eval's docker/tempdir boundary and unusable on our CI hosts). The - # notice fires for EVERY mode, not just bypassPermissions, so operators are - # not misled that plan/acceptEdits/default confine Codex — none of them do. + # Fires for EVERY mode, not just bypassPermissions, so operators are not + # misled that plan/acceptEdits/default confine Codex — none of them do. self._log.warning( "[SECURITY] Codex runs full-access on every permission_mode " + f"(configured: {self.config.permission_mode.value}); permission_mode does not confine it. " @@ -1627,9 +1466,8 @@ def _log_config_enforcement(self) -> None: def _format_turn_result(self, turn_result: Any) -> str: """Format a Codex Turn to a readable string — fallback when no text streamed. - The Turn payload has no ``final_response`` field; assistant text arrives as - agentMessage deltas during streaming. This only fires when streaming produced - nothing, dumping the raw Turn for debugging. + The Turn payload has no ``final_response`` field, so this only fires when + streaming produced nothing, dumping the raw Turn for debugging. """ try: result_dict = turn_result.model_dump() if hasattr(turn_result, "model_dump") else vars(turn_result) @@ -1644,17 +1482,14 @@ async def _run_turn_with_streaming( """Drive ``turn.stream()`` through the per-turn state, emitting the standard event protocol; returns ``(turn_result, latest_token_usage, agent_text)``. - The enclosing ``communicate()`` owns the TurnStart/TurnEnd/AgentEnd - boundaries; this drives the inner notification pump. ``state`` accumulates - commands, the assistant transcript, sub-agent spawns and per-generation - tokens — all mutated in place so a mid-turn crash keeps the partial. + ``communicate()`` owns the TurnStart/TurnEnd/AgentEnd boundaries; this + drives the inner pump. ``state`` is mutated IN PLACE, so a mid-turn crash + keeps the partial. - The cooperative ``should_stop`` poll runs AFTER ``state.dispatch`` (the - emission that lets the watcher latch on the deciding tool call) and BEFORE - the next notification is pulled — the deciding item is kept, the next is - not. No-op when ``should_stop is None`` (behaviorally identical to before). + ``should_stop`` runs AFTER ``state.dispatch`` (the emission the watcher + latches on) and BEFORE the next notification is pulled. """ - # Create the turn handle (starts the turn but doesn't block) + event stream. + # Starts the turn without blocking, and opens the event stream. turn_handle = await self._run_async(self.thread.turn, state.user_input) self._active_turn_handle = turn_handle stream = await self._run_async(turn_handle.stream) @@ -1662,9 +1497,8 @@ async def _run_turn_with_streaming( stream_iter = iter(stream) try: while True: - # Offload the blocking SDK iteration to a worker thread so the event - # loop stays free (parallel agents don't serialize) and the - # watchdog's task.cancel() can actually land at this await point. + # Offloaded so the event loop stays free (parallel agents do not + # serialize) and the watchdog's task.cancel() can land here. notification: Any = await asyncio.to_thread(next, stream_iter, _STREAM_DONE) if notification is _STREAM_DONE: break @@ -1675,10 +1509,9 @@ async def _run_turn_with_streaming( self._log.debug("Cooperative stop requested; ending notification pump at this boundary") self._interrupt_active_turn() # best-effort; stops server-side spend break - # The turn cap shares this boundary: the notification that reached the - # cap is dispatched whole, the next is never pulled. Checked after the - # cooperative stop so an armed early-stop still reports as - # STOPPED_EARLY when both would fire on the same notification. + # The cap shares this boundary: the notification that reached it is + # dispatched whole, the next is never pulled. After the cooperative + # stop, so an armed early-stop wins a tie. if state.max_turns_reached(): state.max_turns_hit = True self._log.debug("max_turns (%s visible turns) reached; ending notification pump", state.max_turns) @@ -1686,9 +1519,8 @@ async def _run_turn_with_streaming( break finally: self._active_turn_handle = None - # Close any orphan tool (item/started without item/completed), flush any - # trailing blocks not closed by a tokenUsage event (e.g. a crash - # mid-generation), then close the stream. Runs on every exit path. + # Close orphan tools, flush trailing blocks no tokenUsage event + # closed, then close the stream. Runs on EVERY exit path. state.close_open_tools() state._flush_message(None) with contextlib.suppress(Exception): @@ -1697,30 +1529,17 @@ async def _run_turn_with_streaming( if state.turn_result is None and not state.ended_cleanly: raise RuntimeError("Turn did not complete (no turn/completed notification received)") - # Belt-and-suspenders: if streaming surfaced no assistant transcript, - # rebuild it from the terminal Turn's ordered item list. + # If streaming surfaced no transcript, rebuild it from the terminal + # Turn's ordered item list. if not state.messages: state.messages.extend(self._messages_from_items(getattr(state.turn_result, "items", None), state.turn_id)) - # Recover each spawned sub-agent's INNER tool calls from its on-disk rollout - # and nest them under the spawning Agent call. The parent stream never - # carries the child's commands (Limited persistence drops them), but its - # rollout always persists the raw function_call/local_shell_call items. - # - # Runs on a turn-cap stop. Recovery is also what carries the children's - # TOKENS: it is the only writer of the ``parent_tool_use_id``-tagged - # messages that ``_fold_subagent_tokens`` sums into the turn total, so - # skipping it drops the child threads' spend from the run's cost entirely - # (Codex bills children on separate threads the parent total never sees). - # A cap is a routine ending, not an exceptional one, so paying ~2s of - # rollout polling beats under-reporting spend on every capped run that - # spawned a sub-agent. The recovered child calls land in the trajectory - # beyond the cap's count, the same way the force-closed orphan does; - # the cap bounds what the model was allowed to DO, not what the record is - # allowed to explain. - # - # Still skipped on a cooperative stop: an armed gate has already decided - # the run, children may have no rollout yet, and that path predates the cap. + # RUNS on a turn-cap stop, because recovery is also the only writer of the + # `parent_tool_use_id`-tagged messages `_fold_subagent_tokens` sums — so + # skipping it drops the child threads' spend from the run's cost entirely. + # Still SKIPPED on a cooperative stop: an armed gate has already decided + # the run, and children may have no rollout yet. + # Rationale: .claude/notes/agents.md § Codex rollout rebuild if state.spawned_children and not state.stopped_early_hit: await self._recover_subagent_tool_calls( state.spawned_children, @@ -1737,10 +1556,9 @@ async def _run_turn_with_streaming( def _messages_from_items(self, items: Any, turn_id: str) -> list[AssistantMessage]: """Rebuild the assistant transcript from a Turn's ``items`` list (fallback). - Same item→block mapping as the streaming path, but Turn items carry no + Same item->block mapping as the streaming path, but Turn items carry no per-item timestamps, so there is no window to measure: the bounds fall back - to now() and ``generation_duration_ms`` is None (unknown), never 0.0 (an - instant generation). Used only when the stream produced no messages. + to now() and ``generation_duration_ms`` is None, never 0.0 (CE058). """ if not items: return [] @@ -1773,8 +1591,7 @@ def _flush() -> None: root_type = getattr(root, "type", None) item_id = getattr(root, "id", "") if root_type is not None and root_type not in _CONTENT_ITEM_TYPES: - # Any tool-like item (generic, not just command/fileChange) → a - # tool_use block, mirroring the streaming path's broad capture. + # Any tool-like item, mirroring the streaming path's broad capture. status = _status_value(getattr(root, "status", "completed")) exit_code = getattr(root, "exit_code", None) is_error = ( @@ -1808,8 +1625,8 @@ def _tool_name(root_type: str | None) -> str: def _tool_parameters(self, root: Any, root_type: str | None) -> dict[str, Any]: """Best-effort ToolStartEvent parameters for any Codex tool item. - Per-kind for the items we understand; an empty dict for unknown tool - kinds (still emitted, just without parameters). + Per-kind for the items we understand; an empty dict for an unknown kind, + which is still emitted, just without parameters. """ if root_type == "commandExecution": return {"command": getattr(root, "command", "")} @@ -1852,13 +1669,10 @@ def _telemetry_for_item( ) -> tuple[CommandTelemetry | None, bool]: """Build (telemetry, is_error) for a completed tool item. - commandExecution/fileChange keep their dedicated rich extractors; every - other tool kind routes through the generic builder so it still produces - countable telemetry. - - The SDK's millisecond stamps arrive as arguments rather than being read - back out of the reducer, so each builder stays a pure function of what - it is given. + commandExecution/fileChange keep their rich extractors; every other kind + routes through the generic builder so it still produces countable + telemetry. The SDK's millisecond stamps arrive as ARGUMENTS rather than + being read back out of the reducer, so each builder stays pure. """ if root_type == "commandExecution": exit_code = getattr(root, "exit_code", None) @@ -1884,8 +1698,8 @@ def _extract_generic_telemetry( ) -> tuple[CommandTelemetry | None, bool]: """CommandTelemetry for any tool item without a dedicated extractor. - Reads status / duration / error generically so MCP calls, web searches, - collab-agent spawns and future tool kinds all render and count uniformly. + Reads status / duration / error generically, so MCP calls, web searches, + collab-agent spawns and future kinds all render and count uniformly. """ try: status_str = _status_value(getattr(root, "status", "") or "") @@ -1941,24 +1755,13 @@ def _handle_collab_completion( Two responsibilities: - 1. SPAWN (``tool == 'spawnAgent'``): remember which Agent call owns each - spawned child thread (so the child's result can nest under it) and the + 1. SPAWN: remember which Agent call owns each spawned child thread, and the spawned model. Follow-up ``wait``/messaging calls reuse the same thread and are NOT new sub-agents. + 2. RESULT: stash the child's returned message as a FALLBACK, used only when + the child's rollout cannot be found later. - Codex emits NO per-sub-agent token breakdown in the parent stream — - every ``thread/tokenUsage/updated`` reports only the PARENT thread's - cumulative usage. The child's real per-generation tokens are recovered - AFTER the turn from its on-disk rollout and reconstructed as nested - ``parent_tool_use_id`` messages (see ``_recover_subagent_tool_calls``); - ``_finalize`` then folds those messages into the turn total. - - 2. RESULT: any collab completion may carry the child's returned message in - ``agents_states[thread].message``. We stash it in ``collab_results`` as - a FALLBACK — used only if the child's rollout can't be found later. When - the rollout IS found, ``_recover_subagent_tool_calls`` rebuilds the - sub-agent's full generation sequence (tool calls + final text) with real - per-generation tokens, so the returned message is just the last of those. + Rationale: .claude/notes/agents.md § Codex rollout rebuild """ tool = _status_value(getattr(root, "tool", "")) receivers = getattr(root, "receiver_thread_ids", None) or [] @@ -1986,29 +1789,15 @@ async def _recover_subagent_tool_calls( ) -> None: """Recover each spawned sub-agent's INNER tool calls AND token usage. - Codex runs every sub-agent on its own child thread whose events never - reach the parent stream, and that child thread persists with *Limited* - rollout policy — which drops ``commandExecution`` events. So neither the - live stream nor ``thread.read`` surfaces the sub-agent's shell commands, - and ``thread/tokenUsage/updated`` only ever reports the PARENT thread, so - per-child tokens never appear in the live stream. - - But the child rollout ALWAYS persists the raw ``function_call`` / - ``local_shell_call`` / ``custom_tool_call`` (+ ``*_output``) ResponseItems - (``should_persist_response_item`` keeps them regardless of mode) AND a - ``token_count`` event with the child thread's cumulative usage. So we - locate the child rollout by thread id and: - - - per inner call, emit one ``CommandTelemetry`` (so the tool row resolves) - plus one nested ``AssistantMessage`` parented to the spawning Agent call - (so the evalboard renders it as an expandable child), carrying that - generation's real per-generation tokens. ``_finalize`` folds these - ``parent_tool_use_id`` messages into the turn total so the run cost - includes the sub-agent, exactly as Claude's total already includes its - bubbled-up sub-agent messages. - - Best-effort: any failure (missing file, parse error) is swallowed so a - recovery hiccup never fails the turn. + Per inner call, emits one ``CommandTelemetry`` (so the tool row resolves) + plus one nested ``AssistantMessage`` parented to the spawning Agent call + (so the evalboard renders it as an expandable child), carrying that + generation's real tokens. ``finalize`` folds those into the turn total. + + Best-effort: any failure is swallowed, so a recovery hiccup never fails the + turn. + + Rationale: .claude/notes/agents.md § Codex rollout rebuild """ home = self._codex_home() for thread_id, parent_tool_id, model in spawned_children: @@ -2057,13 +1846,9 @@ def _codex_home() -> Path: async def _await_rollout_file(self, home: Path, thread_id: str, *, attempts: int = 20) -> Path | None: """Locate a thread's rollout file, polling briefly for the async flush. - The child turn has finished by the time its ``wait`` returns, but the - rollout recorder flushes on a background task, so the file can lag the - parent ``turn/completed`` by a beat. Poll up to ~2s before giving up. - - If ``/sessions`` doesn't exist at all, the binary isn't writing - rollouts there — bail immediately rather than polling for a flush that - can never land (also keeps unit tests with a stub home fast). + The recorder flushes on a background task, so the file can lag the parent + ``turn/completed`` by a beat. A missing ``/sessions`` bails + immediately rather than polling for a flush that can never land. """ if not (home / "sessions").is_dir(): return None @@ -2088,17 +1873,10 @@ def _find_rollout_file(home: Path, thread_id: str) -> Path | None: def _parse_rollout_generations(cls, path: Path) -> list[dict[str, Any]]: """Reconstruct a sub-agent's GENERATIONS from its rollout JSONL. - A ``token_count`` event marks each generation boundary (same as the - parent stream's ``thread/tokenUsage/updated``). We walk the ordered - ``response_item`` lines, accumulating tool calls / assistant text into the - current generation, and close it on each ``token_count`` with that - generation's ``last_token_usage``. Tool CALLS are paired with their OUTPUT - (``*_output``, possibly emitted in a later generation) by ``call_id``. - - Returns ordered generation dicts: ``{"tokens": (input, cached, output, - reasoning) | None, "items": [ordered specs], "tools": [tool-call dicts]}``. - Trailing items with no closing ``token_count`` flush as a final - token-less generation. Partial/corrupt lines are skipped. + A ``token_count`` event marks each generation boundary. Tool CALLS are + paired with their OUTPUT by ``call_id``, since the output can be emitted a + generation later. Trailing items with no closing ``token_count`` flush as a + final token-less generation; corrupt lines are skipped. """ objs: list[dict[str, Any]] = [] for raw in path.read_text(encoding="utf-8").splitlines(): @@ -2110,7 +1888,7 @@ def _parse_rollout_generations(cls, path: Path) -> list[dict[str, Any]]: except json.JSONDecodeError: continue - # Pass 1: tool OUTPUTs by call_id (a call's result can land a generation later). + # Pass 1: tool OUTPUTs by call_id. outputs: dict[str, tuple[str, bool]] = {} for obj in objs: if obj.get("type") == "response_item": @@ -2192,9 +1970,8 @@ def _subagent_tool_name(payload: dict[str, Any]) -> str: def _subagent_parameters(payload: dict[str, Any]) -> dict[str, Any]: """Best-effort parameters for a rollout tool-call ResponseItem. - ``function_call.arguments`` is a JSON string; ``local_shell_call`` carries - an ``action``. Shell-style calls are normalized to ``{"command": ...}`` so - the transcript renders the command line; everything else is passed through. + Shell-style calls are normalized to ``{"command": ...}`` so the transcript + renders the command line; everything else passes through. """ args = payload.get("arguments") if isinstance(args, str) and args: @@ -2235,10 +2012,9 @@ def _subagent_generation_blocks( ) -> tuple[list[ContentBlock], list[CommandTelemetry]]: """Content blocks + tool telemetry for one recovered sub-agent generation. - Blocks are emitted in rollout order (tool calls, assistant text). Each - tool call gets a ``tool_use`` block whose id (``sub::``) - matches a ``CommandTelemetry`` so the evalboard tool row resolves. Inner - tool ids are thread-prefixed to stay unique across the parent's own tools. + Each tool call gets a ``tool_use`` block whose id matches a + ``CommandTelemetry``, so the evalboard tool row resolves. Inner ids are + THREAD-PREFIXED to stay unique across the parent's own tools. """ blocks: list[ContentBlock] = [] telemetries: list[CommandTelemetry] = [] @@ -2275,9 +2051,7 @@ def _subagent_generation_message( """A nested sub-agent generation as an AssistantMessage with real tokens. Parented to the spawning Agent call so it nests in the transcript. Tokens - come from the child's per-generation ``token_count``: the fresh slice - (input - cached) is plain ``input`` and ``cache_creation`` is 0 — Codex - has no separate cache-write fee. + come from the child's per-generation ``token_count``. """ raw_input, cached, output, reasoning = gen["tokens"] or (0, 0, 0, 0) fresh = _fresh_input_tokens(raw_input, cached) @@ -2303,10 +2077,8 @@ def _subagent_text_message( ) -> AssistantMessage: """Fallback nested message: just the sub-agent's returned text, tokenless. - Used only when the child's rollout can't be found, so the answer still - shows under the Agent call even without per-generation detail. ``model`` - is the spawned sub-agent's model (not the parent's), matching the - rollout-found path.""" + Used only when the child's rollout cannot be found. ``model`` is the + SPAWNED sub-agent's model, not the parent's, matching the other path.""" now = datetime.now() return AssistantMessage( started_at=now, @@ -2332,7 +2104,7 @@ def _extract_command_telemetry( ) -> CommandTelemetry | None: """Extract CommandTelemetry from a CommandExecutionThreadItem. - Maps Codex command execution details to the CommandTelemetry format used by Claude Code. + Rationale: .claude/notes/agents.md § Tool-name and argument normalization """ try: @@ -2346,12 +2118,10 @@ def _extract_command_telemetry( # Determine result status from exit code result_status = "success" if exit_code == 0 else "error" if exit_code is not None else "unknown" - # Build result summary with output if available. Store the output WHOLE: - # CommandTelemetry.result_summary is the untruncated tool-result body (its - # length drives CommandTelemetry.result_tokens), so truncating here would - # under-report tool-output size for every command (see CE043). The output is - # already bounded by the Codex harness's own exec-output truncation; any - # further trimming for display belongs in the renderers/reports, not capture. + # Store the output WHOLE: result_summary is the untruncated tool-result + # body and its length drives result_tokens, so trimming here + # under-reports tool-output size for every command (CE043). Display + # trimming belongs in the renderers, not capture. summary_parts = [f"Exit code: {exit_code}" if exit_code is not None else "Command executed"] if output and len(output.strip()) > 0: summary_parts.append(f"Output: {output}") @@ -2396,12 +2166,9 @@ def _extract_file_change_telemetry( ) -> CommandTelemetry | None: """Build CommandTelemetry for a Codex fileChange item. - Recorded as a ``Write`` tool call so cross-agent criteria that count or - match file edits (``command_executed``, ``commands_efficiency``) see the - same signal they get from Claude's Write/Edit tool calls. A failed/declined - apply_patch is recorded as an ``error`` (the old ``status != "error"`` - test never matched the real PatchApplyStatus values, so failed patches - were scored as successful writes). + Recorded as a ``Write`` so cross-agent criteria see the same signal they + get from Claude's Write/Edit calls. A failed or declined apply_patch is an + ``error``, never a successful write. """ try: paths = [str(c.path) for c in changes if hasattr(c, "path")] if changes else [] @@ -2433,34 +2200,20 @@ def _extract_file_change_telemetry( def _token_usage_from_sdk(self, sdk_token_usage: Any) -> TokenUsage | None: """This turn's own slice of the Codex SDK's thread-cumulative total. - Single conversion site for both the TurnEndEvent and the AgentEndEvent, - so cached-input tokens can't be captured in one path but dropped in the - other. The Codex SDK does not surface cost, so we derive it from the - pricing table keyed on the effective model (None if the model is unpriced). - - ``ThreadTokenUsage.total`` counts the whole THREAD, not the turn — that is - the SDK's contract, and ``last`` is the per-generation delta beside it. The - Codex thread is created once per task and reused for every turn (see - ``communicate``), so by turn N ``total`` still carries turns 1..N-1. The - orchestrator sums per-turn usages into the task total, so handing it the - cumulative figure books turn 1 again on turn 2, turns 1-2 again on turn 3, - and so on: the task total becomes a sum of prefix sums, inflating an - N-turn task by roughly (N+1)/2. Subtracting the baseline captured at the - end of the previous turn leaves just this turn. - - Cache-bucket convention (Codex/OpenAI): the SDK's ``input_tokens`` is the - FULL prompt count, *inclusive* of the cached prefix. The fresh slice - (``input_tokens - cached``) is the uncached input (OpenAI bills no separate - cache-write fee), so: - - uncached_input_tokens = input - cached - cache_creation_input_tokens = 0 (no separate cache-write bucket) - cache_read_input_tokens = cached - input_tokens (derived) = uncached + cache_read == the full prompt - - Cost bills the uncached slice at the input rate — identical to the old - "fresh as cache-write" pricing since OpenAI's cache-write rate == input - rate, just labeled honestly. + Single conversion site for both the TurnEndEvent and the AgentEndEvent, so + cached-input tokens cannot be captured in one path and dropped in the + other. The SDK surfaces no cost, so it is rate-carded. + + ``ThreadTokenUsage.total`` counts the whole THREAD, and the thread is + reused for every turn — so the baseline captured at the end of the previous + turn is subtracted to leave just this one. + + Cache-bucket convention (Codex/OpenAI): ``input_tokens`` is the FULL prompt + count, INCLUSIVE of the cached prefix, and there is no separate + cache-write fee. So ``uncached = input - cached``, ``cache_creation = 0``, + ``cache_read = cached``. + + Rationale: .claude/notes/agents.md § Codex rollout rebuild """ if not sdk_token_usage: return None @@ -2474,7 +2227,7 @@ def _token_usage_from_sdk(self, sdk_token_usage: Any) -> TokenUsage | None: ) turn = cumulative.since(self._thread_usage_baseline) self._thread_usage_baseline = cumulative - # Fresh (uncached) prompt slice = full prompt minus the cached prefix. + # Fresh slice = full prompt minus the cached prefix. uncached = _fresh_input_tokens(turn.input, turn.cached) cost = calculate_cost( self._effective_model() or "", @@ -2492,17 +2245,16 @@ def _token_usage_from_sdk(self, sdk_token_usage: Any) -> TokenUsage | None: def _advance_usage_baseline(self, usage: TokenUsage | None) -> None: """Move the thread baseline past a turn whose SDK total never arrived. - The crash/timeout fallback (``_token_usage_from_messages``) reads - per-generation tokens straight off the flushed messages, so the crashed - turn itself is right — but the thread's cumulative total kept climbing on - the SDK side. Without advancing past it here, the next turn's delta would - re-book everything the crashed turn already reported. + The crash fallback reads per-generation tokens off the flushed messages, + so the crashed turn itself is right — but the thread's cumulative total + kept climbing, and without advancing past it the NEXT turn's delta re-books + everything this one already reported. """ if usage is None: return base = self._thread_usage_baseline - # SDK ``input_tokens`` is the full prompt, cached prefix included, so the - # input baseline advances by uncached + cache_read. + # SDK ``input_tokens`` is the full prompt, so the input baseline advances + # by uncached + cache_read. self._thread_usage_baseline = _ThreadTotals( input=base.input + usage.uncached_input_tokens + usage.cache_read_input_tokens, output=base.output + usage.output_tokens, @@ -2513,15 +2265,10 @@ def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[Transc """Add recovered sub-agent (child-thread) tokens to the parent turn total. Codex bills children on separate threads, so the parent's streamed total - (``_token_usage_from_sdk`` / parent-only ``_token_usage_from_messages``) - omits them. The child generations were reconstructed as - ``parent_tool_use_id``-tagged ``AssistantMessage``s carrying their real - per-generation tokens (fresh slice in ``input_tokens``, ``cache_read`` for - the cached prefix, no ``cache_creation`` — Codex has no cache-write fee). - Sum those here as ``uncached_input``, priced per child model, to make the - turn total all-inclusive — the same end state Claude reaches naturally, - where sub-agent messages bubble into the parent stream. A no-op when no - child generations were recovered. + omits them. Summing the recovered ``parent_tool_use_id``-tagged messages + here, priced PER CHILD MODEL (sub-agents may run a different one), makes + the turn total all-inclusive — the same end state Claude reaches naturally. + A no-op when nothing was recovered. """ children = [ m @@ -2534,8 +2281,7 @@ def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[Transc return parent base = parent or TokenUsage() - # Price each child generation on its own model (sub-agents may run a - # different model than the parent), then sum. + # Each child generation on its own model, then sum. child_cost = 0.0 for m in children: child_cost += ( @@ -2560,16 +2306,13 @@ def _fold_subagent_tokens(self, parent: TokenUsage | None, messages: list[Transc def _token_usage_from_messages(self, messages: list[TranscriptMessage]) -> TokenUsage | None: """Sum per-generation tokens off the captured assistant messages. - Crash/timeout fallback for ``_finalize``: when the stream raises before it - returns the SDK ``total`` (so ``_token_usage_from_sdk`` has nothing), the - per-generation tokens were already recorded on the flushed - ``AssistantMessage``s (fresh slice in ``input_tokens``, cached prefix in - ``cache_read``). Summing them recovers the tokens/cost a crashed turn - actually spent. Returns None when nothing was captured, matching - ``_token_usage_from_sdk``'s empty contract. + Crash/timeout fallback: when the stream raises before returning the SDK + ``total``, the per-generation tokens were already recorded on the flushed + messages, so summing them recovers what the crashed turn actually spent. + None when nothing was captured, matching the SDK path's empty contract. """ - # PARENT-thread messages only — sub-agent (separate-thread) tokens are - # added via _fold_subagent_tokens, not summed here (would double-count). + # PARENT-thread messages ONLY: sub-agent tokens are added by + # _fold_subagent_tokens, and summing them here would double-count. assistant = [m for m in messages if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None] if not assistant: return None diff --git a/src/coder_eval/agents/opencode_agent.py b/src/coder_eval/agents/opencode_agent.py index 42fca0d4d..1d3a17a24 100644 --- a/src/coder_eval/agents/opencode_agent.py +++ b/src/coder_eval/agents/opencode_agent.py @@ -1,31 +1,18 @@ """OpenCode agent implementation (the open-source terminal coding agent). -Drives the ``opencode`` CLI in non-interactive mode:: - - opencode run --format json -m --dir [--auto] [--pure] - -which streams **newline-delimited JSON events** on stdout. Each line is one -event; this module reduces that stream into the standardized coder_eval event -protocol (``AgentStart`` / ``TurnStart`` / ``ToolStart`` / ``ToolEnd`` / -``TurnEnd`` / ``AgentEnd``) and lets :class:`EventCollector` build the -``TurnRecord`` — so no telemetry is assembled by hand here. - -Envelope normalization ----------------------- -The CLI emits two envelope shapes on the same stream: the normal form carries -its payload under ``part`` — ``{"type": "tool_use", "sessionID": …, -"part": {…}}`` — while the CLI's own error path emits a flat object with no -``part`` (``{"type": "error", "sessionID": …, "error": {…}}``). :func:`_unwrap` -normalizes both to ``(event_type, payload)`` so the dispatch table is written -once. (The ``session.next.*``/``properties`` envelopes belong to ``opencode -serve``'s HTTP/SSE surface and never appear here — see the note on the event -constants below.) - -Session continuity ------------------- -The ``sessionID`` observed on the first event is retained and replayed via -``--session`` on the next ``communicate()`` call, which is what makes multi-turn -(dialog-mode) evaluation work against a stateless CLI invocation. +Drives the ``opencode`` CLI in non-interactive mode, which streams +newline-delimited JSON events on stdout, and reduces that stream into the +standardized coder_eval event protocol so :class:`EventCollector` builds the +``TurnRecord``. + +The CLI emits TWO envelope shapes on the same stream: the normal form carries +its payload under ``part``, while the CLI's own error path emits a flat object +with none. :func:`_unwrap` normalizes both to ``(event_type, payload)`` so the +dispatch table is written once. + +The ``sessionID`` observed on the first event is replayed via ``--session`` on +the next ``communicate()``, which is what makes dialog mode work against a +stateless CLI invocation. """ from __future__ import annotations @@ -85,9 +72,7 @@ logger = logging.getLogger(__name__) # Grace period between SIGTERM and SIGKILL when tearing down the CLI subprocess. -# Doubles as the post-EOF exit grace in _settle_turn when no turn deadline is -# configured (a CLI that closed its stream but won't exit gets this long to die -# before the turn is crashed). +# Doubles as the post-EOF exit grace in _settle_turn when no deadline is set. _TERM_GRACE_SECONDS = 5.0 # SIGKILL does not exist on Windows (where the process-group sweep is a no-op @@ -95,15 +80,15 @@ # platform, falling back to SIGTERM for the direct-pid kill_sync path. _SIGKILL: signal.Signals = getattr(signal, "SIGKILL", signal.SIGTERM) -# How long to keep draining stdout/stderr after the CLI process has been reaped. -# `opencode run` leaves a local server child holding the inherited pipes open, so -# EOF never arrives on its own and every post-exit read must be bounded. +# How long to keep draining stdout/stderr after the CLI has been reaped: +# `opencode run` leaves a server child holding the pipes open, so EOF never +# arrives on its own. +# Rationale: .claude/notes/agents.md § Reaping the CLI harnesses _DRAIN_SECONDS = 2.0 -# Event type strings emitted by `opencode run --format json`. These are the CLI's -# OWN compact vocabulary, captured from a live run — NOT the `session.next.*` -# names in the server's OpenAPI schema, which describe the HTTP/SSE surface of -# `opencode serve` instead. The two are not interchangeable. +# The CLI's OWN compact vocabulary, captured from a live run — NOT the +# `session.next.*` names in the server's OpenAPI schema, which describe +# `opencode serve`'s HTTP/SSE surface. The two are not interchangeable. _STEP_START = "step_start" _STEP_FINISH = "step_finish" _TEXT = "text" @@ -111,10 +96,8 @@ _ERROR = "error" # The full recognized vocabulary. A zero-exit turn that recognized NOTHING from -# this set captured zero telemetry, and is crashed rather than reported as a -# clean empty success — an earlier version of this harness parsed the wrong -# vocabulary and scored SUCCESS 1.0 with zero turns and zero tokens, which is -# indistinguishable from a real pass in every aggregate. See _settle_turn. +# it captured zero telemetry and is crashed, not scored. +# Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash _RECOGNIZED_EVENTS = frozenset({_STEP_START, _STEP_FINISH, _TEXT, _TOOL_USE, _ERROR}) # How many distinct unrecognized event-type strings to retain for the crash @@ -122,11 +105,8 @@ _MAX_UNRECOGNIZED_TYPES = 8 # OpenCode's native tool names -> the canonical (Claude) vocabulary that every -# criterion is written against. Mirrors codex_agent's _TOOL_ITEM_NAMES: without -# it a `command_executed` criterion with `tool_name: Bash` matches NOTHING on an -# OpenCode run, and the shell-aware `parameters["command"]` extraction in -# criteria/command_executed.py degrades to raw-JSON matching — so the same task -# scores differently per harness. Unknown tools pass through unchanged. +# criterion is written against. Unknown tools pass through unchanged. +# Rationale: .claude/notes/agents.md § Tool-name and argument normalization _TOOL_NAME_MAP: dict[str, str] = { "bash": "Bash", "read": "Read", @@ -141,40 +121,18 @@ "todowrite": "TodoWrite", "todoread": "TodoRead", "task": "Agent", - # The GPT-family edit tool. OpenCode exposes a provider-specific tool set, so - # the vocabulary varies by MODEL within this one harness: a live 174-task run - # showed DeepSeek using write/edit 199 times and apply_patch 0, while GPT-5.6 - # used apply_patch 120 times and write/edit 0. Unmapped, every - # `tool_name: Write` / `tool_name: Edit` criterion scores 0 on a GPT-family - # model that edited the file correctly. Maps to `Write` to match codex_agent's - # `_TOOL_ITEM_NAMES["apply_patch"] = "Write"`, so one criterion reads the same - # on both harnesses. + # The GPT-family edit tool: OpenCode's tool set varies by MODEL within this + # one harness. `Write` matches codex_agent's own mapping. "apply_patch": "Write", - # OpenCode's native skill loader. Without this entry `skill_triggered` (which - # keys on the canonical `Skill`) and any `command_executed` written against - # `tool_name: Skill` read false on every OpenCode run — the engagement happened - # but no criterion could see it. + # OpenCode's native skill loader; `skill_triggered` keys on the canonical name. "skill": "Skill", } -# OpenCode per-tool INPUT-arg key -> canonical (Claude) key. Mirrors -# antigravity_agent's _ANTIGRAVITY_ARG_RENAME and completes what _TOOL_NAME_MAP -# starts: normalizing the tool NAME alone still leaves a `command_executed` with -# a non-Bash `tool_name` matching against a differently-keyed JSON blob (see -# criteria/command_executed.py, which falls back to `json.dumps(parameters)` for -# every tool but Bash), so the same task scores differently per harness. Keyed by -# the canonical tool name (post _TOOL_NAME_MAP); unlisted keys pass through. -# -# `bash` needs no entry: OpenCode already names it `command`, which is why the -# Bash-only shell-aware extraction in command_executed.py was correct as-is. -# `glob`/`grep`/`list` also need none — their `path` already matches Claude's. -# -# BOTH file-path spellings are mapped because the CLI has MOVED: a live capture -# on 2026-08-13 emitted `filePath` (see the fixture in tests/test_opencode_agent.py), -# while the tool schemas registered by the CLI installed at the time of writing -# read `path` (`read`/`write`/`edit` all take `{path, ...}`). Accepting both keeps -# telemetry canonical across the CLI versions a run might use, and neither -# spelling collides with a legitimate parameter of these three tools. +# OpenCode per-tool INPUT-arg key -> canonical (Claude) key, keyed by the +# canonical tool name (post _TOOL_NAME_MAP). Unlisted keys pass through. +# `bash`/`glob`/`grep`/`list` need no entry — their keys already match Claude's. +# BOTH file-path spellings are mapped because the CLI has MOVED between them. +# Rationale: .claude/notes/agents.md § Tool-name and argument normalization _OPENCODE_ARG_RENAME: dict[str, dict[str, str]] = { "Read": {"path": "file_path", "filePath": "file_path"}, "Write": {"path": "file_path", "filePath": "file_path"}, @@ -185,17 +143,15 @@ "newString": "new_string", "replaceAll": "replace_all", }, - # The skill loader's argument. With this rename, `skill_triggered` reads the - # agent-agnostic `parameters["skill"]` on every harness instead of carrying a - # per-harness alternative list in a criterion that must know nothing about - # harnesses. + # So `skill_triggered` reads the agent-agnostic `parameters["skill"]` rather + # than carrying a per-harness alternative list. "Skill": {"name": "skill"}, } # Config fields the OpenCode CLI has no equivalent knob for. `experiments/default.yaml` -# sets `allowed_tools` on every task, so these are silently dropped by default — -# warn once at start() rather than letting a task believe it constrained the agent. -# `plugins` is NOT here: its skills half is honored via _plugin_skill_dirs below. +# sets `allowed_tools` on every task, so start() warns once rather than letting a +# task believe it constrained the agent. `plugins` is NOT here: its skills half is +# honored. Per-harness table: docs/agents/HARNESS_PARITY.md. _UNSUPPORTED_CONFIG_FIELDS: tuple[str, ...] = ( "system_prompt", "system_prompt_file", @@ -203,24 +159,10 @@ "disallowed_tools", ) -# --- skill injection ------------------------------------------------------ -# -# A `plugins:` entry is a Claude-plugin root. Claude Code reads its skills from -# the `skills` field of `/.claude-plugin/plugin.json` (conventionally -# `./skills/`). OpenCode has no plugin knob, but it does load skills from -# `skills.paths` in its config — so mapping the plugin root to that directory is -# what makes one `plugins:` line mean the same thing on both harnesses. -# -# The config is handed over through OPENCODE_CONFIG_CONTENT, which OpenCode -# merges as a final local-scope layer. That was chosen over writing -# `/.opencode/skills/` because it (a) writes nothing into the sandbox -# that is later preserved as run artifacts and inspected by file criteria, and -# (b) does not depend on how the CLI resolves a project root from `--dir`. -# Verified orthogonal to `--pure`, which skips external *plugins*, not -# configured skill paths. -# -# Only the skills half of a plugin is honored. A Claude plugin's agents, hooks, -# commands and MCP servers have no OpenCode equivalent and are still dropped. +# Skill injection: each plugin root's skills dir is merged into `skills.paths` +# through this variable, which OpenCode applies as a final local-scope layer. +# Only the SKILLS half of a plugin is honored. +# Rationale: .claude/notes/agents.md § Skills, per harness _CONFIG_CONTENT_ENV = "OPENCODE_CONFIG_CONTENT" # ToolEndStatus -> CommandTelemetry.result_status (the persisted tri-state). @@ -235,10 +177,9 @@ def _unwrap(obj: dict[str, Any]) -> tuple[str, dict[str, Any]]: """Normalize an OpenCode CLI event to ``(event_type, payload)``. - Every line is ``{type, timestamp, sessionID, part: {...}}`` with the payload - under ``part`` — except the CLI's own error line, which is flat - (``{type: "error", sessionID, error: {...}}``). Returning the top-level dict - for the flat case is safe: the accessors read named keys, never iterate. + Every line carries its payload under ``part`` except the CLI's own error + line, which is flat. Returning the top-level dict for that case is safe: the + accessors read named keys, never iterate. """ event_type = str(obj.get("type") or "") part = obj.get("part") @@ -250,8 +191,7 @@ def _unwrap(obj: dict[str, Any]) -> tuple[str, dict[str, Any]]: def _epoch_ms_to_dt(value: Any) -> datetime | None: """Convert OpenCode's epoch-millisecond timestamps to naive local datetimes. - Naive-local matches what the rest of the telemetry uses (``datetime.now()``), - so durations computed against these stay consistent. + Naive-local matches the rest of the telemetry, so durations stay consistent. """ if not isinstance(value, int | float): return None @@ -264,8 +204,7 @@ def _epoch_ms_to_dt(value: Any) -> datetime | None: def _canonical_params(tool_name: str, params: dict[str, Any]) -> dict[str, Any]: """Rename a tool call's argument keys to the canonical cross-agent vocabulary. - Order is preserved and unlisted keys pass through untouched, so this only ever - re-labels what ``_OPENCODE_ARG_RENAME`` names for this tool. + Order is preserved and unlisted keys pass through untouched. """ rename = _OPENCODE_ARG_RENAME.get(tool_name) if not rename: @@ -299,10 +238,8 @@ def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str self.messages: list[TranscriptMessage] = [] self.text_parts: list[str] = [] self.step_count = 0 - # Steps the CLI reported as FINISHED (`step_finish`), as opposed to - # `step_count`, which counts the ones it started. `_settle_turn` needs the - # distinction: a finished step is the CLI's own claim that a generation - # completed, so one that booked no tokens means the token schema moved. + # Steps the CLI reported as FINISHED, as opposed to `step_count`, which + # counts the ones it started. `_settle_turn` needs the distinction. self.steps_finished = 0 self.turn_id: str = "" # True between a step's `step_start` and its `step_finish`. `finalize` @@ -310,18 +247,9 @@ def __init__(self, *, task_id: str, iteration: int, user_input: str, model: str self.step_open = False self.step_started_at: datetime | None = None # Where the NEXT generation window starts: the previous step's finish. - # The CLI announces a step only once it is already producing one, so a - # window bounded by `step_start` drops the model time that PRODUCED the - # step into the gap before it. Measured on tasks/hello_date with a live - # claude-haiku-4.5: two gaps of 857 ms and 851 ms, carrying no tool - # (the Write inside them took 7 ms), attributed to nothing — 24% of the - # turn's wall clock, enough on its own to hold OpenCode above the - # evalboard's 25% "Unaccounted" red threshold. - # - # None until the first step finishes, and deliberately so: the first - # window keeps its own `step_start`, because everything before it is - # CLI process spawn, not model time. Tiling that in would report Node's - # boot as generation. Same shape as Codex's `gen_mark_ms`. + # None until the first step finishes, and deliberately so — everything + # before the first `step_start` is CLI process spawn, not model time. + # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark: datetime | None = None self.step_text_parts: list[str] = [] self.step_tool_ids: list[str] = [] @@ -362,13 +290,8 @@ def on_step_start(self, part: dict[str, Any]) -> None: self.step_started_at = datetime.now() self.step_text_parts = [] self.step_tool_ids = [] - # There is no per-step span list to reset here any more, and that whole - # class of defect is gone with it: `timing.subtract_tool_time` - # sees every span at once and clips each to the window it overlaps, so - # a call closing in the gap before this `step_start` needs nobody to - # remember it. The reset rule that used to live here was wrong once - # (clearing at `step_start` wiped the span before `step_finish` could - # subtract it — a 100% overstatement of that window). + # No per-step span list to reset here any more: the collector sees every + # span at once and clips each to the window it overlaps. self.emit( TurnStartEvent( task_id=self.task_id, @@ -391,12 +314,11 @@ def on_tool_use(self, part: dict[str, Any]) -> None: """A ``tool_use`` event carries the tool's whole state under ``state``. In practice the CLI emits one already-``completed`` event per call rather - than a call/result pair, so the matching ``ToolStart``/``ToolEnd`` are - both synthesized here. A non-terminal state (``pending``/``running``) is - still handled: the tool is left open and closed by a later event for the - same ``callID``, or force-closed as ``unresolved`` if the turn dies first. - Execution timestamps come from ``state.time``, so ``duration_ms`` reflects - the tool's real runtime rather than our parse instant. + than a call/result pair, so both ``ToolStart`` and ``ToolEnd`` are + synthesized here. A non-terminal state is still handled: the tool is left + open and closed by a later event for the same ``callID``, or force-closed + as ``unresolved``. Execution timestamps come from ``state.time``, so + ``duration_ms`` is the tool's real runtime, not our parse instant. """ state = part.get("state") state = state if isinstance(state, dict) else {} @@ -427,12 +349,10 @@ def on_tool_use(self, part: dict[str, Any]) -> None: ToolStartEvent(task_id=self.task_id, thread_id=self.thread_id, turn_id=self.turn_id, tool=telemetry) ) else: - # A SECOND event for a call already open — the pending/running-then- - # completed lifecycle. The first event routinely carries no `input` - # (the CLI has not finished assembling the call), so freezing the - # first event's view would leave `parameters` permanently `{}` and - # zero every `command_executed` row while the run looked normal. - # Later evidence wins; absent evidence never clears what we have. + # A SECOND event for a call already open. The first routinely carries + # no `input` yet, so freezing its view would leave `parameters` + # permanently `{}` and zero every `command_executed` row while the run + # looked normal. Later evidence wins; absent evidence clears nothing. if params: telemetry.parameters = _canonical_params(telemetry.tool_name, params) if started is not None: @@ -452,10 +372,8 @@ def on_tool_use(self, part: dict[str, Any]) -> None: message = None status = ToolEndStatus.OK - # `times` is the SAME dict read at the top of this function: `state` is - # bound once at the start and nothing between here and there rebinds or - # mutates it, so re-reading `state["time"]` produced an identical value - # from an identical source. One read, one name. + # `times` is the SAME dict read at the top: nothing between rebinds or + # mutates `state`. self._close_tool( call_id, status=status, @@ -505,9 +423,7 @@ def _close_tool( def _rate_card_cost(self) -> float | None: """Price the captured buckets from the static rate card. - ``None`` when the model is unpinned or unpriced, matching "nothing could - be priced". See :meth:`_resolve_cost` for how this composes with the - stream's own ``cost`` reporting. + ``None`` when the model is unpinned or unpriced. """ if not self.model or self.usage.is_empty(): return None @@ -522,19 +438,11 @@ def _rate_card_cost(self) -> float | None: def _resolve_cost(self) -> float | None: """Decide the turn's cost: the stream's own accounting vs the rate card. - A non-zero cost the CLI reported always wins — it is the provider's own - accounting, and (on OpenRouter) per-request routing makes it strictly - better than a static headline rate. The rate card fills two gaps that - would otherwise book tokens with no money and silently understate the - run-level bill: - - - the stream reported no ``cost`` at all (a provider or auth mode that - omits it, or a turn that died before its first ``step_finish``); - - the stream reported ``cost: 0`` for tokens the rate card prices above - zero. OpenCode reports 0 when its own model registry has no price for - the model, or under subscription-style auth — neither means the tokens - were free. A genuinely free model has an all-zero rate entry (or no - entry), so it still resolves to the stream's 0 here. + A non-zero cost the CLI reported always wins. The rate card fills two gaps + that would otherwise book tokens with no money: no ``cost`` field at all, + and ``cost: 0`` for tokens the rate card prices above zero. + + Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ rate = self._rate_card_cost() if not self.saw_cost: @@ -559,19 +467,10 @@ def _warn_token_shape(self, message: str, *args: Any) -> None: def _as_int(self, bucket: str, value: Any) -> int: """Coerce one stream-supplied token count, warning instead of raising. - A bare ``int()`` raises on anything non-numeric (``int("abc")`` -> - ``ValueError``; ``int({...})``/``int([...])`` -> ``TypeError``), which - ``communicate``'s ``except Exception`` turns into an ``AgentCrashError`` - — categorized ``AGENT_CRASH`` with ``max_retries=2``, so ONE mistyped - bucket burns three full attempts and lands the task as ERROR. - - That is the opposite of the policy every neighbouring field follows: - ``_epoch_ms_to_dt`` type-checks, ``state``/``input``/``cost``/``total`` are - all ``isinstance``-gated, and ``_fresh_input_slice`` exists specifically to - warn-once on token-schema drift rather than fail. A changed type in the - very same ``tokens`` dict is drift too, so it is reported the same way and - the turn survives on the buckets it could read. It is also what makes - ``_handle_line``'s advertised "Never raises on bad input" true. + This is what makes ``_handle_line``'s advertised "Never raises on bad + input" true. + + Rationale: .claude/notes/agents.md § Why token-shape drift warns instead of raising """ if isinstance(value, bool) or not isinstance(value, int | float | str): if value is not None: @@ -599,26 +498,19 @@ def _fresh_input_slice( ) -> int: """Decide what ``tokens.input`` means on this stream — per step, from evidence. - coder_eval's ``uncached_input_tokens`` is the fresh slice only (cost bills it - at the input rate and the cache buckets separately), and two conventions for - ``input`` exist in the wild: - - - **flat** — ``input`` already IS the fresh slice and - ``total = input + output + reasoning + cache.read + cache.write``. This is - what a live capture on the current CLI shows (observed 2026-08-13: - ``7966 = 6796 + 128 + 18 + 1024`` exactly). - - **nested** — cached tokens are counted inside ``input`` (the OpenAI - ``prompt_tokens`` convention), so ``total = input + output + reasoning`` - and the fresh slice subtracts the cache buckets. - - The stream's own ``total`` arbitrates per step, so a CLI upgrade that flips - the convention re-classifies itself instead of silently mis-booking a bucket. - With no cache traffic the conventions agree. With no usable ``total`` the - flat (live-verified) reading is taken — but if cache traffic is present that - is an UNVERIFIABLE assumption (the original mapping bug was exactly an - unverified assumption of this kind), so it warns once per turn rather than - defaulting in silence. A ``total`` matching NEITHER warns loudly — the - schema moved, and cost should not be trusted blind. + Two conventions exist in the wild: **flat**, where ``input`` already IS the + fresh slice and ``total = input + output + reasoning + cache``, and + **nested**, where the cache buckets are counted inside ``input`` (the + OpenAI ``prompt_tokens`` convention) and ``total = input + output + + reasoning``. + + The stream's own ``total`` arbitrates PER STEP. With no cache traffic the + two agree. With no usable ``total`` the flat reading is taken, but warns + once if cache traffic is present — that is an unverifiable assumption, and + the original mapping bug was exactly one of those. A ``total`` matching + NEITHER warns loudly. + + Rationale: .claude/notes/agents.md § Token accounting, per harness """ total = tokens.get("total") if not isinstance(total, int): @@ -640,8 +532,8 @@ def _fresh_input_slice( if total == nested: # implies cache traffic, since flat was checked first fresh = raw_in - cr - cw if fresh < 0: - # The stream contradicts itself: `total` says the cache buckets nest - # inside `input`, but `input` is too small to contain them. + # The stream contradicts itself: `total` says the cache buckets + # nest inside `input`, but `input` is too small to hold them. self._warn_token_shape( "tokens.total says the cache buckets nest inside input, but input(%d) < " + "cache.read(%d) + cache.write(%d); keeping `input` as the fresh slice", @@ -675,9 +567,8 @@ def on_step_finish(self, part: dict[str, Any]) -> None: step_cr = self._as_int("cache.read", cache.get("read") or 0) step_in = self._fresh_input_slice(tokens, raw_in, raw_out, step_reasoning, step_cw, step_cr) - # Reasoning tokens are billed at the output rate but reported apart from - # `output`, so fold them in for the turn total; the per-message record - # keeps `reasoning_tokens` separately for visibility. + # Reasoning bills at the output rate but is reported apart from `output`, + # so fold it into the turn total; the per-message record keeps it apart. step_out = raw_out + step_reasoning self.usage = TokenUsage( @@ -704,9 +595,7 @@ def on_step_finish(self, part: dict[str, Any]) -> None: for i, tool_id in enumerate(self.step_tool_ids, start=len(blocks)): blocks.append(ContentBlock(block_type="tool_use", sequence=i, tool_use_id=tool_id)) - # Tile from the previous step's finish. The RAW window only — - # `timing.subtract_tool_time` takes the tool union back out of - # it, once, for every harness. + # Tile from the previous step's finish. The RAW window only. started, generation_ms = close_window( mark=self.gen_mark if self.gen_mark is not None else step_start, now=completed, @@ -729,20 +618,15 @@ def on_step_finish(self, part: dict[str, Any]) -> None: message_id=str(part.get("messageID") or "") or None, ) ) - # A message was appended, so the next window starts where this one - # ended. Only `step_finish` advances the mark: a step that never - # finished published nothing, so tiling past it would attribute its - # time to whichever step finishes next. There is no span list to clear - # alongside it any more — see `on_step_start`. + # A message was appended, so the next window starts where this one ended. + # Only `step_finish` advances the mark. self.gen_mark = completed - # And so is this step's own start stamp, because it has now been SPENT. - # It is passed to `close_window` as `item_start`, whose `min()` pulls - # the window open to cover it; left in place, a second `step_finish` - # with no intervening `step_start` would reopen the next window back at - # the previous step's start and publish that whole span a second time. - # The `min()` still defends a genuinely OPEN step against a backwards - # clock, which is what it is for — this reducer's stamps are raw - # `datetime.now()` and are not on a `TurnClock`. + # SPENT state, cleared HERE and not only in `on_step_start`: a second + # `step_finish` with no intervening start would otherwise republish this + # step's whole span as the next one's. The `min()` in `close_window` still + # defends a genuinely OPEN step against a backwards clock, which is what + # it is for — this reducer's stamps are raw `datetime.now()`. + # Rationale: .claude/notes/agents.md § Per-harness generation marks self.step_started_at = None self.emit( TurnEndEvent( @@ -762,8 +646,8 @@ def on_step_finish(self, part: dict[str, Any]) -> None: def on_error(self, part: dict[str, Any]) -> None: """Record the CLI's own structured error, which ``_settle_turn`` crashes on. - The payload is the flat envelope (no ``part``), and its shape varies: a - nested ``error.data.message`` when the CLI has one, otherwise the error's + The payload is the flat envelope, and its shape varies: a nested + ``error.data.message`` when the CLI has one, otherwise the error's ``name``. Anything else degrades to its string form rather than raising. """ error = part.get("error") @@ -789,11 +673,9 @@ def finalize( """Close orphaned tools and emit the terminal ``AgentEndEvent``. Idempotent: the protocol allows EXACTLY ONE ``AgentEndEvent`` per - ``communicate()``, and the outer ``except Exception`` guard can fire after - a normal finalize (e.g. a failure while building the record). The first - call wins so a late crash cannot emit a second terminal event into the - caller's ``stream_callback``; it still raises, so the failure is not - swallowed. + ``communicate()``. + + Rationale: .claude/notes/agents.md § Shared turn lifecycle """ if self.finalized: return @@ -803,15 +685,9 @@ def finalize( cost = self._resolve_cost() if cost is not None: usage = usage.model_copy(update={"total_cost_usd": cost}) - # A step still open here never received its `step_finish` — the turn - # died between the two (crash, timeout, cancel) or was cut cleanly - # (should_stop, max_turns). Either way its TurnStartEvent must be - # closed, or the protocol's one-pair-per-inner-turn contract - # (Agent.communicate) is broken and every renderer shows a turn that - # opens and never ends. Unlike the siblings, the completed steps have - # already closed themselves in `on_step_finish`, so this fires ONLY for - # the straggler. TurnEndStatus mirrors AgentEndStatus value-for-value - # precisely so this conversion is total. + # A step still open never received its `step_finish`; close it or the + # one-pair-per-inner-turn contract breaks. Completed steps already closed + # themselves, so this fires ONLY for the straggler. if self.step_open: self.step_open = False self.emit( @@ -854,14 +730,11 @@ def finalize( class OpenCodeAgent(Agent[OpenCodeAgentConfig]): """Runs the ``opencode`` CLI as a subprocess, one invocation per turn.""" - # `should_stop` is polled at every event boundary — i.e. tool-call - # granularity — and honored by terminating the CLI subprocess cleanly. + # `should_stop` is polled at every event boundary (tool-call granularity). supports_cooperative_stop: ClassVar[bool] = True - # OpenCode neither appends to nor replaces the system prompt — `system_prompt` - # is in `_UNSUPPORTED_CONFIG_FIELDS` (no CLI knob), so the honest regime is - # `"unknown"` (also the base default). Declared explicitly so the run marker - # is deliberate rather than an unset oversight. + # No CLI knob for `system_prompt`, so the honest regime is `"unknown"`. + # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker system_prompt_semantics: ClassVar[SystemPromptSemantics] = "unknown" def __init__( @@ -873,19 +746,12 @@ def __init__( ) -> None: """Every parameter the agent factory can pass is DECLARED, not absorbed. - ``create_agent`` calls ``agent_class(config, route=route, **kwargs)`` through - a ``cast(Any, ...)``, so pyright checks nothing at the call site; a ``**_`` - sink on this side would mean nothing checks it at runtime either. The - orchestrator depends on that TypeError as a signal — it gates - ``cost_log_tags`` on ``supports_cost_log_tags`` precisely "otherwise the - agent-agnostic factory would forward it into ... constructors that don't - declare it and crash with TypeError" — so a mis-gated kwarg must be loud - here rather than silently dropped. - ``route`` is accepted for factory parity and deliberately unused: the CLI - owns its own provider configuration (see ``docs/agents/OPENCODE.md``), so - the run's Bedrock/Anthropic routing does not apply to it. ``task_id`` only - labels the emitted event stream. + owns its own provider configuration (``docs/agents/OPENCODE.md``), so the + run's Bedrock/Anthropic routing does not apply. ``task_id`` only labels + the event stream. + + Rationale: .claude/notes/agents.md § Why the constructors declare every kwarg """ self.config = config self.route = route @@ -896,10 +762,9 @@ def __init__( self._skill_dirs: list[str] = [] self._session_id: str | None = None self._process: asyncio.subprocess.Process | None = None - # Process-group ids (== the CLI's pid under start_new_session) of every - # invocation this agent spawned, swept on kill()/kill_sync()/stop() — - # `opencode run` leaves a server child alive after the CLI exits, and - # signaling only the CLI pid would orphan it (a slow leak across a batch). + # Process-group ids of every invocation this agent spawned, swept on + # kill()/kill_sync()/stop(): signalling only the CLI pid orphans the + # server child `opencode run` leaves behind. self._spawned_pgids: list[int] = [] self._state = AgentState.WORKING @@ -933,8 +798,8 @@ async def start( self._skill_dirs, ) elif self.config.plugins: - # Plugins were declared but produced nothing — the run is about to - # measure the model without the skills under test. Say so loudly. + # The run is about to measure the model without the skills under + # test. Say so loudly. logger.warning( "opencode: %d plugin(s) declared but 0 skill path(s) resolved — the agent will run " + "WITHOUT them (see docs/agents/OPENCODE.md).", @@ -973,13 +838,10 @@ def kill_sync(self) -> None: def _sweep_process_groups(self) -> None: """SIGKILL every process group this agent spawned (POSIX only). - Each invocation runs in its own session (``start_new_session``), so its - pgid is the CLI's pid and the group contains ONLY what that invocation - spawned — the lingering server child included, a shared daemon we did not - start excluded. The CLI itself gets SIGTERM-then-SIGKILL first (see - ``kill``); this reaps whatever survives it. Sessions are persisted on - disk by OpenCode, so killing a turn's server does not lose ``--session`` - continuity. + Each invocation runs in its own session, so its pgid is the CLI's pid and + the group holds ONLY what that invocation spawned. The CLI itself gets + SIGTERM-then-SIGKILL first (see ``kill``); this reaps what survives. + Sessions persist on disk, so this does not lose ``--session`` continuity. """ if os.name != "posix": return @@ -989,16 +851,16 @@ def _sweep_process_groups(self) -> None: self._spawned_pgids.clear() def get_environment_info(self) -> dict[str, Any]: - # Spread the base first so the `system_prompt_semantics` run marker is - # always present (dashboards read an absent marker as a pre-marker run). + # Base first so the `system_prompt_semantics` run marker is always + # present (an absent marker reads as a pre-marker run). info: dict[str, Any] = { **super().get_environment_info(), "opencode_model": self.config.model, "opencode_pure": self.config.pure, } if self._skill_dirs: - # Recorded per task so a run's report can be checked for whether the - # skills under test actually reached the agent. + # Recorded per task so a report can confirm the skills reached the + # agent. info["opencode_skill_paths"] = list(self._skill_dirs) if self.config.variant: info["opencode_variant"] = self.config.variant @@ -1018,8 +880,8 @@ def _build_argv(self, user_input: str) -> list[str]: argv += ["--variant", self.config.variant] if self.config.pure: argv.append("--pure") - # PLAN mode is the one mode that must not auto-approve side effects; every - # other mode runs unattended, where an approval prompt would simply hang. + # PLAN is the one mode that must not auto-approve side effects; every + # other runs unattended, where an approval prompt would simply hang. if self.config.permission_mode is not PermissionMode.PLAN: argv.append("--auto") if self._session_id: @@ -1037,12 +899,11 @@ def _build_env(self) -> dict[str, str]: real one. ``PLUGIN_TOOLS_DIR`` is advisory and never overrides an inherited value. - Unlike ``CodexAgent._build_codex_env`` — which hands the SDK a partial - dict merged over the real environment, and so must resolve the PATH key - case-insensitively — this returns the WHOLE environment, seeded from - ``os.environ``, whose keys CPython upper-cases on Windows (``os.py``'s - ``encodekey``). ``"PATH"`` is therefore the inherited key on every - platform and cannot duplicate a differently-cased one. + Returns the WHOLE environment, seeded from ``os.environ``, whose keys + CPython upper-cases on Windows — so ``"PATH"`` is the inherited key on + every platform and cannot duplicate a differently-cased one. (Codex hands + the SDK a PARTIAL dict instead, which is why it resolves the key + case-insensitively.) """ env = dict(os.environ) if self._env_path_prepend: @@ -1055,10 +916,9 @@ def _build_env(self) -> dict[str, str]: def _inject_skill_paths(self, env: dict[str, str]) -> None: """Merge the resolved skill directories into ``OPENCODE_CONFIG_CONTENT``. - No plugins means the variable is left exactly as inherited, so a run - without a ``plugins:`` block behaves byte-for-byte as before. An inherited - value is preserved and appended to rather than clobbered, since the host - may legitimately configure OpenCode through the same seam. + No plugins means the variable is left exactly as inherited. An inherited + value is appended to, never clobbered: the host may legitimately configure + OpenCode through the same seam. """ if not self._skill_dirs: return @@ -1129,9 +989,8 @@ def emit(event: StreamEvent) -> None: deadline = None if timeout is None else time.monotonic() + timeout stopped_early = False stderr_drain: asyncio.Future[bytes] | None = None - # Bound OUTSIDE the try so the teardown in `finally` can tell "never - # spawned" (a create_subprocess_exec failure) from "spawned and possibly - # still running". + # Bound OUTSIDE the try so `finally` can tell "never spawned" from + # "spawned and possibly still running". proc: asyncio.subprocess.Process | None = None try: proc = await asyncio.create_subprocess_exec( @@ -1140,13 +999,11 @@ def emit(event: StreamEvent) -> None: stderr=asyncio.subprocess.PIPE, cwd=self.working_directory, env=self._build_env(), - # A single nd-JSON event can carry a whole tool result (a large file - # read), which blows past StreamReader's default 64 KiB line cap and - # would raise ValueError mid-stream, killing the read loop. + # One nd-JSON event can carry a whole tool result, past + # StreamReader's default 64 KiB cap. limit=STDOUT_LINE_LIMIT_BYTES, - # Own session/process group, so teardown can killpg the lingering - # server child without touching anything this invocation didn't - # spawn. POSIX-only knob; harmless False elsewhere. + # Own session/process group, so teardown can killpg the server + # child. POSIX-only knob; harmless False elsewhere. start_new_session=os.name == "posix", ) self._process = proc @@ -1154,21 +1011,14 @@ def emit(event: StreamEvent) -> None: self._spawned_pgids.append(proc.pid) assert proc.stdout is not None - # Drain stderr CONCURRENTLY, from the moment the CLI starts. Reading it - # only after exit (while stdout drives the loop) deadlocks the pair: a - # child that fills the ~64 KiB stderr pipe blocks on write, stops - # emitting stdout, and never exits — so the turn hangs to its deadline. - # docker_runner dodges this by merging stderr into stdout; here that - # would corrupt the nd-JSON, so it gets its own reader instead. + # Drain stderr CONCURRENTLY, or a child that fills the pipe blocks on + # write and hangs the turn to its deadline. + # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses if proc.stderr is not None: stderr_drain = asyncio.ensure_future(proc.stderr.read()) - # `opencode run` spawns a local server child that INHERITS this stdout - # pipe, so the pipe is NOT closed when the CLI itself exits — readline() - # would block until the turn deadline waiting for an EOF that never - # comes. So race each read against process exit: whichever lands first - # wins, and once the process is gone a bounded drain collects whatever - # is still buffered before the loop ends. + # The server child INHERITS this stdout pipe, so it is not closed when + # the CLI exits: race each read against process exit, then drain. exit_waiter = asyncio.ensure_future(proc.wait()) read_task: asyncio.Future[bytes] | None = None try: @@ -1187,9 +1037,8 @@ def emit(event: StreamEvent) -> None: if not done: await self._timeout_turn(state, collector, timeout or 0.0) if not read_task.done(): - # The process exited with the read still pending. Give the - # buffered tail a bounded window, then stop rather than - # waiting on the grandchild's open write end. + # Exited with the read still pending: bound the tail rather + # than wait on the grandchild's open write end. try: await asyncio.wait_for(asyncio.shield(read_task), _DRAIN_SECONDS) except TimeoutError: @@ -1225,8 +1074,7 @@ def emit(event: StreamEvent) -> None: ) state.finalize(status) # Build BEFORE marking the turn clean: a failure in the reduction is a - # failed turn, and `_end_turn_ok` would clear the rollback flag that - # `discard_pending_turn` needs to un-bump `_iteration`. + # failed turn, and `_end_turn_ok` clears the rollback flag. record = collector.build_turn_record() self._end_turn_ok() return record @@ -1239,15 +1087,10 @@ def emit(event: StreamEvent) -> None: self._capture_partial_turn(collector) raise except Exception as e: - # Everything the turn loop does NOT anticipate: a spawn failure - # (OSError/PermissionError from create_subprocess_exec), a StreamReader - # ValueError on a line past `limit`, a malformed-payload TypeError in a - # handler, a pydantic error assembling telemetry. Without this the - # exception escapes raw and breaks the pending-turn contract three ways: - # no AgentEndEvent (an unbalanced event tree for every renderer), the - # captured telemetry dropped instead of parked on `pending_turn`, and - # `_iteration` left incremented because the orchestrator never reaches - # `discard_pending_turn`. Same guard, same reasons, as CodexAgent. + # Everything the loop does NOT anticipate. Without this the exception + # escapes raw and breaks the pending-turn contract three ways: no + # AgentEndEvent, the telemetry dropped rather than parked, and + # `_iteration` left incremented. self._crash_turn(state, collector, f"OpenCode turn failed: {e!s}", cause=e) raise # unreachable (_crash_turn is NoReturn) — makes the no-fall-through explicit finally: @@ -1259,27 +1102,13 @@ def emit(event: StreamEvent) -> None: def _reap_orphaned_cli(self, proc: asyncio.subprocess.Process | None) -> None: """Kill a CLI that is still running as the turn unwinds. No-op otherwise. - Two exits from :meth:`communicate` reach its ``finally`` with the child - ALIVE: the ``except Exception`` crash (a ``StreamReader`` ``ValueError`` - on an over-long line, a malformed-payload ``TypeError`` in a handler) and - an external cancellation — neither passes through the graceful - ``await self.kill()`` that the intentional cuts and ``_settle_turn`` use. - - Abandoning it is not merely a leak. ``AgentCrashError`` is categorized - ``AGENT_CRASH`` (``max_retries=2``) and the orchestrator's attempt-failure - hook only drains ``pending_turn``, so attempt 2 would spawn a SECOND - ``opencode --dir --session `` while attempt 1 is still - editing the files the criteria are about to score — and whichever writer - won would decide the task's result. ``docker_runner`` kills its container - from ``finally`` for the same reason. - - Deliberately synchronous. This runs while a ``CancelledError`` is - propagating, where any await can itself be cut short and leave the child - alive after all; ``Process.kill()`` and the group sweep deliver their - signals with no suspension point. Skipping the SIGTERM courtesy is right - for a turn that is already lost — the graceful escalation in :meth:`kill` - still owns every path that has something left to flush. ``proc`` is - ``None`` when the spawn itself failed, i.e. there is nothing to reap. + Deliberately synchronous: this runs while a ``CancelledError`` is + propagating, where any await can itself be cut short. Skipping the SIGTERM + courtesy is right for a turn that is already lost — :meth:`kill` still + owns every path with something left to flush. ``proc`` is ``None`` when + the spawn itself failed. + + Rationale: .claude/notes/agents.md § Reaping the CLI harnesses """ if proc is None or proc.returncode is not None: return @@ -1300,19 +1129,13 @@ async def _settle_turn( ) -> AgentEndStatus: """Reap the CLI once the read loop is done and decide the turn's end status. - Raises ``AgentCrashError`` (via :meth:`_crash_turn`) when the stream carried - a structured error, when the process died with neither a structured error - nor an intentional stop, or when a clean exit captured no token telemetry - (a zero-telemetry turn must not score — see the guard below). Raises - ``TurnTimeoutError`` when the turn deadline elapses while waiting for the - exit. + Raises ``AgentCrashError`` (via :meth:`_crash_turn`) on a structured error, + on a death with neither a structured error nor an intentional stop, or on + a clean exit that captured no token telemetry. Raises ``TurnTimeoutError`` + when the deadline elapses while waiting for the exit. """ - # Bound the reap: the read loop can end at EOF with the CLI still alive - # (it closed its stream but never exited), and an unbounded wait here - # would outlive the turn deadline — the one window where `timeout` was - # previously unenforced. Give the exit the deadline's remainder, or a - # short fixed grace when no deadline is configured (post-EOF, a healthy - # CLI exits almost immediately). + # Bound the reap: the read loop can end at EOF with the CLI still alive, + # and an unbounded wait here would outlive the turn deadline. remaining = None if deadline is None else max(0.0, deadline - time.monotonic()) try: await asyncio.wait_for(proc.wait(), timeout=_TERM_GRACE_SECONDS if remaining is None else remaining) @@ -1325,10 +1148,8 @@ async def _settle_turn( collector, f"OpenCode closed its event stream but did not exit within {_TERM_GRACE_SECONDS:.0f}s", ) - # Collect what the concurrent reader drained. Bounded for the same reason as - # the read loop: the inherited stderr pipe outlives the CLI, so waiting for - # the reader's own EOF would block. Shielded so the timeout doesn't kill it - # before communicate()'s finally can. + # Bounded for the same reason as the read loop: the inherited stderr pipe + # outlives the CLI. Shielded so the timeout doesn't kill it early. stderr_bytes = b"" if stderr_drain is not None: with contextlib.suppress(TimeoutError): @@ -1337,34 +1158,17 @@ async def _settle_turn( if state.error_message is not None: self._crash_turn(state, collector, f"OpenCode error: {state.error_message}") - # A non-zero exit with no structured error event still means the turn - # died — surface stderr rather than reporting a silent empty success. + # A non-zero exit with no structured error still means the turn died: + # surface stderr rather than reporting a silent empty success. if proc.returncode not in (0, None) and not stopped_early and not state.max_turns_exhausted: detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" self._crash_turn(state, collector, f"OpenCode exited non-zero: {detail}") - # A clean exit that captured NO token telemetry must not score. File-based - # criteria can still pass on whatever the agent did, producing a SUCCESS - # that is silently missing from every aggregate — and, worse, one whose - # `run_limits.max_total_tokens` / `max_usd` gates could never have tripped - # no matter how much the run actually billed. This already happened once - # (the harness parsed the `session.next.*` server vocabulary instead of - # the CLI's), so drift is crashed loudly instead of scored. - # - # The condition is the TELEMETRY, not the event vocabulary. Keying on - # `recognized_events == 0` alone left the identical outcome reachable one - # layer down: a `step_finish` carrying no `tokens` key (a provider or auth - # mode that omits usage) recognizes three events, books an all-zero - # `TokenUsage`, and `EventCollector` then maps that to `token_usage=None` - # — a COMPLETED turn with no tokens, no cost and no warning. - # - # Intentional cuts (should_stop / max_turns) are exempt: both can land - # before the first event, or between a step's start and its `step_finish`. - # - # The second arm keys on a step the CLI reported as FINISHED — its own - # claim that a generation completed — rather than on `usage.is_empty()` - # alone, which would also condemn a stream that was cut before any step - # could finish. + # A clean exit that captured NO token telemetry must not score. The + # condition is the TELEMETRY, not the event vocabulary: both arms below + # reach the same silent-empty-success outcome. Intentional cuts are exempt + # — either can land before the first event, or mid-step. + # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash nothing_recognized = state.recognized_events == 0 finished_without_tokens = state.steps_finished > 0 and state.usage.is_empty() if not stopped_early and not state.max_turns_exhausted and (nothing_recognized or finished_without_tokens): @@ -1381,11 +1185,9 @@ async def _settle_turn( + "event or token schema may have changed — see docs/agents/OPENCODE.md (Telemetry) before " + "trusting any run from this CLI version." ) - # Escape hatch for a provider/auth mode that reports no usage at all, - # where crashing every turn would make the harness unusable rather than - # merely imprecise. Deliberately does NOT cover `nothing_recognized`: - # that arm is vocabulary drift, which has silently zeroed a whole run - # before, and no provider quirk can explain it. + # Escape hatch for a provider/auth mode that reports no usage at all. + # Deliberately does NOT cover `nothing_recognized`: that arm is + # vocabulary drift, which no provider quirk explains. if not self.config.require_token_telemetry and not nothing_recognized: logger.warning("opencode: %s Scored anyway — require_token_telemetry is off.", message) else: @@ -1407,8 +1209,7 @@ def _crash_turn( ) -> NoReturn: """Park the crashed partial record and raise ``AgentCrashError``. - ``cause`` preserves the explicit ``__cause__`` link when called from - inside an ``except ... as e`` block. + ``cause`` preserves the ``__cause__`` link from an ``except ... as e``. """ state.close_open_tools() try: @@ -1424,9 +1225,8 @@ async def _timeout_turn( ) -> NoReturn: """Kill the CLI, park the crashed partial record, raise ``TurnTimeoutError``. - ``_finalize_and_raise_timeout`` emits the terminal event via - ``state.finalize``; the partial record is captured immediately after so - ``pending_turn`` carries everything observed before the deadline. + The partial record is captured immediately after, so ``pending_turn`` + carries everything observed before the deadline. """ await self.kill() state.close_open_tools() @@ -1443,8 +1243,8 @@ def _handle_line(self, line: bytes, state: _OpenCodeTurnState) -> None: try: obj = json.loads(raw) except json.JSONDecodeError: - # OpenCode occasionally interleaves non-JSON notices (e.g. the Bun - # AVX warning) on stdout; a malformed line must not kill the turn. + # OpenCode interleaves non-JSON notices (the Bun AVX warning) on + # stdout; a malformed line must not kill the turn. logger.debug("opencode: skipping non-JSON stdout line: %s", raw[:200]) return if not isinstance(obj, dict): diff --git a/src/coder_eval/agents/pi_agent.py b/src/coder_eval/agents/pi_agent.py index 604eb3590..0f9814266 100644 --- a/src/coder_eval/agents/pi_agent.py +++ b/src/coder_eval/agents/pi_agent.py @@ -1,61 +1,24 @@ """Pi agent implementation (the ``pi`` Node coding agent — https://pi.dev/). -Drives the ``pi`` CLI in JSON print mode:: - - pi -p --mode json --no-context-files --no-approve \ - --session-dir --session-id [--model provider/id] \ - [--thinking L] [--append-system-prompt S] -- - -which streams **newline-delimited JSON events** on stdout. Each line is one -event; this module reduces that stream into the standardized coder_eval event -protocol (``AgentStart`` / ``TurnStart`` / ``ToolStart`` / ``ToolEnd`` / -``TurnEnd`` / ``AgentEnd``) and lets :class:`EventCollector` build the -``TurnRecord`` — so no telemetry is assembled by hand here. The design mirrors -:mod:`coder_eval.agents.opencode_agent` almost verbatim; the differences are -noted inline. - -Event grammar (captured from ``pi`` 0.84.4) -------------------------------------------- -- ``{"type": "session", ...}`` — always line 1 (cwd/id/version). -- ``{"type": "agent_start"}`` — bare; **can appear more than once** per - invocation (Pi auto-retries a transient/provider error internally). -- ``{"type": "turn_start"}`` — bare; one per agent-loop step. This is the unit - ``max_turns`` counts. -- ``{"type": "message_update", "assistantMessageEvent": {...}}`` — streaming - deltas (text/thinking/toolcall). Text deltas drive ``TextChunkEvent``. -- ``{"type": "message_end", "message": {...}}`` — a complete message. Ignored - for token accounting: ``turn_end`` echoes the same assistant usage once per - step, and reading both would double-count. -- ``{"type": "tool_execution_start", "toolCallId": ..., "toolName": ..., - "args": {...}}`` / ``{"type": "tool_execution_end", "toolCallId": ..., - "result": ..., "isError": ...}``. -- ``{"type": "turn_end", "message": {...assistant...}, "toolResults": [...]}`` - — ``message.usage`` is that step's OWN usage (per-generation), summed across - steps for the turn total. -- ``{"type": "agent_end", "messages": [...], "willRetry": }`` — - ``willRetry: true`` means another retry cycle follows in the SAME invocation. -- ``{"type": "agent_settled"}`` — the true terminal event (after all retries); - emit the single ``AgentEndEvent`` here / at EOF, NOT on the first - ``agent_end``. - -Token semantics ---------------- -Per-generation, **SUM** (identical to OpenCode; NOT cumulative). Each -``turn_end.message.usage`` carries ``{input, output, cacheRead, cacheWrite, -[reasoning], totalTokens, cost:{...,total}}`` for that step, where -``totalTokens == input + output + cacheRead + cacheWrite``. Unlike OpenCode, -Pi's ``input`` IS the fresh slice already (no flat/nested arbitration needed), -so it maps straight to ``uncached_input_tokens``. ``reasoning`` bills at the -output rate. ``cost.total`` per step is summed into -``token_usage.total_cost_usd`` (the rate card is only a fallback when the -stream omits cost). - -Session continuity ------------------- -A per-agent ``--session-dir`` + stable ``--session-id`` are assigned in -``start()`` and replayed on every ``communicate()`` — create-if-missing on the -first call, resume after — which is what makes multi-turn (dialog-mode) -evaluation work across separate CLI invocations. +Drives the ``pi`` CLI in JSON print mode, which streams newline-delimited JSON +events on stdout, and reduces that stream into the standardized coder_eval event +protocol so :class:`EventCollector` builds the ``TurnRecord``. The design mirrors +:mod:`coder_eval.agents.opencode_agent`. + +Three grammar facts that are not obvious from the event names (``pi`` 0.84.4): + +- ``agent_start`` can appear MORE THAN ONCE per invocation — Pi auto-retries a + transient provider error internally — and ``agent_end`` is therefore NOT + terminal. ``agent_settled`` (or EOF) is; the single ``AgentEndEvent`` is + emitted there. +- ``turn_start`` is one per agent-loop step, and is the unit ``max_turns`` counts. +- ``message_end`` is ignored for token accounting: ``turn_end`` echoes the same + assistant usage once per step, so reading both would double-count. + +A per-agent ``--session-dir`` + stable ``--session-id``, replayed on every +``communicate()``, are what make dialog mode work across CLI invocations. + +Rationale: .claude/notes/agents.md § Pi """ from __future__ import annotations @@ -117,12 +80,9 @@ logger = logging.getLogger(__name__) # Grace period between SIGTERM and SIGKILL when tearing down the CLI subprocess. -# Doubles as the post-EOF exit grace in _settle_turn when no turn deadline is -# configured. Re-declared here at the same value as OpenCode's rather than shared: -# the full nd-JSON-CLI driver hoist that would unify the two harnesses' teardown -# constants and reducers is a tracked follow-up; the shared plugin->skills resolver -# already lives in `agents/_skills.py`. STDOUT_LINE_LIMIT_BYTES, which IS canonical, -# is imported above. +# Doubles as the post-EOF exit grace in _settle_turn when no turn deadline is set. +# Re-declared at OpenCode's value rather than shared — see the notes. +# Rationale: .claude/notes/agents.md § Reaping the CLI harnesses _TERM_GRACE_SECONDS = 5.0 # SIGKILL does not exist on Windows (where the process-group sweep is a no-op @@ -130,9 +90,9 @@ # platform, falling back to SIGTERM for the direct-pid kill_sync path. _SIGKILL: signal.Signals = getattr(signal, "SIGKILL", signal.SIGTERM) -# How long to keep draining stdout/stderr after the CLI process has been reaped. -# A print-mode CLI may leave an inherited pipe open, so every post-exit read -# must be bounded. +# How long to keep draining stdout/stderr after the CLI has been reaped: a +# print-mode CLI may leave an inherited pipe open, so every post-exit read is +# bounded. _DRAIN_SECONDS = 2.0 # How many distinct unrecognized event-type strings to retain for the crash @@ -140,9 +100,8 @@ _MAX_UNRECOGNIZED_TYPES = 8 # pi's native tool names -> the canonical (Claude) vocabulary every criterion is -# written against. Mirrors opencode_agent._TOOL_NAME_MAP: without it a -# `command_executed` with `tool_name: Bash` matches nothing on a Pi run. Unknown -# tools pass through unchanged. +# written against. Unknown tools pass through unchanged. +# Rationale: .claude/notes/agents.md § Tool-name and argument normalization _TOOL_NAME_MAP: dict[str, str] = { "bash": "Bash", "read": "Read", @@ -150,9 +109,7 @@ "edit": "Edit", "patch": "Edit", "multiedit": "Edit", - # Pi's search tool is `find` (glob-by-pattern), NOT `glob` — mapping it to the - # canonical `Glob` keeps command_executed / commands_efficiency criteria - # comparable across harnesses. There is no `glob` tool in Pi's built-in set. + # Pi's search tool is `find` (glob-by-pattern); there is no `glob` in its set. "find": "Glob", "grep": "Grep", "list": "LS", @@ -163,11 +120,9 @@ "task": "Agent", } -# pi per-tool INPUT-arg key -> canonical (Claude) key. Mirrors -# opencode_agent._OPENCODE_ARG_RENAME. The spike's write/read tools used `path`, -# so map it to `file_path` for Read/Write/Edit (the search tools keep `path`, -# which is already Claude's key). Keyed by the canonical tool name (post -# _TOOL_NAME_MAP); unlisted keys pass through. +# pi per-tool INPUT-arg key -> canonical (Claude) key, keyed by the canonical +# tool name (post _TOOL_NAME_MAP). The search tools keep `path`, which is already +# Claude's key. Unlisted keys pass through. _PI_ARG_RENAME: dict[str, dict[str, str]] = { "Read": {"path": "file_path"}, "Write": {"path": "file_path"}, @@ -179,26 +134,24 @@ }, } -# Config fields the Pi CLI has no equivalent knob for (v1), OR that cannot be -# safely forwarded. `experiments/default.yaml` sets `permission_mode` and -# `allowed_tools` on every task, so warn once at start() rather than let a task -# believe it constrained the agent. NOTE `system_prompt` IS supported (mapped to -# --append-system-prompt) and `plugins` IS supported (each resolved skills dir is -# mapped to a `--skill ` argument), so neither is here. +# Config fields Pi does NOT enforce. `experiments/default.yaml` sets +# `permission_mode` and `allowed_tools` on every task, so start() warns once +# rather than letting a task believe it constrained the agent. `system_prompt` +# and `plugins` ARE supported, so neither is here. Per-harness table: +# docs/agents/HARNESS_PARITY.md. _UNSUPPORTED_CONFIG_FIELDS: tuple[str, ...] = ( "permission_mode", "system_prompt_file", - # Pi's built-in tool names are lowercase (bash/read/write/edit/grep/find/ls) - # and do not match the Claude-namespaced default (Bash/Read/Write/...), so - # forwarding them to --tools would allowlist nonexistent tools and strip the - # agent of ALL tools. Ignored like OpenCode/Codex/Antigravity do. + # Forwarding these to --tools would allowlist nonexistent tools and strip the + # agent of ALL tools: Pi's built-ins are lowercase. + # Rationale: .claude/notes/agents.md § Harness run-limit parity "allowed_tools", "disallowed_tools", ) # The full recognized Pi vocabulary (from `pi` 0.84.4). A clean exit that -# recognized NOTHING from this set is vocabulary drift and is crashed rather than -# scored as a silent empty success (see _settle_turn). +# recognized NOTHING from this set is vocabulary drift and is crashed, not scored. +# Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash _RECOGNIZED_EVENTS = frozenset( { "session", @@ -276,12 +229,10 @@ def __init__( self.user_input = user_input self.model = model - # ONE clock per turn, and every wall stamp below derives from it, so - # the tool spans and the window bounds they are subtracted from cannot - # end up on different bases. Injectable so a test can supply a fake - # rather than monkeypatching this module's `datetime` global — which a - # derived stamp would silently escape, leaving the test passing against - # the real clock instead of failing. + # ONE clock per turn, so the tool spans and the window bounds they are + # subtracted from share a basis. Injectable so a test supplies a fake + # rather than monkeypatching this module's `datetime` global, which a + # derived stamp would silently escape. self.clock = clock or TurnClock() self.started_at = time.monotonic() self.thread_id: str | None = None @@ -304,14 +255,9 @@ def __init__( self.turn_text_parts: list[str] = [] self.turn_tool_ids: list[str] = [] # Where the NEXT generation window starts: the previous turn's end. - # Pi was the only harness measuring from its own `turn_start`, so the - # wall clock between one `turn_end` and the next `turn_start` — the - # model time that PRODUCED that turn — fell into no bucket at all. - # - # None until the first turn finishes, and deliberately so: the first - # window keeps its own `turn_start`, because everything before it is - # CLI process spawn, not model time. Same shape as OpenCode's - # `gen_mark` and Codex's `gen_mark_ms`. + # None until the first turn finishes, and deliberately so — everything + # before the first `turn_start` is CLI process spawn, not model time. + # Rationale: .claude/notes/agents.md § Per-harness generation marks self.gen_mark: datetime | None = None # toolCallId -> telemetry for tools awaiting a result. @@ -327,12 +273,9 @@ def __init__( # message can name what it actually saw. self.recognized_events = 0 self.unrecognized_types: set[str] = set() - # Warn-once guard for token-accounting drift. The event-vocabulary check - # catches renamed EVENT types, but not a renamed/absent `usage` field or a - # bucket whose type changed — those silently coerce to 0 (see `_as_int`) and - # would zero out the run's tokens/cost, blinding max_total_tokens / max_usd - # gates. Mirrors OpenCode's `_warn_token_shape` (its escape hatch shipped - # with this warning; Pi's earlier cut kept the hatch but dropped the warn). + # Warn-once guard for token-accounting drift: the event-vocabulary check + # cannot see inside `usage`. + # Rationale: .claude/notes/agents.md § Why token-shape drift warns instead of raising self.warned_token_shape = False self._emit: Callable[[StreamEvent], None] = lambda _e: None @@ -351,11 +294,9 @@ def agent_output(self) -> str: def on_turn_start(self) -> None: # A prior step's `turn_start` with no `turn_end` — a generation aborted - # mid-turn (the defining willRetry case: a provider error before the - # assistant message completed). Close its dangling TurnStartEvent before - # opening the next, or the stream carries N starts and N-1 ends, breaking - # the one-pair-per-inner-turn contract renderers depend on. `finalize` - # closes only the LAST open turn, so it cannot cover this. + # mid-turn (the willRetry case). Close its dangling TurnStartEvent, or the + # stream carries N starts and N-1 ends and breaks the one-pair-per-inner-turn + # contract. `finalize` closes only the LAST open turn, so it cannot cover this. if self.turn_open: self.turn_open = False self.emit( @@ -373,10 +314,8 @@ def on_turn_start(self) -> None: self.turn_started_at = self.clock.now() self.turn_text_parts = [] self.turn_tool_ids = [] - # No per-turn span list to reset here any more — see the identical note - # in `opencode_agent.on_step_start`. The collector subtracts from final - # bounds with every span known, so nothing has to remember a call that - # closed in the gap before this `turn_start`. + # No per-turn span list to reset here any more: the collector subtracts + # from final bounds with every span known. self.emit( TurnStartEvent( task_id=self.task_id, @@ -427,10 +366,9 @@ def on_tool_execution_end(self, obj: dict[str, Any]) -> None: is_error = bool(obj.get("isError")) if is_error: message = summary or "tool failed" - # Best-effort: Pi does not tag permission denials distinctly, so infer - # from the result text. The persisted tri-state folds both to "error" - # (see _RESULT_STATUS), so a misclassified legit "permission denied" in - # output is cosmetic. Mirrors opencode_agent. + # Best-effort: Pi does not tag permission denials, so infer from the + # text. The persisted tri-state folds both to "error", so a + # misclassification is cosmetic. denied = "permission" in message.lower() or "denied" in message.lower() status = ToolEndStatus.PERMISSION_DENIED if denied else ToolEndStatus.ERROR else: @@ -457,20 +395,11 @@ def _close_tool( timestamp=self.clock.now(), sequence_number=self.sequence, ) - # Only a RESOLVED tool is timed. An orphan force-closed by - # `close_open_tools` was never observed finishing, so the instant the - # sweep runs is not a completion — stamping it manufactures both an - # `execution_completed_at` and the `duration_ms` derived from it, and - # the pair then reads as a measured span that - # `timing.subtract_tool_time` takes back out of a generation - # window it never actually occupied. `execution_started_at` IS kept: - # the CLI really did emit that start, and one bound alone forms no - # span (`main_thread_tool_spans` requires both). This is the guard the - # old comment here claimed and the code did not have — it tested - # `execution_started_at is not None`, which an orphan passes. - # claude-code's `_finalize_commands` leaves the same field `None` for - # the same reason: unknown status and unknown duration are one fact - # (CE058). + # Only a RESOLVED tool is timed: an orphan was never observed finishing, + # so stamping it would manufacture a span the central subtraction then + # takes out of a window it never occupied. `execution_started_at` IS + # kept — one bound alone forms no span (CE058). + # Rationale: .claude/notes/agents.md § Why only a RESOLVED tool is timed if status is not ToolEndStatus.UNRESOLVED: completed = self.clock.now() telemetry.execution_completed_at = completed @@ -500,14 +429,12 @@ def _warn_token_shape(self, message: str, *args: Any) -> None: def _as_int(self, value: Any) -> int: """Coerce one stream-supplied token count; count a non-number as 0. - A bare ``int()`` would raise on a non-numeric value, which - ``communicate``'s ``except Exception`` turns into an ``AgentCrashError`` - (categorized ``AGENT_CRASH``, ``max_retries=2``), burning three attempts - on one mistyped bucket. A bool is never a token count (``int(True) == 1``). + A bool is never a token count (``int(True) == 1``). ``None`` is a legitimately-absent bucket (silent). Any OTHER unparseable - value is a schema drift and warns once — otherwise a changed bucket type - would silently zero the turn's tokens and cost. + value is schema drift and warns once. + + Rationale: .claude/notes/agents.md § Why token-shape drift warns instead of raising """ if value is None: return 0 @@ -532,9 +459,8 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: message = message if isinstance(message, dict) else {} raw_usage = message.get("usage") if not isinstance(raw_usage, dict) or not raw_usage: - # A completed step that booked no usage object at all — a renamed or - # absent `usage` (which the event-vocabulary check cannot see). Its - # tokens/cost silently resolve to 0; say so once. + # A completed step that booked no usage object at all: its tokens and + # cost silently resolve to 0, so say so once. self._warn_token_shape("turn_end carried no usage object; this step's tokens/cost counted as 0") usage = raw_usage if isinstance(raw_usage, dict) else {} @@ -547,12 +473,8 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: # fold it into the turn total (the per-message record keeps it separately). step_out = raw_out + step_reasoning - # A turn_end that DID carry a usage object but whose every bucket resolves - # to 0 is the drift shape the whole-object check above cannot see: keys - # renamed by a CLI upgrade each coerce to 0 (see `_as_int`), tokens and cost - # silently vanish, and `max_usd` / `max_total_tokens` can never trip. Warn - # once (score, don't crash — the documented Pi policy). OpenCode guards the - # same gap with `steps_finished > 0 and usage.is_empty()`. + # A usage object whose every bucket resolves to 0 is the drift shape the + # whole-object check cannot see. Warn once; score, don't crash. if raw_usage and step_in == raw_out == step_reasoning == step_cw == step_cr == 0: self._warn_token_shape("turn_end usage object had all-zero token buckets; this step booked 0 tokens/cost") @@ -562,18 +484,13 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: cache_creation_input_tokens=self.usage.cache_creation_input_tokens + step_cw, cache_read_input_tokens=self.usage.cache_read_input_tokens + step_cr, ) - # Cross-check the stream's OWN `totalTokens` against the buckets we summed. + # Cross-check the stream's OWN `totalTokens` against the summed buckets. # Pi's invariant is totalTokens == input + output + cacheRead + cacheWrite - # (reasoning is billed at the output rate but excluded from this field, so - # compare against raw_out, not step_out). A mismatch means a bucket was - # renamed or its meaning moved under a CLI upgrade — exactly the drift the - # per-bucket `_as_int` coercion would otherwise absorb silently, blinding - # max_total_tokens / max_usd. Warn once, mirroring OpenCode's `tokens.total` - # guard; only when the field is actually present (older streams omit it). + # — reasoning bills at the output rate but is EXCLUDED from this field, so + # compare against raw_out, not step_out. Only when the field is present. reported_total = usage.get("totalTokens") - # Accept int OR float (a `123.0`-shaped total is itself a plausible drift and - # _as_int accepts floats for the buckets); the numeric compare below is - # exact for whole values (123.0 == 123). + # int OR float: a `123.0`-shaped total is itself a plausible drift, and + # the compare below is exact for whole values. if isinstance(reported_total, int | float) and not isinstance(reported_total, bool): expected_total = step_in + raw_out + step_cw + step_cr if reported_total != expected_total: @@ -612,9 +529,7 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: for i, tool_id in enumerate(self.turn_tool_ids, start=len(blocks)): blocks.append(ContentBlock(block_type="tool_use", sequence=i, tool_use_id=tool_id)) - # Tile from the previous turn's end. The RAW window only — - # `timing.subtract_tool_time` takes the tool union back out of - # it, once, for every harness. + # Tile from the previous turn's end. The RAW window only. turn_start = self.turn_started_at if self.turn_started_at is not None else completed started, generation_ms = close_window( mark=self.gen_mark if self.gen_mark is not None else turn_start, @@ -638,26 +553,15 @@ def on_turn_end(self, obj: dict[str, Any]) -> None: message_id=str(message.get("responseId") or "") or None, ) ) - # A message was appended, so the next window starts where this one - # ended. Only a finished turn advances the mark: one that never - # finished published nothing, so tiling past it would attribute its - # time to whichever turn finishes next. There is no span list to clear - # alongside it any more — see `on_turn_start`. + # A message was appended, so the next window starts where this one ended. + # Only a FINISHED turn advances the mark. self.gen_mark = completed - # And so is this turn's own start stamp, because it has now been SPENT. - # It is passed to `close_window` as `item_start`, whose `min()` pulls - # the window open to cover it; left in place, a second `turn_end` with - # no intervening `turn_start` — a duplicate or replayed line, which - # this reducer promises to survive — would reopen the next window back - # at the previous turn's start and publish that whole span a second - # time. Reproduced: 3000 ms of generation for a 2000 ms turn. + # SPENT state, reset HERE and not only in `on_turn_start`: a second + # `turn_end` with no intervening start — a duplicate or replayed line, + # which this reducer promises to survive — would otherwise republish this + # turn's span, text and tool ids as the next turn's. + # Rationale: .claude/notes/agents.md § Per-harness generation marks self.turn_started_at = None - # The CONTENT half of the same reset, and the same argument: both - # lists have now been SPENT into the message appended above. - # Cleared only in `on_turn_start`, a second `turn_end` with no - # intervening start re-emitted the previous turn's text as its own - # assistant message and re-listed the same `tool_use_ids`, so one - # tool call appeared to belong to two generations. self.turn_text_parts = [] self.turn_tool_ids = [] self.emit( @@ -689,15 +593,11 @@ def _rate_card_cost(self) -> float | None: def _resolve_cost(self) -> float | None: """Decide the turn's cost: the stream's own accounting vs the rate card. - Pi reports a real per-call ``cost.total`` (spike-verified), which wins for - any nonzero total. Two conservative fallbacks to the rate card: - - the stream reported no cost field at all (``saw_cost`` False), or - - it reported a cost field but the turn total came out exactly ``$0`` on a - model the rate card DOES price. A true $0 (free/promo response) and a - provider whose cost field is present-but-always-zero are indistinguishable - from the stream alone, so we prefer the rate card: understating cost would - silently defeat ``max_usd`` budget gates, which is the worse failure. A - genuinely free model (no rate-card entry) still resolves to the stream's 0. + Pi reports a real per-call ``cost.total``, which wins for any nonzero + total. It falls back to the rate card when the stream reported no cost at + all, or reported exactly ``$0`` on a model the rate card DOES price. + + Rationale: .claude/notes/agents.md § Cost: the stream versus the rate card """ rate = self._rate_card_cost() if not self.saw_cost: @@ -732,8 +632,8 @@ def finalize( cost = self._resolve_cost() if cost is not None: usage = usage.model_copy(update={"total_cost_usd": cost}) - # A turn still open here never received its `turn_end` — close its - # TurnStartEvent or the one-pair-per-inner-turn contract breaks. + # A turn still open never received its `turn_end`; close it or the + # one-pair-per-inner-turn contract breaks. if self.turn_open: self.turn_open = False self.emit( @@ -779,13 +679,11 @@ def finalize( class PiAgent(Agent[PiAgentConfig]): """Runs the ``pi`` CLI as a subprocess, one invocation per turn.""" - # `should_stop` is polled at every event boundary — i.e. tool-call - # granularity — and honored by terminating the CLI subprocess cleanly. + # `should_stop` is polled at every event boundary (tool-call granularity). supports_cooperative_stop: ClassVar[bool] = True - # Pi maps `system_prompt` to `--append-system-prompt`, so it appends to (does - # not replace) the CLI's default prompt. Declared explicitly (mirrors Codex / - # Antigravity, NOT OpenCode's `"unknown"`). + # `--append-system-prompt` appends to, never replaces, the CLI's own prompt. + # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker system_prompt_semantics: ClassVar[SystemPromptSemantics] = "append" def __init__( @@ -797,14 +695,11 @@ def __init__( ) -> None: """Every parameter the agent factory can pass is DECLARED, not absorbed. - ``create_agent`` calls ``agent_class(config, route=route, **kwargs)`` - through a ``cast(Any, ...)``, so a ``**_`` sink would mean nothing checks - the kwargs at runtime either — a mis-gated kwarg must be loud here rather - than silently dropped. - ``route`` is accepted for factory parity and deliberately unused: the CLI - owns its own provider configuration. ``task_id`` only labels the emitted - event stream. + owns its own provider configuration. ``task_id`` only labels the event + stream. + + Rationale: .claude/notes/agents.md § Why the constructors declare every kwarg """ self.config = config self.route = route @@ -815,11 +710,9 @@ def __init__( # Skills-parent dirs resolved from `agent.plugins`, passed to `pi --skill` # (Pi discovers `/SKILL.md` recursively). Assigned in start(). self._skill_dirs: list[str] = [] - # Per-agent session, reused across communicate() calls for multi-turn / - # simulation continuity (assigned in start(), removed in stop() — NOT in - # kill(), which the orchestrator's mid-turn backstop calls; dropping the - # dir there would break resume across a retried turn. _cleanup always - # calls stop() after any kill(), so the tempdir is still reclaimed). + # Per-agent session, reused across communicate() calls for multi-turn + # continuity. Removed in stop(), deliberately NOT in kill(). + # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses self._session_id: str | None = None self._session_dir: str | None = None self._process: asyncio.subprocess.Process | None = None @@ -865,16 +758,13 @@ async def start( self._env_path_prepend = list(env_path_prepend or []) self._plugin_tools_dir = plugin_tools_dir # A stable pre-assigned id (create-if-missing on turn 1, resume after). - # The tempdir lives OUTSIDE the sandbox working dir and staged reference - # dir, so it never pollutes graded files or trips reference-integrity. - # Drop any session dir from a prior start() first so re-starting the same - # agent instance cannot leak a tempdir. + # The tempdir lives OUTSIDE the sandbox and staged reference dir, so it + # never pollutes graded files. Drop a prior start()'s dir so re-starting + # the same instance cannot leak one. self._cleanup_session_dir() - # Sanitize task_id before it reaches pi's `--session-id`: dataset-row tasks - # have path-shaped ids ("suite/row_3", set in task_loader) and pi derives - # its session file from the id under `--session-dir`, so a raw '/' would - # resolve to a non-existent subdir and fail the row before any work. Keep - # only the safe id charset (mirrors sandbox.py's flatten, but stricter). + # Sanitize task_id before it reaches pi's `--session-id`: a dataset row's + # path-shaped id would resolve to a non-existent subdir under + # `--session-dir` and fail the row before any work. safe_task_id = re.sub(r"[^A-Za-z0-9._-]", "_", self.task_id) self._session_id = f"coder-eval-{safe_task_id}-{uuid4().hex[:8]}" self._session_dir = tempfile.mkdtemp(prefix="pi-session-") @@ -913,10 +803,8 @@ def kill_sync(self) -> None: def _sweep_process_groups(self) -> None: """SIGKILL every process group this agent spawned (POSIX only). - Each invocation runs in its own session (``start_new_session``), so its - pgid is the CLI's pid and the group contains ONLY what that invocation - spawned — a lingering child included, a shared daemon we did not start - excluded. + Each invocation runs in its own session, so its pgid is the CLI's pid and + the group holds ONLY what that invocation spawned. """ if os.name != "posix": return @@ -945,12 +833,9 @@ def get_environment_info(self) -> dict[str, Any]: def _build_argv(self, user_input: str) -> list[str]: # -p exits after the run; --no-context-files + --no-approve isolate the - # sandbox from host AGENTS.md/CLAUDE.md and project-local trust (analogue - # of OpenCode --pure / Claude setting_sources: []). --session-dir + - # --session-id give cross-communicate() continuity (simulation mode) — - # reused every call, created on turn 1, resumed after. NOT --no-session - # (that would defeat continuity). All spike-verified. No --dir flag: the - # working dir is set via the subprocess `cwd`. + # sandbox from host AGENTS.md/CLAUDE.md and project-local trust. + # --session-dir + --session-id give cross-communicate() continuity — NOT + # --no-session, which would defeat it. No --dir: the working dir is `cwd`. assert self._session_dir is not None and self._session_id is not None argv = [ "pi", @@ -969,19 +854,11 @@ def _build_argv(self, user_input: str) -> list[str]: if self.config.thinking_level: argv += ["--thinking", self.config.thinking_level] for skill_dir in self._skill_dirs: - # Additive skill load (from agent.plugins). Pi lists each skill's - # name+description in the system prompt and the agent `read`s the full - # SKILL.md on demand — the OpenCode/Codex `plugins` mechanism, Pi-native. + # Additive skill load (from agent.plugins): Pi lists each skill's + # name+description in the system prompt and reads SKILL.md on demand. argv += ["--skill", skill_dir] - # allowed_tools / disallowed_tools are NOT forwarded. The shared config - # default (experiments/default.yaml) sets Claude-namespaced tool names - # (Bash/Read/Write/Edit/Glob/Grep/Skill), but Pi's built-in tools are - # lowercase and differently named (bash/read/write/edit/grep/find/ls). - # Passing the PascalCase names to `--tools` allowlists tools that do not - # exist in Pi, leaving the agent with ZERO tools ("I don't have tool - # access"). So, like OpenCode/Codex/Antigravity, these fields are treated - # as unenforced (see _UNSUPPORTED_CONFIG_FIELDS) and Pi runs with its full - # native toolset. Warned at start(). + # allowed_tools / disallowed_tools are NOT forwarded — see + # _UNSUPPORTED_CONFIG_FIELDS. Pi runs with its full native toolset. if self.config.system_prompt: argv += ["--append-system-prompt", self.config.system_prompt] # user_input is a distinct argv element after `--` (never shell-interpolated). @@ -994,8 +871,8 @@ def _build_env(self) -> dict[str, str]: The PATH prepend is the mock-shadowing contract (``Agent.start``): the sandbox's mock CLI directories must resolve BEFORE the real binaries. ``PLUGIN_TOOLS_DIR`` is advisory and never overrides an inherited value. - Returns the WHOLE environment (seeded from ``os.environ``) so the CLI - keeps the host's provider credentials (``OPENROUTER_API_KEY``, ...). + Returns the WHOLE environment so the CLI keeps the host's provider + credentials. """ env = dict(os.environ) if self._env_path_prepend: @@ -1040,24 +917,20 @@ def emit(event: StreamEvent) -> None: prompt=user_input, iteration=self._iteration, model=self.config.model, - # One basis with the window bounds this is subtracted - # against — see `timing.TurnClock`. The event model's raw - # `datetime.now()` default put two clocks inside one - # `decompose_turn` subtraction, which clamped a -0.017 ms tail - # to the `0.0` that means "measured, and instant" (CE058). + # One basis with the window bounds this is subtracted against; + # the model's raw `datetime.now()` default put two clocks inside + # one subtraction (CE058). timestamp=state.clock.now(), ) ) - # Deadlines stay on `time.monotonic()` and are deliberately NOT routed - # through the turn clock: a deadline must not move when the wall clock - # steps. `TurnClock` exists to give the RECORDED stamps one basis; this - # is the one place a raw monotonic reading is the right answer. + # Deadlines stay on `time.monotonic()`, deliberately NOT the turn clock: + # a deadline must not move when the wall clock steps. deadline = None if timeout is None else time.monotonic() + timeout stopped_early = False stderr_drain: asyncio.Future[bytes] | None = None - # Bound OUTSIDE the try so the teardown in `finally` can tell "never - # spawned" from "spawned and possibly still running". + # Bound OUTSIDE the try so `finally` can tell "never spawned" from + # "spawned and possibly still running". proc: asyncio.subprocess.Process | None = None try: proc = await asyncio.create_subprocess_exec( @@ -1066,11 +939,10 @@ def emit(event: StreamEvent) -> None: stderr=asyncio.subprocess.PIPE, cwd=self.working_directory, env=self._build_env(), - # A single nd-JSON event can carry a whole tool result, which blows - # past StreamReader's default 64 KiB line cap and would raise - # ValueError mid-stream, killing the read loop. + # One nd-JSON event can carry a whole tool result, past + # StreamReader's default 64 KiB cap. limit=STDOUT_LINE_LIMIT_BYTES, - # Own session/process group, so teardown can killpg any lingering + # Own session/process group, so teardown can killpg a lingering # child without touching anything this invocation didn't spawn. start_new_session=os.name == "posix", ) @@ -1079,16 +951,14 @@ def emit(event: StreamEvent) -> None: self._spawned_pgids.append(proc.pid) assert proc.stdout is not None - # Drain stderr CONCURRENTLY: a child that fills the ~64 KiB stderr pipe - # blocks on write, stops emitting stdout, and never exits — hanging the - # turn to its deadline. It gets its own reader so the nd-JSON on stdout - # stays clean. + # Drain stderr CONCURRENTLY, or a child that fills the pipe blocks on + # write and hangs the turn to its deadline. + # Rationale: .claude/notes/agents.md § Reaping the CLI harnesses if proc.stderr is not None: stderr_drain = asyncio.ensure_future(proc.stderr.read()) - # A print-mode CLI may leave an inherited pipe open, so readline() can - # block on an EOF that never comes. Race each read against process - # exit; once the process is gone a bounded drain collects the tail. + # An inherited pipe may never reach EOF, so race each read against + # process exit; a bounded drain then collects the tail. exit_waiter = asyncio.ensure_future(proc.wait()) read_task: asyncio.Future[bytes] | None = None try: @@ -1142,8 +1012,7 @@ def emit(event: StreamEvent) -> None: ) state.finalize(status) # Build BEFORE marking the turn clean: a failure in the reduction is a - # failed turn, and `_end_turn_ok` would clear the rollback flag that - # `discard_pending_turn` needs. + # failed turn, and `_end_turn_ok` clears the rollback flag. record = collector.build_turn_record() self._end_turn_ok() return record @@ -1156,9 +1025,8 @@ def emit(event: StreamEvent) -> None: self._capture_partial_turn(collector) raise except Exception as e: - # A spawn failure (OSError), a StreamReader ValueError past `limit`, a - # malformed-payload error in a handler, a pydantic error assembling - # telemetry. Funnel to the pending-turn contract like the siblings. + # A spawn failure, a StreamReader ValueError past `limit`, a malformed + # payload, a pydantic error. Funnel to the pending-turn contract. self._crash_turn(state, collector, f"Pi turn failed: {e!s}", cause=e) raise # unreachable (_crash_turn is NoReturn) — makes the no-fall-through explicit finally: @@ -1170,12 +1038,10 @@ def emit(event: StreamEvent) -> None: def _reap_orphaned_cli(self, proc: asyncio.subprocess.Process | None) -> None: """Kill a CLI still running as the turn unwinds. No-op otherwise. - The ``except Exception`` crash and an external cancellation both reach the - ``finally`` with the child possibly alive — neither passes through the - graceful ``kill()``. Abandoning it is not merely a leak: ``AgentCrashError`` - is retried, so attempt 2 would spawn a SECOND ``pi`` editing the very files - the criteria are about to score. Synchronous (no await) so it survives a - ``CancelledError`` in flight. ``proc`` is ``None`` when the spawn failed. + Synchronous (no await) so it survives a ``CancelledError`` in flight. + ``proc`` is ``None`` when the spawn failed. + + Rationale: .claude/notes/agents.md § Reaping the CLI harnesses """ if proc is None or proc.returncode is not None: return @@ -1217,21 +1083,11 @@ async def _settle_turn( with contextlib.suppress(TimeoutError): stderr_bytes = await asyncio.wait_for(asyncio.shield(stderr_drain), timeout=_DRAIN_SECONDS) - # A terminal provider error (stopReason=error that survived pi's internal - # retries) is infrastructure failure, not an agent failure. `pi -p` exits 0 - # after exhausting retries, so without this the turn books as a clean - # COMPLETED (FinalStatus.FAILURE, category "failed") — silently depressing - # the measured pass rate. Crashing routes it through _communicate_with_retry - # and, if unrecovered, to FinalStatus.ERROR (category "error", excluded from - # outcomes). Mirrors opencode_agent._settle_turn. - # - # Gated on intentional cuts like the two crash arms below: `error_message` - # is set at an error `turn_end` and cleared only by a LATER non-error - # `turn_end`, but a `max_turns` / `should_stop` cut can fire at the next - # `turn_start` (before that clearing `turn_end` ever arrives), leaving a - # stale error from a turn pi was still retrying. Without the guard that - # clean, budget-exhausted cut would crash + burn retries, contradicting the - # documented "finalizes cleanly as max_turns_exhausted, no crash" contract. + # A terminal provider error is infrastructure failure, not an agent + # failure, and `pi -p` exits 0 after exhausting retries. GATED on + # intentional cuts: a cut can fire before the clearing `turn_end` arrives, + # leaving a stale error from a turn pi was still retrying. + # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash if state.error_message is not None and not stopped_early and not state.max_turns_exhausted: self._crash_turn(state, collector, f"Pi error: {state.error_message}") @@ -1240,10 +1096,8 @@ async def _settle_turn( detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" self._crash_turn(state, collector, f"Pi exited non-zero: {detail}") - # A clean exit that recognized NO events is vocabulary drift — the CLI's - # schema moved, and scoring a silent empty success would be indistinguishable - # from a real pass in every aggregate. Intentional cuts are exempt (a stop - # can land before the first event). + # A clean exit that recognized NO events is vocabulary drift. Intentional + # cuts are exempt: a stop can land before the first event. if not stopped_early and not state.max_turns_exhausted and state.recognized_events == 0: seen = ", ".join(sorted(state.unrecognized_types)) or "none (stdout carried no JSON events)" self._crash_turn( @@ -1292,10 +1146,8 @@ async def _timeout_turn( def _handle_line(self, line: bytes, state: _PiTurnState) -> None: """Parse one nd-JSON line and dispatch it. Never raises on bad input. - Pi may emit multiple ``agent_start``/``turn_*``/``agent_end`` cycles in one - invocation (auto-retry). ``agent_end`` is NOT terminal — only - ``agent_settled`` / stdout EOF is — so it is recognized and otherwise - ignored, and the read loop keeps going. + ``agent_end`` is NOT terminal — only ``agent_settled`` / stdout EOF is — so + it is recognized, ignored, and the read loop keeps going. """ raw = line.decode("utf-8", "replace").strip() if not raw: @@ -1309,9 +1161,8 @@ def _handle_line(self, line: bytes, state: _PiTurnState) -> None: return event_type = str(obj.get("type") or "") - # `session` is line 1 (cwd/id); `message_start`/`message_end`/`agent_end`/ - # `agent_settled` carry no state we accumulate (usage is read from - # `turn_end`, not the echoing `message_end`). All are recognized vocabulary. + # `session`, `message_start`, `message_end`, `agent_end` and + # `agent_settled` carry no state we accumulate, but are all recognized. if event_type in _RECOGNIZED_EVENTS: state.recognized_events += 1 elif len(state.unrecognized_types) < _MAX_UNRECOGNIZED_TYPES: diff --git a/src/coder_eval/agents/registry.py b/src/coder_eval/agents/registry.py index c53fb45c3..5770e5266 100644 --- a/src/coder_eval/agents/registry.py +++ b/src/coder_eval/agents/registry.py @@ -10,10 +10,9 @@ from typing import TYPE_CHECKING, Any, ClassVar, TypeVar, cast -# Imports kept TYPE_CHECKING-only (with future annotations) so this module imports -# nothing from coder_eval at runtime. That keeps the dependency one-way — the -# plugin loader and models layer import the registry, never the reverse — so there -# is no import cycle (CodeQL py/cyclic-import). +# TYPE_CHECKING-only imports, so this module imports nothing from coder_eval at +# runtime and the dependency edge stays one-way (CodeQL py/cyclic-import). +# Rationale: .claude/notes/agents.md § Why the registry rejects a re-registration if TYPE_CHECKING: from coder_eval.agent import Agent from coder_eval.models import AgentKind, ApiRoute, BaseAgentConfig @@ -36,11 +35,9 @@ class AgentRegistration[ConfigT: BaseAgentConfig]: class AgentRegistry: """Global registry for custom agents. - Provides a decorator-based registration pattern that decouples agent - implementations from the orchestrator factory. Keyed by the agent *kind - string* so a built-in :class:`AgentKind` member and a plugin-supplied raw - string collide on the same key (``AgentKind`` is a ``StrEnum``): an external - plugin can register a brand-new kind that is not an enum member. + Keyed by the agent *kind string*, so a built-in :class:`AgentKind` member and + a plugin-supplied raw string collide on one key — which is what lets a plugin + register a brand-new kind that is not an enum member. """ _registry: ClassVar[dict[str, AgentRegistration[Any]]] = {} @@ -68,11 +65,10 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]): def decorator(agent_cls: type[AgentClassT]) -> type[AgentClassT]: kind = str(agent_kind) existing = cls._registry.get(kind) - # Re-registering the SAME classes is legitimate (idempotent built-in - # reload via load_plugins(force=True)). Re-registering a kind with a - # DIFFERENT implementation is a silent shadow: which agent runs would - # depend on entry-point discovery order, which isn't stable across - # environments — a reproducibility hole. Reject it loudly. + # Re-registering the SAME classes is legitimate (an idempotent + # built-in reload); a DIFFERENT implementation for the same kind is a + # silent shadow, so it is rejected loudly. + # Rationale: .claude/notes/agents.md § Why the registry rejects a re-registration if existing is not None and (existing.agent_class, existing.config_class) != (agent_cls, config_class): raise ValueError( f"Agent kind {kind!r} is already registered to " @@ -136,11 +132,9 @@ def create_agent( Returns: An instance of the requested agent type - Plugins must already be loaded: callers reach a config object through - ``parse_agent_config`` (which loads plugins), and the orchestrator / CLI also - load them up-front. ``create_agent`` deliberately does NOT import - ``coder_eval.plugins`` itself, so ``plugins`` -> ``agents.registry`` stays a - one-way edge (no import cycle). + Plugins must ALREADY be loaded: this deliberately does not import + ``coder_eval.plugins`` itself, so the edge stays one-way. Callers reach a + config through ``parse_agent_config``, which loads them. Raises: ValueError: If the agent_kind is not registered diff --git a/src/coder_eval/agents/watchdog.py b/src/coder_eval/agents/watchdog.py index 081b0915f..7e3c10388 100644 --- a/src/coder_eval/agents/watchdog.py +++ b/src/coder_eval/agents/watchdog.py @@ -24,39 +24,20 @@ class ThreadedWatchdog: """OS-thread-based deadline enforcer. A ``threading.Timer`` fires at ``timeout_seconds`` and invokes ``on_timeout`` - from the timer thread (NOT the asyncio event loop). After it fires, - ``fired`` is True (readable from any thread). - - ``on_timeout`` MUST be synchronous and thread-safe. Exceptions it raises - are logged and swallowed so one bad callback never kills the timer thread - without a trace. - - Typical use in async code: - - def _on_timeout() -> None: - kill_subprocess_by_pid(pid) # sync, thread-safe - - with ThreadedWatchdog( - timeout_seconds=1200, - on_timeout=_on_timeout, - asyncio_task_to_cancel=asyncio.current_task(), - label="turn timeout", - ) as wd: - async for message in query(...): - ... - if wd.fired: - raise TurnTimeoutError(...) - - Notes: - - ``timeout_seconds`` of None or <= 0 → no-op watchdog (no timer started). - This lets callers write a uniform ``with`` block regardless of whether - a timeout is configured. - - When ``asyncio_task_to_cancel`` is provided, the timer thread also - calls ``loop.call_soon_threadsafe(task.cancel)`` after ``on_timeout``. - This delivers cancellation across the thread boundary so mock-based - tests (no real subprocess) still unwind at the deadline. - - Each instance is single-use. Re-entering the ``with`` block after exit - is not supported. + from the TIMER THREAD, not the event loop, so ``on_timeout`` MUST be + synchronous and thread-safe. Exceptions it raises are logged and swallowed, so + one bad callback never kills the timer thread without a trace. After it fires, + ``fired`` is True, readable from any thread. + + ``timeout_seconds`` of None or <= 0 is a no-op watchdog, so a caller can write + a uniform ``with`` block whether or not a timeout is configured. When + ``asyncio_task_to_cancel`` is given, the timer thread also cancels it across + the thread boundary, so mock-based tests with no real subprocess still unwind + at the deadline. + + Each instance is SINGLE-USE: re-entering the ``with`` block is unsupported. + + Rationale: .claude/notes/agents.md § The threaded watchdog """ def __init__( diff --git a/src/coder_eval/streaming/events.py b/src/coder_eval/streaming/events.py index 8e8bf43fa..505a4cddb 100644 --- a/src/coder_eval/streaming/events.py +++ b/src/coder_eval/streaming/events.py @@ -39,10 +39,9 @@ # --- Status codes ----------------------------------------------------------- -# -# One enum per End-event level. Every End event carries the appropriate one so -# success, tool error, permission denial, crash, timeout and orphaned tool calls -# are a single mechanism (no ToolErrorEvent subclass + string scans). +# One enum per End-event level, so success, tool error, permission denial, crash, +# timeout and orphaned calls are ONE mechanism rather than a subclass plus string +# scans. class ToolEndStatus(StrEnum): @@ -74,9 +73,8 @@ class AgentEndStatus(StrEnum): STOPPED_EARLY = "stopped_early" # cooperative early-stop-on-criterion (clean, non-crash) -# Reuse the canonical TranscriptMessage union (defined once in telemetry.py) so -# AgentEndEvent carries per-message telemetry losslessly and stays in lock-step -# with TurnRecord.messages (the deferred token path rides here, not on granular events). +# The canonical TranscriptMessage union, so AgentEndEvent carries per-message +# telemetry losslessly and stays in lock-step with TurnRecord.messages. _MessageList = list[TranscriptMessage] diff --git a/src/coder_eval/streaming/renderers.py b/src/coder_eval/streaming/renderers.py index 0bda6db44..0829488d3 100644 --- a/src/coder_eval/streaming/renderers.py +++ b/src/coder_eval/streaming/renderers.py @@ -95,18 +95,17 @@ def _format_event(self, event: StreamEvent) -> str | None: model = f" (model={escape(event.model)})" if event.model else "" return f"[bold]--- Iteration {event.iteration}{model} ---[/bold]" - # TurnStartEvent is intentionally not rendered on the console: the Rich - # view is a terse live feed and Claude emits one per API call (noisy). - # The full turn tree lives in task.log via LoggingStreamRenderer. + # Intentionally NOT rendered on the console: Claude emits one per API + # call, which is noisy for a terse live feed. The full turn tree lives in + # task.log via LoggingStreamRenderer. if isinstance(event, ToolStartEvent): params_str = escape(format_payload(event.tool.parameters, max_chars=_MAX_PARAMS_LEN)) return f"[cyan]>>> TOOL: {escape(event.tool.tool_name)}[/cyan] | {params_str}" if isinstance(event, ToolEndEvent): - # result_summary is stored WHOLE (untruncated) at capture; cap it here - # for the terse live feed so a large command output doesn't flood the - # console. The reported char count reflects the true (full) length. + # result_summary is stored WHOLE at capture; capping is a display + # concern only, and the reported char count is the true full length. full = event.tool.result_summary or "" preview = escape(_truncate(full, _MAX_RESULT_LEN)) if event.status == ToolEndStatus.OK: diff --git a/src/coder_eval/streaming/wire.py b/src/coder_eval/streaming/wire.py index 71b54816a..78d264112 100644 --- a/src/coder_eval/streaming/wire.py +++ b/src/coder_eval/streaming/wire.py @@ -32,10 +32,10 @@ logger = logging.getLogger(__name__) -# ASCII Record Separator (U+001E) framing the sentinel makes collision -# with real agent / tool / pytest output effectively impossible: control -# chars below 0x20 don't appear in normal stdout. A line that starts with -# this exact byte sequence is a streamed event by construction. +# ASCII Record Separator (U+001E) framing makes collision with real agent, tool +# or pytest output effectively impossible: control chars below 0x20 do not appear +# in normal stdout, so a line starting with this byte sequence is a streamed event +# by construction. LINE_PREFIX = "\x1ecoder-eval-stream\x1e:" _EVENT_CLASSES: dict[str, type[StreamEvent]] = { @@ -104,12 +104,9 @@ class StdoutNDJsonCallback: """ def on_event(self, event: StreamEvent) -> None: - # Plain print + flush -- the in-container Python is line-buffered - # under non-tty stdout, so explicit flush matters. - # BrokenPipeError guard: if the host got SIGKILL'd or docker-kill'd - # the container mid-run, our stdout pipe peer is gone. Letting - # BrokenPipeError propagate would crash whatever in-container code - # path emitted the event (typically the orchestrator's run loop) - # and prevent task.json from being written for partial results. + # Explicit flush: the in-container Python is line-buffered under non-tty + # stdout. The BrokenPipeError guard matters because a host SIGKILL leaves + # our pipe peer gone, and letting it propagate would crash the + # in-container run loop before task.json is written for partial results. with contextlib.suppress(BrokenPipeError, OSError): print(serialize_event(event), flush=True) diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index b810657b4..f550e7fa3 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 73_413 +_ESSAY_BASELINE_WORDS = 60_642 _SRC = Path("src/coder_eval") From d72ac1008cd5e7e9ce8192021b3ec56a835c677d Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 19:00:50 -0700 Subject: [PATCH 04/19] =?UTF-8?q?docs:=204/7=20=E2=80=94=20move=20orchestr?= =?UTF-8?q?ation=20rationale=20into=20.claude/notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cuts orchestrator.py and orchestration/ from 17,724 essay words to 3,980, extending the four sections Phase 1 relocated into orchestration.md rather than opening rival headings for the same topics: the terminal-status chain and the four grading sites under execute-vs-run, the fired-only gate and the ceiling/floor bounds under early stop. New sections cover what the orchestrator alone owns — recording the task as authored, the three separately-resolved routes, interrupt-proof teardown, restoring a PATH from an untrusted run directory, the dialog loop, embedded commands and experiment resolution. regrade's container rationale went to isolation.md, where detached grading already lives. `prose_budget` now resolves a pointer against `##` or `###`, so appending to an existing section does not force the pointer up to its parent. No executable statement changed — proved by `prose_budget --assert-code-unchanged`. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/isolation.md | 131 +++ .claude/notes/orchestration.md | 494 ++++++++ src/coder_eval/orchestration/batch.py | 182 ++- src/coder_eval/orchestration/config.py | 32 +- src/coder_eval/orchestration/early_stop.py | 316 ++--- src/coder_eval/orchestration/evaluation.py | 30 +- src/coder_eval/orchestration/experiment.py | 215 ++-- src/coder_eval/orchestration/regrade.py | 528 +++------ src/coder_eval/orchestration/task_loader.py | 87 +- src/coder_eval/orchestrator.py | 1169 +++++++------------ tests/lint/prose_budget.py | 6 +- tests/test_prose_budget.py | 11 + 12 files changed, 1472 insertions(+), 1729 deletions(-) diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index d5ab9eb73..827ac490e 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -5,3 +5,134 @@ ## Detached grading and `Sandbox.adopt` - **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses is ONE parameter (`grade_in_place`, resolved by `_gate_scope_for_grade`), not two: it shipped beside an `include_setup_phase` every caller passed as its exact complement, and a future caller setting one and forgetting the other would silently drop half of a SECURITY gate. Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and `_assert_regrade_honored` refuses a returned row whose `started_at` moved, because an image predating this change ignores the unknown `regrade` key and RUNS THE AGENT, which the host would otherwise fold back as the recorded row's verdict (the exact sibling of `_assert_grade_honored`, one release later). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. `grant_container_access` now RETURNS what it widened and `run()` restores it in the same `finally`: the two staging dirs are disposable, but the graded workspace is the caller's tree — an operator-supplied `--workspace` was left world-writable permanently. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: every container is launched `TELEMETRY_ENABLED=false` under the invariant "container silent, host emits once", and the grading path had inherited only the silent half. The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `git clone` now passes `--` before the URL (argv position 2, so a value beginning with `-` was parsed as an option). `Sandbox.resolve_files` is containment-checked for the same reason: criterion paths were the one task-authored path skipping `_resolve_within_sandbox`, and `Path(root) / '/etc/passwd'` is `/etc/passwd`. An escaping LITERAL now raises `CheckerMisuseError` rather than resolving to `[]`: returning no match books an eval-CONFIG error as an agent failure — a gating 0.0 reading "file does not exist" for a file that plainly does exist and that no agent behaviour could place inside the sandbox (CE039's exact distinction). `tasks/byod_smoke_test.yaml` was broken that way for several commits, checking `/opt/byod_marker` baked into the BYOD image, with only a task-log warning to show for it; it now asserts on the container with `run_command: test -f …`, which is what a claim about the IMAGE rather than about the agent's workspace should look like. The guard keys on the escaping path EXISTING, so a merely-absent absolute path stays an ordinary failing verdict, and the GLOB branch still warns-and-drops, since filtering some matches out of a search is its normal behaviour. A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (the one legitimate rewrite — we are already inside the container the driver asked for), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. + +### What a graded row inherits, and what it does not + +A task row describes the TASK, so its clock is the agent run's — not the grading pass's. +`started_at`, `completed_at` and `duration_seconds` are all restored from the prior row, +because a 10-minute run re-graded in 2 seconds would otherwise report 2 seconds, and that +figure feeds `average_duration`, the report tables and the evalboard. All three are +restored together, so the row's time fields stay consistent with each other: leaving +`completed_at` at grading wall-clock produces a triple where +`completed_at - started_at != duration_seconds`. + +`setup_ms` is restored for the same reason — a detached grade ADOPTS the workspace rather +than building one, so its own setup is a different activity. `grading_ms` goes the other +way and is deliberately NOT carried: the verdict this row now holds came from THIS pass. + +`pre_run` belongs to the EXECUTE phase and is not re-run against an adopted workspace, so +its recorded outcomes are carried or they vanish from the graded row. `post_run` is the +opposite — it belongs to the GRADING phase, so on a row that came from `execute` the list +is empty and this grade is about to fill it. Both are copied into fresh lists, so +appending can never mutate the prior result. + +`environment_info` is merged, not replaced. The prior capture describes the machine that +RAN the task; ours describes the machine grading it. Prior wins on conflict, and ours +survives as flat `graded_by_*` scalars — flat because `environment_info` is consumed as a +flat map everywhere (the HTML report escapes each value into a table cell, the evalboard +types it as `Record`), so a nested capture renders +as a Python dict repr. Only the facts that identify the grading HOST are kept, and only +when they differ. The same rule covers the grader's API route: writing it into the run's +keys would leave a self-contradictory record — `api_routing: anthropic_direct` beside the +run's stale `aws_region`. + +### Why pre_run and post_run each run exactly once + +`adopt()` guarantees it materializes nothing into the workspace, but that guarantee is +only as strong as its weakest caller: `run()` invokes the hooks unconditionally, with +`cwd = sandbox_dir`. Several in-tree tasks stage fixtures there (`cp -a /app/[!.]* "$PWD/"`), +so re-running `pre_run` during a detached grade would overwrite the agent's deliverables +BEFORE the criteria read them — silently changing the verdict and destroying preserved +artifacts. + +`post_run` runs after the verdict is finalized and is free to mutate the workspace +(`rm -rf node_modules` is the archetype), so it belongs to whichever phase GRADES. +Running it under `execute` inverted its own contract and broke round-trip equivalence: +the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace +`post_run` had already modified. Its two skips are NOT the same condition — `grade=False` +DEFERS it to the grading pass, while an adopted sandbox whose prior row already recorded +`post_run` results has already run it once, and nothing declares those commands +idempotent. + +A run that is never graded therefore never tidies its sandbox — that is the accepted cost +of keeping the verdict honest. + +## Grading a docker row inside a container + +A `driver: docker` row is graded IN a container of its own image, which is the only place +its criteria mean what they meant during the run. The dispatch happens before anything +else — the reference check included — so the container performs every step against +container paths rather than having half of it done against the host's. + +Host-grading a container task is REFUSED rather than downgraded. Its criteria address +container paths (`/verifier`, `/logs/verifier`) and container toolchains; run on the host +they score 0.0 for a trajectory `run` scored 1.0, and the row is written back FAILURE. The +same commands (`rm -rf /verifier`, `mkdir -p /logs/verifier`) also execute unsandboxed on +the grading machine. A silent rewrite additionally neutralized the `docker` refusal in +`Sandbox.adopt`, which exists to catch exactly this. `--allow-host-grading` is the +operator's explicit acceptance of both, and such rows are STAMPED `graded_on_host` so they +are never silently comparable with rows a container graded. + +(The host-grading config's docstring once opened "grading never runs a container: the +docker driver dispatches through DockerRunner, which needs an agent". That premise was +simply wrong — a grading pass needs no agent — and it is why the refusal survived a +release after it stopped being the only answer.) + +### Two known equivalence gaps, stamped rather than refused + +Both are recorded ON the row, for the reason the host-grading stamp exists: a console +warning does not travel with `task.json` into `run.json`, the reports or the evalboard, so +a row it describes cannot be filtered out of a comparison by anything downstream. + +**`graded_without_pre_run`** counts `pre_run` commands that ran in the container which +executed the agent and were NOT re-run. The grading container is a SECOND, fresh one — +only the workspace crosses over — and `pre_run` is not re-run because `Sandbox.adopt` sets +`was_adopted` and the orchestrator skips it (re-running would overwrite the agent's +deliverables before the criteria read them). It is not hypothetical: three of the ten +in-tree `driver: docker` tasks seed state outside the workspace, and `3d-scan-calc` +symlinks `/root/mass_report.json` whose existence is its verifier's first assertion — so +such a row scores 0.000 for a trajectory `run` scores 1.000. Refusing outright was +declined: it would break `run --resume` on rows it grades correctly today whenever +`pre_run` happens to touch only the workspace, which is the common case. A durable, +machine-readable marker lets a consumer decide; a refusal does not. + +**`graded_with_rebuilt_image`** marks the other: the pass re-runs `docker build` under the +run's deterministic tag, so the grading image REPLACES the run's under the same name. A +Dockerfile, build context or base image that moved between the phases means the criteria +read a different filesystem than the agent did. Nothing pins or records image identity on +either side yet, so it cannot be detected after the fact — which is exactly why it is said +at dispatch. A `reference_digest`-style pin is the real fix and needs the identity +recorded on the RUN side too. + +### Why the grading container gets a private scratch directory + +The two callers disagree about what `run_dir` is — `evaluate` passes a freshly prepared +directory, `run --resume` passes the executed row's OWN — and every part of +`DockerRunner`'s result handling assumes an output dir it alone populates: + +- `_parse_result_or_raise` decides "did the container produce a result?" on + `task_json.exists()` and discards the return code. Over the row's own directory the + pre-grade `task.json` is already there, so a grading container that DIED (OOM, exit 137, + or any in-container FATAL guard) was read back as a successful grade — returning the + stale ungraded row as the verdict, with the container's error discarded. +- `run()` opens `run_dir/docker.log` with mode `"w"`, truncating the executed container's + log — the same loss the task.log/grade.log split exists to prevent. +- `grant_container_access(output_dir, writable=True)` would recursively widen the whole + preserved artifacts tree. + +A private directory makes both callers identical. + +Its logs are folded back on BOTH the success and the failure path, and the FAILURE path is +what makes it necessary: the scratch dir is deleted the moment the `with` exits, and +everything explaining a failure lives in it — `docker.log`, a failed build's captured log, +the synthetic `BUILD_FAILED` records. Folding out only on success deleted precisely the +evidence, and the error's own text says "See {log_path} for container output", naming a +path that no longer existed by the time it was printed. `grade.log` is the grading pass's +OWN log, holding the per-criterion detail that is the only durable record of WHY a +criterion scored what it did, and a documented part of the run-directory contract. + +The fold-back refuses to write through a SYMLINK: `shutil.copy2` opens the destination for +writing and follows one, which is an arbitrary-file-overwrite primitive in a run directory +the grader did not create. `docker.log` is renamed for the PHASE, because on the resume +path that name is already taken by the executed container's log. + diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index c5c88b3e6..2377a438f 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -12,6 +12,51 @@ - **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` with grading removed — the agent runs and the full trajectory is captured, but no criterion is checked, `weighted_score` is `None` (never `0.0`, which would be indistinguishable from "graded and scored zero"), and the row finalizes as **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are facts about the *run*, not about grading, and still apply (so `execute` still exits non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog check, and post-failure diagnostics); it crosses the docker boundary in `context.json` (defaulting to `True` in-container, so a host predating `execute` keeps grading). It is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because a task YAML must never declare itself ungraded; only the invoking command decides. `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code path. Three things are refused rather than degraded: `--junit-xml` (a report of verdicts, and there are none — though `reports_junit` still emits `` for an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades nothing), and simulation tasks (their turn-continuation logic reads criteria results, so an ungraded dialog would silently change its own stopping behavior). `stop_early:` blocks are inert under `execute` for the same reason the kill switch exists: the full trajectory is the deliverable. Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and grades with its own tests. +### The terminal-status chain + +`Orchestrator._terminal_status` answers one question and `run()` answers several, which is +why it is extracted; inlining it pushed `run()` past its complexity bound the moment the +grading switch was threaded in. Its ORDER is load-bearing at every step. + +**A detached grade may not overturn an execution fact.** The prior run's terminal status +(TIMEOUT, ERROR, a budget stop) describes an agent phase this pass neither repeated nor +observed. Without that first arm, a crashed run re-graded against its half-finished +workspace reports SUCCESS — with the original `error_message` still attached. + +**The NOT_GRADED arm sits ABOVE `max_turns_exhausted`, and that order is what makes +`execute` + `evaluate` equal a single `run`.** MAX_TURNS_EXHAUSTED reads like an execution +fact but is not one: on the graded path it is subordinate to the verdict — `run` returns +SUCCESS for a max-turns trajectory whose criteria pass, and only falls through to +MAX_TURNS_EXHAUSTED when they do not — so it is not knowable under `grade=False`. +Consuming it first made it terminal AND permanent, so the same agent output scored +SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`; being +category `failed`, `run --resume` then called the row complete and left it forever +unscored. Nothing is lost by deferring: the fact lives on `result.max_turns_exhausted`, +which the seeding carries. The statuses that ARE execution facts differ in kind — they +abort the run before a verdict is reachable, so preserving them overturns nothing. + +### The four grading sites + +`grade=False` is checked in exactly four places, and they do not behave alike: + +1. **Evaluate-only** refuses: with no agent run and no criteria there is nothing left but + an empty `task.json`. +2. **The single-shot loop** stops after capturing the trajectory. Returning False keeps + `FinalStatus` off SUCCESS and the status chain turns it into NOT_GRADED. The + reference-integrity check is skipped too — it protects a grade that is not happening. +3. **The dialog loop** RAISES, and is unreachable today because `execute` rejects + simulation tasks at the CLI: the dialog reads criteria results to decide whether to + keep talking, so an ungraded dialog would silently change its own stopping behavior. + An empty-list "defensive no-op" here would be worse than a refusal — both callers go + straight on to the gate, which treats an empty criteria list as a vacuous pass. +4. **The diagnostics path** records nothing: a `not_evaluated` vector would imply criteria + we were supposed to run and could not. + +Facts about the run are recorded BEFORE the switch, because `execute` withholds the +VERDICT, never the facts — the seeding cannot restore a fact the execute phase never +captured. The budget gate runs AFTER the criteria on the graded path purely for +partial-credit visibility, and there is no partial credit under `execute`. + ## `--resume` is command-relative - **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate ` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`. @@ -19,3 +64,452 @@ ## Early stop on criterion - **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. + +### Gate selection is fired-only + +The weighted armed gate applies IFF the watcher actually cut the run. On a truncated +trajectory the unarmed criteria never had the chance to be satisfied, so they stay +advisory; a run that completed naturally — armed or not, watcher never fired or disarmed +fail-open — has a full trajectory and gates strict-AND over every gating criterion. +Arming a criterion must never change the verdict of a run it did not cut. + +**Both single-shot grading paths must call the same selector.** A detached grade reaches +the verdict through the evaluate-only branch, where `early_stop` arrives from the seeding +rather than from a live watcher; selecting the gate there in a second, hand-written place +is exactly how the seeded field came to be carried but never read, so re-grading an +early-stopped run under the full-run gate flipped its verdict. + +One gate for every early-stopped run, with no per-reason branches: a decision-budget stop +is just a fail-stop whose deciding criterion timed out. The ceiling is an upper bound on +the authoritative armed score only because the watcher reduces the SAME trajectory the +checker scores — it records UNRESOLVED tool ends exactly as the agent's `EventCollector` +does — so the weighted armed gate is correct whether the watcher fired on a pass, a fail +or a timeout. + +**The simulation dialog path does NOT route through this seam**, and `result.early_stop` +is never assigned there, so an armed simulation task gates strict-AND on a possibly +truncated trajectory. Wiring the dialog path through it means also setting `early_stop` +there; until then the limit is stated rather than implied. + +The watcher is built ONCE, in `_setup`, so its turn/tool counters and wall-clock origin +accumulate across retry attempts. It is built before the evaluate-only early return, so an +armed evaluate-only re-grade builds an inert, never-fed watcher — harmless, and one +creation point. Under `execute` it is armed but stays disabled: there is no outcome to +decide and the trajectory is the deliverable, so an armed criterion must not truncate it. + +### Verdicts latch, and the decision happens on the CALL + +Once an armed criterion decides on a RESOLVED round, its `live_verdict` is never polled +again — the checkers' documented monotonicity makes re-polling pure waste. Latching happens +only on resolved rounds, so a dispatched call that never resolves (a crashed attempt) +cannot leave a stale verdict behind across retries; an in-flight round's fresh verdict can +still FIRE a stop, it is just not persisted. + +Deciding on the tool CALL rather than its result is what makes the stop robust: for an +observable criterion the verdict is fully determined by the call's inputs (which skill, +which command), so the watcher latches the instant the call is dispatched — before a +cut-short turn can strip the result and leave the call unresolved. The agent polls +`should_stop` immediately after dispatching each message, so a stop on the call breaks the +loop before the result is ever pulled. The matching `ToolEndEvent` still evaluates, which +covers a verdict that only becomes decidable once the result is known. + +**UNRESOLVED tool ends are RECORDED but never counted or evaluated on.** Agents force-close +orphaned tools as UNRESOLVED *after* the message loop has ended and the terminal status is +already chosen, so they are not live tool activity — treating them as such would let a run +that completed, timed out or crashed without a real in-loop decision latch a false early +stop. + +Live verdicts only TRIGGER the stop; the authoritative scores always come from the standard +check on the frozen trajectory after the cut. + +### The ceiling and floor bounds + +Both the stop rule and the post-hoc gate consult `run_limits.stop_early_gate_threshold` +(default `1.0`) rather than treating every armed criterion as equally decisive. + +A **fail-stop** fires once the armed set's CEILING — best case, every still-undecided or +passing criterion scores 1.0 and every effectively-failed one scores 0 — can no longer +reach the threshold, i.e. the gate is mathematically guaranteed to fail however the +trajectory continues. A `decide_within` timeout participates as an ordinary weighted fail, +so a low-weight criterion's timeout that cannot doom the gate does not stop the run. + +A **pass-stop** fires once the `on_pass: stop` subset's FLOOR — worst case, every +still-undecided member scores 0 — already meets the threshold. + +At the default threshold both bounds collapse exactly to "any single armed criterion's +effective fail stops the run" and "every `on_pass: stop` criterion has live-passed". +Lowering it lets a low-weight armed criterion's failure be absorbed without truncation. + +### Precision is traded, recall is not + +A pass-stop cuts the run the instant the floor locks in, so a fail-armed criterion (a +distractor) that would only misfire on a LATER tool call is never observed, and the frozen +trajectory scores that row a clean pass. That is an intentional precision-for-budget trade +of the opt-in smoke flavor; **authoritative precision must come from a non-early-stop run.** + +Recall is never truncated, because BOTH stops defer on it. The fail-stop is DEFERRED while +any pass-capable armed criterion is still undecided and within its budget, so a distractor +misfire on an early tool call cannot cut a positive row before its expected signal has had +the chance to appear — which would freeze a would-be true positive as a false negative and +deflate recall. Symmetrically the pass-stop is DEFERRED while any pass-capable armed +criterion OUTSIDE the `on_pass: stop` subset is still undecided; subset members are already +accounted for by the floor bound itself. Otherwise an `on_pass: stop` criterion passing +early would freeze a sibling `on_pass: continue` criterion as an unearned fail on the +truncated trajectory. + +**Neither deferral LOSES the trigger.** Verdicts latch monotonically, so a held stop fires +the moment every pass-capable armed criterion decides — the fail-stop is evaluated BEFORE +the pass-stop each round — and if none ever decides, the run simply continues to the cap. +A row with zero pass-capable armed criteria (a negative row stacking only distractors) has +nothing to defer for and fail-stops on the first misfire. + +### Inert triggers are by design, and the watcher fails open + +A trigger whose polarity an instance can never decide is INERT, not an error — one +dataset-fanned YAML line serves both positive rows (pass and timeout live, fail inert) and +distractor rows (fail live, pass and timeout inert) without per-row conditionals. That is +why the validator carries NO per-instance polarity guards. Arming an unobservable criterion +is structurally impossible, since the block exists only on `LiveSuccessCriterion`, so a +`file_exists` criterion carrying one is an `extra='forbid'` error at load. An armed-but- +empty set needs no guard either: with no blocks present there is simply no watcher. + +The watcher keeps its OWN `EventCollector`, independent of the one the agent builds its +returned `TurnRecord` from, so each `live_verdict` sees a fresh single-element partial +trajectory. + +**Fail-open:** a `live_verdict` that raises disarms the watcher, logs loudly, and degrades +to a full run. Because live verdicts are triggers and not truth, this can never produce a +FALSE early stop — it only ever errs toward running more. + +## Recording the task as authored + +`task_config.resolved` and `source_file` describe the task as AUTHORED, which is NOT +always what this process runs. + +`run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the +in-container orchestrator, because it is already inside the container the driver asked +for. Recording that rewrite made the run's own record deny it ever used docker — and a +later `evaluate ` reads the driver back out of the record, so the host-grading +refusal never fired and the `graded_on_host` stamp was never applied. A container task's +criteria ran against the host filesystem silently, which is the exact outcome that gate +exists to prevent. + +The PATH is the same seam for the same reason. `task_file` is what this process resolves +`TASK_DIR` and the reference against; in a container that is `/work/task_dir/task.yaml`, +correct there and meaningless anywhere else. Recording it made a container row's +`source_file` name a path that exists on no host, so a later `evaluate` rebuilt the task +around it: the docker dispatch guard saw a non-None `Path` and let it through, and the +task-dir mount then silently mounted nothing, so every `$TASK_DIR` criterion resolved +against the wrong tree and scored a verdict nobody could explain. + +## Three routes, resolved separately + +The agent's route, the judge's and the simulated user's are resolved independently and +recorded separately, so a run artifact shows what actually ran, graded and talked to it. + +- **`route`** is the agent's own. +- **`eval_route`** (llm_judge / agent_judge) is pinned to a constant Claude backend when + the agent runs on an open-weight LiteLLM model, so grading stays comparable. Its model + is recorded separately from the agent's, so a `checker_context.api_route` override is + visible in run artifacts rather than merely inferable. +- **`simulator_route`** is resolved through the same pinning guard but with NO + `checker_context` overrides: that knob is llm_judge-only and has no bearing on the + simulator. The simulated user is part of the MEASURING INSTRUMENT and must not run on + the agent's own gateway either — but it is also a real Claude Code CLI subprocess, like + the agent under test, not a checker concern. Aliasing it to `route` would drop the + pinning; reading `checker_context` would couple it to the judge. + +All three equal `route` on the Direct and Bedrock backends. + +## Where the orchestrator's own time is booked + +(The bucket definitions live in [timing.md](timing.md); this is only where the +orchestrator starts and stops its own clocks.) + +`setup_ms` covers everything before the agent runs — environment capture, criterion +discovery, sandbox provisioning, `agent.start()` and `pre_run` — as ONE task-level bucket. +Its mark sits at the top of `run()`, not at `_setup()`, because the single largest item is +already behind us by then: `get_version_info()` shells out for the git commit and every +CLI's `--version` and costs 733 ms measured. Starting the mark at `_setup()` put that +outside every named bucket, so it landed in the report's residual — 733 of the 758 ms that +made "Unaccounted" look like a real unknown when it was one nameable call. + +The bucket is roughly harness-independent — measured within ~10 ms of each other for +claude-code and pi on the same machine — which is the tell that it is the orchestrator's +own cost rather than any harness's. Left in the residual it read as 10% of a 19 s task, +and would read 60% of a 3 s one. + +A dimension is OMITTED rather than coalesced to `0`, in telemetry and in the report alike. +Dashboards average with no status filter, so a laundered zero for an ungraded night drags +every tile toward zero and is indistinguishable from a genuinely bad night, or from a +harness that booted instantly. An absent dimension drops out of the average instead. The +same argument makes the log line read `n/a` rather than `0.000`. + +## Teardown must be interrupt-proof + +The task-timeout watchdog can fire while post-run commands are awaiting and deliver its +`CancelledError` inside the `finally` block, which used to abort teardown wholesale — +skipping cleanup (tempdir leaked) AND result finalization (`task.json` lost, so the task +silently drops out of the run). The interrupt is caught, the full teardown runs, and it is +re-raised at the end so callers observe the same exception. The watchdog cancels exactly +once, so the awaits after the catch run normally. + +## Restoring a PATH from a run directory + +A run dir is a shareable artifact — that is the whole point of detached grading — and +under `driver: docker` it is bind-mounted writable into the container the agent runs in. +The PATH recorded in its own `task.json` is PREPENDED ahead of the host PATH, so taken +verbatim it lets a run dir decide which binary `pytest` resolves to on the grader's host. + +Four filters, all about what PATH parity actually needs: + +- **Absolute only.** A relative entry resolves against the grader's current working + directory, which has nothing to do with the run, so `evilbin` becomes `$PWD/evilbin` at + the front of every criterion subprocess's PATH. It also cannot be the toolchain location + it claims to be, since the run resolved it somewhere else. +- Drop anything that is not an existing directory — a dead entry buys no parity. +- Drop any entry inside the WORKSPACE being graded: that tree is agent-writable, so a shim + dropped there would shadow a real tool. +- Drop any entry inside the RUN DIRECTORY as a whole. The workspace is only part of it; + `artifacts/`, a sibling replicate's tree and the run root all travel in the same shared + artifact and are equally attacker-chosen. + +The PATH is captured only on the per-turn happy path, after a successful turn, which +leaves three gaps: an agent crash or turn timeout (the sync never runs, and a crashed +agent's SDK PATH may itself be unreliable), evaluate-only mode, and the window before the +first turn. Persisting it is what closes the evaluate-only gap for a LATER detached grade, +which would otherwise resolve `run_command` criteria against ambient PATH and could reach +a different verdict than the run it claims to be grading. + +A sandbox-setup-time sync was considered and rejected: the agent SDK's effective PATH is +only knowable after the SDK initializes, so it would capture the configured prepends +rather than the full agent environment. + +## The dialog loop + +(The per-site mechanics stay as comments in `_simulation_dialog_loop`; this is only the +shape of the loop and the reasoning that does not fit beside one statement.) + +One invocation runs exactly one dialog trajectory; parallel trials are expanded upstream. +It replaces the criteria-feedback iteration loop for tasks carrying a `simulation` block, +and emits the same streaming events as the single-shot loop, so downstream renderers work +unchanged. + +**It is the one grading path that does not route through the shared gate selector** — see +§ Gate selection is fired-only — so `result.early_stop` is never assigned here and an armed +simulation task gates strict-AND on a possibly truncated trajectory. + +`execute` rejects simulation tasks at the CLI, because the loop reads criteria results to +decide whether to keep talking: an ungraded dialog would silently change its own stopping +behaviour rather than merely skipping the score. + +## Embedded commands + +A recorded config is UNTRUSTED INPUT. `evaluate ` rebuilds the task from the +run's own `task_config.resolved`, and a run directory is a shareable artifact — so +everything the rebuilt config would run executes on the GRADER's host, under the grader's +credentials. `embedded_commands` enumerates that surface and `check_embedded_commands` +refuses, or at minimum names it, before anything runs. + +The enumeration deliberately includes capabilities that are not shell lines, because the +question the consent prompt answers is "what will this do on my machine", not "what will +it exec": + +- **`agent_judge`** has no command string of its own; it spawns a tool-using SDK agent + (Bash included) under the grader's credentials, which is strictly WIDER than one shell + line. +- **`llm_judge`** executes nothing locally, but it spends the grader's model budget and + ships the graded artifacts — optionally the trajectory — to a provider of the recorded + config's choosing. That is a capability an operator should approve. +- **`uipath_eval`** shells out with every argument `shlex`-quoted, so it is disclosure + rather than injection — but it is still a subprocess the recorded config chose to start. + +### Why the container dispatch renders as ONE command + +`check_embedded_commands` joins the list with `"; "` and interpolates `len(commands)` into +the consent prompt, so every entry MUST be a command. The first version appended argv +FRAGMENTS separately, so a task with two build args and one mount asked the operator to +approve "4 shell command(s)" reading `docker build -f Dockerfile; --build-arg FOO=bar; +--build-arg BAZ=qux; -v /a:/b` — one docker invocation described as four commands, three +of which are not commands. The consent prompt is the one place this text has to be exact. + +It also names every HOST PATH the dispatch exposes, not just those under +`sandbox.docker`. Three families reach the record-named image without the record +mentioning them in a `docker` block: the TASK DIRECTORY, copied wholesale from the +recorded `source_file`'s parent (a record whose `source_file` is `~/.ssh/config` copies +all of `~/.ssh` in); every `agent.plugins[].path`, `TemplateDirSource.path` and +`agent.system_prompt_file`, auto-mounted read-only at their host paths; and a WRITABLE +copy of `~/.claude`, `.credentials.json` included. Networking defaults to bridge, so +anything the container can read it can also send. Disclosing only `sandbox.docker.*` +would ask the operator to consent to a strict subset of what actually happens. + +`docker build` deserves its own line in the prompt because it runs every RUN step in the +recorded Dockerfile on this host and expands recorded build args against the GRADER's +environment — so a `${ANTHROPIC_API_KEY}` arg is exfiltratable by a RUN step, and +`extra_args` is spliced into the argv unfiltered. + +A warning is not a control: `_sensitive_source_paths` only warns, and about a fixed list, +and it prints as the command is already being prepared. + +### What the gate covers, and why each part is in scope + +`include_setup_phase` covers the two capability families that exist only on the `--copy` +path: `pre_run`, and the sandbox's own provisioning. Both are SKIPPED when grading in +place, so on that path they are not a capability the run dir has. + +**`post_run` is deliberately NOT behind that flag**, and this is the one place the +distinction bites. It used to be, back when the hooks were skipped as a pair — but +`post_run` belongs to the GRADING phase, so it now runs on EVERY grading path, in place +included. Leaving it inside `include_setup_phase` made the in-place path — the DEFAULT for +a run directory — execute recorded shell with no consent prompt at all. + +It is filtered against the operator's own baseline for a reason worth stating precisely: +the gate asks the operator to approve shell THE RECORD CHOSE, and a single `coder-eval +run` runs `post_run` with no prompt, because the config came from the operator. The +grader's own default experiment appends the same `post_run` to every task it runs, so +finding one of those in a record reveals no choice the record made. Prompting on it would +fire on 100% of run directories, and **a refusal that always fires is read as a formality +and waved through** — which is how the gate would stop protecting the authored commands +that DO represent a choice. + +**Sandbox provisioning is the half the gate originally missed, and the worst one.** The +recorded `sandbox` block is carried through untouched, and the `--copy` branch then calls +`Sandbox.setup`, which reaches `uv pip install `, `npm install +` and `git clone `. A package name is arbitrary code at +install time. Because the scan walked only `success_criteria`, a shared run directory +whose criteria were all `file_exists` sailed through and still ran installers of the +attacker's choosing. + +**`include_container_dispatch` is the same omission again, one layer up**, reintroduced by +the very change that made a docker row gradable. The grade is DISPATCHED INTO A CONTAINER +built from the recorded `sandbox.docker` block — so the record chooses the image that runs +on this host, with the default credential allowlist forwarded into it, a writable copy of +`~/.claude`, and a pinned `--entrypoint` the image itself supplies. That is arbitrary code +execution from a shareable artifact, and strictly WIDER than the `run_command` strings the +gate already refuses. Like `post_run`, it is a capability of the IN-PLACE path, so it +cannot hide behind `include_setup_phase`. + +`grade_in_place` is the ONE lever selecting which families are disclosed, and deliberately +one parameter rather than two. It shipped beside an `include_setup_phase` that every +caller passed as its exact complement — two names for one fact, with nothing rejecting the +incoherent pairings. The flag gates a SECURITY disclosure, so a caller that set one and +forgot the other would drop the container-dispatch half with nothing failing. Both derived +values are computed once. `allow_host_grading` participates only through that derivation: +with it set no container is dispatched, so naming one would ask the operator to approve +something that never runs. + +The scan uses `isinstance` narrowing, never `getattr(c, "command", None)`: an untyped +string probe over a discriminated union is invisible to pyright, so renaming a field +silently degrades the only guard on this path to a permanent no-op. It also cannot reach +`agent_judge`, whose tooling is the widest blast radius of the three. + +### Why the container dispatch requires an EXISTING task file + +The image is built or named by the task's own sandbox config, and the Dockerfile and +reference directory resolve relative to the task file — so without one there is nothing to +build from. + +Testing only for `None` was not enough, and failed on exactly the rows the guard was +written for: the recorded `source_file` of a container row is a real, non-`None` path that +exists on no host, which is the defect § Recording the task as authored describes. Nothing +else here repeats that argument. + +Requiring existence also closes a second hole: on the detached path `task_file` comes +straight from the untrusted record, and its PARENT is what gets copied into the container, +so a recorded `source_file` of `~/.ssh/config` would copy all of `~/.ssh`. Existence alone +does not make the path trusted — that is the consent gate's job — but it removes the +silent-wrong-verdict half. + +## Locating the workspace a finished run left behind + +`sandbox_path` is authoritative when it still exists. Otherwise the preserved artifacts +tree, where preservation nests the workspace under the task id — the EXACT path, not "the +single child of `artifacts/`", because a dataset row's `task_id` contains `/` and the +heuristic resolves one level too high for every row task. + +It RAISES rather than guessing when neither is conclusive. Guessing is worse than failing: +grading the wrong directory makes every path-relative criterion fail as a locating +artifact rather than as a verdict, and reports that as an ordinary score. + +**Every return goes through the containment check, rooted at the RUN DIRECTORY.** Both +`sandbox_path` and `task_id` are unvalidated strings out of the run's own `task.json`, so +`"../../../../home/victim"` joins to a real directory `is_dir()` happily confirms. The +check originally covered one branch of four and rooted the `task_id` case at `artifacts/` +rather than at the run directory, which made it vacuous the moment `artifacts` was ITSELF +a symlink — and `artifacts/` is attacker-supplied for a shared run dir just like the other +two. The escaped tree then became the grading root via `Sandbox.adopt`, `run_command` +criteria ran with it as cwd, and the resulting verdict was written back into the run's own +`task.json`. + +## Experiment resolution + +**Dataset fan-out runs BEFORE variant resolution**, one task per row, each treated as an +independent task for the four-layer merge — which is what locks the invariant that a +variant cannot override the dataset. + +A per-task resolution failure is COLLECTED, not raised inline: a task whose own YAML is +incompatible with the resolved run (Claude-only `sdk_options` surviving a `--type codex` +override, which `CodexAgentConfig` forbids) would otherwise abort the entire run. The +file's resolved tasks are buffered and committed only once the whole file resolves, so a +mid-file failure discards that file's fan-out as a UNIT rather than leaving a partial, +lopsided one behind. + +The `except` sets are deliberately NARROW — `FileNotFoundError`, `OSError`, `ValueError`, +`yaml.YAMLError`. `AttributeError` / `TypeError` / `ImportError` signal a regression in the +loader and must crash loudly rather than silently demote every task to "skipped". Pydantic +`ValidationError` is a `ValueError` subclass in v2, so it is covered. + +**Early-stop arming errors always propagate** and are never demoted to skipped, so a +misconfigured run fails loudly instead of quietly shrinking the suite. + +If EVERY task that reached resolution failed, the run refuses rather than producing an +empty one — and it surfaces the FIRST task's own error rather than a synthesized "global +misconfig" message, because a genuine global cause (a bad `--type`, an invalid `-D`) is +indistinguishable from N tasks each independently incompatible for the same reason. A +`ValueError` is re-raised verbatim so its message stays clean; anything else is normalized +to `ValueError` so it still lands in the caller's clean handler instead of escaping as a +raw traceback. + +Tasks are sorted to run INTERLEAVED — replicate 0 of every (task, variant) first, then +replicate 1 — preserving declaration order within a replicate. Duplicate detection keys on +`(task_id, variant_id, replicate_index)`, because simulation replicates legitimately share +the first two. + +`skip: true` is honored BEFORE dataset expansion, so a quarantined task skips row fan-out, +variant resolution and any further I/O. It is still reported in `skipped_tasks`, so the +suite shows which YAMLs were intentionally excluded rather than failed to load. + +### No-op tasks need no special case anywhere + +`type` is a replace-scalar, so a task-level `type: none` wins over a baseline coding agent +injected by the default experiment, and the merged config validates as `NoneAgentConfig`. +A suite-wide `--model` or `-D agent.*` lands on it harmlessly, and `type: none` already +satisfies the `agent.type` contract. An explicit `--type ` is highest precedence and +replaces it, turning the no-op task into that agent. + +### Aggregation drops ungraded rows rather than zeroing them + +Ungraded replicates — and ONLY those — drop out of a mean entirely: `or 0.0` would average +a clean `execute` run down to a real-looking zero, while dropping an ERRORED one would pay +it a bonus. + +Only graded variants can win or set a spread; including ungraded ones at 0.0 would name an +arbitrary "best" among scores that do not exist. When NOTHING was scored there is no +winner, and the fallback must not invent one — `variants[0]` is whichever arm the input +happened to list first, so swapping the inputs flipped the reported winner while +`is_tie=False` asserted it was a real result. It now sorts by `variant_id` for determinism +and marks a tie among all arms, which is what "no arm outscored another" means. + +`tasks_measured` counts rows with a score, because a TIMEOUT lands in the `failed` bucket +without any criterion having run — so the buckets alone cannot answer "was this measured +at all". `average_score` is `None` rather than `0.000` when nothing was graded, since a +zero beside "Pass Rate: n/a" is indistinguishable from "scored zero". + +The status-priority map is annotated with the SAME `Literal` that `FinalStatus.category` +returns and indexed directly rather than via `.get(..., -1)`. Adding the fourth `ungraded` +bucket was a manual step no checker could verify — an untyped `dict[str, int]` proves +neither that every category is present nor that no stray key is — while the `-1` default it +leaned on was already unreachable, and, though documented as "fail-closed", sorted BELOW +error, so a fifth category would have silently outranked ERROR as the worst status. + + diff --git a/src/coder_eval/orchestration/batch.py b/src/coder_eval/orchestration/batch.py index 2dc2379d0..a3fed140d 100644 --- a/src/coder_eval/orchestration/batch.py +++ b/src/coder_eval/orchestration/batch.py @@ -150,14 +150,13 @@ async def run_single(rt: ResolvedTask) -> TaskResult: try: rt.run_dir.mkdir(parents=True, exist_ok=True) # noqa: CE002 — mkdir on local FS is nanoseconds sandbox_cfg = rt.task.sandbox - # Resolve the driver-derived preservation default HERE, where the - # original driver is still visible (the in-container orchestrator - # sees it forced to tempdir). Explicit --preservation-mode wins. + # HERE, where the original driver is still visible: the + # in-container orchestrator sees it forced to tempdir. driver = sandbox_cfg.driver if sandbox_cfg is not None else "tempdir" preservation_mode = resolve_preservation_mode(config.preservation_mode, driver) - # Explicit DIRECT_WRITE on a non-docker host runs the sandbox under - # run_dir, re-opening the parent-dir node_modules contamination the - # host default (MOVE_ON_WRITE) exists to prevent (MST-9795). + # DIRECT_WRITE on a non-docker host re-opens the parent-dir + # node_modules contamination MOVE_ON_WRITE exists to prevent + # (MST-9795). if config.preservation_mode == PreservationMode.DIRECT_WRITE and driver != "docker": logger.warning( "DIRECT_WRITE on driver=%s runs the sandbox under run_dir; parent-dir " @@ -165,11 +164,9 @@ async def run_single(rt: ResolvedTask) -> TaskResult: driver, ) if sandbox_cfg is not None and sandbox_cfg.driver == "docker": - # Docker isolation: spawn one container per task, parse - # its task.json on completion. The in-container CLI - # serializes stream events as NDJSON on stdout; we - # forward them to the host callback so --stream - # renders identically to the in-process path. + # One container per task; its NDJSON stream events are + # forwarded to the host callback so --stream renders + # identically to the in-process path. from ..isolation.docker_runner import DockerRunner result = await DockerRunner( @@ -179,10 +176,9 @@ async def run_single(rt: ResolvedTask) -> TaskResult: verbose=config.verbose, grade=config.grade, ).run() - # The in-container _finalize_result can't emit task telemetry - # (connection-string env vars aren't forwarded into the - # container), so emit the Task.End event host-side here - # — keeping docker runs at parity with the in-process path. + # The in-container finalize cannot emit telemetry (the + # connection-string env vars aren't forwarded), so the host + # emits Task.End here. from ..orchestrator import build_task_event from ..telemetry import track_event @@ -305,10 +301,8 @@ def _create_error_task_result( TaskResult with error information. """ error_type = type(error).__name__ - # A failed image build is an environment/setup failure — record it as - # BUILD_FAILED (not generic ERROR) so reports/run.json distinguish it. The - # import is lazy + scoped to this except-path use so batch stays import-light - # and the non-docker path never imports the docker runner. + # An environment/setup failure, recorded as BUILD_FAILED rather than generic + # ERROR. The import is lazy so the non-docker path never imports the runner. from ..isolation.docker_runner import DockerBuildError is_build_failure = isinstance(error, DockerBuildError) @@ -357,24 +351,17 @@ class ResumePartition(NamedTuple): def _owes_a_grade(result: EvaluationResult) -> bool: """Whether a finalized row was executed but never scored. - Evidence, not label. ``NOT_GRADED`` is the ordinary shape, but an ``execute`` - row that also tripped a run limit (TIMEOUT, a budget stop) carries an - execution-fact status whose category is ``error``/``failed`` — and no verdict - at all. Both are equally owed a grade; only the first announces it. - - A row that carries a criteria vector or a score has been graded, whatever its - status, so a genuine FAILURE/ERROR from ``run`` is untouched. - - "Executed" is a required half, not decoration. Evidence of *no verdict* alone - was too broad: ``DockerRunner._write_synthetic_task_json`` writes an ERROR / - BUILD_FAILED row for a container that died before producing task.json, and - those carry no verdict either — so a plain ``run --resume`` routed every - dead container and every failed image build into grading, where the fold-back - replaced the real diagnostic ("Container exited with code 137 without - producing task.json") with a wrong-cause grading error, and left the on-disk - record and run.json disagreeing about the same row. ``NOT_GRADED`` announces - itself and is always owed; an execution-fact status must also show that an - agent phase happened at all. + EVIDENCE, not label: ``NOT_GRADED`` is the ordinary shape, but an ``execute`` + row that also tripped a run limit carries an execution-fact status and no + verdict at all. A row with a criteria vector or a score HAS been graded, + whatever its status. + + "Executed" is a required half, not decoration — a synthetic ERROR / + BUILD_FAILED row for a container that died before producing task.json carries + no verdict either, and routing those into grading replaced the real diagnostic + with a wrong-cause grading error. + + Rationale: .claude/notes/orchestration.md § `--resume` is command-relative """ if result.weighted_score is not None or result.success_criteria_results: return False @@ -385,34 +372,18 @@ def partition_for_resume(resolved_tasks: list[ResolvedTask], *, grade: bool = Tr """Split resolved tasks over what ``--resume`` still owes each one. A task is already-complete when its task.json exists, parses, and carries a - final_status. task.json is written atomically at end-of-run, so any parseable - file with a status is a finished task (no partial-write ambiguity). Complete - tasks are reloaded into TaskResults (to fold into run.json) and excluded from - to_run; everything else — including failed-to-parse — re-runs. - - **"Finished" is relative to the resuming command, not absolute.** A - ``NOT_GRADED`` row (written by ``coder-eval execute``) has a final status, so - the naive test calls it complete. That is right for ``execute --resume``, - which owes it nothing — and wrong for ``run --resume``, which was asked to - grade: skipping it would report "already complete", grade nothing, and exit - 0. So under ``grade=True`` those rows go to ``to_grade`` instead, where the - caller runs the criteria against the trajectory and workspace already on - disk rather than paying for the agent a second time. - - The test is the row's **evidence**, not its status: a row is owed a grade - when it was executed but never scored. Keying on ``category == "ungraded"`` - alone missed every ``execute`` row that also carries an execution fact — a - TIMEOUT or budget stop aborts the run before grading, so under ``execute`` it - lands with ``weighted_score is None`` and an empty criteria vector, yet its - category is ``error``/``failed`` and resume filed it as complete. It then - stayed permanently unscored in run.json, the rollup and the evalboard, while - ``evaluate `` graded the identical bytes happily — two entry points - into one feature disagreeing about the same record. - - Note the asymmetry is only for a row that was never scored. FAILURE and ERROR - rows that DO carry a verdict stay complete under both commands — resume has - never retried failures (delete a task's task.json to force that), and this - does not change it. + final_status — written atomically at end-of-run, so any parseable file with a + status is finished. Everything else, including failed-to-parse, re-runs. + + **"Finished" is relative to the resuming command, not absolute.** Under + ``grade=True`` a row that was executed but never scored goes to ``to_grade`` + instead of being called complete, where the caller runs the criteria against + the trajectory already on disk rather than paying for the agent again. + + The asymmetry is only for a row that was NEVER scored: FAILURE and ERROR rows + that DO carry a verdict stay complete under both commands. + + Rationale: .claude/notes/orchestration.md § `--resume` is command-relative Args: resolved_tasks: Fully-resolved tasks for the whole run. @@ -487,23 +458,17 @@ def _load_completed_result(rt: ResolvedTask) -> TaskResult | None: def recover_task_results(run_dir: Path) -> list[TaskResult]: """Reconstruct ``TaskResult``s from every finalized ``task.json`` under ``run_dir``. - The disk half of the run-summary seam: pairs with ``build_run_summary`` to - (re)aggregate a finished run without re-executing it. Each ``task.json`` is the - atomically-written ``EvaluationResult`` for one task, so a file that is missing, - unparseable, or carries no ``final_status`` is skipped as not-yet-finalized - (mirrors ``_load_completed_result``). ``task_id`` and ``variant_id`` come from - the result itself; ``replicate_index`` is recovered from the - ``///`` layout (``build_task_run_dir``). ``suite_id`` / - ``row_id`` are not stored in ``task.json`` and are left ``None`` — they feed - suite rollups, not ``run.json``. - - Results are sorted by ``(variant_id, task_id, replicate_index)`` so the rebuilt - ``run.json`` ordering is deterministic, independent of filesystem walk order. - - A ``task.json`` that lives under a *nested* run dir — a subdirectory carrying its - own ``run.json`` — belongs to that sub-run, not this one, and is excluded so a - parent summary never absorbs a nested suite's tasks. Base runs have no nesting; - this only matters for composed layouts that stack sub-runs under one tree. + The disk half of the run-summary seam. A file that is missing, unparseable or + carries no ``final_status`` is skipped as not-yet-finalized. ``replicate_index`` + is recovered from the ``///`` layout; ``suite_id`` and + ``row_id`` are not stored in ``task.json`` and stay ``None``. + + Sorted by ``(variant_id, task_id, replicate_index)`` so the rebuilt ``run.json`` + ordering is deterministic rather than filesystem-walk order. + + A ``task.json`` under a NESTED run dir — a subdirectory carrying its own + ``run.json`` — belongs to that sub-run and is excluded, so a parent summary + never absorbs a nested suite's tasks. """ nested_roots = [p.parent for p in run_dir.rglob("run.json") if p.parent != run_dir] recovered: list[TaskResult] = [] @@ -513,9 +478,8 @@ def recover_task_results(run_dir: Path) -> list[TaskResult]: try: result = EvaluationResult.model_validate_json(task_json.read_text(encoding="utf-8")) except (OSError, ValueError) as exc: - # Unreadable / malformed / schema-mismatched. Skip so one corrupt file can't - # abort the rebuild, but warn — silently dropping it would shrink tasks_run - # (numerator and denominator) with no signal that a task was lost. + # Skip so one corrupt file cannot abort the rebuild, but WARN: + # silently dropping it shrinks tasks_run with no signal. logger.warning("skipping unreadable/malformed task.json %s: %s", task_json, exc) continue if not result.final_status: @@ -539,13 +503,10 @@ def recover_task_results(run_dir: Path) -> list[TaskResult]: # --- resume config fingerprint ------------------------------------------------ -# The per-task path key (variant_id/task_id/NN) does NOT encode result-affecting -# run config like the model or backend. So --resume, which matches finalized tasks -# purely by that path, would otherwise fold results produced under a *different* -# config into the new run (e.g. resuming a Sonnet run with --model opus keeps the -# Sonnet results for already-finalized tasks). We stamp the config on every run and -# warn (don't refuse) when a resume's config differs (in _run_with_experiment) so -# the resulting mixed-config run.json is surfaced rather than silent. +# The per-task path key does NOT encode result-affecting run config, so --resume +# would otherwise fold results produced under a DIFFERENT config into the new run. +# The config is stamped on every run and a differing resume WARNS rather than +# refuses, so the mixed-config run.json is surfaced rather than silent. RESUME_FINGERPRINT_FILE = "resume_fingerprint.json" @@ -591,10 +552,8 @@ def read_run_fingerprint(run_dir: Path) -> dict[str, object] | None: return data if isinstance(data, dict) else None -# `grade` is excluded from the drift warning: `execute` then `run --resume` is a -# SUPPORTED flow, not a config mistake, and the warning's text ("already-finalized -# tasks keep their original-config results") is actively wrong for it — those rows -# are re-graded with the current config, which is the entire point. +# `grade` is excluded: `execute` then `run --resume` is a SUPPORTED flow, and the +# warning's text is actively wrong for it — those rows ARE re-graded. _FINGERPRINT_DIFF_EXEMPT = frozenset({"grade"}) @@ -626,10 +585,8 @@ def _override_uip_versions_from_tasks(version_info: dict[str, Any], task_results ``"unknown"``, and for ``tool_plugins`` no non-empty plugin entry at all (legacy results, or every task errored before env capture). """ - # Defence-in-depth alongside the chokepoint fix in _uip_version: filter to - # version-shaped strings so junk already on disk (older runs captured a - # CLI that printed a JSON envelope instead of a version) can't leak into - # the aggregated chip when those results are re-summarised on --resume. + # Defence-in-depth alongside the chokepoint fix: filter to version-shaped + # strings so junk already on disk cannot leak in on --resume. cli_versions = sorted( { v @@ -650,9 +607,8 @@ def _override_uip_versions_from_tasks(version_info: dict[str, Any], task_results for name, plugin_version in plugins.items(): if isinstance(plugin_version, str) and plugin_version: plugin_versions.setdefault(name, set()).add(plugin_version) - # Gate on collected entries, not on "some task had a tool_plugins dict": - # an all-empty consensus ({} from every task) must not stomp the host - # fallback — symmetric with the ""/"unknown" filter on cli_version above. + # Gate on COLLECTED entries: an all-empty consensus must not stomp the host + # fallback. if plugin_versions: drifted = {name: sorted(versions) for name, versions in plugin_versions.items() if len(versions) > 1} if drifted: @@ -727,17 +683,14 @@ def build_run_summary( statuses = [r.result.final_status for r in task_results] version_info = get_version_info() - # The run-level cli/tool versions must describe what the tasks executed, - # not this process's host installs: under --driver docker each task runs - # in its own container, which auto-installs the latest alpha tool plugins - # at first use — the host's `uip` tree can differ arbitrarily (#366 - # recorded host values by mistake). Aggregate the per-task (in-container) - # captures; host values survive only as a fallback when no task reported. + # Must describe what the TASKS executed, not this process's host installs: + # under docker each task's container auto-installs the latest alpha plugins, + # so the host's tree can differ arbitrarily (#366 recorded host values by + # mistake). Host values survive only as a fallback. _override_uip_versions_from_tasks(version_info, task_results) host_coder_eval = version_info.get("coder_eval", "unknown") - # Surface host↔container version drift: under --driver docker the agent - # ran against the image's version, not the host's. Without this warning - # framework_version silently mis-attributes the runtime. + # Without this, framework_version silently mis-attributes the runtime: under + # docker the agent ran against the image's version, not the host's. container_versions = { (r.result.environment_info or {}).get("coder_eval") for r in task_results if r.result.environment_info } @@ -761,9 +714,8 @@ def build_run_summary( tasks_not_graded=sum(1 for s in statuses if s.category == "ungraded"), tasks_token_budget_exceeded=sum(1 for s in statuses if s == FinalStatus.TOKEN_BUDGET_EXCEEDED), tasks_cost_budget_exceeded=sum(1 for s in statuses if s == FinalStatus.COST_BUDGET_EXCEEDED), - # Verdict evidence for pass_rate / error_share. A TIMEOUT lands in the - # `failed` bucket without any criterion having run, so the buckets alone - # cannot answer "was this run measured at all". + # Verdict evidence: a TIMEOUT lands in `failed` with no criterion having + # run, so the buckets alone cannot answer this. tasks_measured=sum(1 for r in task_results if r.result.weighted_score is not None), skipped_tasks=skipped_tasks or [], max_parallel=max_parallel, diff --git a/src/coder_eval/orchestration/config.py b/src/coder_eval/orchestration/config.py index 0a287fbd6..79252a188 100644 --- a/src/coder_eval/orchestration/config.py +++ b/src/coder_eval/orchestration/config.py @@ -55,14 +55,12 @@ class BatchRunConfig(BaseModel): ), ) - # Agent type override stays a dedicated field: it requires re-parsing the - # discriminated union (not a simple field-merge), so it is injected into the - # generic agent patch by apply_overrides rather than living in `overrides`. + # A dedicated field because it requires re-parsing the discriminated union, + # not a simple field-merge; apply_overrides injects it into the agent patch. agent_type: str | None = Field(default=None, description="Override agent type for all tasks (e.g., 'claude-code')") - # Generic layer-5 task-config overrides. Built from -D/--set and the surviving - # flag aliases (--model, --driver) in run_command, then applied to the resolved - # TaskDefinition by orchestration.overrides. + # Layer-5 overrides, built from -D/--set plus the surviving flag aliases. + # Rationale: .claude/notes/orchestration.md § Config merging and CLI overrides overrides: dict[str, Any] = Field( default_factory=dict, description=( @@ -94,12 +92,10 @@ class BatchRunConfig(BaseModel): description="CLI override for replicates per (task, variant). None = defer to experiment layers.", ) - # Grading switch: `coder-eval run` (True) vs `coder-eval execute` (False). - # It lives HERE and nowhere else on purpose — it is deliberately NOT part of - # the 5-layer task merge, so there is no `-D grade=...` path and no - # MergeField (CE014 does not apply to a scalar bool outside the merged - # roots). A task YAML must never be able to declare itself ungraded; only - # the invoking command decides. + # HERE and nowhere else on purpose: deliberately NOT part of the 5-layer task + # merge, so there is no `-D grade=...` path. A task YAML must never be able to + # declare itself ungraded; only the invoking command decides. + # Rationale: .claude/notes/orchestration.md § Execute vs. run: the grading switch grade: bool = Field( default=True, description=( @@ -111,14 +107,10 @@ class BatchRunConfig(BaseModel): # Logging verbose: bool = Field(default=False, description="Enable verbose (DEBUG level) logging for Docker output") - # Docker WORKDIR alignment for the non-docker-driver dispatch path (host - # process, or already inside a container someone else built — e.g. a Harbor - # trial container running `coder-eval execute` as its agent). Mirrors what - # `DockerRunner`/`_run-task-internal` already do for `sandbox.driver: docker` - # (see `Orchestrator.workspace_dir`'s docstring); this is the same mechanism, - # exposed publicly for the case where coder-eval's OWN docker driver isn't - # the one building the container. Only meaningful for a single resolved task - # — `run_batch` raises if more than one task would collide on it. + # Docker WORKDIR alignment for the NON-docker-driver dispatch path — a host + # process, or a container someone else built. The same mechanism the docker + # driver already uses, exposed for when coder-eval's own driver is not the one + # building the container. Only meaningful for a single resolved task. workspace_dir: Path | None = Field( default=None, description=( diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index 68fa2f216..6666a8af2 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -1,102 +1,23 @@ """Early-stop-on-criterion: resolution-time validation + runtime watcher. -Opt-in mechanism that ends a single-shot run early once its *armed* criteria -decide the outcome mid-run. Arming lives ENTIRELY on the criterion — there is -no run-level master switch. A ``stop_early:`` block on a live-observable -criterion (``LiveSuccessCriterion`` only, so arming an unobservable criterion -is unrepresentable) alone activates the run's watcher; -``run_limits.stop_early: false`` is the run-level KILL SWITCH that force- -disarms every block (the one-line experiment-variant override for an -authoritative, non-truncated run), and ``run_limits.stop_early: true`` — the -removed master arm — is rejected at resolution: - -* ``stop_early: {}`` — the block's PRESENCE is the arming; it carries one - implicit trigger: a native live FAIL may end the run (fail-stop). -* ``stop_early: {on_pass: stop}`` — a live PASS may also end the run - (pass-stop); the default ``on_pass: continue`` just latches the verdict. +Opt-in. Arming lives ENTIRELY on the criterion — there is no run-level master +switch. A ``stop_early:`` block on a ``LiveSuccessCriterion`` (so arming an +unobservable criterion is unrepresentable) alone activates the run's watcher; +``run_limits.stop_early: false`` is the run-level KILL SWITCH, and +``stop_early: true`` — the removed master arm — is rejected at resolution. + +* ``stop_early: {}`` — the block's PRESENCE is the arming, carrying one implicit + trigger: a native live FAIL may end the run. +* ``stop_early: {on_pass: stop}`` — a live PASS may also end it. * ``stop_early: {decide_within: N}`` — still undecided after N tool-call steps - latches an *effective* FAIL, fed through the same fail-stop rule (reported - as ``decision_budget_exceeded``). - -A trigger whose polarity the instance can never decide (see -``live_decidable_polarities``) is INERT BY DESIGN, not an error — one -dataset-fanned YAML line (same block on every row) serves both positive rows -(pass/timeout live, fail inert) and distractor rows (fail live, pass/timeout -inert) without per-row conditionals. - -This module owns the whole feature: - -* ``validate_early_stop`` — resolution-time guardrails. Rejects every - configuration v1 cannot honor as a hard error, so an unsupported arming is - never a silent no-op. -* ``EarlyStopWatcher`` — the runtime observer. A ``StreamCallback`` composed - into the agent's event stream that maintains its own ``EventCollector``, - evaluates the armed criteria's ``live_verdict`` on each tool *call* (and on - its result), applies the stop rule, and exposes ``should_stop()`` (the - cooperative interrupt the agent polls) plus ``info`` (the ``EarlyStopInfo`` - the orchestrator records). Fail-open: a raising ``live_verdict`` disarms the - watcher and degrades to a full run — a verdict bug can never cause a *false* - early stop. - -Verdicts LATCH: once an armed criterion decides (pass or fail) on a resolved -round, its ``live_verdict`` is never polled again for the rest of the run — -the checkers' documented monotonicity makes re-polling pure waste. Latching -happens only on resolved rounds (``ToolEndEvent``); an in-flight round's fresh -verdict can fire a stop but is not persisted, so a dispatched call that never -resolves (e.g. a crashed attempt) cannot leave a stale verdict behind across -retries. - -Deciding on the tool *call* (``ToolStartEvent``), not the result, is what makes -the stop robust: for an observable criterion the verdict is fully determined by -the call's inputs (which skill / which command), so the watcher can latch the -instant the call is dispatched — before a cut-short turn (e.g. a timeout) can -strip the result and leave the call unresolved. The agent polls ``should_stop`` -immediately after dispatching each message, so a stop on the call breaks the -loop before the result message is ever pulled. - -Live verdicts only *trigger* the stop; the authoritative scores always come -from the standard ``check_all_async`` on the frozen trajectory after the cut. - -Weighting: both the stop rule and the post-hoc gate -(``EvaluationResult.armed_criteria_passed``) consult ``run_limits. -stop_early_gate_threshold`` (default ``1.0``) rather than treating every armed -criterion's pass/fail as equally decisive. A fail-stop fires once the armed -set's CEILING (best case: every still-``undecided``/``pass`` criterion scores -1.0, every effectively-failed one scores 0) can no longer reach the threshold — -i.e. the gate is mathematically guaranteed to fail regardless of how the -trajectory continues. A ``decide_within`` timeout participates as an -ordinary weighted fail: a low-weight criterion's timeout that cannot doom the -gate does not stop the run (it is absorbed, exactly like a low-weight native -fail). A pass-stop fires once the ``on_pass: stop`` subset's FLOOR (worst -case: every still-undecided one scores 0) already meets the threshold. At the -default threshold of 1.0 both bounds collapse exactly to "any single armed -criterion's effective fail stops the run" / "every on_pass=stop criterion has -live-passed". Lowering the threshold lets a low-weight armed criterion's -failure be absorbed without truncating the run. - -Precision trade-off: a pass-stop cuts the run the instant the on_pass=stop -floor locks in, so a fail-armed criterion (e.g. a distractor) that would -only misfire on a LATER tool call is never observed — the frozen trajectory -then scores that row as a clean pass. This is an intentional -precision-for-budget trade of the opt-in "smoke" flavor; the authoritative -precision/recall must come from a non-early-stop (``stop_early: false``) run. - -Recall, by contrast, is never truncated — BOTH stops defer on it. The -fail-stop is DEFERRED while any pass-capable armed criterion is still -undecided (and within its budget), so a distractor misfire on an early tool -call cannot cut a positive row before its expected signal has had the chance -to appear (which would freeze a would-be TP as an FN and deflate recall/F1). -Symmetrically, the pass-stop is DEFERRED while any pass-capable armed -criterion OUTSIDE the on_pass=stop subset is still undecided (members of the -subset are already accounted for by the floor bound itself) — otherwise an -on_pass=stop criterion passing early would freeze a sibling ``on_pass: -continue`` criterion (e.g. one armed via ``decide_within``) as an unearned -fail on the truncated trajectory. Neither deferral loses the trigger — -verdicts latch monotonically, so the held stop fires the moment every -pass-capable armed criterion decides (fail-stop is evaluated before pass-stop -each round), and if none ever decides the run simply continues to the cap. A -row with zero pass-capable armed criteria (e.g. a negative row stacking only -distractors) has nothing to defer for and fail-stops on the first misfire. + latches an effective FAIL, fed through the same rule. + +A trigger whose polarity an instance can never decide is INERT BY DESIGN. + +This module owns the whole feature: ``validate_early_stop`` (resolution-time +guardrails) and ``EarlyStopWatcher`` (the runtime ``StreamCallback``). + +Rationale: .claude/notes/orchestration.md § Early stop on criterion """ from __future__ import annotations @@ -128,10 +49,8 @@ from coder_eval.criteria.base import BaseCriterion, LiveVerdict from coder_eval.models import CommandTelemetry, TaskDefinition - # Armed pair the watcher holds: (criterion model, its checker). Lives in the - # TYPE_CHECKING block (only annotations reference it, and those are lazy under - # `from __future__ import annotations`), so the names are real references - # rather than quoted strings static analyzers cannot resolve. + # In the TYPE_CHECKING block (only lazy annotations reference it), so the + # names are real references rather than strings analyzers cannot resolve. _ArmedPair = tuple[LiveSuccessCriterion, BaseCriterion[Any]] @@ -165,29 +84,22 @@ def early_stop_active(task: TaskDefinition) -> bool: def validate_early_stop(task: TaskDefinition) -> None: """Validate an armed early-stop task at resolution time; no-op when unarmed. - Called after the config layers have merged (``resolve_all_tasks`` post-CLI - overrides, the ``plan`` per-variant loop, and defensively in - ``Orchestrator._setup``). ``run_limits.stop_early: true`` (the removed - master arm) is always rejected; everything else is skipped unless the task - is actually armed (``early_stop_active``), so default runs — and runs - force-disarmed via the ``stop_early: false`` kill switch — are entirely - unaffected. - - Raise order (matters for which error a multiply-invalid task reports first): - 1. ``run_limits.stop_early: true`` -> error (master arm removed) - 2. armed together with ``simulation.enabled`` -> error - 3. agent does not declare ``supports_cooperative_stop`` -> error - 4. degenerate ``stop_early_gate_threshold`` (``<= 0.0``) -> error - - There are deliberately NO per-instance polarity guards: a trigger whose - polarity this instance cannot decide is inert by design (documented on each - trigger field), which is what lets one dataset-fanned YAML line serve both - positive and distractor rows. Arming an unobservable criterion type is - structurally impossible — the ``stop_early`` block exists only on - ``LiveSuccessCriterion``, so a ``file_exists`` criterion carrying one is a - pydantic ``extra='forbid'`` error at load time, not a case this validator - needs to catch. And an armed-but-empty set needs no guard either: with no - blocks present there is simply no watcher, byte-for-byte default behavior. + Called after the config layers have merged, and defensively in + ``Orchestrator._setup``. ``run_limits.stop_early: true`` is always rejected; + everything else is skipped unless the task is actually armed, so default runs + — and runs force-disarmed by the kill switch — are entirely unaffected. + + RAISE ORDER matters for which error a multiply-invalid task reports first: + + 1. ``run_limits.stop_early: true`` (master arm removed) + 2. armed together with ``simulation.enabled`` + 3. agent does not declare ``supports_cooperative_stop`` + 4. degenerate ``stop_early_gate_threshold`` (``<= 0.0``) + + There are deliberately NO per-instance polarity guards, and no armed-but-empty + guard. + + Rationale: .claude/notes/orchestration.md § Inert triggers are by design, and the watcher fails open Raises: EarlyStopConfigError: on any unsupported armed configuration. @@ -273,22 +185,18 @@ def validate_early_stop(task: TaskDefinition) -> None: class EarlyStopWatcher: """Observes the agent event stream and trips the cooperative interrupt. - A ``StreamCallback`` composed into the agent's callback chain (alone when - ``--stream`` is off, else beside the ``TaskScopedCallback``). It maintains - its OWN ``EventCollector`` — independent of the one the agent builds its - returned ``TurnRecord`` from — so each ``live_verdict`` sees a fresh, - single-element partial-trajectory list. On every tool call it evaluates the - armed criteria still undecided and applies the stop rule; once a stop fires - (or the watcher disarms on a raising verdict) the decision is latched and - further events are ignored. - - The orchestrator polls ``should_stop`` (passed to ``agent.communicate``) and, - after the turn, reads ``info`` to populate ``EvaluationResult.early_stop``. - - Fail-open: a ``live_verdict`` that raises disarms the watcher, logs - loudly, and degrades the run to a full run (``info`` stays ``None``). Because - live verdicts are triggers — not truth — this can never produce a *false* - early stop; it only ever errs toward running more. + A ``StreamCallback`` composed into the agent's callback chain. It maintains its + OWN ``EventCollector``, so each ``live_verdict`` sees a fresh single-element + partial trajectory. On every tool call it evaluates the armed criteria still + undecided and applies the stop rule; once a stop fires the decision is LATCHED + and further events are ignored. + + The orchestrator polls ``should_stop`` and afterwards reads ``info``. + + FAIL-OPEN: a raising ``live_verdict`` disarms the watcher and degrades to a + full run, which can never produce a FALSE early stop. + + Rationale: .claude/notes/orchestration.md § Inert triggers are by design, and the watcher fails open """ def __init__( @@ -303,9 +211,8 @@ def __init__( self._armed = armed self._gate_threshold = gate_threshold self._armed_weight = sum(c.weight for c, _ in armed) - # Per-instance decidable polarities, aligned with ``_armed``. Static for - # the run. Each trigger below is resolved against this set — a trigger - # whose polarity the instance cannot decide is inert by design. + # Per-instance decidable polarities, aligned with `_armed`, static for the + # run. A trigger whose polarity the instance cannot decide is inert. self._decidable: list[frozenset[LivePolarity]] = [ criterion.live_decidable_polarities() for criterion, _checker in armed ] @@ -321,38 +228,32 @@ def __init__( self._pass_trigger: list[bool] = [ block.on_pass == "stop" and "pass" in pol for block, pol in zip(blocks, self._decidable, strict=True) ] - # The fail trigger is IMPLICIT in arming: an armed criterion's native - # live-fail may always stop the run (ceiling-gated) — that is what the - # block's presence means. Inert when the instance can't live-fail. + # IMPLICIT in arming: an armed criterion's native live-fail may always + # stop the run (ceiling-gated). Inert when it cannot live-fail. self._fail_trigger: list[bool] = ["fail" in pol for pol in self._decidable] - # A timeout only means anything for an instance actively waiting to - # observe a pass — a fail-only instance's 'undecided' is its success - # state (the forbidden event hasn't happened), so its budget is inert. + # A timeout only means anything for an instance waiting to observe a + # PASS: a fail-only instance's 'undecided' IS its success state. self._budget: list[int | None] = [ block.decide_within if "pass" in pol else None for block, pol in zip(blocks, self._decidable, strict=True) ] if not any(self._pass_trigger) and not any(self._fail_trigger) and all(b is None for b in self._budget): - # Legal (a fanned row whose armed lines are all inert for this row's - # role) but user-visible: on a non-fanned task this is dead config — - # the row can never stop early and will simply run to the cap, - # gating on the armed subset if the watcher somehow fires. + # Legal for a fanned row whose armed lines are all inert for its role, + # but on a non-fanned task this is dead config. logger.warning("[%s] all armed stop triggers are inert for this row; run cannot stop early", task_id) self._max_turns = max_turns self._collector = EventCollector() self._sdk_turn_index = 0 self._tool_call_index = 0 self._started_monotonic: float | None = None - # Latched verdicts, aligned with ``_armed``. Once an entry leaves - # "undecided" (on a RESOLVED round) its checker is never polled again — - # the checkers' documented monotonicity makes re-polling pure waste. - # ``_budget_expired`` marks a latched fail as timeout-driven (reported - # as DECISION_BUDGET_EXCEEDED instead of CRITERION_FAILED). + # Once an entry leaves "undecided" on a RESOLVED round its checker is + # never polled again. `_budget_expired` marks a latched fail as + # timeout-driven, reported as DECISION_BUDGET_EXCEEDED. + # Rationale: .claude/notes/orchestration.md § Verdicts latch, and the decision happens on the CALL self._latched: list[LiveVerdict] = ["undecided"] * len(armed) self._budget_expired: list[bool] = [False] * len(armed) - # Previous round's verdicts, for the "which criterion flipped to pass - # this round" attribution on pass-stop. Reassigned ONLY at the end of a - # non-firing evaluation, so it always holds the PREVIOUS round when a - # stop fires. Starts all-"undecided". + # For the "which criterion flipped to pass" attribution. Reassigned ONLY + # at the end of a non-firing evaluation, so it always holds the PREVIOUS + # round when a stop fires. self._prev_verdicts: list[LiveVerdict] = ["undecided"] * len(armed) self._info: EarlyStopInfo | None = None self._disarmed = False @@ -409,39 +310,20 @@ def on_event(self, event: StreamEvent) -> None: def _on_event_impl(self, event: StreamEvent) -> None: """Forward the event to the internal collector; evaluate on each tool call. - Counts - ``TurnStartEvent`` for ``sdk_turn_index`` and each dispatched tool call - for the 1-based ``tool_call_index``, and stamps the wall-clock origin at - the FIRST ``AgentStartEvent`` only (a retry's second AgentStart does not - reset it). - - The decision is evaluated on the tool *call* (``ToolStartEvent``): for an - observable criterion the verdict is fully determined by the call's inputs, - so latching here lets the agent's post-dispatch ``should_stop`` poll break - the loop before a cut-short turn can strip the result. The call is not in - the collector yet (it reduces commands from ``ToolEndEvent``), so it is - passed to ``_evaluate_impl`` as the in-flight command, reported at - ``tool_call_index + 1`` (it has no ``ToolEndEvent`` to count yet). The - matching ``ToolEndEvent`` still evaluates, which covers a verdict that only - becomes decidable once the result is known and is a no-op once a call has - already latched the stop. ``tool_call_index`` is incremented on the - resolved ``ToolEndEvent`` so it stays a count of completed tool calls. - - UNRESOLVED tool ends are RECORDED but never counted or evaluated on. - ``_ClaudeTurnState.finalize`` force-closes orphaned tools as UNRESOLVED - *after* the message loop has ended and the terminal status is already - chosen (COMPLETED / TIMEOUT / crash) — those are not live tool activity - and must not trip the stop rule, or a run that ran to completion (or timed - out / crashed) without a real, in-loop decision would latch a false early - stop. A legitimate stop always fires on the in-loop call, so skipping the - evaluation can never suppress a real stop. They DO land in the collector: - the agent's own ``EventCollector`` records force-closed commands into the - ``TurnRecord`` that ``check_all_async`` later scores (including a crashed - attempt's drained partial turn), so the watcher must reduce the same - trajectory — otherwise its verdicts (and the fail-stop's ceiling bound) - would be computed over a strictly smaller command set than the - authoritative check, and a ``decide_within`` timeout could latch an - effective fail on a criterion the frozen trajectory scores as a pass. + Counts ``TurnStartEvent`` for ``sdk_turn_index`` and each dispatched call + for the 1-based ``tool_call_index``, stamping the wall-clock origin at the + FIRST ``AgentStartEvent`` only, so a retry does not reset it. + + The decision is evaluated on the tool CALL. It is not in the collector yet + (which reduces commands from ``ToolEndEvent``), so it is passed in as the + in-flight command; ``tool_call_index`` increments on the resolved end, so + it stays a count of COMPLETED calls. + + UNRESOLVED tool ends are RECORDED but never counted or evaluated on — they + must still land in the collector, or the watcher would reduce a strictly + smaller command set than the authoritative check. + + Rationale: .claude/notes/orchestration.md § Verdicts latch, and the decision happens on the CALL """ if isinstance(event, AgentStartEvent): if self._started_monotonic is None: @@ -455,9 +337,8 @@ def _on_event_impl(self, event: StreamEvent) -> None: return elif isinstance(event, ToolEndEvent): if event.status == ToolEndStatus.UNRESOLVED: - # Trajectory parity with the agent's collector (see docstring): - # record, but don't count a round or evaluate — a force-closed - # orphan is not live tool activity and must not fire a stop. + # Trajectory parity with the agent's collector: record, but do + # not count a round or evaluate. self._collector.on_event(event) return self._tool_call_index += 1 @@ -531,10 +412,8 @@ def _collect_verdicts(self, in_flight: CommandTelemetry | None, tool_call_index: """ record = self._collector.build_turn_record() if in_flight is not None: - # The in-flight call has no ToolEnd yet, so the collector (which - # reduces commands from ToolEnd) has not captured it. Append it so its - # engagement is visible to the verdict; re-sort by sequence to keep the - # partial trajectory in emission order. + # No ToolEnd yet, so the collector has not captured it. Append and + # re-sort by sequence to keep the partial trajectory in order. record.commands = sorted([*record.commands, in_flight], key=lambda c: c.sequence_number) records = [record] verdicts: list[LiveVerdict] = [] @@ -545,10 +424,8 @@ def _collect_verdicts(self, in_flight: CommandTelemetry | None, tool_call_index: try: verdict: LiveVerdict = checker.live_verdict(criterion, records) except Exception: - # Re-raise to the wrapping try/except in on_event, which sets - # _disarmed — but log the specific criterion here first, since - # that context (which criterion's live_verdict raised) would - # otherwise be lost once the exception is caught generically. + # Log WHICH criterion raised before re-raising to on_event's + # generic handler, where that context would be lost. logger.error( "[%s] early-stop live_verdict raised for criterion %r", self._task_id, @@ -594,32 +471,17 @@ def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: tool_call_index = self._tool_call_index + (1 if in_flight is not None else 0) verdicts = self._collect_verdicts(in_flight, tool_call_index) - # Recall deferral: while any pass-capable armed criterion is still - # undecided (within its budget — an expired budget is already an - # effective fail), a fail-stop is HELD. Cutting a positive row on a - # distractor misfire before its expected signal could appear would - # freeze a would-be TP as an FN (truncating the suite's recall); the - # misfire latches via the criterion's own monotone semantics, so the - # deferred fail still fires the moment every pass-capable criterion - # decides, and a row with zero pass-capable criteria (a negative row) - # defers nothing. + # RECALL DEFERRAL: a fail-stop is HELD while any pass-capable armed + # criterion is still undecided and within budget. A row with zero + # pass-capable criteria defers nothing. + # Rationale: .claude/notes/orchestration.md § Precision is traded, recall is not pass_capable_undecided = any( v == "undecided" and "pass" in pol for v, pol in zip(verdicts, self._decidable, strict=True) ) - # Fail-stop: a criterion whose effective verdict is "fail" is a - # CANDIDATE — the fail trigger is implicit in arming (native fail on a - # fail-capable instance, or a decide_within timeout). The stop only fires once the ceiling bound (best case: - # every still-undecided or already-passed armed criterion ends up - # scoring 1.0, every failed one scores 0) can no longer reach - # ``gate_threshold``, i.e. the armed gate (``EvaluationResult. - # armed_criteria_passed``) is GUARANTEED to fail no matter what happens - # on the rest of the trajectory. At the default ``gate_threshold=1.0`` - # this is equivalent to firing on the first candidate (any armed - # criterion's weight is > 0 by construction, so a single fail already - # drops the ceiling below 1.0) — below 1.0 a low-weight candidate's - # failure (or timeout) may not be enough to doom the gate, so the run - # keeps going: the failure is absorbed. + # A criterion whose effective verdict is "fail" is a CANDIDATE; the stop + # fires only once the ceiling bound can no longer reach `gate_threshold`. + # Rationale: .claude/notes/orchestration.md § The ceiling and floor bounds if not pass_capable_undecided: # Deterministic precedence: a native live-fail candidate always wins # over a budget-driven one, so the persisted/telemetry reason cannot diff --git a/src/coder_eval/orchestration/evaluation.py b/src/coder_eval/orchestration/evaluation.py index eca96a85b..91dfe6fc4 100644 --- a/src/coder_eval/orchestration/evaluation.py +++ b/src/coder_eval/orchestration/evaluation.py @@ -64,30 +64,24 @@ def resolve_reference_dir(task: TaskDefinition, task_file: Path | None) -> Path if not task.reference: return None - # Under driver: docker the host bind-mounts the reference at a fixed container - # path and layers an empty tmpfs over its original location inside the - # task-dir mount, so the agent cannot reach it via $TASK_DIR. Resolving - # relative to task_file would therefore find that empty mask, not the - # solution — so the container mount wins whenever it is present. - # - # Gated on the env var AS WELL AS the path, and for the same reason - # Sandbox.enforces_permission_windows is: a bare `/work/references` probe - # silently hijacks every task's reference on any host that happens to have - # that directory (a Linux box using /work as a workspace root is entirely - # plausible, and this package is going open-source). The failure would be - # invisible — wrong reference content, wrong reference_comparison scores, - # wrong judge prompts, no error. + # Under docker the host bind-mounts the reference at a fixed container path + # and masks its original location with an empty tmpfs, so resolving relative + # to task_file would find the MASK. Gated on the env var as well as the path, + # for the reason Sandbox.enforces_permission_windows is: a bare + # `/work/references` probe would silently hijack every task's reference on any + # host that happens to have that directory, invisibly. + # Rationale: .claude/notes/permissions.md § Reference solutions and the anti-cheat window container_mount = Path(CONTAINER_REFERENCE_DIR) if os.environ.get(IN_CONTAINER_ENV) == "1": if container_mount.is_dir(): logger.debug("Reference resolved from the container mount at %s", container_mount) return container_mount - # Hard fail rather than falling back to task_file.parent. In-container + # HARD FAIL rather than falling back to task_file.parent: in-container # that fallback resolves to the UN-masked reference under the `:ro` - # task-dir bind — which the mode-000 window then cannot chmod (EROFS), so - # the run would complete with the solution readable by the agent for the - # whole turn, reporting a normal pass/fail. A missing mount means the - # host-side wiring is broken; that must be loud, not silently unprotected. + # task-dir bind, which the mode-000 window cannot chmod (EROFS) — so the + # run would complete with the solution readable for the whole turn, + # reporting a normal pass/fail. A missing mount means the host-side wiring + # is broken, and that must be LOUD rather than silently unprotected. raise FileNotFoundError( f"Task declares reference.directory={task.reference.directory!r} but {CONTAINER_REFERENCE_DIR} " + "is not mounted in this container; refusing to run unprotected. Most likely that path does not " diff --git a/src/coder_eval/orchestration/experiment.py b/src/coder_eval/orchestration/experiment.py index 6fd5192c9..fb8a583f1 100644 --- a/src/coder_eval/orchestration/experiment.py +++ b/src/coder_eval/orchestration/experiment.py @@ -402,28 +402,22 @@ def resolve_task_for_variant( Returns: Tuple of (resolved TaskDefinition, config lineage dict, effective_repeats). """ - # Layer 1-4 raw agent dicts. Experiment-side dicts are dict[str, Any] (passed - # through verbatim); the task agent is dumped with exclude_unset so Pydantic - # defaults don't leak into the merge. Timing (max_turns/turn_timeout) belongs - # under run_limits — a legacy max_turns under agent: now fails loudly via the - # agent model's extra="forbid" rather than being silently hoisted. + # Experiment-side dicts pass through verbatim; the task agent is dumped with + # exclude_unset so Pydantic defaults don't leak into the merge. Timing belongs + # under run_limits — a legacy `max_turns` under `agent:` fails loudly. default_agent = default_experiment.defaults.agent if default_experiment.defaults else None exp_defaults_agent = experiment.defaults.agent if experiment.defaults else None variant_agent_clean = variant.agent task_agent = task.agent.model_dump(exclude_unset=True) if task.agent else None - # Resolve all three `-D`-reachable roots through the SAME generic resolver - # the CLI layer uses (config_merge.resolve_root) — one merge implementation, - # one set of per-field strategies, lineage emitted as a side effect into the - # shared `lineage` dict. Type is enforced after CLI overrides (layer 5) so - # `--type` can satisfy the contract for tasks that omit `agent.type`. + # All three `-D`-reachable roots through the SAME generic resolver the CLI + # layer uses, with lineage emitted as a side effect. Type is enforced AFTER + # CLI overrides, so `--type` can satisfy the contract for tasks that omit it. + # Rationale: .claude/notes/orchestration.md § Config merging and CLI overrides lineage: dict[str, ConfigLineageEntry] = {} # --- agent: layers default -> exp-defaults -> task -> variant --- - # No-op (agent: {type: none}) tasks need no special-casing here: `type` is a - # replace-scalar, so a task-level `type: none` wins over a baseline coding - # agent injected by the default experiment, and the merged config validates - # as NoneAgentConfig. The orchestrator then dispatches to NoOpAgent. + # Rationale: .claude/notes/orchestration.md § No-op tasks need no special case anywhere resolved_agent: AgentConfig | BaseAgentConfig | None agent_layers: list[Layer] = [] agent_specs: list[tuple[ConfigSource, dict[str, Any] | None]] = [ @@ -465,10 +459,9 @@ def _add_rl(rl: RunLimits | None, source: ConfigSource) -> None: # task's own (default) sandbox so resolved_task.sandbox is never None. resolved_sandbox = resolve_root("sandbox", sandbox_layers, lineage=lineage) or task.sandbox - # Resolve pre_run / post_run through the same engine via their declared append - # strategies: pre_run appends in layer order (exp-defaults setup first, then the - # task's); post_run uses append_order="reverse" (the task's commands first, then - # experiment-defaults cleanup-last). Only exp-defaults + task contribute. + # Through the same engine via their declared append strategies: pre_run + # appends in layer order, post_run uses append_order="reverse" so the task's + # commands run first and experiment-defaults cleanup last. prepost_layers: list[Layer] = [] exp_patch: dict[str, Any] = {} if experiment.defaults and experiment.defaults.pre_run: @@ -488,9 +481,8 @@ def _add_rl(rl: RunLimits | None, source: ConfigSource) -> None: resolved_pre_run = prepost.get("pre_run", []) resolved_post_run = prepost.get("post_run", []) - # Resolve simulation: shallow-merge across default → experiment-defaults → task → variant. - # Mirrors agent merge semantics — a later layer's keys overwrite earlier ones, and - # the final dict is validated by building a SimulationConfig from it. + # Shallow-merge across all four layers, mirroring agent merge semantics; the + # final dict is validated by building a SimulationConfig from it. resolved_simulation = _resolve_simulation(default_experiment, experiment, task, variant, lineage) resolved_checker_context = _resolve_checker_context(default_experiment, experiment, task, variant, lineage) @@ -511,9 +503,8 @@ def _add_rl(rl: RunLimits | None, source: ConfigSource) -> None: _config = config if config is not None else BatchRunConfig(run_dir=Path(".")) effective_repeats = _resolve_repeats(default_experiment, experiment, variant, _config, lineage) - # When no config was supplied (direct callers / tests), enforce the agent.type - # contract here — _apply_cli_overrides won't run to do it later. A no-op task's - # `type: none` satisfies it (type is set), so only a truly type-less agent trips. + # Direct callers / tests: enforce the agent.type contract here, since + # _apply_cli_overrides won't run to do it later. if config is None and resolved_agent is not None and resolved_agent.type is None: raise ValueError( "Agent 'type' is required but was not set by any layer (default experiment, " @@ -542,11 +533,7 @@ def _apply_cli_overrides( """ from .overrides import apply_overrides - # No-op (agent: {type: none}) tasks need no special-casing: the resolved agent - # is a NoneAgentConfig, so a suite-wide `--model` / `-D agent.*` lands on it - # harmlessly (NoOpAgent ignores them) and the `type: none` already satisfies the - # agent.type contract. An explicit `--type ` is highest precedence and, as for - # any task, replaces the type — turning the no-op task into that agent. + # Rationale: .claude/notes/orchestration.md § No-op tasks need no special case anywhere assert task.agent is not None, f"Task '{task.task_id}' has no agent config" apply_overrides(task, config.overrides, agent_type=config.agent_type, lineage=lineage) @@ -572,9 +559,8 @@ def resolve_task_files( """ exp_dir = experiment_file.parent if experiment_file is not None else task_file.parent - # Resolve system_prompt_file (may be injected by variant as relative or absolute path). - # Rebind: the resolver returns a new config so the prompt/file swap is atomic - # (see resolve_agent_system_prompt). + # REBIND: the resolver returns a new config, so the prompt/file swap is + # atomic. task.agent = resolve_agent_system_prompt(task.agent, exp_dir) # Resolve relative template_sources paths @@ -591,40 +577,25 @@ def resolve_all_tasks( ) -> tuple[list[ResolvedTask], list[SkippedTask]]: """Resolve all (task x variant) combinations into typed, run-ready entries. - Applies all 5 config layers in one place: - 1. default experiment base - 2. task YAML - 3. experiment base - 4. variant overrides - 5. CLI overrides - - Also handles tag filtering and unique task ID validation. - - Task YAMLs that fail to load (YAML parse error, Pydantic validation, - dataset expansion error) are recorded in the returned ``skipped`` list and - excluded from the resolved set rather than aborting the suite. - - Per-task config-resolution failures (a task whose own YAML is incompatible - with the resolved run — e.g. Claude-only ``sdk_options`` surviving a - ``--type codex`` override, which ``CodexAgentConfig`` forbids) are likewise - demoted to ``skipped`` — but only when other tasks resolve. If EVERY task - that reaches resolution fails, the cause is a global invocation error (a bad - ``--type`` / ``-D`` value, repeats over the cap) rather than a per-task - incompatibility, so it is re-raised and aborts the run. Early-stop arming - errors always propagate. The caller surfaces ``skipped`` in the run summary - so per-task failures are loud but recoverable. + Dataset fan-out runs BEFORE variant resolution, so a variant cannot override + the dataset. Per-task resolution failures are demoted to ``skipped`` — unless + EVERY task that reached resolution failed, which is re-raised so the CLI + aborts rather than producing an empty run. + + Rationale: .claude/notes/orchestration.md § Experiment resolution Args: - task_files: Paths to task YAML files. - experiment: The active experiment definition. - default_experiment: The default experiment (experiments/default.yaml). - config: Batch run configuration (provides CLI overrides, tags, run_dir). - experiment_file: Path to the experiment YAML file. Used to resolve - relative paths injected by experiment variants. Falls back to task - file directory when None. + task_files: Task YAML paths to resolve. + experiment: The experiment supplying variants and defaults. + default_experiment: The baseline experiment merged under it. + config: Run config, carrying the layer-5 ``-D`` overrides. + experiment_file: The experiment YAML's own path, used to resolve + experiment-relative paths. Falls back to each task file's directory + when None. Returns: - Tuple of (resolved tasks ready for run_batch, skipped task records). + ``(resolved, skipped)`` — every runnable (task x variant) entry, and the + tasks excluded by ``skip:`` or a per-task resolution failure. Raises: ValueError: If duplicate task IDs are found after resolution. @@ -633,12 +604,8 @@ def resolve_all_tasks( resolved: list[ResolvedTask] = [] skipped: list[SkippedTask] = [] - # Per-task config-resolution failures are collected here rather than raised - # inline. After the loop they are demoted to ``skipped`` — UNLESS every task - # that reached resolution failed, which signals a global invocation error - # (bad --type, an invalid -D value, repeats over the cap) that trips every - # task identically rather than a per-task incompatibility; that case is - # re-raised so the CLI aborts cleanly instead of producing an empty run. + # Collected rather than raised inline, and decided after the loop. + # Rationale: .claude/notes/orchestration.md § Experiment resolution resolution_errors: list[tuple[Path, Exception]] = [] attempted = 0 @@ -656,47 +623,32 @@ def resolve_all_tasks( for task_file in task_files: try: task, source_yaml = load_task(task_file) - # Honor `skip: true` before dataset expansion — quarantined tasks - # skip row fan-out, variant resolution, and any further I/O. The - # task is reported in RunSummary.skipped_tasks so the suite shows - # which YAMLs were intentionally excluded vs. failed to load. - # Bypassed by --include-skipped (config.include_skipped) so on-demand / - # local runs can execute quarantined or opt-in tasks; the nightly/CI - # leave the flag off and keep excluding them. + # BEFORE dataset expansion, so a quarantined task skips row fan-out, + # variant resolution and any further I/O. Still reported in + # RunSummary.skipped_tasks. Bypassed by --include-skipped. if task.skip and not config.include_skipped: reason = f"skip: true (task_id={task.task_id!r})" logger.info("Skipping task %s — skip: true in YAML", task.task_id) skipped.append(SkippedTask(path=str(task_file), reason=reason)) continue - # Dataset fan-out BEFORE variant resolution: one task per row, each - # treated as an independent task for the 4-layer merge below. This - # locks the invariant that variants cannot override the dataset. + # BEFORE variant resolution, which locks the invariant that a variant + # cannot override the dataset. expanded_tasks = expand_dataset( task, task_file.parent, max_rows=config.max_rows, sample_per_stratum=config.sample_per_stratum, ) - # Narrow set: real load failures only. We deliberately don't catch - # AttributeError / TypeError / ImportError — those signal a regression - # in load_task / expand_dataset and should crash loudly rather than - # silently demote every task to "skipped". Pydantic ValidationError - # is a ValueError subclass in v2, so it's covered. + # NARROW: real load failures only. AttributeError / TypeError / + # ImportError signal a regression and must crash loudly. except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) as exc: reason = f"{type(exc).__name__}: {exc}"[:500] logger.warning("Skipping task file %s — %s", task_file, reason) skipped.append(SkippedTask(path=str(task_file), reason=reason)) continue - # Isolate layer 1-5 config resolution per task file. A task whose own - # YAML config is incompatible with the resolved run — e.g. Claude-only - # agent fields (`sdk_options`) surviving a `--type codex` override, which - # `CodexAgentConfig` forbids (`extra="forbid"`) — raises here. Without - # isolation that single task aborts the entire coder-eval run. Buffer the - # file's resolved tasks and commit them only once the whole file - # resolves, so a mid-file failure discards this file's fan-out as a unit - # (mirroring the load/expand isolation above) rather than leaving a - # partial, lopsided fan-out behind. + # Isolated per task file, and the file's tasks are buffered so a mid-file + # failure discards its fan-out as a UNIT. attempted += 1 file_resolved: list[ResolvedTask] = [] try: @@ -716,9 +668,8 @@ def resolve_all_tasks( # Apply layer 5 (CLI overrides) _apply_cli_overrides(resolved_task, config, lineage) - # Early-stop guardrails: run once the task is fully resolved (all 5 - # layers merged, incl. the -D run_limits.stop_early kill switch). No-op unless armed; - # a bad arming raises EarlyStopConfigError (a ValueError). + # Once the task is fully resolved, so the -D kill switch is + # already merged. No-op unless armed. validate_early_stop(resolved_task) # Fan-out: simulation n_trials takes precedence over experiment repeats @@ -743,34 +694,20 @@ def resolve_all_tasks( config_lineage=dict(lineage), ) ) - # Early-stop arming errors are a deliberate hard stop: they always - # propagate (never demoted to skipped) so a misconfigured run fails loudly - # instead of quietly shrinking the suite. + # A deliberate hard stop: never demoted to skipped, so a misconfigured + # run fails loudly instead of quietly shrinking the suite. except EarlyStopConfigError: raise - # Narrow set, matching the load/expand block above: config-resolution - # and IO failures are collected (decided after the loop, below); - # AttributeError / TypeError / ImportError still crash loudly as - # regressions. Pydantic ValidationError is a ValueError subclass in v2, - # so it's covered. + # NARROW, matching the load/expand block above. except (FileNotFoundError, OSError, ValueError, yaml.YAMLError) as exc: resolution_errors.append((task_file, exc)) continue resolved.extend(file_resolved) - # Decide the fate of collected per-task resolution failures. If EVERY task - # that reached resolution failed, refusing to proceed (rather than producing - # an empty run) is the right call. We surface the first task's own error — - # not a synthesized "global misconfig" message — because we can't actually - # tell a genuine global cause (a bad --type / -D value that trips every task - # identically) from N tasks each independently incompatible for the same - # per-task reason. A ValueError (incl. Pydantic ValidationError and the "no - # agent registered" guard) is re-raised verbatim so its message stays clean; - # only a non-ValueError (FileNotFoundError/OSError/yaml.YAMLError — e.g. a - # missing system_prompt_file) is normalized to ValueError so it still lands - # in the caller's `except ValueError` (clean typer.BadParameter) instead of - # escaping as a raw traceback. + # If EVERY task that reached resolution failed, refuse rather than produce an + # empty run, surfacing the FIRST task's own error. + # Rationale: .claude/notes/orchestration.md § Experiment resolution if resolution_errors and len(resolution_errors) == attempted: first_exc = resolution_errors[0][1] if isinstance(first_exc, ValueError): @@ -790,9 +727,8 @@ def resolve_all_tasks( filtered_ids = {t.task_id for _, t in filtered} resolved = [rt for rt in resolved if rt.task.task_id in filtered_ids] - # Validate no duplicate (task_id, variant_id, replicate_index) combinations. - # Simulation replicates legitimately share (task_id, variant_id); the tuple - # is only a duplicate when the replicate_index also matches. + # Simulation replicates legitimately share (task_id, variant_id), so the + # replicate_index is part of the key. seen: dict[tuple[str, str, int], list[Path]] = {} for rt in resolved: key = (rt.task.task_id, rt.variant_id, rt.replicate_index) @@ -805,9 +741,8 @@ def resolve_all_tasks( ] raise ValueError("Duplicate task IDs found:\n" + "\n".join(lines)) - # Sort so tasks run interleaved: replicate 0 of every (task, variant) first, - # then replicate 1, etc. Within the same replicate, preserve original - # task-file and variant declaration order. + # INTERLEAVED: replicate 0 of every (task, variant) first. Declaration order + # is preserved within a replicate. task_order = {tf: i for i, tf in enumerate(dict.fromkeys(rt.task_file for rt in resolved))} variant_order = {v.variant_id: i for i, v in enumerate(experiment.variants)} resolved.sort(key=lambda rt: (rt.replicate_index, task_order[rt.task_file], variant_order[rt.variant_id])) @@ -832,15 +767,9 @@ def _pick_worst_status(statuses: list[FinalStatus]) -> FinalStatus: while its sibling is not, and the ordering above is what keeps that from absorbing an unmeasured replicate into a pass. """ - # Annotated with the SAME Literal `FinalStatus.category` returns, and indexed - # directly rather than via `.get(..., -1)`. Adding the fourth `ungraded` - # bucket here was a manual step no checker could verify — an untyped - # `dict[str, int]` proves neither that every category is present nor that no - # stray key is — while the `-1` default it leaned on was already unreachable - # (the `assert set(_STATUS_CATEGORIES) == set(FinalStatus)` in models/enums.py - # makes `category` total). Worse, that default was documented as - # "fail-closed", but -1 sorts BELOW error, so a fifth category would have - # silently outranked ERROR as the worst status. + # The SAME Literal `FinalStatus.category` returns, indexed directly rather + # than via `.get(..., -1)`. + # Rationale: .claude/notes/orchestration.md § Aggregation drops ungraded rows rather than zeroing them priority: dict[Literal["succeeded", "failed", "error", "ungraded"], int] = { "error": 0, "failed": 1, @@ -896,9 +825,8 @@ def _fold_replicates(task_id: str, variant_id: str, reps: list[TaskResult]) -> V Extracted from ``aggregate_results``, which was already the largest function in the module before the ungraded bucket added another filter to it. """ - # Ungraded replicates — and ONLY those — drop out entirely rather than - # contributing 0.0: `or 0.0` would average a clean `execute` run down to a - # real-looking zero, while dropping an errored one would pay it a bonus. + # Ungraded replicates — and ONLY those — drop out rather than contributing + # 0.0. scores = _measured_scores(reps) non_errored = [r for r in reps if r.result.final_status.category != "error"] durations = [r.result.duration_seconds for r in non_errored] @@ -962,15 +890,9 @@ def aggregate_results( # Build task summaries task_summaries: list[TaskExperimentSummary] = [] for task_id, variants in task_variants.items(): - # Only graded variants can win or set a spread. Including ungraded ones - # at 0.0 would name an arbitrary "best" among scores that do not exist. - # - # When NOTHING was scored there is no winner, and the fallback must not - # invent one: `variants[0]` is whichever arm the input happened to list - # first, so swapping the two inputs flipped the reported winner — with - # `is_tie=False` asserting it was a real result. Sort by variant_id (so - # the field is at least deterministic) and mark it a tie among all arms, - # which is what "no arm outscored another" actually means. + # Only GRADED variants can win or set a spread, and when nothing was + # scored the fallback must not invent a winner. + # Rationale: .claude/notes/orchestration.md § Aggregation drops ungraded rows rather than zeroing them scored = [(v, v.weighted_score) for v in variants if v.weighted_score is not None] if scored: best = max(scored, key=lambda pair: (pair[1], pair[0].variant_id))[0] @@ -1018,13 +940,10 @@ def aggregate_results( tasks_not_graded=sum(1 for v in vr_list if v.final_status.category == "ungraded"), tasks_token_budget_exceeded=sum(1 for v in vr_list if v.final_status == FinalStatus.TOKEN_BUDGET_EXCEEDED), tasks_cost_budget_exceeded=sum(1 for v in vr_list if v.final_status == FinalStatus.COST_BUDGET_EXCEEDED), - # Verdict evidence for pass_rate. A TIMEOUT lands in the `failed` - # bucket without any criterion having run, so the buckets alone - # cannot answer "was this variant measured at all". + # Verdict evidence: a TIMEOUT lands in `failed` with no criterion + # having run, so the buckets alone cannot answer this. tasks_measured=sum(1 for v in vr_list if v.weighted_score is not None), - # Mean over GRADED rows only, and None when there are none: a clean - # execute run has no average score, and reporting 0.000 next to - # "Pass Rate: n/a" is a number indistinguishable from "scored zero". + # GRADED rows only, and None when there are none. average_score=_mean_graded_score(vr_list), average_duration=sum(v.duration_seconds / v.replicate_count for v in vr_list) / len(vr_list), total_tokens=total_tokens, diff --git a/src/coder_eval/orchestration/regrade.py b/src/coder_eval/orchestration/regrade.py index ae4b19aa5..ee0a13232 100644 --- a/src/coder_eval/orchestration/regrade.py +++ b/src/coder_eval/orchestration/regrade.py @@ -79,31 +79,16 @@ def task_from_prior( """Rebuild the executed task from the run's own recorded config. Rebuilding from ``task_config.resolved`` rather than re-reading the YAML is - what makes the grade describe the run that happened: ``resolved`` is the - post-merge definition, so variant overrides, ``-D`` flags and dataset row - expansion are all already baked in. Re-loading the source YAML would silently - grade a DIFFERENT task whenever any of those were used. - - Falls back to the source YAML only when ``resolved`` will not validate (a - schema change since the run), and says so loudly — a quiet fallback would - reintroduce exactly the drift above. - - ``allow_recorded_commands`` gates the shell half. See - :func:`check_embedded_commands`. - - ``grade_in_place`` is the ONE lever that selects which capability families - the gate discloses, and it is deliberately one parameter rather than two. - It shipped beside an ``include_setup_phase`` that every caller passed as its - exact complement -- two names for one fact, with nothing rejecting the - incoherent pairings. That is not cosmetic here: the flag gates a SECURITY - disclosure, so a future caller that set one and forgot the other would drop - the container-dispatch half of the untrusted-config gate with nothing - failing. Both derived values are computed once, in - :func:`_gate_scope_for_grade`. - - ``allow_host_grading`` participates only through that derivation: with the - flag set, no container is dispatched, so naming one in the consent prompt - would ask the operator to approve something that never runs. + what makes the grade describe the run that happened: ``resolved`` is + post-merge, so variant overrides, ``-D`` flags and dataset expansion are + already baked in. Falls back to the source YAML only when ``resolved`` will not + validate, and says so LOUDLY — a quiet fallback reintroduces that drift. + + ``allow_recorded_commands`` gates the shell half (see + :func:`check_embedded_commands`); ``grade_in_place`` is the ONE lever selecting + which capability families the gate discloses. + + Rationale: .claude/notes/orchestration.md § What the gate covers, and why each part is in scope """ record = prior.task_config if record is None: @@ -126,9 +111,8 @@ def task_from_prior( task, run_dir, allow_recorded_commands=allow_recorded_commands, - # The recorded source path, so the prompt can name the task DIRECTORY - # the dispatch copies in. It is the same untrusted value - # `_grade_in_container` resolves the image from. + # The recorded source path, so the prompt can name the task DIRECTORY the + # dispatch copies in. The same untrusted value the image resolves from. task_file=Path(record.source_file) if record.source_file else None, **_gate_scope_for_grade(task, grade_in_place=grade_in_place, allow_host_grading=allow_host_grading), ) @@ -202,64 +186,20 @@ def embedded_commands( ) -> list[str]: """Every shell command a rebuilt task definition would run on this host. - ``include_setup_phase`` covers the two capability families that exist only on - the ``--copy`` path: ``pre_run``, and the sandbox's own provisioning. Both are - SKIPPED when grading in place (``Sandbox.adopt`` runs no installer, and - re-running ``pre_run`` would overwrite the agent's deliverables before the - criteria read them), so on that path they are not a capability the run dir - has. - - ``post_run`` is deliberately NOT behind that flag, and this is the one place - the distinction bites. It used to be, back when the hooks were skipped as a - pair — but ``post_run`` belongs to the GRADING phase (``execute`` defers it), - so it now runs on EVERY grading path, in place included. Leaving it inside - ``include_setup_phase`` would have made the in-place path — the DEFAULT for a - run directory — execute recorded shell with no consent prompt at all. - - It is filtered against ``_operator_baseline_post_run()`` for a reason worth - stating precisely: this gate asks the operator to approve shell **the record - chose**, and a single ``coder-eval run`` — the behaviour the split must - reproduce — runs ``post_run`` with no prompt at all, because the config came - from the operator. The grader's own default experiment appends the same - ``post_run`` to every task it ever runs, so finding one of those commands in - a record reveals no choice the record made and grants no capability the - operator's own config does not already exercise on every run. Prompting on it - would fire on 100% of run directories, and a refusal that always fires is - read as a formality and waved through — which is how the gate would stop - protecting the authored commands that DO represent a choice. - - Sandbox provisioning is the half this gate originally missed, and it was the - worst one. ``grading_sandbox_config`` carries the recorded ``sandbox`` block - through untouched, and the ``--copy`` branch then calls ``Sandbox.setup``, - which reaches ``uv pip install ``, ``npm install `` and ``git clone ``. A package name is arbitrary - code at install time. Because the scan walked only ``success_criteria``, a - shared run directory whose criteria were all ``file_exists`` sailed through - the gate and still ran installers of the attacker's choosing. - - ``include_container_dispatch`` is the same omission again, one layer up, and - it was reintroduced by the very change that made a docker row gradable. When - a ``driver: docker`` row is graded, the grade is DISPATCHED INTO A CONTAINER - built from the recorded ``sandbox.docker`` block -- so the record chooses the - image that runs on this host, with the default credential allowlist - (``ANTHROPIC_API_KEY``, ``UIPATH_ACCESS_TOKEN``, ``AWS_BEARER_TOKEN_BEDROCK`` - ...) forwarded into it, a writable copy of ``~/.claude``, and a pinned - ``--entrypoint`` the image itself supplies. That is arbitrary code execution - from a shareable artifact, and it is a strictly WIDER capability than the - ``run_command`` strings this gate already refuses. It reached the host - unprompted because the scan walked only ``success_criteria`` and ``post_run`` - -- the identical blind spot described in the paragraph above, which is the - argument for naming it here rather than trusting the next reader to notice. - - Like ``post_run``, it is a capability of the IN-PLACE path (the default for a - run directory), so it cannot hide behind ``include_setup_phase``. + ``include_setup_phase`` covers the two families that exist only on the + ``--copy`` path: ``pre_run`` and the sandbox's own provisioning. + + ``post_run`` and ``include_container_dispatch`` are deliberately NOT behind + that flag — both are capabilities of the IN-PLACE path, which is the DEFAULT + for a run directory. ``post_run`` is filtered against the operator's own + baseline, because a refusal that fires on every run directory is read as a + formality and waved through. ``isinstance`` narrowing, never ``getattr(c, "command", None)``: an untyped - string probe over a discriminated union is invisible to pyright, so renaming - a field silently degrades the only guard on this path to a permanent no-op — - the exact hazard ``models/tasks.py`` already documents in prose. It also - cannot reach ``agent_judge``, whose ``bash`` tooling is the widest blast - radius of the three. + probe over a discriminated union is invisible to pyright, so a renamed field + would silently degrade the only guard on this path to a no-op. + + Rationale: .claude/notes/orchestration.md § Embedded commands """ from coder_eval.models import ( AgentJudgeCriterion, @@ -274,20 +214,16 @@ def embedded_commands( if isinstance(c, RunCommandCriterion): commands.append(c.command) elif isinstance(c, AgentJudgeCriterion): - # No command string of its own: it spawns a Claude Code SDK agent - # with tool access (Bash included) under the grader's credentials, - # which is a strictly wider capability than one shell line. + # No command string of its own: it spawns a tool-using agent under + # the grader's credentials, which is WIDER than one shell line. commands.append(f"") elif isinstance(c, LLMJudgeCriterion): - # No shell, but it spends the grader's model budget and ships the - # graded artifacts (and optionally the trajectory) to a provider of - # the recorded config's choosing. That is a capability the operator - # should approve, even though nothing executes locally. + # No shell, but it spends the grader's budget and ships the graded + # artifacts to a provider the recorded config chose. commands.append(f"") elif isinstance(c, UiPathEvalCriterion): - # Builds and shells `uv run uipath eval …`. Every argument is - # shlex-quoted, so this is disclosure rather than injection — but it - # is still a subprocess the recorded config chose to start. + # Every argument is shlex-quoted, so this is disclosure rather than + # injection — but still a subprocess the record chose to start. commands.append(f"uv run uipath eval {c.agent_name} {c.eval_set}") # Unconditional: post_run runs on every path that grades (see the docstring). # Minus the operator's own universal baseline, which the record did not choose. @@ -311,42 +247,24 @@ def embedded_commands( def _container_dispatch_commands(task: TaskDefinition, task_file: Path | None) -> list[str]: """The container dispatch, rendered as the ONE shell command it is. - Every string this returns is a command, because that is what the caller - promises: :func:`check_embedded_commands` joins the list with ``"; "`` and - interpolates ``len(commands)`` into the consent prompt. The first version - appended argv FRAGMENTS as separate entries, so a `dockerfile_path` task with - two build args and one mount asked the operator to approve "4 shell - command(s)" reading ``docker build -f Dockerfile; --build-arg FOO=bar; - --build-arg BAZ=qux; -v /a:/b`` -- one docker invocation described as four - commands, three of which are not commands. The consent prompt is the one - place this text has to be exact. - - It also names every HOST PATH the dispatch exposes, not just the ones under - ``sandbox.docker``. Three families reach the record-named image without the - record ever mentioning them in a ``docker`` block: - - * the TASK DIRECTORY, copied wholesale from the recorded ``source_file``'s - parent (``DockerRunner._prepare_task_dir_mount``) -- a record whose - ``source_file`` is ``~/.ssh/config`` copies all of ``~/.ssh`` in; - * every ``agent.plugins[].path``, every ``TemplateDirSource.path`` and - ``agent.system_prompt_file``, auto-mounted read-only at their host paths by - ``_build_argv``. ``_sensitive_source_paths`` only *warns* about a fixed - list of these, and this module's own gate docstring states the governing - principle: a warning is not a control, because it prints as the command is - already being prepared; - * a WRITABLE copy of ``~/.claude``, ``.credentials.json`` included. - - Networking defaults to ``--network bridge``, so anything the container can - read it can also send. Disclosing only ``sandbox.docker.*`` would ask the - operator to consent to a strict subset of what actually happens. + Every string returned is a command, because that is what the caller promises: + :func:`check_embedded_commands` joins them with ``"; "`` and interpolates + ``len(commands)`` into the consent prompt. The consent prompt is the one place + this text has to be exact. + + It names every HOST PATH the dispatch exposes, not just the ones under + ``sandbox.docker`` — the task directory, every auto-mounted plugin and + template path, and a writable copy of ``~/.claude``. + + Rationale: .claude/notes/orchestration.md § Embedded commands """ docker = task.sandbox.docker parts: list[str] = [] if docker.dockerfile_path: - # `docker build` runs every RUN step in the recorded Dockerfile on this - # host, and expands recorded build args against the GRADER's environment, - # so a `${ANTHROPIC_API_KEY}` arg is exfiltratable by a RUN step. - # `extra_args` is spliced into the argv unfiltered. + # Runs every RUN step in the recorded Dockerfile on this host, and + # expands recorded build args against the GRADER's environment, so a + # `${ANTHROPIC_API_KEY}` arg is exfiltratable by a RUN step. `extra_args` + # is spliced into the argv unfiltered. parts.append(f"docker build -f {docker.dockerfile_path}") parts += [f"--build-arg {key}={value}" for key, value in docker.build.args.items()] parts += [f"--secret {spec}" for spec in docker.build.secrets] @@ -397,20 +315,17 @@ def check_embedded_commands( ) -> None: """Refuse — or at minimum name — the shell a rebuilt config will run here. - ``task_config.resolved`` is data that travels inside a run directory, and a - run directory is a shareable artifact — the detached-grading flow exists so - one machine can execute and another can grade. Rebuilding the task from it - means the *run dir* decides what ``run_command`` criteria the grader runs, - with the grader's environment (API keys, cloud credentials, SSH agent). + ``task_config.resolved`` travels inside a run directory, and a run directory is + a SHAREABLE ARTIFACT — the detached-grading flow exists so one machine can + execute and another can grade. Rebuilding from it means the RUN DIR decides + what runs on the grader's host, with the grader's environment. - A warning is not a control: it is printed as the command is already being - prepared, and nobody reads a log line fast enough to stop it. So a recorded - config that carries shell is REFUSED unless the operator opted in. The common - case — ``execute`` then ``evaluate`` on your own machine — is unaffected - whenever the criteria are file/JSON checks, and the opt-in is one flag. + A warning is not a control: it prints as the command is already being prepared. + So a recorded config carrying shell is REFUSED unless the operator opted in. + Passing the task file explicitly (``evaluate ``) bypasses + this — that config came from the operator, not from the artifact. - Passing the task file explicitly (``evaluate ``) also - bypasses this: that config came from the operator, not from the artifact. + Rationale: .claude/notes/orchestration.md § Embedded commands """ commands = embedded_commands( task, @@ -475,24 +390,17 @@ def _fall_back_to_source( def default_workspace(run_dir: Path, prior: EvaluationResult) -> Path: """Locate the workspace a finished run left behind. - ``sandbox_path`` is authoritative when it still exists — it is where the run - actually worked. Otherwise fall back to the preserved artifacts tree, where - preservation nests the workspace under the task id. + ``sandbox_path`` is authoritative when it still exists; otherwise the + preserved artifacts tree, where preservation nests the workspace under the + task id. - Raises rather than guessing when neither is conclusive. Guessing is worse - than failing here: grading the WRONG directory makes every path-relative - criterion fail as a locating artifact rather than as a verdict, and it + RAISES rather than guessing when neither is conclusive — grading the WRONG + directory makes every path-relative criterion fail as a locating artifact and reports that as an ordinary score. **Every** return goes through ``_contained``, checked against ``run_dir``. - The containment check originally covered one branch of four and rooted the - ``task_id`` case at ``artifacts/`` rather than at the run directory, which - made it vacuous the moment ``artifacts`` was ITSELF a symlink — and - ``artifacts/`` is attacker-supplied for a shared run dir just like - ``sandbox_path`` and ``task_id``. The escaped tree then became the grading - root via ``Sandbox.adopt``, `run_command` criteria ran with it as cwd, and - the resulting verdict — criterion detail text included — was written back - into the run's own ``task.json``. + + Rationale: .claude/notes/orchestration.md § Locating the workspace a finished run left behind """ def _contained(candidate: Path, description: str) -> Path: @@ -508,10 +416,8 @@ def _contained(candidate: Path, description: str) -> Path: if prior.sandbox_path: recorded = Path(prior.sandbox_path) if recorded.is_dir(): - # An absolute path out of the run's own task.json, which is - # untrusted input for a shared run dir. Criteria execute with cwd - # there and may mutate it, so an out-of-tree location has to be the - # operator's explicit choice. + # Untrusted input for a shared run dir, and criteria execute with cwd + # there and may mutate it. return _contained(recorded, f"The recorded sandbox_path ({recorded})") artifacts = run_dir / ARTIFACTS_DIRNAME @@ -525,13 +431,10 @@ def _contained(candidate: Path, description: str) -> Path: # every check rooted at `artifacts` tautological. _contained(artifacts, f"The artifacts directory ({artifacts})") - # The exact path, not a heuristic. `task_id` may contain "/" (dataset rows - # are "/"), so "the single child of artifacts/" resolves one - # level too high for every row task. - # - # `task_id` is an unvalidated string out of the run's own task.json, so - # `"../../../../home/victim"` joins to a real directory that `is_dir()` - # happily confirms. + # The EXACT path, not a heuristic: `task_id` may contain "/" (dataset rows are + # "/"), so "the single child of artifacts/" resolves one level too + # high for every row task. It is also an unvalidated string out of the run's + # own task.json. by_task_id = artifacts / prior.task_id if by_task_id.is_dir(): return _contained(by_task_id, f"The recorded task_id ({prior.task_id!r})") @@ -682,34 +585,19 @@ def grading_sandbox_config(task: TaskDefinition, *, allow_host_grading: bool = F """The sandbox config a grading pass runs under. This is the HOST-grading config, and reaching it with ``driver: docker`` means - the container route was declined. A docker row is normally graded IN a - container of its own image (:func:`_should_grade_in_container`), which is - dispatched before this function is called; what is left here is the ``--copy`` - path, which cannot adopt a container workspace, and the two-argument + the container route was declined: what is left is the ``--copy`` path, which + cannot adopt a container workspace, and the two-argument ``evaluate `` form. - (This docstring once opened "grading never runs a container: the docker - driver dispatches through DockerRunner, which needs an agent". That premise - was simply wrong — a grading pass needs no agent — and it is the reason the - refusal below survived for a release after it stopped being the only answer.) - - Grading a container task on the host is refused rather than downgraded. A container task's criteria - address container paths (``/verifier``, ``/logs/verifier``) and container - toolchains; run on the host they score 0.0 for a trajectory ``run`` scored - 1.0, and the row is written back FAILURE. The same commands (``rm -rf - /verifier``, ``mkdir -p /logs/verifier``) also execute unsandboxed on the - grading machine. A silent rewrite additionally neutralized the ``docker`` - refusal in ``Sandbox.adopt``, which exists to catch exactly this. - - ``allow_host_grading`` is the operator's explicit acceptance of both. Rows - graded that way are stamped ``graded_on_host`` in ``environment_info`` - (:func:`stamp_host_grading`) so they are never silently comparable with rows - a container graded. + Host-grading a container task is REFUSED, not downgraded. + ``allow_host_grading`` is the operator's explicit acceptance, and such rows are + stamped ``graded_on_host``. Re-validated rather than ``model_copy(update=...)``: ``update`` skips both pydantic validation and pyright, so a typo would produce a SandboxConfig - violating its own ``Literal`` and surface much later at an unrelated - ``if driver == "docker"`` branch. + violating its own ``Literal`` and surface much later. + + Rationale: .claude/notes/isolation.md § Grading a docker row inside a container """ if task.sandbox.driver != "docker": return task.sandbox.model_copy(deep=True) @@ -770,33 +658,22 @@ def _should_grade_in_container(task: TaskDefinition, *, allow_host_grading: bool def _fold_back_container_logs(container_run_dir: Path, run_dir: Path) -> None: """Rescue the grading container's logs from the scratch dir before it dies. - Called on BOTH the success and the failure path, and the failure path is the - one that makes it necessary. ``_grade_in_container`` runs the whole dispatch - inside a ``TemporaryDirectory``, and every ``DockerRunner`` diagnostic is - written into it: ``docker.log`` (the container's merged stdout+stderr, where - the in-container FATAL guards land, since ``_grade_in_container`` passes no - ``stream_callback`` and nothing is echoed), the captured build log a failed - ``docker build`` persists there, and the synthetic ``BUILD_FAILED`` / - ``ERROR`` records. Folding out only on success deleted precisely the evidence - -- and DockerRunError's own text says ``See {log_path} for container - output``, naming a path that no longer existed by the time it was printed. - - ``grade.log`` is the grading pass's OWN log, written by the in-container - orchestrator (``task_log_path(run_dir, regrade=True)``) and holding the - per-criterion detail -- ``run_command`` stdout/stderr, judge prompts and - verdicts -- which is the only durable record of WHY a criterion scored what - it did. It is a documented part of the run-directory contract, so a - ``driver: docker`` row must not be the one shape that silently lacks it. - - Best-effort throughout: a side-car log is not the verdict, and this runs - where an exception is already in flight. + Called on BOTH the success and the failure path, and the FAILURE path is what + makes it necessary: the scratch dir is deleted the moment the dispatch's + ``with`` exits, and everything explaining a failure lives in it. + + ``grade.log`` is the grading pass's OWN log, holding the per-criterion detail + that is the only durable record of WHY a criterion scored what it did, and a + documented part of the run-directory contract. + + Best-effort throughout: a side-car log is not the verdict, and this runs where + an exception is already in flight. + + Rationale: .claude/notes/isolation.md § Grading a docker row inside a container """ for name, dest_name in ((DOCKER_LOG_FILENAME, GRADE_DOCKER_LOG_FILENAME), (GRADE_LOG_FILENAME, GRADE_LOG_FILENAME)): - # `docker.log` is renamed for the PHASE: on the resume path that name is - # already taken by the executed container's log, and overwriting it would - # repeat the task.log/grade.log truncation bug one layer down. - # `grade.log` does not collide -- a detached grade is the only thing that - # ever writes it into that directory -- so it keeps its name. + # Renamed for the PHASE: on the resume path that name is already taken by + # the executed container's log. `grade.log` does not collide. source = container_run_dir / name if not source.is_file(): continue @@ -806,13 +683,10 @@ def _fold_back_container_logs(container_run_dir: Path, run_dir: Path) -> None: run_dir.mkdir(parents=True, exist_ok=True) dest = run_dir / dest_name if dest.is_symlink(): - # `shutil.copy2` opens the destination for writing, which FOLLOWS a - # symlink there -- an arbitrary-file-overwrite primitive in a run - # directory the grader did not create (`run --resume` passes the - # executed row's own directory; `--run-dir` can point anywhere). The - # sibling verdict write goes through `write_text_atomic` precisely - # for this, and the `suppress(OSError)` below would have made the - # redirect leave no trace at all. + # `shutil.copy2` opens the destination for writing and FOLLOWS a + # symlink there — an arbitrary-file-overwrite primitive in a run + # directory the grader did not create. The `suppress(OSError)` below + # would have made the redirect leave no trace. logger.warning("Refusing to write %s: it is a symlink.", dest) continue with contextlib.suppress(OSError): @@ -839,9 +713,8 @@ def _fold_back_container_grade(container_run_dir: Path, run_dir: Path, task_id: """ graded = container_run_dir / TASK_JSON_FILENAME if not graded.is_file(): - # `_parse_result_or_raise` already raised in this case; if we are here - # the runner returned a result, so the file exists. Guard anyway rather - # than raise a confusing FileNotFoundError from the copy. + # `_parse_result_or_raise` already raised if the runner returned no + # result, so guard rather than raise a confusing FileNotFoundError. return try: run_dir.mkdir(parents=True, exist_ok=True) @@ -859,24 +732,16 @@ def _stamp_container_grading(result: EvaluationResult, task: TaskDefinition) -> """Record the container grade's known equivalence gaps ON the row. The sibling of :func:`stamp_host_grading`, and written for the reason that - function's own docstring gives: a console warning does not travel with + function's docstring gives: a console warning does not travel with ``task.json`` into ``run.json``, the reports or the evalboard, so a row it describes cannot be filtered out of a comparison by anything downstream. - ``graded_without_pre_run`` is the count of ``pre_run`` commands that ran in - the container which executed the agent and were NOT re-run here. It is not a - hypothetical: three of the ten in-tree ``driver: docker`` tasks seed state - outside the workspace in ``pre_run`` (``3d-scan-calc`` symlinks - ``/root/mass_report.json``, and its verifier's first assertion is that the - path exists), so such a row scores 0.000 for a trajectory ``coder-eval run`` - scores 1.000. Refusing outright was considered and declined: it would make - ``run --resume`` fail on rows it grades correctly today whenever ``pre_run`` - happens to touch only the workspace, which is the common case. A durable, - machine-readable marker lets a consumer decide, which a refusal does not. - - ``graded_with_rebuilt_image`` marks the other gap: the grading pass re-ran - ``docker build`` under the run's deterministic tag, so the image is only as - stable as the Dockerfile and its base were between the two phases. + ``graded_without_pre_run`` counts ``pre_run`` commands that ran in the + container which executed the agent and were not re-run here; + ``graded_with_rebuilt_image`` marks that the pass re-ran ``docker build`` under + the run's deterministic tag. + + Rationale: .claude/notes/isolation.md § Two known equivalence gaps, stamped rather than refused """ if task.pre_run: result.environment_info["graded_without_pre_run"] = len(task.pre_run) @@ -921,36 +786,16 @@ async def _grade_in_container( ) -> EvaluationResult: """Grade ``workspace`` inside a container built from ``task``'s own image. - The container gets two separate mounts, and keeping them separate is the - point: ``run_dir`` (this GRADING pass's fresh directory) at the standard - output location, and ``workspace`` (the ORIGINAL run's output) at - ``CONTAINER_GRADE_WORKSPACE``. The grade writes its ``task.json`` into the - former, which the caller then folds back into the row — preserving - ``task.execute.json`` exactly as on the host path — while the latter is - adopted and never written over. - - ``task_file`` is required, and must EXIST here. The image is built or named - by the task's own sandbox config, and DockerRunner resolves the Dockerfile - and reference directory relative to the task file; without one there is - nothing to build from. - - Testing only for ``None`` was not enough, and failed on exactly the rows this - guard was written for. A ``driver: docker`` run records its ``source_file`` - from the IN-CONTAINER orchestrator, so the value is - ``/work/task_dir/task.yaml`` — a real, non-``None`` ``Path`` that does not - exist on the grading host. The guard was skipped, and the failure then went - QUIET where it matters: ``_prepare_task_dir_mount`` does ``if not - source.is_dir(): return``, so the grading container got no ``TASK_DIR`` mount - at all and any ``$TASK_DIR`` criterion silently resolved against a different - tree than during the run — a wrong verdict with no error anywhere. - - Requiring the file to exist also closes a second hole: on the detached path - ``task_file`` comes straight from the untrusted record, and its PARENT is - what ``_prepare_task_dir_mount`` copies into the container. A recorded - ``source_file`` of ``~/.ssh/config`` would copy the whole of ``~/.ssh``. - Existence alone does not make the path trusted — that is what the - ``--allow-recorded-commands`` gate is for, and it now names the container - dispatch — but it removes the silent-wrong-verdict half. + Two separate mounts, and keeping them separate is the point: ``run_dir`` (this + GRADING pass's fresh directory) at the standard output location, and + ``workspace`` (the ORIGINAL run's output) at ``CONTAINER_GRADE_WORKSPACE``. + The grade writes its ``task.json`` into the former, which the caller folds back + into the row; the latter is adopted and never written over. + + ``task_file`` is required and must EXIST here — testing only for ``None`` was + not enough, and failed on exactly the rows this guard was written for. + + Rationale: .claude/notes/orchestration.md § Why the container dispatch requires an EXISTING task file """ from coder_eval.isolation.docker_runner import DockerRunError, DockerRunner @@ -970,24 +815,11 @@ async def _grade_in_container( task.task_id, ) if task.pre_run: - # KNOWN EQUIVALENCE GAP, made loud because it cannot be closed here. - # - # This is a SECOND, fresh container. Only `workspace` crosses from the - # one that ran the agent; everything that container's `pre_run` did - # OUTSIDE the workspace is gone -- and `pre_run` is not re-run, because - # `Sandbox.adopt` sets `was_adopted` and the orchestrator skips it (it - # would otherwise overwrite the agent's deliverables before the criteria - # read them, which is the defect that skip exists for). - # - # It is not hypothetical: `tasks/samples/skillsbench/3d-scan-calc`'s - # pre_run does `ln -sfn "$PWD/mass_report.json" /root/mass_report.json` - # and its verifier's first assertion is that /root/mass_report.json - # exists. In a fresh container /root is pristine, so the row scores 0.000 - # for a trajectory `run` scores 1.000. - # - # Re-running pre_run here would trade this bug for the deliverable- - # clobbering one, so the honest move is to name it at dispatch and let - # the operator use --allow-host-grading or a single `run`. + # KNOWN EQUIVALENCE GAP, made loud because it cannot be closed here: this + # is a SECOND, fresh container, so everything the first one's `pre_run` did + # OUTSIDE the workspace is gone. Re-running pre_run here would trade this + # bug for the deliverable-clobbering one. + # Rationale: .claude/notes/isolation.md § Two known equivalence gaps, stamped rather than refused logger.warning( "Task %r declares %d pre_run command(s). They ran in the container that executed the " + "agent and are NOT re-run here: this is a second container, and only the workspace " @@ -998,22 +830,10 @@ async def _grade_in_container( len(task.pre_run), ) if task.sandbox.docker.dockerfile_path: - # The SECOND known equivalence gap, and the one the user guide already - # promised was warned about while nothing emitted it. - # - # `_build_image` re-runs `docker build` on every dispatch under the - # deterministic tag `coder-eval-task-:built`, so the grading image - # REPLACES the run's under the same name. A Dockerfile, build context or - # base image that moved between the two phases means the criteria read a - # different filesystem than the agent did, and the score changes for - # identical agent output. - # - # Nothing pins or records image identity on either side yet, so this - # cannot be detected after the fact -- which is exactly why it is said - # at dispatch. (A `reference_digest`-style pin on the resolved image is - # the real fix and is deliberately NOT attempted here; it needs the - # identity recorded on the RUN side too, which is a change to the run - # path rather than to grading.) + # The SECOND known gap: `_build_image` re-runs `docker build` under the + # deterministic tag, so the grading image REPLACES the run's under the same + # name. Nothing pins image identity on either side yet, which is exactly + # why it is said at dispatch. logger.warning( "Task %r builds its image from %s, so this grading pass re-runs `docker build`. If the " + "Dockerfile, its build context or its base image changed since the run, the criteria " @@ -1023,25 +843,10 @@ async def _grade_in_container( task.task_id, task.sandbox.docker.dockerfile_path, ) - # A SCRATCH output dir, never the caller's. The two callers disagree about - # what `run_dir` is -- `evaluate` passes a freshly prepared directory, while - # `run --resume` passes the executed row's OWN directory -- and every part of - # DockerRunner's result handling assumes an output dir it alone populates: - # - # * `_parse_result_or_raise` decides "did the container produce a result?" - # on `task_json.exists()` and discards `returncode`. Over the row's own - # directory the pre-grade `task.json` is already there, so a grading - # container that DIED (OOM, exit 137, or any of `_grade_recorded_run`'s - # own FATAL guards) was read back as a successful grade -- returning the - # stale ungraded row as the verdict, with the container's error discarded. - # * `run()` opens `run_dir/docker.log` with mode "w", truncating the - # executed container's log -- the same loss the task.log/grade.log split - # was introduced to prevent. - # * `grant_container_access(output_dir, writable=True)` would recursively - # widen the whole preserved artifacts tree. - # - # Giving the container a private directory makes both callers identical and - # makes the docstring above true, rather than true of one caller. + # A SCRATCH output dir, never the caller's: the two callers disagree about what + # `run_dir` is, and DockerRunner's result handling assumes an output dir it + # alone populates. + # Rationale: .claude/notes/isolation.md § Why the grading container gets a private scratch directory with tempfile.TemporaryDirectory(prefix="coder-eval-grade-") as scratch: container_run_dir = Path(scratch) rt = ResolvedTask( @@ -1055,20 +860,16 @@ async def _grade_in_container( try: result = await DockerRunner( rt, - # The grading pass owns its scratch dir and nothing else. The - # workspace is a bind mount of the ORIGINAL run's output and must - # survive untouched. + # The grading pass owns its scratch dir and nothing else: the + # workspace is a bind mount of the ORIGINAL run's output. preservation_mode=PreservationMode.NONE, prior_result=prior, grade_workspace=workspace, ).run() except (DockerRunError, OSError) as e: - # Wrapped, because `orchestration/` must not leak an isolation-layer - # exception to the CLI, and because the actionable next step is the - # host-grading escape hatch rather than a docker stack trace. OSError - # joins it because the staging copies (`_prepare_task_dir_mount`, - # `_prepare_reference_mount`) and the log open raise it unwrapped, - # and a raw traceback would drop the guidance below. + # `orchestration/` must not leak an isolation-layer exception to the + # CLI, and the actionable next step is the host-grading escape hatch. + # OSError joins it because the staging copies raise it unwrapped. raise RegradeError( f"Grading {task.task_id!r} in a container failed: {e}. The container's own output was " + f"kept at {run_dir / GRADE_DOCKER_LOG_FILENAME}. Re-run with --allow-host-grading " @@ -1076,13 +877,11 @@ async def _grade_in_container( + "score differently, and the row is stamped graded_on_host)." ) from e finally: - # ALWAYS, not only on success: the scratch dir is deleted the moment - # this `with` exits, and everything that explains a failure lives in - # it. See `_fold_back_container_logs`. + # ALWAYS, not only on success: the scratch dir dies with this `with`, + # and everything that explains a failure lives in it. _fold_back_container_logs(container_run_dir, run_dir) - # Fold the grade back into the row the caller asked about, mirroring what - # the host path does in place. `back_up_pre_grade_record` has already - # preserved task.execute.json, so this write is the graded record. + # Fold the grade back into the row the caller asked about. + # `back_up_pre_grade_record` already preserved task.execute.json. _fold_back_container_grade(container_run_dir, run_dir, task.task_id) # Outside the `with`: the scratch dir has served its purpose, and both of # these act on the returned result, which the caller writes back last. @@ -1107,54 +906,41 @@ async def regrade_in_place( ) -> EvaluationResult: """Run ``task``'s criteria against an already-executed ``workspace``. - The workspace is *adopted*, never copied: it is the run's own output, and the + The workspace is ADOPTED, never copied: it is the run's own output, and the template-copy path filters out ``node_modules`` / ``dist`` / ``build`` / ``.venv``, which would make a criterion reading those fail as a copying artifact rather than as a verdict. - ``prior`` supplies the trajectory and the run's execution facts (see - ``Orchestrator._seed_from_prior_result``), so criteria that read the agent's - tool calls score exactly as they would have during the run. - - ``recorded_task`` / ``recorded_task_file`` are the two halves of one seam: - what the row RECORDS, as distinct from what this process runs and resolves - paths against. Both matter only in the container, where the task is rewritten - to ``driver: tempdir`` and every path is a container path. Omitting the file - half made every container-graded row re-record ``/work/task_dir/task.yaml`` - as its ``source_file`` -- a path that exists on no host, which is the exact - defect ``Orchestrator.recorded_task_file`` was added to fix, reintroduced one - caller down. A later ``evaluate `` on such a row then hits - ``_grade_in_container``'s own "a task file that is not on this host" refusal. + ``prior`` supplies the trajectory and execution facts, so criteria that read + the agent's tool calls score as they would have during the run. + + ``recorded_task`` / ``recorded_task_file`` are two halves of one seam — what + the row RECORDS, as distinct from what this process runs. Both matter only in + the container, where the task is rewritten to ``driver: tempdir``. + + Rationale: .claude/notes/orchestration.md § Recording the task as authored """ from coder_eval.orchestrator import Orchestrator - # Every path through this function grades, so an empty `success_criteria` - # is never legal here the way it is for `execute` (which never calls this - # function at all). Checked once, at the single choke point every re-grade - # entry point (`evaluate `, `run --resume`'s `to_grade` set, and - # the container-dispatch branch below) shares -- a criteria-free task would - # otherwise finalize as `FinalStatus.SUCCESS` at `weighted_score: 0.0` - # (`all_criteria_passed([])` is vacuously `True`, - # `calculate_weighted_score([])` writes `0.0`), an internally contradictory - # "successful" result for what is actually a misconfigured task. + # Every path through this function grades, so an empty `success_criteria` is + # never legal here the way it is for `execute`. Checked at the single choke + # point every re-grade entry point shares — a criteria-free task would + # otherwise finalize SUCCESS at `weighted_score: 0.0`, since + # `all_criteria_passed([])` is vacuously True. if not task.success_criteria: raise RegradeError( f"task {task.task_id!r} has no `success_criteria` and cannot be graded (it would silently " + "score SUCCESS at weighted_score 0.0). Add at least one criterion before re-grading it." ) - # A `driver: docker` row is graded INSIDE a container of the same image, - # which is the only place its criteria mean what they meant during the run. - # Dispatched before anything else here, including the reference check, so the - # container performs every step against container paths rather than having - # half of it done against the host's. + # Graded INSIDE a container of the same image, dispatched before anything else + # here — the reference check included — so the container performs every step + # against container paths. + # Rationale: .claude/notes/isolation.md § Grading a docker row inside a container if _should_grade_in_container(task, allow_host_grading=allow_host_grading): - # `recorded_task` is NOT forwarded, and that is deliberate rather than an - # omission: the container re-derives it from the staged task.yaml (see - # `run_task_internal_command`'s `authored_task`), which IS this `task`. - # Accepting a DIFFERENT one and dropping it would leave no evidence, so - # say so instead -- the whole point of the seam is that the record must - # not quietly disagree with what was authored. + # NOT forwarded, deliberately: the container re-derives it from the + # staged task.yaml, which IS this `task`. Accepting a DIFFERENT one and + # dropping it would leave no evidence, so say so instead. if recorded_task is not None and recorded_task != task: raise RegradeError( "recorded_task cannot be honored when grading in a container: the container rebuilds " @@ -1162,10 +948,8 @@ async def regrade_in_place( + "--allow-host-grading." ) if recorded_task_file is not None and recorded_task_file != task_file: - # Same rule as `recorded_task` above. The container receives this - # value over `context.json`'s `host_task_file`, which DockerRunner - # fills from `rt.task_file` -- i.e. from `task_file` here. A - # different one could not be honored, so say so rather than drop it. + # Same rule: the container receives this over `context.json`, filled + # from `task_file` here, so a different one could not be honored. raise RegradeError( "recorded_task_file cannot be honored when grading in a container: the container " + "records the host task file the dispatch forwards to it, which is `task_file`. " diff --git a/src/coder_eval/orchestration/task_loader.py b/src/coder_eval/orchestration/task_loader.py index 9cf62c585..02e6b5d1b 100644 --- a/src/coder_eval/orchestration/task_loader.py +++ b/src/coder_eval/orchestration/task_loader.py @@ -72,22 +72,15 @@ def load_task(task_file: Path) -> tuple[TaskDefinition, str]: def resolve_template_source_paths(sources: list[TemplateSource], base_dir: Path) -> None: """Resolve TemplateDirSource paths to absolute, in place. - Expands $VAR / ${VAR} environment variables, then normalizes the path: - relative paths are resolved against ``base_dir``; absolute paths are - used as-is (but still go through ``Path(...)`` for string normalization). - - Undefined env variables raise ``ValueError`` — a template directory is a - load-bearing config field and an unresolved variable would otherwise - surface as a cryptic "Template directory not found" error at sandbox - setup, far from the actual configuration mistake. - - Scope: only environment variables (``$VAR`` / ``${VAR}``) are expanded - here. Dataset row substitution (``${row.field}`` in ``expand_dataset``) - runs over ``initial_prompt`` and ``success_criteria`` only — it does - NOT touch ``sandbox.template_sources``. The two regexes are disjoint - (env requires ``[A-Za-z_][A-Za-z0-9_]*``, row-var requires the dot) - but a ``${row.X}`` left inside a template path will not be substituted - and will fail at sandbox setup. + Expands ``$VAR`` / ``${VAR}``, then resolves a relative path against + ``base_dir``. An UNDEFINED env variable raises rather than deferring: a + template directory is load-bearing config, and an unresolved variable would + otherwise surface as a cryptic "Template directory not found" at sandbox + setup, far from the actual mistake. + + Only ENV variables are expanded here. Dataset row substitution runs over + ``initial_prompt`` and ``success_criteria`` only, so a ``${row.X}`` left in a + template path will not be substituted and fails at sandbox setup. Skips non-TemplateDirSource entries. @@ -257,15 +250,12 @@ def resolve_agent_system_prompt[T: AgentConfig | BaseAgentConfig | None](agent_c prompt_path = (base_dir / prompt_path).resolve() if not prompt_path.exists(): raise FileNotFoundError(f"system_prompt_file not found: {prompt_path}") - # A whitespace-only file is no prompt at all — mirror the normalization - # _blank_prompt_is_no_prompt applies to inline prompts (model_copy skips - # validators, so this seam has to apply it itself). + # A whitespace-only file is no prompt at all. `model_copy` skips validators, + # so this seam applies the normalization itself... content = prompt_path.read_text(encoding="utf-8").strip() or None - # ...which means model_copy also skips check_replace_mode_has_prompt, so a - # blank file under `replace` would reach the agent as (replace, no prompt) - # and silently downgrade to the append preset at runtime. Reject it here - # instead: the file is the only prompt the config had, and the docs promise - # this combination fails at load. + # ...and also skips check_replace_mode_has_prompt, so a blank file under + # `replace` would reach the agent as (replace, no prompt) and silently + # downgrade to the append preset at runtime. if content is None and getattr(agent_config, "system_prompt_mode", "append") == "replace": raise ValueError( f"system_prompt_file {prompt_path} is empty; system_prompt_mode='replace' requires a " @@ -379,29 +369,22 @@ def expand_dataset( Tasks without ``dataset:`` pass through unchanged as ``[task]``. - Each expanded task: - - has task_id rewritten to ``"/"`` - - has ``dataset`` cleared (prevents re-expansion downstream) - - has ``${row.}`` substituted in ``initial_prompt`` and in all - string leaves of ``success_criteria`` entries + Each expanded task has its task_id rewritten to + ``"/"``, its ``dataset`` cleared (preventing + re-expansion downstream), and ``${row.}`` substituted in + ``initial_prompt`` and every string leaf of ``success_criteria``. - Row ids are validated against a safe pattern so they're filesystem-safe - when used as directory names under the run_dir. + Row ids are validated against a safe pattern, since they become directory + names under the run dir. Args: task: Task that may carry a dataset. - task_file_dir: Directory of the source task YAML (for resolving dataset.paths). - max_rows: Optional CLI cap on rows used (for cheap smoke runs). A - fixed-seed uniform-random N-row sample over the whole dataset - (reproducible, but unbiased across ``dataset.paths`` — unlike a raw - slice). When provided, overrides both ``sample_per_stratum`` args. - Absent it, ``sample_per_stratum`` (stratified random) applies. - sample_per_stratum: Optional CLI override (``--sample-per-stratum``) for - ``dataset.sample_per_stratum`` — keep up to N rows per stratum - (stratum = ``dataset.stratify_field``, default ``expected_skill``). - Lets a runner cap a stratified dataset without editing the task YAML - (the nightly activation suite uses this). Ignored when ``max_rows`` - is set. When None, falls back to ``dataset.sample_per_stratum``. + task_file_dir: Directory of the source task YAML. + max_rows: CLI cap (``--sample``) — a fixed-seed uniform-random sample, + reproducible but unbiased across ``dataset.paths``. Overrides both + ``sample_per_stratum`` args. + sample_per_stratum: CLI override for ``dataset.sample_per_stratum``, so a + runner can cap without editing the YAML. Ignored under ``max_rows``. Returns: Expanded list of TaskDefinitions. Length is 1 when dataset is None. @@ -418,19 +401,15 @@ def expand_dataset( if not rows: raise ValueError(f"Dataset for task '{task.task_id}' is empty") - # Row selection precedence: - # 1. CLI --sample (max_rows): flat uniform-random N over the whole dataset. - # Fixed seed => reproducible across runs, but (unlike a first-N slice) - # unbiased across the concatenated dataset.paths. - # 2. sample_per_stratum: stratified random N-per-stratum. CLI - # --sample-per-stratum (the arg) overrides dataset.sample_per_stratum - # (the YAML), so a runner can cap a dataset without editing its task. + # Row selection precedence: CLI --sample (flat uniform-random N over the whole + # dataset, fixed seed so it is reproducible but unbiased across the + # concatenated paths) wins over sample_per_stratum, whose CLI arg in turn + # overrides the YAML so a runner can cap a dataset without editing its task. ds = task.dataset n_per_stratum = sample_per_stratum if sample_per_stratum is not None else ds.sample_per_stratum - # Stratified sampling is seeded only by dataset.sample_seed. When that is None the sample is - # deliberately nondeterministic — re-drawn every run — regardless of whether the CLI - # --sample-per-stratum flag or the YAML supplied the count (see Dataset.sample_seed). The - # nightly activation suite relies on this to broaden coverage across runs. + # Seeded ONLY by dataset.sample_seed. When that is None the sample is + # deliberately nondeterministic — re-drawn every run — and the nightly + # activation suite relies on this to broaden coverage. stratum_seed = ds.sample_seed if max_rows is not None and max_rows < len(rows): rows = random.Random(_SMOKE_SAMPLE_SEED).sample(rows, max_rows) diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 51f8fb7ec..39af0480b 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -151,12 +151,10 @@ async def _pump_stream( log_fn("[%s] %s", label, line) -# Structural tags emitted by ClaudeCodeAgent._format_messages. Other -# bracketed words (markdown footnotes, pylint codes, unknown SDK message types -# like [TaskStartedMessage]) are intentionally NOT matched — they pass through -# as content. Source of truth for the tag vocabulary is -# ``ClaudeCodeAgent._format_messages``; this regex is telemetry-only (utterance -# extraction for the per-task log), not a correctness-critical parser. +# Structural tags emitted by ClaudeCodeAgent._format_messages, which is the SSOT +# for the vocabulary. Other bracketed words (markdown footnotes, pylint codes, +# unknown SDK message types) are intentionally NOT matched — they pass through as +# content. Telemetry-only, not a correctness-critical parser. _UTTERANCE_TAG_RE = re.compile(r"^\[(ASSISTANT|RESULT - SUCCESS|RESULT - ERROR|TOOL USE)\](?: (.*))?$") @@ -192,34 +190,22 @@ def _format_routing(route: ApiRoute, effective_model: str | None = None) -> str: def _extract_utterance(raw: str) -> str: """Collapse a ClaudeCodeAgent-formatted transcript to a clean utterance. - Input looks like: + Input looks like:: + [ASSISTANT] Sure, I'll do X. [TOOL USE] Read - [ASSISTANT] Here is the answer... [RESULT - SUCCESS] Here is the answer... - The SDK's ``ResultMessage`` duplicates the final assistant text, which - makes conversation.log read as if every message is repeated. Prefer the - ``[RESULT - ...]`` payload when it is non-empty (it is the canonical - final utterance); otherwise fall back to concatenated ``[ASSISTANT]`` - blocks. ``[TOOL USE]`` lines are dropped. Input that does not look - tagged at all (plain user text like a pinned initial_prompt) is - returned unchanged. - - Pre-tag content handling: any content appearing BEFORE the first - tagged line is collected into an implicit ``[ASSISTANT]`` block. It - survives in the output only on the ASSISTANT-fallback path (no - ``[RESULT - ...]`` in the transcript). When a ``[RESULT - SUCCESS]`` - is present, it supersedes all ``[ASSISTANT]`` content — including - any pre-tag content — because the ResultMessage is the SDK's - canonical final utterance and ASSISTANT lines are chain-of-thought - that the RESULT already incorporates. ClaudeCodeAgent always begins - its output with a tag in practice, so this mostly matters for - defensive handling of upstream format drift. - - Asymmetry note: ``[RESULT - SUCCESS]`` strips its label (it is the - canonical answer); ``[RESULT - ERROR]`` keeps a ``[RESULT - ERROR]`` - prefix in the output so the error state remains visible in the log. + Prefers a non-empty ``[RESULT - ...]`` payload — the SDK's canonical final + utterance, which duplicates the final assistant text and otherwise makes + conversation.log read as if every message is repeated. Falls back to + concatenated ``[ASSISTANT]`` blocks, including any content appearing before + the first tag. ``[TOOL USE]`` lines are dropped, and untagged input (a pinned + ``initial_prompt``) is returned unchanged. + + Asymmetric on purpose: ``[RESULT - SUCCESS]`` strips its label, while + ``[RESULT - ERROR]`` KEEPS its prefix so the error state stays visible in the + log. """ if not raw: return "" @@ -279,25 +265,19 @@ def _extract_failure_reason(result: CriterionResult) -> str | None: def build_task_event(result: EvaluationResult, *, driver: str, variant_id: str) -> tuple[str, dict[str, Scalar]]: """Build the (event_name, properties) for a finalized task's telemetry event. - Shared by the in-process path (``Orchestrator._finalize_result``) and the - docker path (``orchestration/batch.py``) so both drivers emit an identical - ``CoderEval.Task.End`` event. Carries only enums/counts/durations/config-derived - ids — no user content. None-safe. - - Every task emits the SAME event name (``CoderEval.Task.End``); the outcome - lives in dimensions, never the name. ``Status`` carries the exact - ``FinalStatus`` and ``Category`` carries the canonical ``FinalStatus.category`` - bucket (``succeeded`` / ``failed`` / ``error``) — the single source of truth - shared with reports. Slicing belongs in dimensions (the App Insights idiom), - so dashboards group by ``Status``/``Category`` rather than matching event - names, and the telemetry bucketing can never drift from ``category``. - - The return type is the scalar event contract (``dict[str, Scalar]``) so a - non-scalar property is caught here by pyright, not just str()-coerced at - runtime by the telemetry layer. Token counts are intentionally NOT emitted — - this is usage telemetry, not eval analytics. Task/variant ids are emitted as - stable one-way hashes (``hash_identifier``) so an author-defined free-text id - that could encode sensitive data never reaches the telemetry store verbatim. + Shared by the in-process path and the docker path so both drivers emit an + identical ``CoderEval.Task.End`` event. Carries only + enums/counts/durations/config-derived ids — no user content. None-safe. + + Every task emits the SAME event name; the outcome lives in DIMENSIONS, never + the name, so dashboards group by ``Status``/``Category`` and the telemetry + bucketing can never drift from ``FinalStatus.category``. + + The return type is the scalar event contract, so a non-scalar property is + caught by pyright rather than str()-coerced at runtime. Token counts are + deliberately NOT emitted — this is usage telemetry, not eval analytics — and + task/variant ids are one-way hashes, so an author-defined free-text id never + reaches the telemetry store verbatim. """ props: dict[str, Scalar] = { "TaskId": hash_identifier(result.task_id), @@ -312,19 +292,13 @@ def build_task_event(result: EvaluationResult, *, driver: str, variant_id: str) "EarlyStopped": result.early_stop is not None, "EarlyStopReason": (result.early_stop.reason.value if result.early_stop is not None else ""), } - # `Score` is OMITTED, never coalesced to 0.0, when the row was not graded. - # Dashboards compute `avg(todouble(customDimensions.Score))` with no status - # filter, so a laundered zero for a `coder-eval execute` night drags every - # score tile toward zero and is indistinguishable from a genuinely bad night. - # An absent dimension drops out of the average instead. + # OMITTED, never coalesced to 0.0, when the row was not graded. + # Rationale: .claude/notes/orchestration.md § Where the orchestrator's own time is booked if result.weighted_score is not None: props["Score"] = float(result.weighted_score) # The four wall-clock buckets, from the ONE canonical summation — this - # function does not add anything up itself. Each is OMITTED rather than - # coalesced to 0, for the same reason as `Score` above: a dashboard - # averaging `StartupMs` with no filter would read a laundered zero as a - # harness that booted instantly, which is indistinguishable from a run that - # predates the capture. An absent dimension drops out of the average. + # function adds nothing up itself. Each is omitted rather than zeroed, for the + # reason `Score` above is. from .reports_stats import turn_time_buckets buckets = turn_time_buckets(result) @@ -403,44 +377,34 @@ def __init__( """Initialize the orchestrator. Args: - task: Task definition to evaluate - run_dir: Per-task directory within a run (e.g., runs/2025-10-09_15-30-45/default/hello_date/00/) - preservation_mode: How to persist the sandbox after completion - (NONE / MOVE_ON_WRITE / DIRECT_WRITE). The driver-derived - default is resolved upstream at the batch dispatch seam. - task_file: Path to task YAML file (for resolving reference file paths) - stream_callback: Optional callback for real-time event streaming - sandbox: Pre-built Sandbox to use directly; if None, creates one from task config and runs the agent - variant_id: Experiment variant identifier for this task - source_yaml: Raw YAML text from the task file - config_lineage: Config lineage dict (dotted-path -> ConfigLineageEntry) - replicate_index: Zero-indexed trial number (for simulation tasks with n_trials > 1). - Defaults to 0, which covers single-shot tasks and single-trial simulations. - workspace_dir: Docker WORKDIR alignment. When set, the agent runs - in-place at this absolute container path (the task image's WORKDIR) instead of - run_dir/artifacts/, and the workspace is copied out to run_dir/artifacts/ - at cleanup. Resolved host-side by DockerRunner; None keeps standard behavior. - Takes precedence over preservation_mode when set. - grade: Whether to evaluate success criteria after execution. False is - `coder-eval execute`: the agent runs and the full trajectory is - captured, but no criterion is checked, ``weighted_score`` stays - None, and the row finalizes as ``FinalStatus.NOT_GRADED``. It is - deliberately NOT a task-config field — a task YAML must never be - able to declare itself ungraded — so it arrives only from - ``BatchRunConfig.grade``, never from the 5-layer merge or -D. - prior_result: A completed run's ``EvaluationResult`` to re-grade - (evaluate-only mode). Its trajectory and execution facts are - carried onto the fresh result so the grade describes the run that - actually happened instead of an empty one — see - ``_seed_from_prior_result`` for the field-by-field rationale. + task: Task definition to evaluate. + run_dir: Per-task run directory. + preservation_mode: How to persist the sandbox (NONE / MOVE_ON_WRITE / + DIRECT_WRITE). The driver-derived default resolves upstream. + task_file: Task YAML path, for resolving reference file paths. + stream_callback: Optional real-time event callback. + sandbox: Pre-built Sandbox; if None, one is created from task config. + variant_id: Experiment variant identifier. + source_yaml: Raw YAML text from the task file. + config_lineage: Dotted-path -> ConfigLineageEntry. + replicate_index: Zero-indexed trial number for n_trials > 1. + workspace_dir: Docker WORKDIR alignment — the agent runs in-place + there and the workspace is copied out at cleanup. Takes + precedence over ``preservation_mode``. + grade: False is ``coder-eval execute``. Deliberately NOT a task-config + field — a task YAML must never declare itself ungraded. + prior_result: A completed run's result to re-grade. + recorded_task / recorded_task_file: What the run RECORDS, which is not + always what this process runs. + + Rationale: .claude/notes/orchestration.md § Recording the task as authored """ self.task = task self.run_dir = run_dir - # Per-attempt nonce for the LiteLLM cost-log join. The proxy log is + # Per-attempt nonce for the LiteLLM cost-log join: the proxy log is # append-only and the run_id is a deterministic hash of run_dir, so a - # re-run into the same --run-dir would otherwise re-match (and double-count) - # a prior attempt's rows. A fresh nonce per Orchestrator (one per process - # invocation) scopes the join to THIS attempt's records. + # re-run into the same --run-dir would otherwise re-match a prior + # attempt's rows. self._cost_attempt_nonce = uuid.uuid4().hex self.preservation_mode = preservation_mode self.workspace_dir = workspace_dir @@ -452,36 +416,23 @@ def __init__( self.config_lineage = config_lineage or {} self.replicate_index = replicate_index self.grade = grade - # What `task_config.resolved` records, which is NOT always what we RUN. - # `run_task_internal_command` rewrites `driver: docker` -> `tempdir` - # before building the in-container Orchestrator (we are already inside - # the container the driver asked for), and recording that rewrite made - # the run's own record deny it ever used docker. A later - # `evaluate ` reads the driver back out of the record, so the - # host-grading refusal never fired and the `graded_on_host` stamp was - # never applied: a container task's criteria ran against the host - # filesystem silently, which is the exact outcome that gate exists to - # prevent. The record must describe the task as AUTHORED. + # What `task_config.resolved` records, which is NOT always what we RUN: + # the in-container path rewrites `driver: docker` -> `tempdir` before + # building its Orchestrator, and recording that made the run's own record + # deny it ever used docker. + # Rationale: .claude/notes/orchestration.md § Recording the task as authored self.recorded_task = recorded_task if recorded_task is not None else task - # Same seam, same reason, for the PATH. `task_file` is what this process - # resolves TASK_DIR and the reference against -- in a container that is - # `/work/task_dir/task.yaml`, which is correct here and meaningless - # anywhere else. Recording it made a container row's `source_file` name a - # path that exists on no host, so a later `evaluate ` rebuilt the - # task around it: the docker dispatch guard saw a non-None Path and let it - # through, and `_prepare_task_dir_mount` then silently mounted nothing - # (`if not source.is_dir(): return`), so every `$TASK_DIR` criterion - # resolved against the wrong tree and scored a verdict nobody could - # explain. The host forwards its own path for the record. + # Same seam, same reason, for the PATH: in a container `task_file` is + # `/work/task_dir/task.yaml`, correct here and meaningless anywhere else. + # The host forwards its own path for the record. self.recorded_task_file = recorded_task_file if recorded_task_file is not None else task_file self.prior_result = prior_result # Derived paths self.report_path = self.run_dir / TASK_JSON_FILENAME self.html_report_path = self.run_dir / "task.html" - # Clean user<->agent transcript for simulation runs. Written alongside - # task.log so a human can follow the conversation without the - # orchestrator noise in between. + # Clean user<->agent transcript for simulation runs, written alongside + # task.log so a human can follow it without the orchestrator noise. self.conversation_log_path = self.run_dir / "conversation.log" # Note: artifacts directory (run_dir/artifacts) is created on-demand during sandbox preservation @@ -491,50 +442,37 @@ def __init__( # API routing (initialized in _setup) self.route: ApiRoute | None = None - # Route for the simulated user, a real Claude Code CLI subprocess like - # the agent under test. Resolved via resolve_evaluation_route(settings, - # self.route) with NO checker_context overrides — decoupled from - # checker_context.api_route (that override is llm_judge-only and has no - # bearing on the simulator) while still inheriting the LiteLLM-agent -> - # pinned-Claude-backend guard, since the simulated user is part of the - # measuring instrument and must not run on the agent's own open-weight - # gateway either. Equals self.route for the Direct/Bedrock backends. + # The simulated user's route: pinned like eval_route, but resolved with NO + # checker_context overrides. + # Rationale: .claude/notes/orchestration.md § Three routes, resolved separately self.simulator_route: ApiRoute | None = None - # Route for the evaluation side (llm_judge / agent_judge): - # pinned to a constant Claude backend so grading stays comparable when the - # agent runs on an open-weight (LiteLLM) model. Equals self.route for the - # Direct/Bedrock backends. + # The judge's route: pinned to a constant Claude backend so grading stays + # comparable when the agent runs on an open-weight model. self.eval_route: ApiRoute | None = None # Result tracking self.result: EvaluationResult | None = None - # Per-run private copy of task.reference.directory, staged in _setup and - # removed in _cleanup. Criteria address it as $REFERENCE_DIR / REFERENCE_DIR. - # It is a COPY, not the checked-out path, so the mode-000 anti-cheat window - # around each agent turn can't block a sibling task's judge mid-read, and a - # crashed run can only leave a throwaway directory unreadable. + # Per-run private COPY of task.reference.directory, staged in _setup and + # removed in _cleanup; criteria address it as $REFERENCE_DIR. A copy, not + # the checked-out path, so the anti-cheat window cannot block a sibling + # task's judge mid-read. + # Rationale: .claude/notes/permissions.md § Reference solutions and the anti-cheat window self._reference_dir: Path | None = None - # The mkdtemp root that holds ``_reference_dir``, recorded the moment it - # is created and BEFORE the copy runs, so a copy that raises part-way - # (unreadable source file, ENOSPC) still gets cleaned up. Keying cleanup - # on ``_reference_dir.parent`` instead would leak the partial copy of the - # reference solution, because ``_reference_dir`` is only assigned on the - # success path. None under docker, where the reference is a host-owned - # bind mount rather than a tempdir of ours. + # Recorded BEFORE the copy runs, so a copy that raises part-way still gets + # cleaned up: keying cleanup on `_reference_dir.parent` would leak the + # partial copy, since that field is only assigned on the success path. + # None under docker, where the reference is a host-owned bind mount. self._reference_staging_root: Path | None = None - # SHA-256 of the staged reference tree, taken right after staging and - # re-verified before grading. The window is per-turn, so between turns - # the (necessarily writable) docker mount is back at its normal mode; an - # agent-backgrounded process could overwrite the reference and drive - # ``reference_comparison`` to 1.0. See _verify_reference_integrity. + # SHA-256 of the staged reference tree, re-verified before grading: the + # window is per-turn, so between turns an agent-backgrounded process could + # overwrite the reference and drive `reference_comparison` to 1.0. self._reference_digest: str | None = None - # Early-stop watcher (created in _setup only when a criterion carries a - # stop_early: block and the kill switch is not thrown; None otherwise, - # so the default path is entirely unaffected). + # Created in _setup only when armed; None otherwise, so the default path + # is entirely unaffected. self._early_stop_watcher: EarlyStopWatcher | None = None # One-shot flag: emit the "cost budget configured but no cost data" warning @@ -568,38 +506,13 @@ def _agent_name(self) -> str: def _terminal_status(self, success: bool) -> FinalStatus: """The status a normally-completed evaluation loop lands on. - Extracted from ``run()`` because the chain answers one question and - ``run()`` answers several; inlining it grew ``run()`` past the - complexity bound the moment the grading switch was threaded in. - - Order matters at every step: - - * A detached grade may NOT overturn an execution fact. The prior run's - terminal status (TIMEOUT, ERROR, a budget stop) describes an agent - phase this pass neither repeated nor observed. Without the first arm a - crashed run re-graded against its half-finished workspace reports - SUCCESS — with the original ``error_message`` still attached. - * The NOT_GRADED arm sits ABOVE ``max_turns_exhausted``, and that order - is what makes ``execute`` + ``evaluate`` equal a single ``run``. - MAX_TURNS_EXHAUSTED reads like an execution fact but is not one: on the - graded path it is subordinate to the verdict — ``run`` returns SUCCESS - for a max-turns trajectory whose criteria pass, and only falls through - to MAX_TURNS_EXHAUSTED when they do not. So it is not knowable under - ``grade=False``. Consuming it here first made it terminal *and* - permanent (``is_execution_fact`` pins it in the first arm), so the same - agent output scored SUCCESS/1.0 under ``run`` and MAX_TURNS_EXHAUSTED - under ``execute`` → ``evaluate`` — and, being category ``failed``, - `run --resume` called the row complete and left it forever unscored. - Nothing is lost by deferring: the fact lives on - ``result.max_turns_exhausted``, which ``_seed_from_prior_result`` - carries, so the detached grade walks this identical chain and reaches - the arm below. - - The statuses that ARE execution facts (TIMEOUT, ERROR, the budget - stops) differ in kind: they abort the run before a verdict is - reachable, so preserving them overturns nothing. + ORDER MATTERS at every step: a detached grade may not overturn an + execution fact, and the NOT_GRADED arm sits ABOVE ``max_turns_exhausted`` + so that ``execute`` + ``evaluate`` equals a single ``run``. With ``grade=True`` and no prior result the chain is the original one. + + Rationale: .claude/notes/orchestration.md § The terminal-status chain """ assert self.result is not None, "Result not initialized" inherited = self.prior_result.final_status if self.prior_result is not None else None @@ -628,9 +541,8 @@ async def run(self) -> EvaluationResult: """ from .logging_config import task_log_handler - # Agent must be resolved before reaching the orchestrator. No-op (type: none) - # tasks are resolved to a NoneAgentConfig like any other agent, so there is - # no separate "no agent" branch here. + # No-op (type: none) tasks resolve to a NoneAgentConfig like any other + # agent, so there is no separate "no agent" branch here. assert self.task.agent is not None, ( f"Task '{self.task.task_id}' has no agent config. Ensure experiment resolution ran before orchestration." ) @@ -641,14 +553,10 @@ async def run(self) -> EvaluationResult: agent_type = self.task.agent.type start_time = time.time() - # The monotonic twin of `start_time`, and the mark `setup_ms` measures - # from. It sits HERE rather than at `_setup()` because the phase is - # defined as everything before the agent runs, and the single largest - # item is already behind us by then: `get_version_info()` shells out for - # the git commit and every CLI's `--version` and costs 733 ms measured. - # Starting the mark at `_setup()` put that outside every named bucket, - # so it landed in the report's residual — 733 of the 758 ms that made - # "Unaccounted" look like a real unknown when it was one nameable call. + # The mark `setup_ms` measures from. It sits HERE and not at `_setup()`: + # the phase is everything before the agent runs, and its single largest + # item (`get_version_info`, 733 ms measured) is already behind us by then. + # Rationale: .claude/notes/orchestration.md § Where the orchestrator's own time is booked setup_started = time.monotonic() started_at = datetime.now() @@ -666,9 +574,9 @@ async def run(self) -> EvaluationResult: self._seed_from_prior_result() - # Calculate task log path. A re-grade gets its own file: `prior_result` - # means the agent phase already ran and its trajectory log is sitting in - # this very directory, and `task_log_handler` opens `mode="w"`. + # A re-grade gets its OWN file: `prior_result` means the agent phase + # already ran and its trajectory log is in this directory, and the handler + # opens `mode="w"`. task_log_file = task_log_path(self.run_dir, regrade=self.prior_result is not None) task_log_file.parent.mkdir(parents=True, exist_ok=True) # noqa: CE002 — mkdir on local FS is nanoseconds @@ -678,28 +586,15 @@ async def run(self) -> EvaluationResult: # Setup components await self._setup() - # Run pre-run commands inside the sandbox before the agent starts. - # A failing command with fail_on_error=True raises RuntimeError, - # which propagates to the outer except Exception below and lands - # the run as FinalStatus.ERROR; _run_post_run_commands and - # _cleanup still execute via the finally block. + # A failing command with fail_on_error=True raises, landing the + # run as ERROR; post_run and _cleanup still run via the finally. await self._run_pre_run_commands() - # Everything before the agent phase, booked as ONE task-level - # bucket: the environment capture, criterion discovery, sandbox - # provisioning, agent start() and pre_run. Roughly harness- - # independent — measured within ~10 ms of each other for - # claude-code and pi on the same machine — which is the tell - # that it is the orchestrator's own cost rather than any - # harness's. It used to land in the report's residual, where a - # known constant reads as unexplained time: 10% of a 19s task, - # and it would read 60% of a 3s one. + # Everything before the agent phase, as ONE task-level bucket. self.result.setup_ms = (time.monotonic() - setup_started) * 1000.0 - # Enforce task-level timeout via an OS-thread watchdog that - # SIGKILLs the in-flight CLI subprocess AND cancels this - # task. The threaded approach is immune to anyio cancel - # scopes that were silently swallowing asyncio.wait_for - # cancellations during long rate-limited API calls. + # An OS-thread watchdog, immune to the anyio cancel scopes that + # silently swallowed asyncio.wait_for cancellations during long + # rate-limited API calls. task_timeout = self.task.run_limits.task_timeout if self.task.run_limits else None def _kill_agent_subprocess_sync() -> None: @@ -720,9 +615,8 @@ def _kill_agent_subprocess_sync() -> None: task_timeout=task_timeout, start_time=start_time, ) - # Belt-and-suspenders: if the loop returned normally but the - # watchdog fired during post-loop work or the inner coro - # swallowed the cancel, still classify as TIMEOUT. + # The loop can return normally while the watchdog fired during + # post-loop work, or the inner coro swallowed the cancel. if wd.fired and task_timeout is not None: raise TaskTimeoutError( task_timeout, @@ -750,10 +644,9 @@ def _kill_agent_subprocess_sync() -> None: logger.error(f"Task timed out: {e}") - # Recover the turn in flight when the watchdog killed the agent. - # Nothing else on this path does: the cancel arrives as a - # BaseException, so it never reaches the retry executor's - # per-attempt hook that drains the slot on a turn-level timeout. + # Nothing else on this path recovers the in-flight turn: the + # cancel arrives as a BaseException, so it never reaches the retry + # executor's per-attempt hook. await self._drain_killed_turn() except BudgetExceededError as e: # Map token-budget breaches and cost-budget breaches to distinct @@ -796,15 +689,10 @@ def _kill_agent_subprocess_sync() -> None: logger.error(f"Evaluation failed: {e}", exc_info=True) finally: - # Teardown must be interrupt-proof: the task-timeout watchdog can - # fire while post-run commands are awaiting and deliver its - # CancelledError right here in the finally block, which used to - # abort it wholesale — skipping _cleanup() (tempdir leaked) AND - # _finalize_result() (task.json lost, so the task silently drops - # out of the run). Catch the interrupt, finish the full teardown, - # then re-raise it at the end so callers observe the same exception - # as before. The watchdog cancels exactly once, so the teardown - # awaits below run normally after the CancelledError is caught. + # INTERRUPT-PROOF: the task-timeout watchdog can deliver its + # CancelledError right here. Catch it, finish the teardown, then + # re-raise so callers observe the same exception. + # Rationale: .claude/notes/orchestration.md § Teardown must be interrupt-proof teardown_interrupt: BaseException | None = None try: # BEFORE post-run/cleanup: needs the live sandbox to resolve @@ -819,14 +707,10 @@ def _kill_agent_subprocess_sync() -> None: e, ) await self._cleanup() - # Capture the sanitised log tail AFTER teardown so any errors - # logged during post-run / cleanup also land in the report, - # but BEFORE _finalize_result so task.json includes the field. - # Allowlist non-success terminal statuses; SUCCESS and - # MAX_TURNS_EXHAUSTED skip the tail to keep task.json compact. - # NOT_GRADED is deliberately absent: like SUCCESS and - # MAX_TURNS_EXHAUSTED it is not a diagnosis of something going - # wrong, so it keeps task.json compact. + # AFTER teardown, so post-run and cleanup errors land in the + # report, but BEFORE finalization so task.json includes it. An + # ALLOWLIST: SUCCESS, MAX_TURNS_EXHAUSTED and NOT_GRADED all skip + # it, none being a diagnosis of something going wrong. if self.result.final_status in { FinalStatus.ERROR, FinalStatus.TIMEOUT, @@ -856,31 +740,22 @@ def _seed_from_prior_result(self) -> None: if prior is None or self.result is None: return - # A task row describes the TASK, so its clock is the agent run's, not the - # grading pass's. Left alone, a 10-minute run re-graded in 2 seconds would - # report 2 seconds — and that figure feeds average_duration, the report - # tables and the evalboard, so harness-vs-harness comparisons would be - # quietly wrong. _finalize_result restores the duration after its own - # timing write; the grading pass's cost is recorded separately there. + # A task row describes the TASK, so its clock is the agent run's. + # Rationale: .claude/notes/isolation.md § Detached grading and `Sandbox.adopt` self.result.started_at = prior.started_at - # completed_at too, so the row's three time fields stay consistent with - # each other: leaving it at grading wall-clock produces a triple where - # completed_at - started_at != duration_seconds, which misleads anyone - # deriving a duration from the timestamps. + # completed_at too, or the row's three time fields disagree. self.result.completed_at = prior.completed_at - # The trajectory itself. Every derived figure in _finalize_result — - # token totals, cost, command_stats, model_used, assistant turns — - # recomputes from `iterations`, so seeding it reproduces them exactly. + # Every derived figure recomputes from `iterations`, so seeding it + # reproduces them exactly. self.result.iterations = list(prior.iterations) # Evaluate-only hardcodes 1; a multi-turn run must not be reported as # single-turn just because the re-grade ran once. self.result.iteration_count = prior.iteration_count or len(prior.iterations) - # LOAD-BEARING for the verdict: gate selection is FIRED-ONLY. When - # early_stop is not None the checker gates on the weighted ARMED subset - # instead of strict-AND over every criterion. Dropping it would re-grade a - # truncated trajectory under the full-run gate and flip the verdict. + # LOAD-BEARING for the verdict: gate selection is FIRED-ONLY, so dropping + # this re-grades a truncated trajectory under the full-run gate. + # Rationale: .claude/notes/orchestration.md § Gate selection is fired-only self.result.early_stop = prior.early_stop # Execution facts that outlive the agent process. @@ -892,44 +767,23 @@ def _seed_from_prior_result(self) -> None: self.result.agent_config = prior.agent_config self.result.expected_commands = prior.expected_commands self.result.simulation = prior.simulation - # The run's own setup cost, for the reason `duration_seconds` is - # restored: it is a fact about the run, not about this pass. A detached - # grade ADOPTS the workspace rather than building one - # (`Sandbox.adopt`), so its own setup is a different activity — writing - # it here would report the re-grade's cheap adoption as the run's - # provisioning. `grading_ms` goes the other way and is deliberately NOT - # carried: the verdict this row now holds came from THIS pass's grading. + # The run's own setup cost, for the reason `duration_seconds` is restored. + # `grading_ms` goes the other way and is deliberately NOT carried. self.result.setup_ms = prior.setup_ms - # pre_run belongs to the execute phase and is NOT re-run against an - # adopted workspace (see _skip_pre_run_for_adopted), so its recorded - # outcomes would otherwise vanish from the graded row. - # - # post_run is the opposite: it belongs to the GRADING phase, so on a row - # that came from `execute` this list is EMPTY and this grade is about to - # fill it (see _skip_post_run). Copied into a fresh list either way, so - # appending here can never mutate the prior result. + # pre_run belongs to the EXECUTE phase and post_run to the GRADING phase, + # so one is carried and the other is about to be filled. Fresh lists + # either way, so appending cannot mutate the prior result. + # Rationale: .claude/notes/isolation.md § Detached grading and `Sandbox.adopt` self.result.pre_run_results = list(prior.pre_run_results) self.result.post_run_results = list(prior.post_run_results) - # The artifacts pointer. An adopted sandbox is not deleted by cleanup(), - # so the path stays valid — and a SECOND grade needs it, since without it - # the caller falls back to guessing the workspace. + # An adopted sandbox is not deleted by cleanup(), so the path stays valid + # — and a SECOND grade needs it, or the caller guesses the workspace. self.result.sandbox_path = prior.sandbox_path - # environment_info: the prior run's capture describes the machine that - # RAN the task (installed_tools, api route, coder_eval version). Ours - # describes the machine grading it. Prior wins on conflict, and ours is - # preserved as flat `graded_by_*` scalars rather than being interleaved — - # a report that shows the grader's tool versions as the run's is worse - # than one that shows neither. - # Flattened to scalars rather than nested wholesale: environment_info is - # a flat map everywhere it is consumed (the HTML report `_esc`apes each - # value into a table cell; the evalboard types it as - # Record>), so a - # whole nested env capture renders as a Python dict repr. Only the three - # facts that identify the grading HOST are kept, and only when they - # differ from the run's. + # Merged, never replaced: prior wins on conflict, and ours survives as + # flat `graded_by_*` scalars. grader = self.result.environment_info provenance = { f"graded_by_{key}": grader[key] @@ -1029,9 +883,9 @@ async def _evaluate_post_failure_criteria(self) -> None: if self.result is None: return if not self.grade: - # Grading site 4 of 4. Under `execute` no criterion is checked on any - # path, diagnostics included — recording a not_evaluated vector here - # would imply criteria we were supposed to run and couldn't. + # Grading site 4 of 4: recording a not_evaluated vector would imply + # criteria we were supposed to run and couldn't. + # Rationale: .claude/notes/orchestration.md § The four grading sites return if self.success_checker is None or self.sandbox is None: self._record_post_failure_not_evaluated("the sandbox or success checker was unavailable") @@ -1121,11 +975,9 @@ def _finalize_weighted_score(self) -> None: if self.grade: self.result.calculate_weighted_score(self.task.success_criteria) else: - # Explicit None, NOT the 0.0 calculate_weighted_score writes for - # an empty results list — that value is indistinguishable from a - # task that was graded and scored zero, and every downstream - # `score or 0.0` would launder it into a real-looking failure - # (CE049). + # Explicit None, NOT the 0.0 written for an empty results list — + # that value is indistinguishable from a graded row that scored + # zero, and every downstream `score or 0.0` launders it (CE049). self.result.weighted_score = None except ValueError as e: logger.error("Weighted-score computation failed; marking row ERROR: %s", e, exc_info=True) @@ -1182,32 +1034,28 @@ def _finalize_result(self, start_time: float) -> None: if not self.result: return - # Every resolved task carries an agent config (no-op tasks resolve to a - # NoneAgentConfig); a missing one is a resolution bug. The evaluate-only - # path doesn't reach here with task.agent unset. + # Every resolved task carries an agent config; a missing one is a + # resolution bug. if self.task.agent is None: logger.error("Cannot finalize result: task.agent is None") return self.result.completed_at = datetime.now() self.result.duration_seconds = time.time() - start_time - # Read off the checker, which accumulated it across every call site it - # served. Stays None when nothing was graded (`coder-eval execute`), - # which is the distinction CE058 is about: no criteria ran, so no - # measurement exists — as opposed to one that came back instant. + # Accumulated by the checker across every call site it served. Stays None + # when nothing was graded: no criteria ran, so no measurement exists, as + # opposed to one that came back instant (CE058). if self.success_checker is not None: self.result.grading_ms = self.success_checker.grading_ms - # Re-grade: the row keeps the agent run's duration (see - # _seed_from_prior_result). The grading pass's own cost is preserved - # alongside rather than discarded, so a slow judge is still visible. + # The row keeps the agent run's duration; the grading pass's own cost is + # preserved alongside, so a slow judge is still visible. self._finalize_regrade_timing() - # Weighted score. This call site is wrapped because _finalize_result runs - # inside run()'s finally — an unguarded raise here would skip persistence and - # lose task.json. The other calculate_weighted_score calls (the simulation - # path) run inside run()'s try, whose broad `except Exception` already converts - # a raise into a populated ERROR result, so they intentionally stay unwrapped. + # Wrapped because _finalize_result runs inside run()'s finally, where an + # unguarded raise would skip persistence and lose task.json. The + # simulation-path calls run inside run()'s try, whose broad handler already + # converts a raise into a populated ERROR result. self._finalize_weighted_score() # Command statistics @@ -1223,9 +1071,8 @@ def _finalize_result(self, start_time: float) -> None: if not self.result.model_used and self.task.agent is not None and self.task.agent.model: self.result.model_used = self.task.agent.model - # Open-weight (LiteLLM) backend: replace per-turn cost with the ACTUAL - # per-call OpenRouter cost captured proxy-side. Runs BEFORE aggregation so - # the run total re-derives from the corrected per-turn costs. + # Replace per-turn cost with the ACTUAL proxy-side cost. BEFORE + # aggregation, so the run total re-derives from the corrected values. self._join_litellm_actual_cost() # Aggregate token usage @@ -1268,18 +1115,15 @@ def _finalize_result(self, start_time: float) -> None: "Task finished: status=%s duration=%.1fs score=%s iterations=%d", self.result.final_status.value, self.result.duration_seconds or 0.0, - # "n/a", not 0.000: this line is read when diagnosing a run, and a - # zero here would say the criteria scored nothing rather than that - # nothing was scored. + # "n/a", not 0.000: a zero here would say the criteria scored nothing + # rather than that nothing was scored. "n/a" if self.result.weighted_score is None else f"{self.result.weighted_score:.3f}", self.result.iteration_count, ) - # Usage telemetry (non-fatal; placed before persistence since track_event - # cannot raise). For the docker driver this in-process emit runs INSIDE the - # container where telemetry is off (the connection-string env vars aren't - # forwarded), so the host emits the event from batch.py instead — see - # build_task_event. Non-docker tasks finalize on the host and emit here. + # Non-fatal, and before persistence since track_event cannot raise. Under + # the docker driver this runs INSIDE the container, where telemetry is off, + # so the host emits the event from batch.py instead. from .telemetry import track_event driver = self.task.sandbox.driver if self.task.sandbox else "" @@ -1289,33 +1133,28 @@ def _finalize_result(self, start_time: float) -> None: # Persist self.report_path.parent.mkdir(parents=True, exist_ok=True) # noqa: CE002 — mkdir on local FS is nanoseconds - # Spill any judge transcripts to sibling YAML files BEFORE - # we dump task.json, so transcript_path is set on each judge result. - # The inline `transcript` field stays in memory — HTML rendering below - # uses it directly. We strip it from the JSON dump via `exclude=...`. + # BEFORE the task.json dump, so transcript_path is set on each judge + # result. The inline `transcript` stays in memory for HTML rendering and is + # stripped from the JSON dump. from .evaluation.judge_persistence import TASK_JSON_TRANSCRIPT_EXCLUDE, spill_judge_transcripts spill_judge_transcripts(self.result, self.report_path.parent) - # Atomic write: tmp file + os.replace. A SIGKILL mid-write (e.g. the - # docker-driver host-heartbeat watchdog firing) would otherwise leave - # a truncated task.json that the host parses as malformed-JSON rather - # than as "no result", conflating two distinct failure modes. + # A SIGKILL mid-write would otherwise leave a truncated task.json that + # parses as malformed-JSON rather than as "no result". + # Rationale: .claude/notes/persistence.md § write_text_atomic write_text_atomic( # noqa: CE002 — small JSON write at end of run self.report_path, self.result.model_dump_json( indent=2, - # Strip inline transcripts: they live in sibling YAML files - # next to task.json, referenced by transcript_path. Excluding - # `transcript` here avoids ~20-100 KB of bloat per judge result - # in the row record without losing any data. + # They live in sibling YAML files, referenced by transcript_path; + # this avoids ~20-100 KB of bloat per judge result. exclude=TASK_JSON_TRANSCRIPT_EXCLUDE, ), ) - # Also emit an HTML trace/report alongside task.json. HTML failure must - # never mask the underlying run outcome — write_task_html logs and - # returns None on failure. + # HTML failure must never mask the run outcome — write_task_html logs and + # returns None. from .reports_html import write_task_html write_task_html(self.result, self.html_report_path) @@ -1451,11 +1290,9 @@ def _join_litellm_actual_cost(self) -> None: if not (isinstance(self.route, LiteLLMRoute) and settings.litellm_cost_log and self.result is not None): return if self.prior_result is not None: - # Re-grading someone else's trajectory. The join keys on THIS - # Orchestrator's per-attempt nonce, which the original turns were - # never tagged with, so it would match nothing and overwrite the - # already-corrected per-turn costs with a warning about a missing - # bill. The prior run's cost is the real one; leave it alone. + # The join keys on THIS Orchestrator's per-attempt nonce, which the + # original turns were never tagged with — it would match nothing and + # overwrite the already-correct per-turn costs. logger.debug("Re-grade of a prior trajectory: keeping its recorded cost, skipping the LiteLLM join.") return try: @@ -1469,9 +1306,8 @@ def _join_litellm_actual_cost(self) -> None: if applied: logger.info("LiteLLM actual-cost join: real per-call cost applied to %d turn(s)", applied) else: - # Tags were stamped but nothing matched (file absent, proxy never - # wrote, wrong path, or a run/task/attempt mismatch). The run stays - # on the static rate card — warn so it isn't mistaken for the real bill. + # Stamped but unmatched: the run stays on the static rate card, so + # warn rather than let it pass as the real bill. logger.warning( "LiteLLM actual-cost join found no matching records in %s (run=%s task=%s); cost stays static", settings.litellm_cost_log, @@ -1541,10 +1377,9 @@ async def _stage_reference(self) -> None: destination = staging / "reference" self._reference_dir = await asyncio.to_thread(stage_reference_dir, source, destination) self._reference_digest = await asyncio.to_thread(digest_tree, self._reference_dir) - # Persist it: a DETACHED grade happens in a different process with no - # access to this instance, and refuses to score old work against a new - # answer key by comparing the tree it stages against this recorded hash - # (orchestration/regrade.py::verify_reference_unchanged). + # Persisted because a DETACHED grade runs in another process and refuses + # to score old work against a new answer key by comparing the tree it + # stages against this recorded hash. if self.result is not None: self.result.environment_info["reference_digest"] = self._reference_digest self._validate_reference_consumers() @@ -1608,11 +1443,8 @@ def _arm_early_stop(self) -> None: if self.grade: self._early_stop_watcher = EarlyStopWatcher.for_task(self.task) return - # Early stop cuts the run once coder-eval's own criteria decide the - # outcome. Under `execute` there is no outcome to decide and the - # trajectory is the deliverable (an external harness grades it), so an - # armed criterion must not truncate it. Same effect as the - # run_limits.stop_early kill switch, decided one layer up. + # Under `execute` there is no outcome to decide and the trajectory IS the + # deliverable, so an armed criterion must not truncate it. logger.info( "Grading disabled (execute mode): early-stop is armed but stays disabled; " + "the full trajectory is the deliverable." @@ -1639,22 +1471,19 @@ async def _setup(self) -> None: Raises: RuntimeError: If setup fails """ - # Defensive early-stop guardrails for the library-use and in-container - # paths (the CLI already validated during resolution). No-op unless - # some criterion carries a stop_early: block. + # Guardrails for the library-use and in-container paths; the CLI already + # validated during resolution. validate_early_stop(self.task) self._warn_on_ineffective_task_timeout() - # Build the early-stop watcher once, up front, when armed (>= 1 criterion - # with a stop_early: block and the run_limits.stop_early kill switch not - # thrown). This sits BEFORE the evaluate-only early return below, so an - # armed evaluate-only re-grade builds an inert (never-fed) watcher — - # harmless, and keeps a single creation point. + # ONCE, up front, and BEFORE the evaluate-only early return: an armed + # evaluate-only re-grade builds an inert watcher, which is harmless and + # keeps a single creation point. + # Rationale: .claude/notes/orchestration.md § Gate selection is fired-only self._arm_early_stop() - # Stage the reference BEFORE either branch returns: judge criteria with - # include_reference=true (and any $REFERENCE_DIR/... file entry) expect it - # populated in evaluate-only re-grades too, where no agent ever runs. + # BEFORE either branch returns: judge criteria with include_reference + # expect it populated in evaluate-only re-grades too. await self._stage_reference() if self.sandbox is not None: @@ -1669,9 +1498,8 @@ async def _setup(self) -> None: self._record_route_environment_info() return - # Validate API keys (agent guaranteed non-None after experiment resolution). - # validate_api_keys exempts the no-op agent (type: none) internally — it - # makes no API call, so it needs no agent keys. + # validate_api_keys exempts the no-op agent internally — it makes no API + # call, so it needs no agent keys. assert self.task.agent is not None and self.task.agent.type is not None settings.validate_api_keys(str(self.task.agent.type)) @@ -1680,13 +1508,11 @@ async def _setup(self) -> None: self.sandbox = Sandbox(self.task.sandbox, task_id=self.task.task_id, task_dir=task_dir) self.sandbox.reference_dir = self._reference_dir - # workspace_dir (docker WORKDIR alignment) wins: run the agent in-place at - # the image's own WORKDIR so its inputs/verifier paths line up, then copy - # the workspace out to run_dir/artifacts in _cleanup. Otherwise: - # DIRECT_WRITE runs the sandbox straight in run_dir/artifacts/ - # (no end-of-run copy/move); MOVE_ON_WRITE / NONE run in a tempdir — - # this keeps the run off run_dir on shared hosts, where parent-dir - # node_modules can contaminate Node tool resolution (MST-9795). + # workspace_dir (docker WORKDIR alignment) WINS: run the agent in-place at + # the image's own WORKDIR so its paths line up, then copy out in _cleanup. + # Otherwise DIRECT_WRITE runs straight in run_dir/artifacts and the rest + # run in a tempdir, which keeps the run off run_dir on shared hosts where + # a parent-dir node_modules contaminates Node tool resolution (MST-9795). if self.workspace_dir is not None: if self.preservation_mode == PreservationMode.DIRECT_WRITE: logger.debug( @@ -1698,19 +1524,12 @@ async def _setup(self) -> None: direct_target = self.run_dir / "artifacts" / self.task.task_id else: direct_target = None - # DIRECT_WRITE deliberately does NOT clear the target dir, so a reused - # --run-dir (or --resume) can leave a prior run's files alongside this - # run's outputs and silently perturb file-based criteria. Surface it. - # Suppressed in workspace_dir mode ONLY inside a container - # (IN_CONTAINER_ENV, per CE056 -- never on the field itself): the - # original writer was exclusively `run_task_internal_command`, where - # the WORKDIR is a fresh container filesystem every run, so a WORKDIR - # (/root, /app) legitimately holds the image's baked inputs there, not - # stale prior-run files. `--workspace-dir` is now also a host-reachable - # CLI flag on `run`/`execute`, where the named directory persists - # across invocations exactly like DIRECT_WRITE's own target -- keying - # the suppression on `workspace_dir is None` silently disabled the - # warning on precisely the new path where it is needed. + # DIRECT_WRITE deliberately does NOT clear the target, so a reused + # --run-dir can leave a prior run's files beside this one's and perturb + # file-based criteria. Suppressed in workspace_dir mode ONLY inside a + # container (CE056 — never on the field itself), where the WORKDIR is a + # fresh filesystem holding the image's baked inputs; `--workspace-dir` is + # also a host flag, where the directory persists and the warning is needed. in_container = os.environ.get(IN_CONTAINER_ENV) == "1" if ( not (self.workspace_dir is not None and in_container) @@ -1741,10 +1560,8 @@ async def _setup_sandbox() -> Any: # Determine API routing from settings.api_backend enum self._resolve_routes() - # Create and start the agent. For a no-op (type: none) task this dispatches - # to NoOpAgent, whose start/communicate/stop are no-ops — the orchestrator - # runs the normal lifecycle without any agentless branching, and the - # criteria are checked against the pre_run-prepared sandbox. + # A no-op (type: none) task dispatches to NoOpAgent, whose lifecycle is + # no-ops, so the orchestrator needs no agentless branching. assert self.task.agent is not None self.agent = await self._create_agent() @@ -1768,18 +1585,11 @@ async def _start_agent() -> None: # Save agent config on result (copy to prevent mutation of shared reference) self.result.agent_config = self.task.agent.model_copy(deep=True) - # Re-capture environment_info with sandbox path (for CLAUDE.md hash). - # # UPDATE, never rebind. `get_version_info` returns a FRESH dict, so - # assigning it here discarded every key written earlier in `_setup` — - # and `_stage_reference` runs earlier and writes `reference_digest` - # there. The digest therefore never reached task.json, which left - # `regrade.verify_reference_unchanged` taking its "recorded no digest" - # early return on every real run: the answer-key anti-cheat was a - # permanent no-op that CE054 could not see, because a write DID exist - # in `src/` — it was just dead. Merging keeps the sandbox-derived - # capture authoritative for the keys it owns without deleting anyone - # else's. + # assigning it discarded every key written earlier in `_setup` — including + # `reference_digest`, which left the answer-key anti-cheat a permanent + # no-op that CE054 could not see, because a write DID exist in `src/`; it + # was just dead. self.result.environment_info.update( get_version_info( sandbox_path=Path(self.result.sandbox_path) if self.result.sandbox_path else None, @@ -1796,35 +1606,17 @@ async def _start_agent() -> None: def _sync_sandbox_command_path_with_agent(self) -> None: """Align criteria command PATH with the PATH used for the last agent query. - Scope: called from the per-turn happy path in ``run_iteration`` / - ``run_simulation`` *after* a successful ``_communicate_with_retry``. - That means three pre-existing gaps remain (none introduced by this - change): - - - **Agent crash / turn timeout** — the sync is skipped because the - method never returns; criteria fall back to ambient - ``os.environ['PATH']``. Acceptable: a crashed agent's SDK PATH may - itself be unreliable. - - **Evaluate-only mode** (``orchestrator.run_evaluation_only``) — no - agent turn runs, so no sync. Criteria use ambient PATH, same as - before this change. - - **Before the first turn** — same reason; first criterion check - always runs after at least one turn under the normal flow. - - Sandbox-setup-time sync (using ``SandboxConfig.mock_path_dirs``) was - considered but rejected: the agent SDK's effective PATH is only - knowable after the SDK initializes, so a setup-time sync would - capture only the configured prepends, not the full agent env. - - ``Agent.get_sdk_options()`` is declared synchronous on the ABC - (``dict[str, Any] | None``). ``AsyncMock``-based test fixtures - return a coroutine for *any* attribute access regardless of the - declared signature; ``isawaitable`` plus ``coroutine.close()`` - prevents leaking ``RuntimeWarning: coroutine was never awaited`` - from those fixtures into unrelated tests. That is a test-fixture - concern, not a production contract violation — logged at DEBUG. - Returning a non-dict-and-non-None *is* a production contract - violation and is logged at WARNING. + Called from the per-turn happy path AFTER a successful + ``_communicate_with_retry``, which leaves three gaps: an agent crash or + turn timeout, evaluate-only mode, and the window before the first turn. In + each, criteria fall back to ambient ``os.environ['PATH']``. + + ``Agent.get_sdk_options()`` is declared synchronous on the ABC, but + ``AsyncMock`` fixtures return a coroutine for ANY attribute access, so it + is closed rather than awaited — a test-fixture concern, logged at DEBUG. A + non-dict, non-None return IS a production contract violation and warns. + + Rationale: .claude/notes/orchestration.md § Restoring a PATH from a run directory """ if self.agent is None or self.sandbox is None: return @@ -1834,10 +1626,8 @@ def _sync_sandbox_command_path_with_agent(self) -> None: if isawaitable(sdk_options): close = getattr(sdk_options, "close", None) if callable(close): - # ``coroutine.close()`` only documents ``RuntimeError`` - # (raised when invoked on a currently-running coroutine, - # which cannot apply here). Narrow the suppress accordingly - # so genuine unexpected exceptions still propagate. + # `close()` only documents RuntimeError, which cannot apply here, + # so narrow the suppress and let real exceptions propagate. with suppress(RuntimeError): close() logger.debug( @@ -1857,11 +1647,8 @@ def _sync_sandbox_command_path_with_agent(self) -> None: path = sdk_env.get("PATH") if isinstance(path, str) and path: self.sandbox.set_command_base_path(path) - # Persist it so a LATER detached grade (`coder-eval evaluate` over a - # finished run dir) can restore the same PATH. Without this the - # "evaluate-only mode" gap named above is permanent: the re-grade - # would resolve `run_command` criteria against ambient PATH and could - # reach a different verdict than the run it claims to be grading. + # Persisted so a LATER detached grade can restore the same PATH; + # otherwise it resolves run_command criteria against ambient PATH. if self.result is not None: self.result.environment_info["command_base_path"] = path @@ -1900,12 +1687,7 @@ def _resolve_routes(self) -> None: assert self.sandbox is not None self.route = resolve_route(settings) overrides = self._eval_route_overrides() - # Decoupled from checker_context.api_route (no overrides passed) so the - # simulator never reads the litellm-judge-only knob, but still routed - # through resolve_evaluation_route (not aliased to self.route) so the - # LiteLLM-agent -> pinned-Claude-backend guard still applies to it: the - # simulated user is part of the measuring instrument and must not run on - # the agent's own open-weight gateway either. + # Rationale: .claude/notes/orchestration.md § Three routes, resolved separately self.simulator_route = resolve_evaluation_route(settings, self.route) self.eval_route = resolve_evaluation_route( settings, @@ -1975,35 +1757,21 @@ def _record_route_environment_info(self) -> None: assert self.result is not None assert self.route is not None if self.prior_result is not None: - # A detached grade resolves routes for ITS OWN host, which may be a - # different backend from the one that ran the task. Writing them into - # the run's keys contradicts the "prior wins" contract in - # _seed_from_prior_result and leaves a self-contradictory record — - # `api_routing: anthropic_direct` beside the run's stale `aws_region` - # and `bedrock_model`. Keep the run's routing; record the grader's - # alongside it, under the same `graded_by_` provenance prefix the - # seeding uses, and only when it actually differs. + # A detached grade resolves routes for ITS OWN host: writing them into + # the run's keys leaves a self-contradictory record, so they go under + # the `graded_by_` prefix and only when they differ. self._record_grader_route_provenance() return self.result.environment_info["api_routing"] = ROUTE_NAMES[type(self.route)] - # The judge side (llm_judge / agent_judge) may run on a different, - # constant backend — pinned to Claude when the agent is on LiteLLM — so - # record it: a run then shows what actually graded it, distinct from the + # Recorded so a run shows what actually graded it, distinct from the # agent's api_routing. if self.eval_route is not None: self.result.environment_info["eval_routing"] = ROUTE_NAMES[type(self.eval_route)] - # bedrock_model/litellm_model below are sourced from self.route (the - # AGENT's route) — record the judge's own model separately so a - # checker_context.api_route.model override (or the LiteLLM-agent - # pinned-to-Bedrock default) is visible in run artifacts, not just - # inferable from the agent's model. + # The models below are the AGENT's; record the judge's separately so a + # checker_context override is visible rather than merely inferable. if self.eval_route.model: self.result.environment_info["eval_model"] = self.eval_route.model - # The simulator is pinned the same way eval_route is (LiteLLM agent -> - # constant Claude backend) but resolved independently of - # checker_context.api_route — record it separately so a run shows what - # the simulated user actually talked to, distinct from both api_routing - # and eval_routing above. + # Recorded separately so a run shows what the simulated user talked to. if self.simulator_route is not None: self.result.environment_info["simulator_routing"] = ROUTE_NAMES[type(self.simulator_route)] if self.simulator_route.model: @@ -2056,10 +1824,9 @@ def _refresh_runtime_tool_versions(self) -> None: self.sandbox.refresh_plugin_tools_dir() versions = runtime_uip_versions(self.sandbox.plugin_tools_dir, self.sandbox.uip_search_path) # Keep the setup-time values when post-task resolution comes back - # empty (e.g. `uip` gone from PATH) — they are the better estimate - # of what the task ran than "unknown"/{}. Gate on looks_like_version - # (not a "" / "unknown" denylist) so this sink shares the one - # version-shape contract with _uip_version and the run-level join. + # empty — they estimate what the task ran better than "unknown". Gated + # on looks_like_version, not a denylist, so this shares one + # version-shape contract with the run-level join. if looks_like_version(versions.get("cli_version")): self.result.environment_info["cli_version"] = versions["cli_version"] if versions.get("tool_plugins"): @@ -2082,23 +1849,17 @@ async def _create_agent(self) -> Agent[Any]: from coder_eval.agents import AgentRegistry, create_agent from coder_eval.plugins import ensure_plugins_loaded - # Safety net for the production agent-construction path: create_agent no - # longer self-loads (to keep plugins -> registry a one-way import edge), so - # ensure plugin kinds are registered here before dispatch. + # create_agent no longer self-loads (keeping plugins -> registry one-way), + # so plugin kinds are registered here before dispatch. ensure_plugins_loaded() assert self.task.agent is not None assert self.task.agent.type is not None - # LiteLLM (open-weight) route only: give the agent correlation headers so a - # proxy-side cost-logging callback can attribute each call's real cost + - # cache buckets back to this task-run. x-ce-run-id is a stable per-task-run - # key (the join, in _finalize_result, recomputes it identically); x-ce-task-id - # is the human-readable canonical id. - # - # Gate on AGENT CAPABILITY, not the route: the route is settings-derived and - # independent of agent type, but only agents whose __init__ accepts the kwarg - # (supports_cost_log_tags) may receive it — otherwise the agent-agnostic - # factory would forward it into NoOp/Codex/Antigravity/plugin constructors - # that don't declare it and crash with TypeError under API_BACKEND=litellm. + # LiteLLM only: correlation headers so a proxy-side cost callback can + # attribute each call back to this task-run. GATED ON AGENT CAPABILITY, not + # on the route — the route is settings-derived and independent of agent + # type, so the agent-agnostic factory would otherwise forward the kwarg + # into constructors that do not declare it. + # Rationale: .claude/notes/agents.md § Why the constructors declare every kwarg kwargs: dict[str, Any] = {} registration = AgentRegistry.get(self.task.agent.type) if ( @@ -2145,11 +1906,9 @@ async def _communicate_with_retry( if self.stream_callback is not None: agent_callback = TaskScopedCallback(self.stream_callback, self._log_task_id) - # Compose the early-stop watcher into the stream when armed. It is the - # sole callback when --stream is off, else it runs alongside the - # TaskScopedCallback. The same watcher instance persists across retry - # attempts (created once in _setup), so its turn/tool counters and - # wall-clock origin accumulate correctly. The closures below read `watcher`. + # The sole callback when --stream is off, else alongside the + # TaskScopedCallback. The same instance persists across retry attempts, so + # its counters and wall-clock origin accumulate. watcher = self._early_stop_watcher if watcher is not None: agent_callback = ( @@ -2225,29 +1984,14 @@ async def _communicate_attempt() -> TurnRecord: iteration=iteration, ) from None - # ANTI-CHEAT WINDOW. The agent shares a filesystem with the harness, so - # without this it can simply read the reference solution (and the task YAML - # with its criteria) instead of solving the task. Both directories sit at - # mode 000 for the whole of every communicate attempt — including retries, - # since the wrapper is outside execute_with_retry — and are restored on - # every exit path, so criteria and judges that run afterwards read normally. - # - # Routed through the SANDBOX, which owns whether a chmod window means - # anything for its driver. - # - # task_dir is shielded ALONGSIDE reference_dir. It previously was not, - # because under docker it was bind-mounted `:ro` and the chmod returned - # EROFS -- producing only a per-turn "could not chmod" warning. It is now - # a read-write throwaway copy (docker_runner._prepare_task_dir_mount), so - # the window applies. That matters because the task dir holds grading - # material beyond the reference: run_command fixtures, expected outputs, - # and -- for a task laid out flat, whose parent is the whole `tasks/` - # tree -- every SIBLING task's reference solution. - # - # What this does NOT do is hide the task DEFINITION. `task.yaml` is also - # staged at /work/input for the in-container orchestrator to read, and - # that mount is untouched by this window. Hiding the criteria from the - # agent remains a separate, unsolved problem. + # ANTI-CHEAT WINDOW. Both the reference and the task dir sit at mode 000 + # for the whole of every communicate attempt — retries included, since this + # wrapper is outside execute_with_retry — and are restored on every exit + # path. Routed through the SANDBOX, which owns whether a chmod window means + # anything for its driver. It does NOT hide the task DEFINITION: task.yaml + # is also staged at /work/input, and hiding the criteria from the agent is + # a separate, unsolved problem. + # Rationale: .claude/notes/permissions.md § Reference solutions and the anti-cheat window assert self.sandbox is not None async with self.sandbox.set_permissions([self._reference_dir, self.sandbox.task_dir]): turn_record = await execute_with_retry( @@ -2304,30 +2048,16 @@ def _accumulate_judge_usage( def _sanitize_restored_path(self, recorded: str) -> str: """Filter a PATH restored from a run's own ``task.json`` before prepending it. - The restored value is PREPENDED ahead of the host PATH, and it arrives - from a file inside the directory being graded — a run dir is a shareable - artifact (that is the whole point of the detached-grading flow), and under - ``driver: docker`` it is bind-mounted writable into the container the agent - runs in. Prepending it verbatim lets a run dir decide which binary + The restored value arrives from inside the directory being graded — a + shareable artifact, bind-mounted writable into the agent's container under + ``driver: docker`` — so verbatim it lets a run dir decide which binary ``pytest`` resolves to on the grader's host. - Four filters, all cheap and all about what PATH parity actually needs: - - * **Absolute only.** A relative entry resolves against the grader's - *current working directory*, which has nothing to do with the run — so - ``evilbin`` in a recorded PATH becomes ``$PWD/evilbin`` at the front of - every criterion subprocess's PATH. It also cannot be the toolchain - location it claims to be, since the run resolved it somewhere else. - * Drop anything that is not an existing directory (a dead entry buys no - parity). - * Drop any entry inside the **workspace** being graded — that tree is - agent-writable, so a shim dropped there would shadow a real tool. - * Drop any entry inside the **run directory** as a whole. The workspace - is only part of it; ``artifacts/``, a sibling replicate's tree and the - run root itself all travel in the same shared artifact and are all - equally attacker-chosen. - - What remains is the run's genuine toolchain locations. + Four filters: absolute paths only, existing directories only, nothing + inside the workspace, nothing inside the run directory. What remains is the + run's genuine toolchain locations. + + Rationale: .claude/notes/orchestration.md § Restoring a PATH from a run directory """ workspace = self.sandbox.sandbox_dir.resolve() if self.sandbox and self.sandbox.sandbox_dir else None run_root = self.run_dir.resolve() @@ -2361,43 +2091,19 @@ def _sanitize_restored_path(self, recorded: str) -> str: def _select_gate(self) -> bool: """Apply the verdict gate to the criteria results already on ``self.result``. - Gate selection is FIRED-ONLY: the weighted armed gate applies iff the - watcher actually cut the run (``early_stop is not None``) — on a truncated - trajectory the unarmed criteria never had the chance to be satisfied, so - they stay advisory. A run that completed naturally (armed or not, watcher - never fired or disarmed fail-open) has a full trajectory and gates - strict-AND over every gating criterion, exactly like an unarmed run — - arming a criterion (e.g. adding a ``decide_within`` fail-fast timeout) - must never change the verdict of a run it didn't cut. - - BOTH SINGLE-SHOT grading paths must call this — the live one and the - evaluate-only one, which are its two call sites. A detached grade - (``evaluate `` / ``run --resume``) reaches the verdict through - the evaluate-only branch, where ``early_stop`` arrives via - ``_seed_from_prior_result`` rather than from a live watcher; selecting - the gate there in a second, hand-written place is exactly how the - seeded field came to be carried but never read — re-grading an - early-stopped run under the full-run strict-AND gate flips its verdict. - - The simulation dialog path does NOT route through here, and saying "both - grading paths" without that qualifier read as though it did. It is - benign only because ``result.early_stop`` is never assigned on the - dialog path, so an armed simulation task silently gates strict-AND on a - possibly-truncated trajectory. Wiring the dialog path through this seam - means also setting ``early_stop`` there; until then the limit is stated - rather than implied. + Gate selection is FIRED-ONLY: the weighted armed gate applies IFF the + watcher actually cut the run. BOTH single-shot grading paths must call + this — the live one and the evaluate-only one — or a re-graded + early-stopped run is scored under the full-run gate and flips its verdict. + + The simulation dialog path does NOT route through here. + + Rationale: .claude/notes/orchestration.md § Gate selection is fired-only """ assert self.result is not None if self.result.early_stop is not None: - # One gate for every early-stopped run, no per-reason branches: a - # decision-budget stop is just a fail-stop whose deciding criterion - # timed out (the watcher only fires once the weighted ceiling - # proves the armed gate cannot pass). The ceiling is an upper bound - # on the authoritative armed score only because the watcher reduces - # the SAME trajectory the checker scores — it records UNRESOLVED - # tool ends exactly like the agent's EventCollector does (see - # EarlyStopWatcher._on_event_impl) — so the weighted armed gate is - # correct whether the watcher fired on a pass, a fail, or a timeout. + # ONE gate for every early-stopped run, with no per-reason branches. + # Rationale: .claude/notes/orchestration.md § Gate selection is fired-only gate_threshold = ( self.task.run_limits.stop_early_gate_threshold if self.task.run_limits is not None @@ -2432,17 +2138,15 @@ async def _evaluation_loop(self) -> bool: assert self.task.agent is not None if self.agent is None: - # Grading site 1 of 4. Evaluate-only with grading off would neither - # run an agent nor check anything — a no-op that still writes a - # task.json. Refuse instead of producing an empty row. + # Grading site 1 of 4: refuse rather than write an empty row. + # Rationale: .claude/notes/orchestration.md § The four grading sites if not self.grade: raise ValueError( "grade=False is meaningless on the evaluate-only path (no agent attached): " + "the run would neither execute nor grade." ) - # No agent attached: evaluate-only re-grade of a completed sandbox. - # (No-op tasks have a NoOpAgent here, so they take the normal path - # below.) Check the criteria directly against the sandbox. + # Evaluate-only re-grade of a completed sandbox. (No-op tasks have a + # NoOpAgent here, so they take the normal path below.) assert self.success_checker is not None assert self.result is not None unsupported = [c.type for c in self.task.success_criteria if c.requires_agent] @@ -2454,14 +2158,12 @@ async def _evaluation_loop(self) -> bool: unsupported, ) # A bare `evaluate ` has no trajectory, so one nominal - # iteration stands for the single grading pass. A re-grade seeded - # from a prior result already carries the real count (and the turns - # the trajectory-reading criteria need) — do not flatten it to 1. + # iteration stands for the grading pass. A seeded re-grade already + # carries the real count — do not flatten it. if self.prior_result is None: self.result.iteration_count = 1 - # Load reference in evaluate-only mode too: judge criteria with - # include_reference=true expect this populated even when no agent - # runs. The agent-driven branch below has the same call. + # Evaluate-only loads the reference too: include_reference judges + # expect it populated even when no agent runs. criteria_results = await self.success_checker.check_all_async( self.task.success_criteria, reference_dir=self._reference_dir, @@ -2475,17 +2177,15 @@ async def _evaluation_loop(self) -> bool: assert self.sandbox is not None and self.sandbox.sandbox_dir is not None sandbox_dir = self.sandbox.sandbox_dir - # When a SimulationConfig is present and enabled, replace the - # criteria-feedback iteration loop with a multi-turn dialog between - # the agent and an LLM-simulated user. The single-shot loop below is - # skipped entirely — simulated tasks run exactly one dialog per call. + # A simulation block replaces the single-shot loop entirely; a simulated + # task runs exactly one dialog per call. if self.task.simulation is not None and self.task.simulation.enabled: # initial_prompt is optional in simulation mode — when unset, the # simulator produces the opening utterance itself. return await self._simulation_dialog_loop(self.task.initial_prompt, sandbox_dir) - # initial_prompt is guaranteed for real agents (check_prompt_fields); a - # no-op (type: none) task runs with no prompt — send empty, NoOpAgent + # Guaranteed for real agents by check_prompt_fields; a no-op task runs + # with no prompt, which NoOpAgent # ignores it and returns an empty turn. current_prompt = self.task.initial_prompt or "" @@ -2512,16 +2212,12 @@ async def _evaluation_loop(self) -> bool: logger.debug(f"Agent response received ({len(turn_record.agent_output)} chars)") - # Facts about the RUN, recorded before the grading switch. `execute` - # withholds the verdict, never the facts: `_seed_from_prior_result` - # cannot restore a fact the execute phase never captured, so a later - # `evaluate` would inherit the wrong terminal status. - # - # Recording the fact is NOT the same as finalizing on it. `max_turns` - # exhaustion decides the status only when the criteria fail (see - # `_terminal_status`), so under `grade=False` this flag is carried into - # task.json and consumed by the detached grade, not turned into a - # terminal status here. + # Facts about the RUN, recorded BEFORE the grading switch: `execute` + # withholds the verdict, never the facts. Recording the fact is not + # finalizing on it — max_turns decides the status only when the criteria + # fail, so under grade=False this is carried into task.json for the + # detached grade rather than turned into a terminal status. + # Rationale: .claude/notes/orchestration.md § The four grading sites if turn_record.max_turns_exhausted: self.result.max_turns_exhausted = True logger.warning( @@ -2531,16 +2227,15 @@ async def _evaluation_loop(self) -> bool: # Soft cumulative-turn check (logs once; never aborts). self._check_expected_turns(iteration=iteration) - # Grading site 2 of 4. `execute` stops here: the trajectory is captured - # and persisted exactly as on a graded run, but nothing is scored. - # Returning False keeps FinalStatus off SUCCESS; run()'s status chain - # turns it into NOT_GRADED. The reference-integrity check is skipped too - # — it exists to protect a grade that is not happening. + # Grading site 2 of 4. The trajectory is captured and persisted exactly as + # on a graded run, but nothing is scored; returning False keeps FinalStatus + # off SUCCESS and the status chain turns it into NOT_GRADED. The + # reference-integrity check is skipped — it protects a grade that is not + # happening. if not self.grade: logger.info("Grading disabled (execute mode): skipping success criteria.") - # The budget gate is a run limit, not a verdict. Its only reason to - # sit after the criteria on the graded path is partial-credit - # visibility, and there is no partial credit here. + # A run limit, not a verdict: its only reason to sit after the + # criteria on the graded path is partial-credit visibility. self._check_run_limits(iteration=iteration) return False @@ -2554,9 +2249,8 @@ async def _evaluation_loop(self) -> bool: ) self.result.success_criteria_results = criteria_results - # Determine if all criteria passed their thresholds. all_passed is - # single-sourced via the model gate; passed_count/total_count are kept - # only for the human-readable log line below. + # all_passed is single-sourced via the model gate; the counts below are + # only for the log line. pairs = list(zip(criteria_results, self.task.success_criteria, strict=True)) passed_count = sum(1 for r, c in pairs if r.score >= c.pass_threshold) total_count = len(pairs) @@ -2564,18 +2258,15 @@ async def _evaluation_loop(self) -> bool: # Reuse the model method for weighted score (single source of truth) self.result.calculate_weighted_score(self.task.success_criteria) - # calculate_weighted_score just ran, so a score exists; the fallback is - # for the type, not for an unmeasured row (this branch only runs when - # grading did). + # A score exists (calculate_weighted_score just ran); the fallback is for + # the type, not for an unmeasured row. current_score = self.result.weighted_score or 0.0 # noqa: CE049 — graded here by construction logger.info(f"Success criteria: {passed_count}/{total_count} passed, weighted score: {current_score:.3f}") self._emit_criteria_event(criteria_results) - # Budget gate runs AFTER criteria so partial-credit visibility is preserved. - # (max_turns capture and the soft turn check are recorded above, before - # the grading switch — they are facts about the run, not verdicts.) + # AFTER the criteria, so partial-credit visibility is preserved. self._check_run_limits(iteration=iteration) return all_passed @@ -2620,21 +2311,11 @@ async def _run_dialog_criteria_check( """ assert self.result is not None assert self.success_checker is not None - # Grading site 3 of 4. Unreachable today — `execute` rejects simulation - # tasks at the CLI, because the dialog's turn-continuation logic reads - # criteria results to decide whether to keep talking, so an ungraded - # dialog would silently change its own stopping behavior. - # - # RAISES rather than returning an empty list, matching the evaluate-only - # path's refusal. The empty-list version described itself as a - # "defensive no-op so the gate holds", and it was neither: both callers - # go straight on to `all_criteria_passed`/`calculate_weighted_score`, - # which treat an empty criteria list as a vacuous pass/0.0 rather than - # raising, so a silent no-op here would produce a criteria-free dialog - # that scores as though nothing had been asked of the agent. If the - # simulation restriction ever lifts, that "no-op" turns every ungraded - # dialog into a silently-passing one. A loud refusal here is honest - # about the fact that this path has no ungraded semantics yet. + # Grading site 3 of 4. Unreachable today, and RAISES rather than returning + # an empty list: both callers go straight on to the gate, which treats an + # empty criteria list as a vacuous pass, so a silent no-op here would + # produce a criteria-free dialog scoring as though nothing was asked. + # Rationale: .claude/notes/orchestration.md § The four grading sites if not self.grade: raise ValueError( "Grading is disabled but the simulation dialog path requires criteria results to " @@ -2768,30 +2449,23 @@ async def _acquire_opener( async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: Path) -> bool: # noqa: PLR0915 — sequential dialog driver; decomposed into helpers, residual length is irreducible without a _DialogState rewrite (see plan Phase 5). Statement count ratcheted by CE022. """Run the task as a multi-turn dialog driven by an LLM user simulator. - This replaces the criteria-feedback iteration loop for tasks that - define a ``simulation`` block. One invocation runs exactly one - dialog trajectory (trial). Parallel trials are handled upstream by - the batch expander — this method is per-trial. - - Lifecycle: - 1. Obtain the opening user utterance. If the task pinned one via - ``initial_prompt``, use it verbatim; otherwise ask the simulator - to produce it from persona + goal (pure-simulation mode). - 2. Send the opening utterance to the agent as turn 1. - 3. After each agent reply, optionally check success criteria. - Break with ``criteria_passed`` if they pass and - ``stop_on_criteria_pass`` is set. - 4. Evaluate stop conditions (turn cap, token budget). - 5. Ask the simulator for the next user message. If the simulator - emits the stop token, break with ``stop_token``. - 6. Loop. On any simulator exception, terminate with ``error``. - 7. After the dialog ends, run a final criteria check unless one - just happened, and return pass/fail. - - Emits the same streaming events as the single-shot loop (the agent emits - its ``AgentStart``/``Turn``/``Tool``/``AgentEnd`` lifecycle; the orchestrator - adds ``CriteriaCheckEvent``) so downstream UI renderers work unchanged. - Simulator telemetry is recorded on ``self.result.simulation``. + Replaces the criteria-feedback iteration loop for tasks that define a + ``simulation`` block. One invocation runs exactly ONE dialog trajectory; + parallel trials are expanded upstream. + + Lifecycle: obtain the opening utterance (pinned ``initial_prompt``, else + ask the simulator), send it as turn 1, then after each agent reply + optionally check criteria, evaluate the stop conditions (turn cap, token + budget), and ask the simulator for the next message — breaking on a passing + criteria check with ``stop_on_criteria_pass``, on the simulator's stop + token, or on a simulator exception. A final criteria check runs afterwards + unless one just did. + + Emits the same streaming events as the single-shot loop, so downstream + renderers work unchanged. Simulator telemetry lands on + ``self.result.simulation``. + + Rationale: .claude/notes/orchestration.md § The dialog loop """ assert self.result is not None assert self.task.simulation is not None @@ -2804,17 +2478,14 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: config=sim_config, task_description=self.task.description, initial_prompt=initial_prompt, - # simulator_route is resolved independently of eval_route/ - # checker_context.api_route (see _resolve_routes) — the simulator - # is a real Claude Code CLI subprocess, same as the agent under - # test, not a checker/judge concern. + # Resolved independently of eval_route and checker_context: the + # simulator is a real CLI subprocess, not a checker concern. route=self.simulator_route, ) await simulator.start() - # stop_reason is left unset until the loop picks a concrete reason; - # the final assertion before telemetry-write catches any exit path - # that forgot to set it, instead of silently defaulting. + # Left UNSET until the loop picks a concrete reason, so the assertion + # before the telemetry write catches an exit path that forgot one. stop_reason: DialogStopReason | None = None simulator_input_tokens = 0 simulator_output_tokens = 0 @@ -2828,8 +2499,7 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: # UserMessage captured for the upcoming agent call; prepended to the # next turn_record.messages. None outside simulation paths. pending_user_turn: UserMessage | None = None - # The RESOLVED simulator model (backend-translated), not the configured id — - # it labels each simulator UserMessage and is persisted on the telemetry so + # The RESOLVED model (backend-translated), not the configured id, so # simulator cost prices from the model that actually served the call. sim_model_id = simulator.model # Track whether we entered the agent-call loop — used by the finally @@ -2856,24 +2526,20 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: current_prompt = initial_prompt pending_user_turn = UserMessage(text=initial_prompt) self._log_conversation("USER", 1, current_prompt, metadata="pinned initial_prompt") - # Parallel history of clean (user, agent) pairs for the simulator. - # This intentionally excludes the working-directory prefix that gets - # prepended to agent prompts — the simulator should see the user's - # actual utterances, not framework wrapping. + # Clean (user, agent) pairs, deliberately without the + # working-directory prefix the framework prepends to agent prompts. dialog_pairs: list[tuple[str, str]] = [] check_every_turn = sim_config.check_criteria in ("every_turn", "both") - # Dialog-wide running total of each judge's token usage. The per-turn - # check replaces ``success_criteria_results`` wholesale, so we fold - # each turn's judge slice into this accumulator (see - # ``_accumulate_judge_usage``) to avoid dropping earlier judge calls. + # The per-turn check replaces `success_criteria_results` WHOLESALE, so + # each turn's judge slice folds into this or earlier calls are lost. # Keyed by (position, criterion_type) — a stable criterion identity. judge_usage_accum: dict[tuple[int, str], TokenUsage] = {} - # turns_completed advances in lockstep with the agent's _iteration: - # one _communicate_with_retry call per sim turn keeps partials - # and the successful retry on the same iteration number. + # In lockstep with the agent's _iteration — one + # _communicate_with_retry per sim turn — so a partial turn and its + # successful retry share an iteration number. while True: turns_completed += 1 self.result.iteration_count = turns_completed @@ -2931,10 +2597,8 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: except BudgetExceededError: stop_reason = DialogStopReason.RUN_LIMIT_EXCEEDED if not criteria_checked_this_turn: - # Run for partial-credit side effects only (stores results - # on self.result + recomputes score). This fallback site - # deliberately neither sets all_passed nor emits an event, - # so the return value is unused. + # Partial-credit side effects only; this fallback site + # neither sets all_passed nor emits an event. await self._run_dialog_criteria_check(judge_usage_accum) raise @@ -2949,9 +2613,8 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: stop_reason = stop_decision.reason break - # Soft cumulative-turn check (logs once; never aborts the dialog). - # Runs BEFORE the max_turns break so a single turn that trips both - # the hard cap and the soft target still emits the expected_turns + # Soft check (logs once, never aborts). BEFORE the max_turns + # break, so a turn tripping both still emits the expected_turns # warning before the dialog terminates. self._check_expected_turns(iteration=turns_completed) @@ -3019,16 +2682,13 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: ) return all_passed finally: - # If an agent turn was attempted but crashed before attaching pending_user_turn to a turn_record, - # append it as a standalone entry so its telemetry is not lost. (If the simulator's opener - # had stop_requested, we never entered the agent loop, so don't record it.) + # An agent turn attempted but crashed before attaching its user turn + # to a record: append it standalone or its telemetry is lost. if agent_turn_attempted and pending_user_turn is not None and self.result is not None: - # TurnRecord.duration_seconds defaults to 0.0 and no caller passes - # it, so this turn used to report 0s for a simulator call that - # really took seconds — halving `avg_turn` in the HTML report for - # every simulation task. A pinned opener has no simulator call - # behind it (generation_duration_ms is None by design), so 0.0 - # there is the real duration, not a placeholder. + # `duration_seconds` defaults to 0.0 and no caller passes it, so + # this used to report 0 s for a simulator call that took seconds, + # halving `avg_turn` for every simulation task. A PINNED opener has + # no simulator call behind it, so 0.0 there is real. sim_ms = pending_user_turn.generation_duration_ms standalone_turn = TurnRecord( iteration=turns_completed, @@ -3039,12 +2699,9 @@ async def _simulation_dialog_loop(self, initial_prompt: str | None, sandbox_dir: ) self.result.iterations.append(standalone_turn) - # When the dialog bails out via exception (TurnTimeoutError, - # TaskTimeoutError, etc.) before reaching the explicit telemetry - # write above, the happy-path write never happens — record partial - # telemetry here so analytics still see the run. ``stop_reason`` - # being None at this point means "exit was not an in-band stop - # decision" (i.e., exception-driven), which we classify as ERROR. + # The happy-path telemetry write never happens when the dialog bails + # out through an exception. A None `stop_reason` here means the exit + # was not an in-band stop decision, so it is classified ERROR. if self.result is not None and self.result.simulation is None: self.result.simulation = self._build_simulation_telemetry( n_trials=sim_config.n_trials, @@ -3158,13 +2815,11 @@ async def _run_command_list( limit=self._POST_RUN_STREAM_LIMIT, ) # No `# nosec` here: bandit does not flag - # asyncio.create_subprocess_shell at all (B602 is - # subprocess.Popen(shell=True)), so the suppression this line - # used to carry was inert -- and an inert id silently - # pre-suppresses a real finding if a flagged construct is ever - # added here. The shell IS intentional: pre/post_run commands are - # authored in the task YAML, which is already a trusted artifact - # (it can run anything via a run_command criterion). + # asyncio.create_subprocess_shell at all, so the suppression this + # line used to carry was inert — and an inert id pre-suppresses a + # real finding if a flagged construct is ever added. The shell IS + # intentional: pre/post_run commands are authored in the task YAML, + # already a trusted artifact. stdout_chunks: list[str] = [] stderr_chunks: list[str] = [] @@ -3232,11 +2887,9 @@ async def _run_command_list( raise RuntimeError(f"{human} command failed: {cmd.command!r}") from e logger.warning("%s command '%s' failed: %s", human, cmd.command, e) finally: - # Every exit from this iteration -- normal, `continue` from the - # timeout branch, or a raised RuntimeError -- must release the - # two pipes this command opened. Without it they survive until - # GC, which on Windows reports as an unraisable ResourceWarning - # against an unrelated test. See _close_subprocess_transport. + # EVERY exit must release the two pipes this command opened, or + # they survive until GC and surface on Windows as an unraisable + # ResourceWarning against an unrelated test. _close_subprocess_transport(proc) async def _run_pre_run_commands(self) -> None: @@ -3271,24 +2924,16 @@ async def _run_post_run_commands(self) -> None: def _skip_pre_run_for_adopted(self, commands: Sequence[PreRunCommand]) -> bool: """True when ``pre_run`` must not run against an adopted sandbox. - ``adopt()`` guarantees it materializes nothing into the workspace, but - that guarantee is only as strong as its weakest caller: ``run()`` invokes - the hooks unconditionally, and those commands run with - ``cwd = sandbox_dir``. Several in-tree tasks stage fixtures there - (``cp -a /app/[!.]* "$PWD/"``), so re-running them during a detached - grade would overwrite the agent's deliverables *before* the criteria read - them — silently changing the verdict and destroying preserved artifacts. - - ``pre_run`` prepares the environment the AGENT needs, so it belongs to - the execute phase; the prior run already ran it, and its recorded results - are carried over by ``_seed_from_prior_result``. Its ``post_run`` sibling - is the opposite case — see ``_skip_post_run``. - - ``commands`` is passed in rather than looked up from a phase name. The - lookup was a stringly-typed branch whose only consumer was a log line, so - a typo (``"prerun"``) would silently report the wrong count with - nothing — not pyright, not ruff — able to see it, in the same module - CE050 was written to protect from exactly that. + ``pre_run`` prepares the environment the AGENT needs, so it belongs to the + execute phase; the prior run already ran it, and its recorded results are + carried over by the seeding. Re-running it would overwrite the agent's + deliverables before the criteria read them. + + ``commands`` is passed in rather than looked up from a phase NAME: that + lookup was a stringly-typed branch whose only consumer was a log line, so a + typo would silently report the wrong count with nothing able to see it. + + Rationale: .claude/notes/isolation.md § Why pre_run and post_run each run exactly once """ if self.sandbox is None or not self.sandbox.was_adopted: return False @@ -3303,29 +2948,15 @@ def _skip_pre_run_for_adopted(self, commands: Sequence[PreRunCommand]) -> bool: def _skip_post_run(self) -> bool: """True when ``post_run`` must not run in THIS phase. - ``post_run`` runs after the verdict is finalized and is free to mutate - the workspace (``rm -rf node_modules`` is the archetype), so it belongs - to whichever phase GRADES — never to the phase that merely executes. - Running it under ``execute`` inverted its own contract and broke - round-trip equivalence: the criteria had not read the tree yet, so - ``execute`` + ``evaluate`` graded a workspace ``post_run`` had already - modified and could return a different verdict than a single ``run`` for - the identical trajectory. - - Two skips, and they are NOT the same condition: - - * ``grade=False`` (``execute``) — deferred, not cancelled. The commands - run later, when ``evaluate`` / ``run --resume`` grades the row. A run - that is never graded therefore never tidies its sandbox; that is the - accepted cost of keeping the verdict honest. - * an adopted sandbox whose prior row ALREADY recorded ``post_run`` - results — that phase graded, so the commands have run once. Nothing - declares them idempotent, and ``_seed_from_prior_result`` has already - copied those results onto this row, so a second pass would both re-run - the side effects and double-count the records. - - The two combined mean each command runs exactly once, in the grading - phase, whichever command that turns out to be. + ``post_run`` is free to mutate the workspace, so it belongs to whichever + phase GRADES — never to the phase that merely executes. + + TWO SKIPS, and they are NOT the same condition: ``grade=False`` DEFERS the + commands to the grading pass, while an adopted sandbox whose prior row + already recorded ``post_run`` results has run them once already. Together + they mean each command runs exactly once, in the grading phase. + + Rationale: .claude/notes/isolation.md § Why pre_run and post_run each run exactly once """ assert self.result is not None commands = self.task.post_run @@ -3367,21 +2998,18 @@ async def _cleanup(self) -> None: except Exception as e: logger.warning(f"Failed to stop agent: {e}") - # Drop the staged reference copy. Deliberately NOT preserved into - # run_dir/artifacts: run directories get archived, uploaded, and shared, - # and the reference solution must not ride along. + # Deliberately NOT preserved into run_dir/artifacts: run directories get + # archived, uploaded and shared, and the reference must not ride along. # - # Keyed on _reference_staging_root, recorded before the copy — NOT on - # _reference_dir.parent, which is only set once the copy succeeds and so - # would leak a half-written reference when copytree raises. The field is - # None under docker, where the reference is the host-owned bind mount: - # that one is not ours to delete, and rmtree'ing its parent would take - # /work with it. + # Keyed on the staging root recorded BEFORE the copy — NOT on + # `_reference_dir.parent`, which is only set once the copy succeeds. The + # field is None under docker, where the reference is the host-owned bind + # mount: that one is NOT OURS TO DELETE, and rmtree'ing its parent would + # take `/work` with it. # - # rmtree_restrictive, not rmtree(ignore_errors=True): a run killed - # mid-turn leaves the tree at mode 000, where scandir raises - # PermissionError and plain rmtree silently declines — orphaning a - # tempdir that holds the reference solution, with no log line. + # rmtree_restrictive, because a run killed mid-turn leaves the tree at + # mode 000, where plain rmtree silently declines. + # Rationale: .claude/notes/persistence.md § rmtree_restrictive staging_root = self._reference_staging_root self._reference_dir = None self._reference_staging_root = None @@ -3391,16 +3019,14 @@ async def _cleanup(self) -> None: except Exception as e: logger.warning("Failed to remove staged reference dir %s: %s", staging_root, e) - # Cleanup sandbox. Preservation and cleanup() are SIBLING try blocks: - # a preservation failure (e.g. disk full during preserve_to) must never - # skip cleanup(), or the tempdir leaks. + # SIBLING try blocks: a preservation failure must never skip cleanup(), + # or the tempdir leaks. if self.sandbox: try: if self.workspace_dir is not None and self.result: - # Docker WORKDIR alignment: the agent ran in-place at the image - # WORKDIR; copy that workspace out to run_dir/artifacts/ - # (capture_to grants cross-uid read on the COPY and tolerates - # dangling symlinks). Takes precedence over preservation_mode. + # Docker WORKDIR alignment: the agent ran in-place at the + # image WORKDIR, so copy that workspace out. Takes precedence + # over preservation_mode. artifacts_dir = self.run_dir / "artifacts" preserved_path = await asyncio.to_thread(self.sandbox.capture_to, artifacts_dir) self.result.sandbox_path = str(preserved_path) @@ -3412,11 +3038,9 @@ async def _cleanup(self) -> None: self.result.sandbox_path = str(preserved_path) logger.info(f"Sandbox preserved to: {preserved_path}") elif self.preservation_mode == PreservationMode.DIRECT_WRITE and self.result: - # Sandbox already lives in run_dir/artifacts — nothing to move. - # Set sandbox_path first, then grant a+rX (a fallible chmod) so - # artifacts written by a root-owned docker container stay - # traversable across the host uid boundary (MOVE_ON_WRITE gets - # this via preserve_to; DIRECT_WRITE skips it, so apply it here). + # Already in run_dir/artifacts — nothing to move. Set the path + # first, then grant a+rX so artifacts written by a root-owned + # container stay traversable across the host uid boundary. self.result.sandbox_path = str(self.sandbox.sandbox_dir) await asyncio.to_thread(self.sandbox.grant_read_access) logger.info(f"Sandbox preserved (in-place): {self.sandbox.sandbox_dir}") @@ -3430,10 +3054,9 @@ async def _cleanup(self) -> None: except Exception as e: logger.warning(f"Failed to preserve sandbox (continuing with cleanup): {e}") if self.result and self.preservation_mode == PreservationMode.MOVE_ON_WRITE: - # The artifacts were never moved — don't point at the tempdir - # cleanup() is about to delete. DIRECT_WRITE keeps its path: - # that sandbox is persistent (cleanup() is a no-op) and the - # artifacts still exist even if the a+rX chmod failed. + # Never moved — do not point at the tempdir cleanup() is about + # to delete. DIRECT_WRITE keeps its path: that sandbox is + # persistent and the artifacts exist even if the chmod failed. self.result.sandbox_path = None try: diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index f550e7fa3..3d65f466f 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 60_642 +_ESSAY_BASELINE_WORDS = 46_898 _SRC = Path("src/coder_eval") @@ -52,7 +52,9 @@ _POINTER = re.compile(r"Rationale:\s*(\S+\.md)\s*§\s*(.+?)\s*$") -_HEADING = re.compile(r"^##\s+(.+?)\s*$") +# ``##`` or ``###``: a pointer may target a SUBSECTION, so appending to an existing +# section (the single-home rule) does not force the pointer up to the parent heading. +_HEADING = re.compile(r"^#{2,3}\s+(.+?)\s*$") # Executable directives that live in comments and therefore never enter the AST. _DIRECTIVE = re.compile(r"^#\s*(?:noqa\b|type:\s*ignore\b|pyright:|nosec\b|pragma:|fmt:)") diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py index e527ffe98..9ead3a6b9 100644 --- a/tests/test_prose_budget.py +++ b/tests/test_prose_budget.py @@ -181,6 +181,17 @@ def test_a_heading_with_backticks_resolves(self, tmp_path: Path) -> None: root = self._root(tmp_path, source, "# Timing\n\n## `--resume` is command-relative\n") assert prose_budget.check_pointers(root) == [] + def test_a_subsection_heading_resolves(self, tmp_path: Path) -> None: + """A pointer may target a ``###``: the single-home rule appends to a section.""" + source = '"""Rationale: .claude/notes/timing.md § The clamp"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n\n### The clamp\n\nWhy.\n") + assert prose_budget.check_pointers(root) == [] + + def test_a_level_one_heading_does_not_resolve(self, tmp_path: Path) -> None: + source = '"""Rationale: .claude/notes/timing.md § Timing"""\n' + root = self._root(tmp_path, source, "# Timing\n\n## close_window\n") + assert len(prose_budget.check_pointers(root)) == 1 + def test_a_heading_with_a_colon_resolves(self, tmp_path: Path) -> None: source = '"""Rationale: .claude/notes/timing.md § Execute vs. run: the grading switch"""\n' root = self._root(tmp_path, source, "# Timing\n\n## Execute vs. run: the grading switch\n") From fcad6ae58a08bf06be7df993437ee155b5464556 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 19:50:04 -0700 Subject: [PATCH 05/19] =?UTF-8?q?docs:=205/7=20=E2=80=94=20move=20isolatio?= =?UTF-8?q?n,=20sandbox=20and=20CLI=20rationale=20into=20.claude/notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 13,289 words of essay-shaped prose across 11 files down to 2,960, with no code change (AST + directive multiset verified against the phase-start SHA). docker_runner.py 4,993 -> ~1,100 and sandbox.py 3,259 -> ~590 are the bulk. The repo's largest comment block, 34 lines on --cap-drop, is now the four COUNTERPART lines that state the coupling plus a pointer; the capability argument it carried is rationale and moved. isolation.md gains Capability drops and the anti-cheat window, What crosses into the container, Trusting what the container sends back, and The sandbox the criteria run in. Detached grading from the CLI joins the existing detached-grading section rather than rivalling it, and the in-container driver rewrite stays in orchestration.md, which already owned it. The Phase-1 relocated bullet covered nine of these topics already, so the single-home rule needed excision as well as writing: seven spans moved out of it, each left as a cross-reference to the section that now owns them. The --cap-drop/FOWNER known gap stays in docs/DOCKER_ISOLATION.md; the notes link it and say so. All eight --help outputs are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/isolation.md | 543 ++++++++++++- .claude/notes/orchestration.md | 94 +++ src/coder_eval/cli/__init__.py | 6 +- src/coder_eval/cli/aggregate_command.py | 14 +- src/coder_eval/cli/evaluate_command.py | 166 ++-- src/coder_eval/cli/evaluate_target.py | 18 +- src/coder_eval/cli/execute_command.py | 22 +- src/coder_eval/cli/plan_command.py | 21 +- src/coder_eval/cli/run_command.py | 285 +++---- src/coder_eval/cli/run_helpers.py | 27 +- .../cli/run_task_internal_command.py | 175 ++-- src/coder_eval/isolation/docker_runner.py | 756 ++++++------------ src/coder_eval/sandbox.py | 481 ++++------- tests/lint/prose_budget.py | 2 +- 14 files changed, 1262 insertions(+), 1348 deletions(-) diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index 827ac490e..8b7d9dfa6 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -4,7 +4,7 @@ ## Detached grading and `Sandbox.adopt` -- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses is ONE parameter (`grade_in_place`, resolved by `_gate_scope_for_grade`), not two: it shipped beside an `include_setup_phase` every caller passed as its exact complement, and a future caller setting one and forgetting the other would silently drop half of a SECURITY gate. Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and `_assert_regrade_honored` refuses a returned row whose `started_at` moved, because an image predating this change ignores the unknown `regrade` key and RUNS THE AGENT, which the host would otherwise fold back as the recorded row's verdict (the exact sibling of `_assert_grade_honored`, one release later). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. `grant_container_access` now RETURNS what it widened and `run()` restores it in the same `finally`: the two staging dirs are disposable, but the graded workspace is the caller's tree — an operator-supplied `--workspace` was left world-writable permanently. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: every container is launched `TELEMETRY_ENABLED=false` under the invariant "container silent, host emits once", and the grading path had inherited only the silent half. The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `git clone` now passes `--` before the URL (argv position 2, so a value beginning with `-` was parsed as an option). `Sandbox.resolve_files` is containment-checked for the same reason: criterion paths were the one task-authored path skipping `_resolve_within_sandbox`, and `Path(root) / '/etc/passwd'` is `/etc/passwd`. An escaping LITERAL now raises `CheckerMisuseError` rather than resolving to `[]`: returning no match books an eval-CONFIG error as an agent failure — a gating 0.0 reading "file does not exist" for a file that plainly does exist and that no agent behaviour could place inside the sandbox (CE039's exact distinction). `tasks/byod_smoke_test.yaml` was broken that way for several commits, checking `/opt/byod_marker` baked into the BYOD image, with only a task-log warning to show for it; it now asserts on the container with `run_command: test -f …`, which is what a claim about the IMAGE rather than about the agent's workspace should look like. The guard keys on the escaping path EXISTING, so a merely-absent absolute path stays an ordinary failing verdict, and the GLOB branch still warns-and-drops, since filtering some matches out of a search is its normal behaviour. A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (the one legitimate rewrite — we are already inside the container the driver asked for), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. +- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses follows from ONE parameter (see § Detached grading from the CLI). Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and the dispatch is guarded against an image that ignores the `regrade` key (see § The two honored-request guards). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: the grading path had inherited only the silent half of the container-silent invariant (§ Environment forwarding). The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `Sandbox.resolve_files` is containment-checked for the same reason (see § Criterion paths are contained, quietly). A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (see .claude/notes/orchestration.md § The in-container driver rewrite), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. ### What a graded row inherits, and what it does not @@ -57,6 +57,70 @@ idempotent. A run that is never graded therefore never tidies its sandbox — that is the accepted cost of keeping the verdict honest. +### Detached grading from the CLI + +`evaluate` decides its two shapes with a **pure** resolver over one filename probe, so a +plain work directory that merely happens to hold a file called `task.json` reads as a run +directory. The resolver cannot tell a real record from a namesake, so the caller re-reads +the record and falls back to WORK_DIR when it does not parse — without that fallback the +pre-existing `evaluate ` form aborted on a pydantic wall the user could +escape only by renaming their own file. The two-argument form over a *run* directory is +allowed on purpose: it is "iterate on my criteria against an expensive run I already paid +for", which is most of the reason `execute` and `evaluate` are separate commands. + +The grade is dispatched by branching on `prior is not None` directly, and the sandbox is +built inside the branch that uses it, so neither value is `Optional` at its use site. +Building a host `sandbox_config` first called `grading_sandbox_config`, whose whole job is +to REFUSE a `driver: docker` row — so the refusal fired before the branch that no longer +needs it, and no docker row could be graded properly at all. (A bare `assert` plus a +comment asserting an invariant the type checker can hold structurally is the weakest +narrowing available; it is stripped entirely under `-O`.) + +The orchestrator-direct branch — fresh work-dir grading, or `--copy` — is the sibling of +the delegating one and needs the same two things stated only once for the delegating path: +a criteria-free task must be refused rather than finalized SUCCESS at `weighted_score` +0.0, and a host-graded row must carry the `graded_on_host` stamp. CLAUDE.md, the user +guide and CE051's own `noqa` all state that stamp as unconditional, so +`evaluate --copy --allow-host-grading` was writing an unstamped host verdict +nothing downstream could tell apart from a container-graded one. + +Errors are rendered unwrapped, like the sibling handlers: the delegating branch raises for +a missing or unresolvable task file and for a failed grading container, and both messages +carry the operator's next step, which arrived as the tail of a stack trace instead. The +grading-crash check must run BEFORE the criteria-count guard: `Orchestrator.run()` converts +internal failures into a populated ERROR result with an EMPTY criteria list, so the count +check fired first and the user was told only "Result count mismatch: got 0, expected 2" — +the real error never printed, and the "still re-gradeable" notice unreachable on exactly +the path it was written for. On that path the pre-grade record is restored: leaving ERROR +on disk replaces a re-gradeable NOT_GRADED row with one BOTH commands treat as permanently +complete. + +`_write_back` is gated on RUN_DIR mode, not merely on `prior is not None`. `--format +harbor` seeds a SYNTHETIC prior on the WORK_DIR shape from the supplied `--trajectory`; +that target is not a run directory and has no `task.execute.json` sibling to preserve, so +writing there planted a spurious `task.json` into the Harbor-synced workdir and wedged a +later `evaluate` on it into RUN_DIR mode. The write refuses to follow a symlink — a run +directory is a shareable artifact, so its `task.json` is untrusted input and following a +link turns `evaluate ` into an arbitrary-file-overwrite primitive on the grader's +host — and it is atomic, matching the orchestrator's own writer, because a torn write makes +the row parse as malformed, which a later `--resume` reads as "not complete" and re-pays +for the agent. + +The pre-grade snapshot is taken BEFORE anything grades. Taking it inside `_write_back` +captured an ALREADY-GRADED record whenever `--run-dir` pointed at the target run dir (the +orchestrator writes there first), destroying the evidence the copy exists to preserve. + +The recorded-shell gate takes ONE lever — whether the grade happens in place — passed once +and derived through the same function `run_evaluation` uses (`_gate_scope_for_grade`). A +second copy of the rule keeps answering the old question the moment the default moves: the +lever shipped beside an `include_setup_phase` every caller passed as its exact complement, +and a caller setting one and forgetting the other would silently drop half of a SECURITY +gate. In place, the grade may dispatch a +CONTAINER built from the recorded sandbox block, a wider capability than any recorded shell +string; on `--copy` instead, `pre_run` and the sandbox's own installers, neither of which an +adopted workspace reaches. `post_run` is in NEITHER set — it belongs to the grading phase +and runs on both paths, so it is scanned unconditionally. + ## Grading a docker row inside a container A `driver: docker` row is graded IN a container of its own image, which is the only place @@ -136,3 +200,480 @@ writing and follows one, which is an arbitrary-file-overwrite primitive in a run the grader did not create. `docker.log` is renamed for the PHASE, because on the resume path that name is already taken by the executed container's log. +## Capability drops and the anti-cheat window + +The container runs as root, and root bypasses ordinary file permissions via +`CAP_DAC_OVERRIDE` / `CAP_DAC_READ_SEARCH`. Without dropping both, the mode-000 window +`fs_permissions.py` puts around every agent turn is a NO-OP on native Linux — verified: a +`chmod 000` directory is still readable by root in a default container, and Permission +denied once the two caps are dropped. It appears to work on macOS Docker Desktop even +without the drop, because virtiofs enforces host-side; that is a platform accident, not +the rule. Nothing in a sandbox legitimately needs to override discretionary access +control, so dropping these costs the task nothing. + +**FOWNER and CHOWN are deliberately NOT dropped**, though an earlier revision dropped +them. That decision, the deliberate re-chmod it leaves open, and the measured evidence for +both are owned by +[docs/DOCKER_ISOLATION.md § Architecture](../../docs/DOCKER_ISOLATION.md#architecture); +they are not restated here. + +The window is a real control only inside a container, where the filesystem is private to +one task. On the host (`driver: tempdir`) it is a deliberate no-op: parallel tasks in one +batch share the checked-out `tasks//` tree, so chmod-ing it is a cross-task side +effect on the user's own working copy for no isolation benefit — there is no boundary to +enforce when the agent is just another process with the same uid. The predicate is the +`CODER_EVAL_IN_CONTAINER` env var, NOT `config.driver`, because the in-container entry +point rewrites `driver: docker` to `tempdir` before constructing the orchestrator, so a +driver-keyed predicate would read "tempdir" inside the container and disable the window on +exactly the path that needs it. + +### grant_container_access + +Dropping DAC_OVERRIDE revokes root's bypass on EVERY framework-owned bind mount, not just +the reference. The container runs as root but OWNS none of them: on native Linux the mount +preserves the uid that ran `coder-eval`, so every access root makes is an "other" access +that only ever succeeded via the capability. Dropping it therefore also revoked the +container's ability to write its own output — the in-container orchestrator died on the +very first `open('/work/output/task.log', 'w')` with EACCES, taking every `driver: docker` +task with it. + +Widening the *host* side restores that access through the `other` bits instead of through a +capability, which is what keeps the drop affordable. Semantics match `chmod -R o+rwX` +(`o+rX` when `writable=False`): the `X` form adds execute only to directories and to files +that are already executable, so a copied hook script stays runnable and a data file does +not silently become one. Symlinks are skipped via `lstat` — `chmod` follows them, and for +the `~/.claude` copy the target can be an arbitrary path outside the staging tree. + +`writable=False` is not cosmetic. The container only ever *reads* and `chmod`s the +reference copy, so withholding the write bit keeps `_verify_reference_integrity` from being +the sole guard against tampering during the gaps between windows. The task-dir copy is +read-only for the same reason: criteria read fixtures there, nothing legitimately writes +them, and withholding `o+w` keeps an agent from rewriting the expectations it is graded +against. The `~/.claude` copy IS writable — the CLI rewrites settings and state in place, +`copytree` preserves host modes, and `~/.claude` is routinely 0700 with 0600 files, which +without DAC_OVERRIDE is unreadable to the container, so the agent cannot authenticate. + +The function returns `(path, original_mode)` for every entry it changed so a caller that +widened a tree it does not own can put it back. The framework-created staging dirs are +disposable and ignore it; the graded workspace is not. That workspace is the one mount +whose files the harness did NOT create, so the owner bits cannot be assumed — it happens to +work when the executing container (as root) wrote the tree, which is what makes the broken +case expensive: an operator-supplied `--workspace`, or artifacts re-created host-side, are +owned by the host uid, and container root without DAC_OVERRIDE reaches them only through +`other`. Criteria then fail EACCES and book a gating 0.0 that reads as an agent failure — +the CE039 shape this feature exists to end. Restoring it matters because the tree survives +the dispatch: an operator-supplied `--workspace` left world-writable forever is a real, +permanent exposure on a shared host. + +No-op on Windows, where POSIX mode bits are not the access-control mechanism. + +### Why the framework mounts are writable copies + +The in-container orchestrator holds the reference and task directories at mode 000 for the +duration of every agent turn, and neither obvious alternative works: + +* `:ro` rejects the chmod outright — verified: `chmod: /ro: Read-only file system`. No + window is expressible at all. +* Read-write *without* a copy chmods the operator's REAL `tasks/` tree. Verified: the host + directory came back 0600 and even the harness's own cleanup then failed with `Permission + denied`. A crashed run would strand a checkout at 000. + +Shielding the whole task tree, rather than masking just `reference.directory` with a tmpfs +as the old symmetric `-v ::ro` mount required, also closes a +leak the mask could not: a task at `tasks/foo.yaml` has parent `tasks/`, so the old mount +exposed every SIBLING task's directory, reference solutions included. + +Symmetry was never load-bearing. The container is told where the task dir is via +`--task-dir`, and that path only seeds `TASK_DIR` — it is never re-read. `TASK_DIR` is +exposed solely to criterion subprocesses, so the agent has no legitimate need for this tree +mid-turn. The reference gets its own dedicated mount at `/work/references` and an empty +tmpfs is layered over its path inside the task-dir mount; docker applies mounts by +target-path depth, so the deeper tmpfs wins regardless of argv order. The original is +deliberately NOT auto-mounted at its host path — that would re-expose it through `$TASK_DIR`, +the exact hole the mask closes. + +The staging tree is removed with `rmtree_restrictive`, not `rmtree(ignore_errors=True)`: it +holds the references copy, which stays at mode 000 for the whole of every turn, and a +container killed mid-turn never restores it. `scandir` on a 000 directory raises +PermissionError, which `ignore_errors` swallows — orphaning a tempdir that holds the +reference solution. + +The graded workspace is the exception that is NOT copied: criteria legitimately mutate what +they grade (a `run_command` that compiles, a `post_run` that cleans up), and copying is what +the host path proved wrong — for the reason `Sandbox.adopt` exists at all, above. + +## What crosses into the container + +### The lean ~/.claude copy + +The host's `~/.claude` is copied to a throwaway tmp dir and that copy is mounted read-WRITE +at the host's own `~/.claude` path (HOME is forwarded, so the path is symmetric), so the +in-container CLI can write settings, session ephemera and cache without ever touching the +host's real one. + +The container needs only auth, settings and plugins; everything else under `~/.claude` is +heavy, transient, or host-local state it never reads. On a real host the skip list is the +difference between a ~300 MB copy and a few MB — `security/` alone is often hundreds of MB, +and `projects/`, `cache/`, `file-history/`, `backups/`, `sessions/`, `telemetry/`, +`downloads/` and `shell-snapshots/` all accumulate without bound. The last group is volatile +per-session churn the *running* CLI rewrites continuously (this harness itself runs inside +Claude Code, so the live host tree mutates while it is copied): dropping it also shrinks the +window for a mid-walk vanish/rewrite race under `--max-parallel`, whose residual is covered +by a bounded retry. Patterns match by basename at every level, so the list is a DENYLIST: +anything not named — `settings.json`, `.credentials.json`, `plugins/` — is copied through. + +Symlinks are copied AS symlinks. A plugin marketplace cache can contain a self-referential +link (`plugins/uipath -> ..`) that makes a following walk recurse infinitely and abort the +copy; copying them verbatim is correct and loop-proof. + +### Environment forwarding + +Forwarding is an explicit allowlist, extendable per task. `--env VAR` (name-only) tells +docker to copy the value from the current environment at run time, so secrets stay out of +the rendered argv that gets logged. + +The run's backend rides that same path: `API_BACKEND` is allowlisted and `--backend` syncs +it into `os.environ` at the CLI, so it forwards like any other var. A flag that only mutated +in-process `Settings` would be dropped at the container boundary and the in-container +`Settings` would silently default to DIRECT, downgrading the judge's and the agent's route. + +`LITELLM_BASE_URL` points at a proxy on the HOST, which a bridge-network container cannot +reach on loopback, so localhost is rewritten to the docker host alias and that alias is +published with `--add-host` for Linux parity (automatic on Docker Desktop). It is only a +URL, so an explicit `--env VAR=value` is safe to render in the logged argv — unlike the auth +token, which stays name-only. `LITELLM_COST_LOG` is the proxy's per-call cost log, written +on the host and READ by the in-container actual-cost join, so its directory is bind-mounted +at the same host path read-only (the host proxy is the sole writer) and the resolved +ABSOLUTE path is forwarded. Both are skipped when the container has no network. + +Telemetry is hard-disabled inside the container with an explicit value, not name-only, so it +overrides any inherited or baked-in setting. The app ships a baked-in default connection +string, so without this the in-container orchestrator emits `CoderEval.Task.End` and the +host re-emits the same event after parsing the result, double-counting every docker task. +The invariant is "container silent, host emits once". + +`IN_CONTAINER_ENV` tells in-container agents the harness already provides OS-level +isolation. The Codex agent reads it to fall back to its full-access sandbox: Codex's +Landlock-backed sandboxes cannot initialize inside a container and otherwise fail writes +silently. + +### The entrypoint and the image contract + +The framework entrypoint is pinned at run time rather than trusted from the image, so the +orchestrator launch survives a task Dockerfile that sets its own ENTRYPOINT/CMD or clears it +with `ENTRYPOINT []`. `--entrypoint` resets the image CMD, which is fine — the run command is +passed explicitly after the image and forwarded to the entrypoint. A task Dockerfile must +start `FROM coder-eval-agent:` so the runtime is present; the built image is +asserted to carry the `org.coder-eval.version` label, because a bare `FROM ubuntu` builds +fine and then dies at `docker run` with a cryptic `exec: ".../coder_eval_entrypoint.sh": no +such file`. + +### Extra mounts and reserved destinations + +An `extra_mounts` entry is `src:dst[:mode]`, and both halves are task-authored strings. A +leading Windows drive letter is split off first so the colon in `C:\foo` is not misread as +the separator (the container side is always POSIX, so only the source can carry one); a bare +`C:` is deliberately not matched, being malformed. After variable expansion the spec is +re-checked for a `:`, because a variable whose value carries one would add fields to the +rebuilt spec and silently move the destination or widen the mode. The mode defaults to `ro` +when omitted: mounting host paths RW by default is the wrong sandbox stance, and the few RW +cases are better stated than implied by silence. + +Destinations that would shadow framework-owned mounts are rejected, in expanded form, +against the same reserved set the workspace-dir validator uses. `/work/…` substrings are +caught too — `/work/foo` lands underneath the staging dir and shadows the input/output tree. + +Auto-mounted sources that look like credential directories get a loud warning rather than a +refusal. Task YAMLs typically come from in-house suite authors, but `plugin.path`, +`reference.directory` and `template_sources` are user-controlled strings and a typo, or a +hostile suite, can silently expose `~/.ssh`; legitimate uses exist (a task that really does +want `~/.aws/config`), so the warning surfaces the surprise instead of blocking it. + +The container gets a stable, UNIQUE name so cancellation can target it. PID alone collides +under `--max-parallel > 1`, so a uuid suffix and the replicate index disambiguate. Task ids +are sanitized and truncated to 80 characters — dataset row ids like `suite/row` break docker +name validation, and an earlier 30-character cap collided visibly on long shared prefixes. +The same sanitizing applies to `mkdtemp` prefixes (the `/` would need a parent dir that does +not exist) and to the deterministic build tag, which must be lowercase. + +### The heartbeat watchdog is armed only inside a container + +The host touches a heartbeat file in the output dir every couple of seconds while alive, and +the in-container watchdog exits if it goes stale. It is the only defence against the host +being SIGKILL'd — Claude Code's Escape, say — before the asyncio cleanup that would +`docker kill` the container, which would otherwise keep burning LLM budget orphaned. + +The thread's whole authority is `os._exit(137)` on the process it runs in, and the only +process that may be reaped that way is the container's disposable main. Outside a container +it can do nothing but harm, and it did: a test invoked the command in-process (legitimately +— the command must refuse a malformed `context.json`, and proving that means calling it) and +the pytest worker inherited the thread, which found no heartbeat and 40s later exited the +worker mid-way through an unrelated test file. It named a different test on each run and on +each platform, carried no traceback, and took that worker's coverage data with it — so the +gate reported "65.13 < 80.00", naming neither the test nor the cause. It is therefore gated +on `CODER_EVAL_IN_CONTAINER`, not on `driver`, for the same reason the permission window is. + +`os._exit` skips atexit and IO flushing, so the error line explaining the suicide would +routinely be lost, making a genuine stale-heartbeat exit indistinguishable from an external +SIGKILL in the archived logs. Flush best-effort first; never let a flush failure stop the +exit. + +### The context payload is untrusted input + +`context.json` is the host→container boundary, and every value crossing it is COERCED, not +merely annotated. `json.loads` returns `Any`, so pyright accepts `variant_id: str = +context["variant_id"]` for a value that may be anything at all — the annotation reads like a +guarantee and enforces nothing, and a `"replicate_index": "00"` reached `build_task_run_dir` +typed as `int`. `grade` was once the only value coerced: a hand-edited or older-format +`"grade": "false"` arrives as a truthy `str` typed as `bool` and silently grades a run that +asked not to be graded. `regrade` is coerced for the same reason, and getting that one wrong +re-RUNS the agent against a workspace the operator asked only to grade, destroying the +trajectory being graded. + +Keys absent on an older host fall back to the pre-existing behaviour rather than failing: +`grade` defaults to True, `preservation_mode` to the docker default (a deliberate default, +not back-compat — this command only ever runs under the docker driver), `host_task_file` and +`workspace_dir` to None, and `source_yaml` to the staged post-override YAML. + +The host always serialises the POST-override `TaskDefinition` into the staged `task.yaml`, +never `source_yaml`, because the raw on-disk text predates `--model` and `-D` mutations the +container must see. `source_yaml` is forwarded separately so `task.json`'s audit trail +matches the in-process driver's. + +## Trusting what the container sends back + +### The stdout line limit + +asyncio's `StreamReader` caps a single line at 64 KiB by default. The container streams +events as one NDJSON line each, and a single event carrying a large tool input — an agent +writing a whole `.flow` file, say — serialises well past that. The default-limit reader then +raised `ValueError` mid-stream, which tore the container down before it wrote `task.json`: +the entire task was lost and the host recorded a bare ERROR with no per-task report. The +limit is raised to 64 MiB, mirroring the orchestrator's own guard on post-run subprocesses, +and the read loop is an explicit `readline` (not `async for`) so a single over-limit line +degrades to a dropped line instead of a teardown. `readline` drains the offending bytes and +resyncs at the next newline; the dropped line is a host-side render event or a log line, and +`task.json` crosses via the bind mount rather than stdout, so the result is unaffected. + +Each line is then a three-way split: a wire-prefixed line that parses goes to the host +callback and is NOT echoed to `docker.log` (the callback is the canonical destination); a +prefixed line that parses badly is a wire bug and is preserved raw so it is not lost; an +unprefixed line is an ordinary log line. + +### A container that produced no task.json + +The container can die before its orchestrator's `finally` writes the file — torn down after +a host-side stream failure, or killed externally. A synthetic ERROR `task.json` is persisted +so the row stays visible on dashboards and timelines instead of vanishing; the batch layer's +in-memory skeleton never reaches the per-task directory. A file that is present but +unparseable (schema skew from a stale image, a truncated write) degrades the same way rather +than crashing with an uncaught `ValidationError`. + +That synthetic record goes through `write_text_atomic` like every other writer of the file. +The hand-rolled tmp+replace it replaced used `Path.write_text`, which FOLLOWS symlinks — so a +pre-planted `task.json.synthetic.tmp` in a run directory (a shareable artifact, bind-mounted +writable into the agent's own container) redirected this harness-privileged write to any path +the grading user could reach. It also falsified the helper's "one writer, so the crash +semantics cannot differ" claim, which is the property future readers rely on. + +The build runs before `run_dir/docker.log` and `task.json` exist, so a build failure would +otherwise leave an empty result dir with no trace: the build log is persisted to `docker.log` +and a synthetic BUILD_FAILED record written, then the error re-raised for the batch +dispatcher to record run-level. + +Cancellation is handled in a `finally` because `docker run --rm` does NOT propagate a kill +daemon-side: without it, Ctrl-C on the host leaves the container running and burning budget. +Suppression is narrowed to `CancelledError` throughout, so a genuine `KeyboardInterrupt` or +`SystemExit` from a parallel sibling still propagates. A non-zero `docker kill` usually means +the container was already gone (a race with `--rm`) or the daemon refused, so stderr is +surfaced to keep the ambiguity debuggable. + +### The two honored-request guards + +`grade` and `regrade` cross the boundary only through `context.json`, and an image that +predates either key ignores it and falls through to its old behaviour. The image-version +preflight only warns, so version skew would change what a command MEANS. + +For `grade`: a stale image grades anyway, so `execute --driver docker` would silently +produce SUCCESS/FAILURE rows indistinguishable from a normal graded run. For `regrade`: a +stale image ignores the staged `prior.json` and the workspace mount and falls through to the +ordinary orchestrator branch, which **starts an agent** from `initial_prompt` — so the host +would fold a fabricated trajectory back over the recorded row as its "grade", publishing a +verdict for work it never looked at and billing the model for it. Nothing else catches that: +`_assert_grade_honored` early-returns because a grading container is dispatched with +`grade=True`. + +Both are keyed on EVIDENCE, not on the label. For `grade`, "did it grade" is +`success_criteria_results` or a non-None `weighted_score`: exempting every execution-fact +status let a stale image return a fully graded MAX_TURNS_EXHAUSTED row — criteria vector, +weighted score and all — unchallenged, because that exemption exists for statuses a *fresh* +image also produces, and a fresh one produces them with neither. For `regrade`, a container +that honored the request seeds from `prior` and never runs the agent, so a DIFFERENT +`started_at` is the tell: `_seed_from_prior_result` restores the agent run's `started_at` +verbatim, so a fresh run is the only way that field can move. + +The refusal quarantines the on-disk record before raising. Refusing in memory only left the +graded `task.json` sitting in the bind-mounted host run dir, where a later `execute --resume` +read it back as a completed row (its category is `succeeded`, so the resume partition files +it under prior results) and plain `aggregate` folded it straight into `run.json` — publishing +exactly the row the guard declined to publish. Refusing in memory while leaving contradictory +bytes on disk is not a refusal. + +## The sandbox the criteria run in + +### Why the venv gets system site packages + +`--system-site-packages` is load-bearing. The sandbox venv goes on the criterion PATH, which +governs every `run_command` criterion plus `pre_run`/`post_run`, so inside a task image that +provisions packages globally an ISOLATED venv shadowed the interpreter while providing +nothing: `python` resolved to the empty venv and could not import them, while `pip` — which +`uv venv` does not place in the venv at all — fell through to the image's global pip and +reported them present. Measured in a task image: `import langchain` raised +`ModuleNotFoundError` while `pip list` showed `langchain 1.3.14`. An agent verifying its own +work chased that contradiction for ten turns and ran out of budget. + +The venv is NOT on the agent's own PATH — the orchestrator prepends only the resolved mock +dirs there — so the contradiction is a property of criterion and pre/post-run subprocesses. +System site packages fixes it in the direction that keeps both halves: the image's globals +stay importable, `python` and `pip` agree, and installs still land in the venv (`sys.prefix` +remains the sandbox), so a task's `env_packages` cannot leak into the image. The `uv venv` +and stdlib `venv` paths do not produce the same artifact — the latter seeds pip — so which +shape this host got is logged rather than left to be inferred. + +`adopt` DISCOVERS an existing venv instead of creating one, so criteria get the same +`VIRTUAL_ENV`/PATH the agent had, and it is gated on `config.python` for the same reason +`setup` is: discovering a venv a task never asked for grades it under a PATH it never ran +under, and would let an agent shadow binaries by writing `.venv/bin/` into its own workspace. + +### The criterion environment, layer by layer + +Each layer is independent — none breaks if another is absent. + +1. Inherit the parent environment, so agent tools and credentials remain reachable. +2. If the orchestrator captured the agent's SDK PATH, **prepend** it ahead of the host PATH + rather than replacing it: the agent's PATH only needs to win the lookup race for its + bundled toolchain, and system binaries must stay reachable to criteria. Prepending also + stays symmetric with the venv and node_bin prepends below. +3. Activate the sandbox virtualenv, first-hit-wins. If the agent's PATH already contains the + venv scripts dir — likely, since it inherits this process's environment — the prepend + duplicates the entry, which is harmless everywhere and left explicit so the order does not + depend on what the agent SDK injects. +4. Prepend `/node_modules/.bin`. +5. Pin `NODE_PATH=""` so Node's fallback search cannot pick up contaminated parent-dir + installs. This does NOT disable parent-walking from cwd — that is hard-wired in Node — but + it eliminates `NODE_PATH`-mediated leaks. +6. Pin `NPM_CONFIG_PREFIX` to a sandbox-scoped directory, so an `npm install` from inside the + sandbox writes into the sandbox rather than `$HOME/node_modules`, where concurrent + sandboxes would shadow each other. +7. Expose `TASK_DIR` for criterion scripts. +8. Expose `REFERENCE_DIR`, the per-run staged copy, when the task declares a reference. + Safe here because `run_command` criteria execute AFTER the agent's turn, outside the + mode-000 window; absent for a task with no `reference:` block. + +The parent `node_modules` check is detection-only. Concurrent tasks, or anything else running +`npm install --save` in a shared parent, drop packages where Node's parent-walking resolver +finds them before the sandbox-local install. The failure mode is generic to Node module +resolution rather than specific to one npm scope, so the check stays scope-agnostic — +`coder_eval` is a generic framework and should not single out one ecosystem's namespace — and +auto-remediation is avoided because those directories may legitimately belong to the user. + +Generated `record_cli` recorders go on PATH FIRST, and refuse to generate a shim whose name a +user mock dir already provides, so the order can never silently shadow a task's own mock — it +only fixes which directory wins for names the harness owns. The clash check covers every name +the feature generates, not just the bare one: on Windows PATHEXT resolves `uip` to the +generated `uip.cmd` ahead of the task's own `mocks/uip.cmd`. The recorder dir is wiped rather +than reused, because DIRECT_WRITE does not clear the target dir and a reused `--run-dir` would +leave a previous run's log to be scored as this one's. The log file is seeded empty: +`cli_called` treats a MISSING log as a harness fault (score 0 even for a negative guard), +which is right when a mock never ran and wrong for a correct run that legitimately called +nothing. + +### Criterion paths are contained, quietly + +`Path('/tmp/sandbox') / '/etc/passwd'` is `/etc/passwd` — pathlib discards the prefix on an +absolute right operand — so criterion paths, the one consumer that skipped the containment +helper every other task-authored path goes through, were a pass-fail oracle over any file the +grading user could read, and `json_check` could surface parsed values in `details`. That was +defensible while a task YAML was operator-supplied; it stopped being so when +`evaluate ` began rebuilding the criteria list from a shareable run directory. + +The predicate returns False rather than raising: an out-of-sandbox path is indistinguishable +to the criterion from a file that is not there, which is the same answer the template and +mock-dir paths give, and raising would book a config error as an agent crash (CE039). It is +silent by design — the escape is reported ONCE per criterion, naming the pattern the task +author actually wrote, because logging at the predicate named a resolved absolute path +(uninformative: the author's own string joined onto a tempdir) once per rejected glob match, +so a wide pattern produced a burst of near-identical warnings. + +An escaping path that names an EXISTING file is a different case and raises +`CheckerMisuseError`. Returning `[]` booked an eval-CONFIG error as an agent failure: the +criterion scored a gating 0.0 with "file does not exist" for a file that plainly does exist, +with only a WARNING in the task log. `tasks/byod_smoke_test.yaml` was broken exactly that way +— it checks `/opt/byod_marker`, baked into the BYOD image — and the suite reported a 0.0 +nobody could explain from the score alone. No agent behaviour can ever satisfy such a path, +so it is not a verdict about the agent, which is precisely the distinction CE039 enforces. A +merely-absent absolute path still resolves to "no match", an ordinary failing verdict, and +the GLOB branch still warns and drops rather than raising, since filtering some matches out of +a search is its normal behaviour. + +Resolution prefers the literal: a path naming an existing file resolves to itself **even when +it contains a glob metacharacter**, so a real `report[2024].json` is graded as itself rather +than as a character class that would silently match `report2.json`. Only when the literal does +not exist is it expanded, so a criterion can address a file whose exact location the prompt +does not pin. Glob matches are filtered through the sandbox's ignore patterns, because the +sandbox root holds harness-created content the agent never authored and grading off it is +neither fair nor deterministic; only segments the glob *discovered* are filtered, so a segment +the pattern names literally (`dist/**/*.js`) is an explicit opt-in and survives. Matches are +sorted for determinism and directories dropped so a glob cannot resolve to something +unreadable. The ambiguity error enumerates a bounded number of matches, because the message is +persisted to `task.json` and injected into judge prompts, where an unbounded listing over a +wide pattern is a real payload. + +### preserve_to, capture_to, and the capture denylist + +`mkdtemp` creates the sandbox root at 0700. Under `driver: docker` the container runs as root, +so the preserved tree lands on the host bind-mount owned by root with that 0700 top dir — the +host user, a different uid, then cannot traverse it, so the blob upload and any `ls` see an +empty dir and silently skip the artifacts. Both paths therefore grant `a+rX` on the tree the +host reads. + +`preserve_to` MOVES and repoints `sandbox_dir` so a later `cleanup()` is a no-op; absolute +paths inside the venv are not rewritten, matching the prior copy-based behaviour. +`capture_to` COPIES instead, because the sandbox there is the container's own WORKDIR (`/root`, +say), which is discarded with `--rm` and may contain the orchestrator's own cwd — so a copy is +safe and non-destructive. It does not repoint `sandbox_dir`: the workspace persists +in-container and is reaped with the container. `symlinks=True` plus `ignore_dangling_symlinks` +makes a dangling link a no-op rather than a failure, the exact breakage the old +`cp -a "$PWD/." "/root/"` reconciliation prelude hit. + +Because the WORKDIR can BE `$HOME`, capture excludes two classes of entry. The SECURITY +denylist is credential stores that must never leak into artifacts that get uploaded — most +importantly `.claude`, the RW lean copy that carries `.credentials.json`, plus `.aws`, `.ssh`, +`.gnupg`, `.docker`, `.azure`, `.netrc` and `.gitconfig` (which can embed PATs via +`credential.helper`). It is defense-in-depth: the eval images bake no credentials, but a future +image that does should not silently expose them. The NOISE class is sandbox-created bulk and +home-dir infrastructure written by uv, pip, npm and the shell when WORKDIR overlaps HOME; those +are never task deliverables. Both match by basename at every level. + +Cleanup on a failed `setup` removes ONLY a temp dir the sandbox created itself. A +caller-supplied `target_dir` (DIRECT_WRITE) may be a pre-existing artifacts dir whose contract +is never to be cleared, and `_cleanup_on_exit` already distinguishes the two. + +### Materializing a template into the sandbox + +Template copying matches ignore patterns against the template-RELATIVE path, because checking +the absolute path lets an ancestor directory named `dist`, `build`, `env`, `venv` or +`node_modules` filter out the entire template — a repo cloned under `~/build/…`, say. Symlinks +are handled before `is_dir()`/`is_file()`, which follow them: a `tools/node_modules/x -> ../y` +link would look like a directory and produce an empty dir at the destination, breaking npm +workspace resolution. At the destination, `is_symlink()` comes before `exists()` because +`exists()` follows the link and a *broken* symlink is still an overwrite to clear; only a real +directory needs `rmtree`. Link targets are preserved verbatim, relative and absolute alike — +absolute targets remain live links into the host filesystem, which is intended for trusted +template authors, not a defense boundary. + +`git clone` is invoked with `--` before the URL: it is argv position 2, so without the +separator a value beginning with `-` is parsed as an OPTION rather than a repository +(`--upload-pack=…` runs a command of the caller's choosing). That URL is task-authored, and +since `evaluate ` rebuilds the task from a shareable run directory it is no longer +necessarily the operator's own string. diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index 2377a438f..0a79fe32d 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -57,10 +57,80 @@ VERDICT, never the facts — the seeding cannot restore a fact the execute phase captured. The budget gate runs AFTER the criteria on the graded path purely for partial-credit visibility, and there is no partial credit under `execute`. +### Refusing a criteria-free task under grade + +`TaskDefinition.success_criteria` accepts an empty list at the model level, because the +Harbor agent-phase `task.yaml` is criteria-free by design and must round-trip through +`coder-eval execute`, which never grades. But scoring is vacuous over an empty list — +`all_criteria_passed` returns True and `calculate_weighted_score` returns 0.0 — so such a +task graded under `run` finalizes as SUCCESS at `weighted_score: 0.0`, an internally +contradictory result for what is actually a typo, a bad merge, or a `-D` override that +cleared the list. The refusal is therefore scoped to `grade`, exactly as the sibling +simulation refusal is. + +It is checked against the POST-`--resume` set, never the full resolved list: an +already-finalized row is folded back from `prior_results` and never re-executed or +re-graded, so its own possibly-empty criteria are moot and must not block a run that is not +going to grade it. The `to_grade` rows ARE about to be graded, so they get the same check — +explicitly, rather than relying on `regrade_in_place`'s own per-row guard, so the whole +batch is refused up front instead of one row at a time turning into a mid-resume warning. +`evaluate` needs the same guard on its orchestrator-direct branch, which never calls +`regrade_in_place` at all. + +### What the exit code counts + +The command exits non-zero when any task failed, errored, or any suite missed its +thresholds — and, under `run` only, when any row came back ungraded. `run` was asked for a +verdict and did not produce one (the grade crashed, or `--resume` could not grade the row), +which is a failure of the command even though the row is neither `failed` nor `error`. +Under `execute` an ungraded row is the expected outcome for every task and must not fail +the command. + +The JUnit report is written BEFORE that gate, so a failing run still produces one; a write +error propagates rather than being swallowed. Telemetry is flushed in a `finally` so it runs +on both the success and the raised path, without catching the `typer.Exit` decided after it. + +Per-suite rollups are skipped entirely under `execute`: a rollup aggregates per-criterion +results and there are none, so running it would gate a suite on an empty aggregate and +report a threshold failure for a run that was never measured. The ungraded bucket is named +explicitly in the aggregate line for the same reason — `coder-eval aggregate ` is the +step right after `coder-eval execute`, so an ungraded run is the FIRST thing it renders, and +without the term it reads "Aggregated 12 task(s) (0 ok / 0 fail / 0 err)": four numbers that +no longer sum to `tasks_run`, with nothing on screen to say where the rest went. The +end-of-run summary likewise reports what happened instead of "0/N succeeded", which for a +clean `execute` reads as a total failure, and points at the run-dir resume form rather than +`evaluate ` — the two-argument shape grades a bare directory with NO +trajectory, so `command_executed` / `skill_triggered` / trajectory-reading judges score +differently from what `run` would have produced. + ## `--resume` is command-relative - **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate ` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`. +### When a resumed grade crashes + +A row that could not even be READ has no recorded result to fold back, but dropping it +removes it from `run.json` AND from `tasks_not_graded`, which is what the exit gate counts — +so a resume whose rows were all unreadable reported success. A minimal ungraded placeholder +stands in instead, keeping the task visible and the command non-zero. The read happens +inside the `try`: outside it, one bad `task.json` propagates out of the loop and aborts the +whole resume before `run_batch`, so none of the `to_run` tasks execute either — the opposite +of "one bad row never aborts". + +An orchestrator-level grading crash is not a verdict about the run, and `Orchestrator.run()` +converts internal failures into a populated ERROR result rather than raising, so the +`except` above never sees them and the ERROR row replaced a perfectly re-gradeable +NOT_GRADED one — ERROR being "complete" for both commands, the row could then never be +graded again. Fixing the in-memory result is only half of it: `_finalize_result` has already +written the ERROR `task.json` into that same directory, so `run.json` would say NOT_GRADED +while the row on disk says ERROR, and the on-disk one is what a later `--resume` reads. The +pre-grade record is put back. + +The `--resume` config-drift warning is best-effort and informational: the per-task path key +does not encode the run config, so resumed tasks keep their original-config results, and +surfacing the mismatch makes the resulting mixed-config `run.json` visible instead of +silent. A missing stamp (a run predating the feature) is tolerated. + ## Early stop on criterion - **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. @@ -202,6 +272,18 @@ around it: the docker dispatch guard saw a non-None `Path` and let it through, a task-dir mount then silently mounted nothing, so every `$TASK_DIR` criterion resolved against the wrong tree and scored a verdict nobody could explain. +### The in-container driver rewrite + +CE051 forbids rewriting `sandbox.driver`, and this is its single exemption: the process is +already inside the container the docker driver asked for, so the isolation the driver names +is present rather than bypassed, and a nested docker would be both wrong and impossible (no +docker CLI in the image). The rewrite goes through `model_validate` rather than +`model_copy(update=...)`, matching its sibling in `regrade.grading_sandbox_config`: `update` +skips BOTH pydantic and pyright, so a typo produces a `SandboxConfig` violating its own +`Literal` and only surfaces far downstream. Two driver-rewrite sites landing in one change +with two different levels of type safety is how the weaker one becomes the pattern people +copy. + ## Three routes, resolved separately The agent's route, the judge's and the simulated user's are resolved independently and @@ -479,6 +561,18 @@ the first two. variant resolution and any further I/O. It is still reported in `skipped_tasks`, so the suite shows which YAMLs were intentionally excluded rather than failed to load. +### How a resolution failure reaches the operator + +A GLOBAL failure raises and is surfaced as a clean CLI error instead of a traceback; +`run_batch`'s own resolution-time guards raise plain `ValueError` and are converted to that +same error, so every refusal in the pipeline reads alike. Per-task failures never raise -- +see above. + +Every task-file pattern the caller wrote is checked for a match, not just their union. +Accumulating and checking only the total meant one stale entry among several -- a renamed or +moved suite -- silently ran the surviving subset and exited 0, so a CI gate reported green +over tasks it never measured. + ### No-op tasks need no special case anywhere `type` is a replace-scalar, so a task-level `type: none` wins over a baseline coding agent diff --git a/src/coder_eval/cli/__init__.py b/src/coder_eval/cli/__init__.py index 62eda48ab..6b95d1088 100644 --- a/src/coder_eval/cli/__init__.py +++ b/src/coder_eval/cli/__init__.py @@ -75,9 +75,9 @@ def main( raise typer.Exit(0) -# Register core commands. Each public command is wrapped with track_command so it -# emits a CoderEval.Cli. event (Status/DurationMs/ErrorType) on completion; -# functools.wraps preserves the signature so Typer still parses each command's flags. +# Each public command is wrapped with track_command so it emits a +# CoderEval.Cli. event; functools.wraps preserves the signature so Typer +# still parses each command's flags. app.command(name="run")(track_command("run")(run_command)) app.command(name="execute")(track_command("execute")(execute_command)) app.command(name="plan")(track_command("plan")(plan_command)) diff --git a/src/coder_eval/cli/aggregate_command.py b/src/coder_eval/cli/aggregate_command.py index 942c6f0e0..511cbc3ec 100644 --- a/src/coder_eval/cli/aggregate_command.py +++ b/src/coder_eval/cli/aggregate_command.py @@ -59,10 +59,8 @@ def aggregate_command( out_dir = output_dir or run_dir - # tags / source-path / run-level fields are inputs (static task metadata), not - # results, so carry them from an existing run.json when present — reading them - # from a stale summary is safe. Absent → empty maps + a window derived from the - # recovered results. + # Inputs (static task metadata), not results, so carrying them from an existing + # run.json is safe even when that summary is stale. task_tags, task_paths, prior = _read_prior_metadata(run_dir) start_time, end_time = _resolve_window(results, prior) skipped = _recover_skipped_tasks(prior) @@ -78,11 +76,9 @@ def aggregate_command( skipped_tasks=skipped, ) write_run_summary(summary, out_dir) - # The fourth bucket is named here too. `coder-eval aggregate ` is the - # step right after `coder-eval execute`, so an ungraded run is the FIRST - # thing this line renders — and without the term it reads - # "Aggregated 12 task(s) (0 ok / 0 fail / 0 err)", four numbers that no - # longer sum to tasks_run with nothing on screen to say where the rest went. + # The fourth bucket is named here too: `aggregate` is the step right after + # `execute`, so an ungraded run is the FIRST thing this line renders. + # Rationale: .claude/notes/orchestration.md § What the exit code counts counts = f"{summary.tasks_succeeded} ok / {summary.tasks_failed} fail / {summary.tasks_error} err" if summary.tasks_not_graded: counts += f" / {summary.tasks_not_graded} not graded" diff --git a/src/coder_eval/cli/evaluate_command.py b/src/coder_eval/cli/evaluate_command.py index d8f47ad54..d6db81719 100644 --- a/src/coder_eval/cli/evaluate_command.py +++ b/src/coder_eval/cli/evaluate_command.py @@ -134,10 +134,9 @@ def _resolve_run_dir_or_work_dir( prior: EvaluationResult | None = None if target.mode is EvaluateMode.RUN_DIR and target.task_file is not None: # `is_run_dir` is a filename probe, so a plain work directory holding an - # unrelated file called task.json lands here. That would abort the - # pre-existing `evaluate ` form on a pydantic wall the - # user can only escape by renaming their own file. The task file is - # already in hand, so fall back to the shape they asked for. + # unrelated file called task.json lands here. The task file is already in + # hand, so fall back to the shape the user asked for. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI try: load_prior_result(target.target) except RegradeError as e: @@ -155,23 +154,10 @@ def _resolve_run_dir_or_work_dir( task, source_yaml = load_task(target.task_file) console.print(f"[dim]Grading with {target.task_file} (overrides the run's recorded config).[/dim]") else: - # ONE lever, passed once, and derived through the SAME function - # `run_evaluation` uses rather than restated. It decides which - # capability families the recorded-shell gate discloses, so a second - # copy of the rule would keep answering the old question the moment - # the default moved — and silently stop covering commands that then - # do run. - # - # In place: the grade may dispatch a CONTAINER built from the - # recorded sandbox block, a wider capability than any recorded shell - # string. On --copy instead: pre_run and the sandbox's own - # installers, neither of which an adopted workspace reaches. post_run - # is in NEITHER set — it belongs to the grading phase and runs on - # both paths, so `embedded_commands` scans it unconditionally. - # - # Both answers follow from this single boolean, so the gate derives - # them itself (`_gate_scope_for_grade`) rather than taking two - # arguments a caller could set incoherently. + # ONE lever, passed once and derived through the SAME function + # `run_evaluation` uses: the gate works out both answers itself rather + # than taking two arguments a caller could set incoherently. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI task, source_yaml = task_from_prior( prior, target.target, @@ -196,11 +182,8 @@ def _resolve_run_dir_or_work_dir( console.print(f"[red]✗ Work directory is not a directory:[/red] {escape(str(work_dir))}") raise typer.Exit(1) - # Evaluate-only mode bypasses experiment resolution + CLI overrides, so - # `agent` may be None or `agent.type` may be unset for tasks that defer - # those to the experiment / CLI layers. The orchestrator only uses - # `agent.type` for result labeling here (no agent is created), so a - # default is safe. + # Evaluate-only mode bypasses experiment resolution and CLI overrides, so + # `agent.type` may be unset. It is used only for result labeling here. if task.agent is None: task.agent = parse_agent_config(type=AgentKind.CLAUDE_CODE) elif task.agent.type is None: @@ -208,10 +191,9 @@ def _resolve_run_dir_or_work_dir( if prior is not None: verify_reference_unchanged(prior, task, task_file) - # Snapshot the ungraded record BEFORE anything grades. Taking it inside - # _write_back instead would capture an ALREADY-GRADED record whenever - # --run-dir points at the target run dir (the orchestrator writes there - # first), destroying the very evidence the copy exists to preserve. + # BEFORE anything grades: taken inside _write_back it would capture an + # ALREADY-GRADED record whenever --run-dir points at the target run dir. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI back_up_pre_grade_record(target.target) return _ResolvedInputs( @@ -239,18 +221,16 @@ def _replicate_index_of(run_dir: Path) -> int: def evaluate_command( task_or_run_dir: Path = typer.Argument( # noqa: B008 ..., - # One metavar per positional, so the usage line reads as Click renders - # it. A composite metavar on the first ("[TASK_FILE] TARGET") plus an - # empty one on the second produced `[TASK_FILE] TARGET []`. + # One metavar per positional: a composite on the first plus an empty one + # on the second rendered `[TASK_FILE] TARGET []`. metavar="TASK_FILE_OR_RUN_DIR", help="Task YAML file, or (when it is the only argument) a finished run directory.", exists=True, ), work_dir: Path | None = typer.Argument( # noqa: B008 None, - # No metavar="" here: an empty one leaks a bare `[]` into both the usage - # line and the arguments table. The first positional's metavar already - # spells out the two shapes. + # No metavar="" here: an empty one leaks a bare `[]` into the usage line + # and the arguments table. help="Directory containing the code to evaluate. Omit when TASK_FILE is a run directory.", ), workspace: Path | None = typer.Option( # noqa: B008 @@ -432,12 +412,10 @@ def run_evaluation( ) if not task.success_criteria: - # `evaluate` always grades -- unlike `execute`, there is no legal reason - # for a zero-criteria task to reach here. `regrade_in_place` guards its - # own delegating branch; this guard covers the sibling orchestrator-direct - # branch below (fresh work-dir grading, or `--copy`), which never calls - # `regrade_in_place` and would otherwise finalize a criteria-free task as - # SUCCESS at weighted_score 0.0. + # This guard covers the orchestrator-direct branch below, which never + # calls `regrade_in_place` and would otherwise finalize a criteria-free + # task as SUCCESS at weighted_score 0.0. + # Rationale: .claude/notes/orchestration.md § Refusing a criteria-free task under grade console.print( f"[red]✗ Task {task.task_id!r} has no `success_criteria` and cannot be graded " + "(it would silently score SUCCESS at weighted_score 0.0). Add at least one criterion.[/red]" @@ -452,25 +430,15 @@ def run_evaluation( console.print(f"[red]✗ Failed to prepare run directory:[/red] {escape(str(e))}") raise typer.Exit(1) from e - # `regrade_in_place` owns the sandbox on the delegating path — and for a - # `driver: docker` row it owns rather more than that, dispatching a grading - # CONTAINER of the task's own image. Building a host sandbox_config here - # first would call `grading_sandbox_config`, whose whole job is to REFUSE - # that driver, so the refusal fired before the branch that no longer needs - # it and no docker row could ever be graded properly. - # - # Branching on ``prior is not None`` directly, and building the sandbox - # inside the branch that uses it, so NEITHER value is Optional at its use - # site. Both were, briefly, re-narrowed by a bare `assert` plus a comment - # asserting an invariant the type checker could hold structurally — and - # `assert` is the weakest narrowing available, stripped entirely under -O. + # Branching on ``prior is not None`` directly, and building the sandbox inside + # the branch that uses it, so NEITHER value is Optional at its use site -- + # `grading_sandbox_config` REFUSES a docker driver, so building one up front + # fired that refusal before the branch that no longer needs it. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI async def _setup_and_run() -> EvaluationResult: if grade_in_place and prior is not None: - # Delegate to the shared re-grade core. Restating its body here is - # how this path and `run --resume` came to differ (replicate_index, - # error semantics) while CLAUDE.md called regrade.py the single - # implementation — two copies of "how to re-grade" drift into two - # verdicts for the same run. + # Delegate to the shared re-grade core rather than restating it: two + # copies of "how to re-grade" drift into two verdicts for one run. return await regrade_in_place( task=task, prior=prior, @@ -510,23 +478,18 @@ async def _setup_and_run() -> EvaluationResult: prior_result=prior, ) graded = await orchestrator.run() - # Same stamp the delegating branch gets from `regrade_in_place`. The - # `grading_sandbox_config` call above accepted the docker->host - # downgrade for THIS branch too, and - # CLAUDE.md, the user guide and CE051's own noqa all state the stamp as - # unconditional — so `evaluate --copy --allow-host-grading` - # was writing an unstamped host verdict that nothing downstream could - # tell apart from a container-graded one. + # HAZARD: the same stamp the delegating branch gets from + # `regrade_in_place`. CLAUDE.md, the user guide and CE051's own noqa all + # state it as unconditional. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI stamp_host_grading(graded, task) return graded try: result = asyncio.run(_setup_and_run()) except RegradeError as e: - # The delegating branch raises this for the missing/unresolvable task - # file and for a failed grading container, and both messages carry the - # operator's next step. Rendered like the three sibling handlers above -- - # unwrapped, they arrived as the tail of a stack trace. + # Rendered like the three sibling handlers above: unwrapped, these + # operator-facing messages arrived as the tail of a stack trace. console.print(f"[red]✗ {escape(str(e))}[/red]") raise typer.Exit(1) from e _report_and_exit(result, task=task, prior=prior, target=target, prepared_run_dir=prepared_run_dir) @@ -548,33 +511,23 @@ def _report_and_exit( status handling landed. Always raises ``typer.Exit``. """ - # BEFORE the count guard below. A grading crash returns a populated ERROR - # result with an EMPTY criteria list (Orchestrator.run() converts internal - # failures into a result rather than raising), so the count check fires - # first and the user is told only "Result count mismatch: got 0, expected 2" - # — the real error is never printed, and the "still re-gradeable" notice is - # unreachable on exactly the path it was written for. - # Whether the terminal status describes THIS pass or was carried over from - # the run being graded. `Orchestrator._terminal_status` preserves a prior - # execution fact (TIMEOUT / ERROR / BUILD_FAILED / a budget stop) because - # grading may not overturn it — so reading `result.final_status` as this - # pass's own outcome misreports both arms below. It made a preserved ERROR - # print the ORIGINAL run's crash message as though grading had crashed, - # claim the row was "left ungraded" (it was not — the restored record still - # reads ERROR), and throw away a verdict that had just been computed at - # 1.000; and it made a preserved TIMEOUT exit 0 under "All criteria passed", - # so a CI wrapper reading the exit code goes green on a row run.json counts - # as failed. + # BEFORE the count guard below: a grading crash returns a populated ERROR + # result with an EMPTY criteria list, so the count check fired first and the + # real error was never printed. + # + # Whether the terminal status describes THIS pass or was carried over from the + # run being graded. `Orchestrator._terminal_status` preserves a prior execution + # fact, so reading `result.final_status` as this pass's own outcome misreports + # both arms below. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI inherited = prior is not None and prior.final_status.is_execution_fact if result.final_status is FinalStatus.ERROR and not inherited: console.print(f"\n[red]✗ Evaluation error: {result.error_message}[/red]") if prior is not None: - # A grading-time crash (a failing checker, an unreachable judge) is - # not a verdict about the run. Leaving ERROR on disk would replace a - # perfectly re-gradeable NOT_GRADED row with one BOTH commands treat - # as permanently complete, so the run could never be graded again - # without hand-restoring task.execute.json. + # A grading-time crash is not a verdict about the run. Leaving ERROR + # on disk replaces a re-gradeable NOT_GRADED row with one BOTH + # commands treat as permanently complete. restore_pre_grade_record(target.target) console.print( f"[yellow]⚠[/] Grading errored; {target.target / TASK_JSON_FILENAME} is left " @@ -625,14 +578,10 @@ def _report_and_exit( if result.sandbox_path: console.print(f"[dim]Artifacts: {result.sandbox_path}[/dim]") - # `prior is not None` alone is not enough: `--format harbor` seeds a - # SYNTHETIC prior on the WORK_DIR shape (from the supplied - # `--trajectory`), which is not a run directory and carries no - # `task.execute.json` sibling to preserve. `_write_back` is documented as - # "replace the graded RUN's task.json" and writes into `target.target`, - # which in WORK_DIR mode is the directory being graded, not a run dir -- - # writing there planted a spurious task.json into the Harbor-synced - # workdir and wedged a later `evaluate` on it into RUN_DIR mode. + # RUN_DIR mode, not merely `prior is not None`: `--format harbor` seeds a + # SYNTHETIC prior on the WORK_DIR shape, which is not a run directory and has + # no `task.execute.json` sibling to preserve. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI if prior is not None and target.mode is EvaluateMode.RUN_DIR: console.print( f"[dim]Re-graded {prior.final_status.value} → {result.final_status.value} " @@ -641,10 +590,9 @@ def _report_and_exit( _write_back(target.target, result) if result.final_status.is_execution_fact: - # The criteria tally is real and worth printing — it is why the table - # above still renders — but it is not the row's outcome. run.json will - # count this row under its preserved status, and the exit code must - # agree with run.json rather than with the tally. + # The criteria tally is real -- it is why the table above still renders -- + # but it is not the row's outcome, and the exit code must agree with + # run.json rather than with the tally. console.print( f"\n[red]Criteria: {passed}/{total} passed, but the run itself ended as " + f"{result.final_status.value} — grading cannot overturn that.[/red]" @@ -672,15 +620,13 @@ def _write_back(run_dir: Path, result: EvaluationResult) -> None: target = run_dir / TASK_JSON_FILENAME backup = run_dir / PRE_GRADE_JSON_FILENAME if target.is_symlink(): - # A run directory is a shareable artifact, so its task.json is untrusted - # input. Following a symlink here turns `evaluate ` into an - # arbitrary-file-overwrite primitive on the grader's host. + # HAZARD: a run directory is a shareable artifact, so following a symlink + # here is an arbitrary-file-overwrite primitive on the grader's host. console.print(f"[yellow]⚠[/] {target} is a symlink; refusing to write through it.") return try: - # Atomic, matching the orchestrator's own task.json writer: a torn write - # here makes the row parse as malformed, which a later --resume reads as - # "not complete" and re-pays for the agent. + # Atomic, matching the orchestrator's own writer: a torn write makes the + # row parse as malformed, which a later --resume re-pays for. write_text_atomic(target, result.model_dump_json(indent=2, exclude=TASK_JSON_TRANSCRIPT_EXCLUDE)) except OSError as e: # Never fail the grade over the write-back: the verdict was computed and diff --git a/src/coder_eval/cli/evaluate_target.py b/src/coder_eval/cli/evaluate_target.py index b16e88576..5151479fc 100644 --- a/src/coder_eval/cli/evaluate_target.py +++ b/src/coder_eval/cli/evaluate_target.py @@ -88,20 +88,12 @@ def resolve_evaluate_target(first: Path, second: Path | None) -> EvaluateTarget: ) return EvaluateTarget(mode=EvaluateMode.RUN_DIR, target=first, task_file=None) - # Two arguments. The second is the place; the first is the task file. When - # that place turns out to be a run directory the caller is re-grading it with - # a DIFFERENT task file than the one it ran with — the "iterate on my - # criteria against an expensive run I already paid for" case, which is the - # main reason to keep `execute` and `evaluate` separate at all. Allow it, and - # let the caller be told which config won. - # - # This probe is a filename test, so a plain work directory that merely - # happens to contain a file called `task.json` is read as a run directory — - # and the pre-existing two-argument form would abort on a pydantic wall with - # no way to override it. It is not repaired here (this function is pure and - # cannot tell a real record from a namesake); the caller re-reads the record - # and falls back to WORK_DIR when it does not parse. See + # The second argument is the place; the first is the task file. A run directory + # there means re-grading with a DIFFERENT task file, which is allowed. This + # probe is a filename test, and this function is pure, so the caller re-reads + # the record and falls back to WORK_DIR when it does not parse -- see # ``evaluate_command._resolve_run_dir_or_work_dir``. + # Rationale: .claude/notes/isolation.md § Detached grading from the CLI mode = EvaluateMode.RUN_DIR if second.is_dir() and is_run_dir(second) else EvaluateMode.WORK_DIR return EvaluateTarget(mode=mode, target=second, task_file=first) diff --git a/src/coder_eval/cli/execute_command.py b/src/coder_eval/cli/execute_command.py index 8047dcdb3..51df9af06 100644 --- a/src/coder_eval/cli/execute_command.py +++ b/src/coder_eval/cli/execute_command.py @@ -1,28 +1,20 @@ """Execute command - run evaluation tasks WITHOUT grading them. ``coder-eval execute`` is ``coder-eval run`` with the grading half removed: the -sandbox is built, the agent runs, and the full trajectory is captured into the -usual ``task.json`` / ``run.json`` layout — but no success criterion is checked, +sandbox is built, the agent runs, and the full trajectory is captured into the usual +``task.json`` / ``run.json`` layout -- but no success criterion is checked, ``weighted_score`` stays ``None``, and each row finalizes as ``FinalStatus.NOT_GRADED``. -It exists so an *external* harness can own the verdict. The motivating case is -Harbor (Terminal-Bench 2.0), which builds its own container, calls coder-eval as -the agent, and grades with its own ``tests/test.sh``. Grading twice there would -be worse than not grading at all: coder-eval's verdict would be reported -alongside Harbor's without being the one that counts. - -Every flag on ``run`` is available here except ``--junit-xml``, which is a report -of verdicts and there are none. - -``--resume`` IS supported, because ``partition_for_resume`` now takes the -resuming command into account: a ``NOT_GRADED`` row owes ``execute`` nothing (it -finished executing) but owes ``run`` a grade, so ``run --resume`` grades those -rows in place rather than skipping them as "already complete". +Every flag on ``run`` is available here except ``--junit-xml``, which is a report of +verdicts and there are none. ``--resume`` IS supported: a ``NOT_GRADED`` row owes +``execute`` nothing but owes ``run`` a grade. The command shares ``run``'s entire body (``run_command.run_pipeline``); only the Typer signature is restated, because Typer builds its parser from the signature. ``tests/test_execute_command.py`` asserts the two signatures stay in step. + +Rationale: .claude/notes/orchestration.md § Execute vs. run: the grading switch """ from pathlib import Path diff --git a/src/coder_eval/cli/plan_command.py b/src/coder_eval/cli/plan_command.py index 61d2da8b0..b994d74d2 100644 --- a/src/coder_eval/cli/plan_command.py +++ b/src/coder_eval/cli/plan_command.py @@ -106,14 +106,11 @@ def run_plan(*, task_files: list[Path] | None = None, experiment: Path | None = all_valid = True for task_file in resolved_task_files: try: - # Capture warnings so unknown-field UnknownTaskFieldWarnings - # (emitted by TaskDefinition._warn_on_unknown_fields while the - # top-level schema stays in soft-launch mode) surface inline - # below \u2014 they don't fail the run, but they're visible to the - # author and to any CI log scraper. Other DeprecationWarnings - # raised during load (legacy-timing migrations, pydantic, - # transitive libs) are re-emitted through warnings.showwarning - # so they still reach stderr instead of getting swallowed. + # Captured so unknown-field warnings surface inline below -- NON-blocking, + # and the way a stale top-level field (`max_iterations`, `llm_reviewer`) + # the soft-launch validator drops silently becomes visible. Every other + # warning raised during load is re-emitted through warnings.showwarning so + # it still reaches stderr. with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always", DeprecationWarning) task, _source_yaml = load_task(task_file) @@ -132,12 +129,8 @@ def run_plan(*, task_files: list[Path] | None = None, experiment: Path | None = console.print(f" [dim]Success criteria: {len(task.success_criteria)}[/dim]") - # Surface unknown-field warnings as inline notices (non-blocking; - # catches stale top-level fields like max_iterations / llm_reviewer - # that the soft-launch validator otherwise drops silently). Match - # by category, not message text, so a reworded warning string - # doesn't silently break this rendering. Anything else captured - # gets re-emitted to stderr so non-target deprecations stay visible. + # Matched by CATEGORY, not message text, so a reworded warning string + # does not silently break this rendering. for w in caught: if issubclass(w.category, UnknownTaskFieldWarning): console.print(f" [yellow]⚠[/yellow] [yellow]{w.message}[/yellow]") diff --git a/src/coder_eval/cli/run_command.py b/src/coder_eval/cli/run_command.py index 8e234de7a..247ac5bda 100644 --- a/src/coder_eval/cli/run_command.py +++ b/src/coder_eval/cli/run_command.py @@ -57,10 +57,9 @@ def _resolve_experiment_path(experiment: Path | None) -> Path | None: if experiment.exists(): return experiment - # Try resolving bare name under experiments/ (project-root-relative, not CWD-relative). - # Path: cli/run_command.py → cli/ → coder_eval/ → src/ → project_root (4 levels). - # NOTE: This assumes a source checkout. If installed into site-packages, this won't resolve. - # That's acceptable since experiments/ lives in the repo, not the installed package. + # Project-root-relative, not CWD-relative: cli/ -> coder_eval/ -> src/ -> root is the + # four `.parent`s below. Assumes a source checkout; acceptable because experiments/ + # lives in the repo, not the installed package. _project_root = Path(__file__).resolve().parent.parent.parent.parent experiments_dir = _project_root / "experiments" for candidate in [ @@ -94,10 +93,9 @@ def _litellm_preflight_error(current_settings: Settings) -> str | None: if current_settings.api_backend != ApiBackend.LITELLM or not current_settings.litellm_base_url: return None base_url = current_settings.litellm_base_url - # Reject a scheme-less/non-http(s) URL with a clear message instead of letting - # urlopen raise a bare ValueError ("unknown url type") that escapes as a - # traceback. Also makes the `# nosec B310` below honest — the scheme is now - # constrained to http(s), which is exactly what B310 audits. + # HAZARD: constrains the scheme to http(s), which is what makes the `# nosec + # B310` below honest. Also avoids a bare urlopen ValueError escaping as a + # traceback. if urllib.parse.urlsplit(base_url).scheme not in ("http", "https"): return ( f"LITELLM_BASE_URL must be an http(s) URL, got {base_url!r}. " @@ -105,9 +103,8 @@ def _litellm_preflight_error(current_settings: Settings) -> str | None: ) url = f"{base_url.rstrip('/')}/health/liveliness" try: - # B310: url is built from the operator-configured LITELLM_BASE_URL, whose - # scheme is validated to http(s) just above — not untrusted input; this - # only probes reachability of that proxy endpoint. + # B310: the URL is operator-configured and scheme-validated just above; + # this only probes reachability of that proxy endpoint. urllib.request.urlopen(url, timeout=5).close() # nosec B310 except urllib.error.HTTPError: return None # server responded (up), just not 200 on this path @@ -272,10 +269,9 @@ def run_command( None, "--type", "-T", - # Open string, not a closed click.Choice: the agent registry (incl. plugin - # kinds discovered at startup) is the source of truth, and it isn't populated - # at CLI-definition time. An unregistered kind fails at parse_agent_config with - # a clear "No agent registered for type ...; Registered kinds: [...]" message. + # Open string, not a closed click.Choice: the registry is the source of + # truth and is not populated at CLI-definition time. An unregistered kind + # fails at parse_agent_config with a clear message. help="Override agent type for all tasks (e.g. 'claude-code', 'codex', or a plugin kind)", ), model: str | None = typer.Option( @@ -330,10 +326,8 @@ def run_command( help="Run each (task, variant) N times. Overrides experiment/variant `repeats:`. Must be >=1.", min=1, ), - # typer types this as str|None at signature level; click.Choice narrows - # the runtime value to {"tempdir","docker"}. BatchRunConfig.driver - # expects the Literal; the field validator accepts any str and the - # Choice constraint plus the experiment-layer Literal hint keep us safe. + # typer types this str|None; click.Choice narrows the runtime value to the + # Literal BatchRunConfig.driver expects. driver: str | None = typer.Option( None, "--driver", @@ -474,11 +468,8 @@ def run_pipeline( # --resume needs an explicit run dir to resume into (auto-generated dirs are always fresh). if resume and run_dir is None: raise typer.BadParameter("--resume requires --run-dir pointing at the run to continue.") - # --allow-host-grading only reaches anything from inside the `if resume:` - # branch below, so without --resume it parsed, was accepted, and did nothing - # at all — no warning, no error. The sibling mode-scoped flag on the same - # feature (`evaluate --workspace`) hard-errors on exactly this misuse; two - # new flags behaving differently for one user mistake is the inconsistency. + # Without --resume this flag parsed, was accepted, and did nothing at all. Its + # sibling mode-scoped flag (`evaluate --workspace`) hard-errors on exactly this. if allow_host_grading and not resume: raise typer.BadParameter( "--allow-host-grading applies to --resume only (it decides how an executed-but-ungraded " @@ -497,12 +488,9 @@ def run_pipeline( set_overrides=set_overrides, ) - # Override API backend if --backend was passed. The flag is shorthand for the - # API_BACKEND env var, so mirror it into os.environ as well: the docker driver - # forwards the backend into the container via the standard env passthrough - # (name-only `--env API_BACKEND`, which reads os.environ). A flag that only - # mutated `settings` would be dropped at the container boundary and the - # in-container Settings would silently default to DIRECT. + # Mirrored into os.environ, not just `settings`: the docker driver forwards the + # backend via name-only `--env API_BACKEND`, which reads os.environ. + # Rationale: .claude/notes/isolation.md § Environment forwarding if backend is not None: from coder_eval.models import ApiBackend @@ -577,40 +565,25 @@ async def _run_all_tasks( ) -> None: """Async entry point for running all tasks (optionally in parallel). - Tasks are resolved through the experiment layer (defaulting to - experiments/default.yaml) and executed via run_batch. + Tasks resolve through the experiment layer and execute via run_batch. Args: - task_files: List of task file paths or glob patterns - preservation_mode: Sandbox preservation mode, or None for the driver-derived default - run_dir: Custom run directory (or None for auto-generated) - max_parallel: Maximum number of concurrent tasks - include_tags: Only run tasks matching any of these tags - exclude_tags: Skip tasks matching any of these tags - agent_type: Optional override for agent type (re-parses the union) - overrides: Generic layer-5 task-config overrides (path -> typed value) - from -D/--set and the bespoke flag aliases - stream_mode: Optional stream mode ('full' or 'minimal') for real-time output - experiment_path: Optional path to experiment YAML (default: experiments/default.yaml) - junit_xml: Optional path to write a JUnit XML report to, after the run - summary is persisted and before the failure exit-code gate. - grade: False for `coder-eval execute` — run and capture, score nothing. - format: 'harbor' writes a trajectory.json (ATIF) sibling for every - task.json once the run finishes — see `harbor.atif_emit.emit_trajectories_for_run`. - When the run wrote exactly ONE trajectory (the shape a `CoderEvalAgent` - Harbor agent invocation always produces — one fixed-path agent-phase - task.yaml, no dataset/experiment fan-out), it is additionally copied to - `/trajectory.json` so a caller that pointed `--run-dir` at a - fixed discovery path (e.g. Harbor's `self.logs_dir`) can find it there - without knowing coder-eval's internal `///` - nesting. Multi-task runs are left nested only — there is no single - trajectory to promote. - workspace_dir: Run the single resolved task's agent in-place at this path - instead of run_dir/artifacts (see `BatchRunConfig.workspace_dir` and - `Orchestrator.workspace_dir`). Meant for a `CoderEvalAgent` invocation - inside a container someone else already built (Harbor's), so the - agent's writes land where that container's own verifier looks for - them, rather than in a throwaway tempdir the verifier never sees. + task_files: Task file paths or glob patterns + preservation_mode: Preservation mode, or None for the driver default + run_dir: Run directory, or None for auto-generated + max_parallel: Maximum concurrent tasks + include_tags / exclude_tags: Tag filters + agent_type: Agent-type override (re-parses the union) + overrides: Layer-5 task-config overrides + stream_mode: 'full' or 'minimal' real-time output + experiment_path: Experiment YAML (default: experiments/default.yaml) + junit_xml: Where to write a JUnit XML report, written after the run summary + is persisted and before the failure exit-code gate + grade: False for `coder-eval execute` -- run and capture, score nothing + format: 'harbor' writes a trajectory.json (ATIF) sibling per task.json, and + promotes it to `/trajectory.json` when the run wrote exactly one + (a multi-task run is left nested; there is nothing to promote) + workspace_dir: Run the agent here instead of run_dir/artifacts """ # Prepare run directory run_dir = prepare_run_directory(run_dir) @@ -642,9 +615,8 @@ async def _run_all_tasks( from ..telemetry import flush_telemetry, track_event - # TaskFileCount is the pre-expansion file count (dataset fan-out and variant - # resolution happen later); per-task counts are reconstructable from the - # CoderEval.Task.End events. + # Pre-expansion: dataset fan-out and variant resolution happen later, and + # per-task counts are reconstructable from the CoderEval.Task.End events. track_event( "CoderEval.Run.Start", { @@ -697,28 +669,21 @@ async def _run_all_tasks( # Print execution summary print_execution_summary(run_dir, summary) - # Write the JUnit report (if requested) BEFORE the exit-code gate below, - # so a failing run still produces the report. suite.json + run.json are - # already on disk (written inside _run_with_experiment). A write error - # propagates (loud failure, exit != 0) rather than being swallowed. + # BEFORE the exit-code gate, so a failing run still produces the report. + # Rationale: .claude/notes/orchestration.md § What the exit code counts if junit_xml is not None: from ..reports_junit import write_junit_xml written = write_junit_xml(run_dir, junit_xml) console.print(f"[green][OK]JUnit report written to {written}[/green]") finally: - # Explicit flush before process exit (belt-and-suspenders with atexit). - # In a `finally` so it runs on the success path and on any raised - # exception, but never catches/swallows the typer.Exit decided below. + # In a `finally` so it runs on both paths, without swallowing the + # typer.Exit decided below. flush_telemetry() - # Exit with non-zero code if any tasks failed, errored, or any suite failed its thresholds. - # - # An ungraded row counts too, but only under `run`: `run` was asked for a - # verdict and did not produce one (the grade crashed, or --resume could not - # grade the row), which is a failure of the command even though the row is - # neither `failed` nor `error`. Under `execute` an ungraded row is the - # expected outcome for every task, so it must not fail the command. + # Failures, errors, missed suite thresholds -- and, under `run` only, a row that + # came back ungraded. Under `execute` an ungraded row is the expected outcome. + # Rationale: .claude/notes/orchestration.md § What the exit code counts ungraded_but_asked_to_grade = grade and summary.tasks_not_graded > 0 if summary.tasks_failed > 0 or summary.tasks_error > 0 or failed_suite_gates > 0 or ungraded_but_asked_to_grade: raise typer.Exit(1) @@ -798,25 +763,16 @@ async def _grade_resumed_tasks( """Grade the rows ``coder-eval execute`` left NOT_GRADED, in place. Each task's trajectory and workspace are already on disk, so this runs the - criteria against them instead of re-running the agent — that reuse is the - whole reason to split ``execute`` from ``run``. - - The task config comes from ``rt.task`` (this run's own 5-layer resolution), - not from the recorded one: ``--resume`` re-resolves the same task files, and - a config that drifted since the execute is already surfaced by the run - fingerprint warning above. + criteria against them instead of re-running the agent -- that reuse is the whole + reason to split ``execute`` from ``run``. The task config comes from ``rt.task`` + (this run's own 5-layer resolution), not from the recorded one. A task that cannot be graded is reported and folded back in with its ORIGINAL - ungraded result, so one bad row neither aborts the resume nor silently - vanishes from run.json — it stays visible as ``tasks_not_graded``, with the - reason on its ``error_message``. That covers three shapes: a helper raising, - a row too broken to read at all (skipped entirely — there is nothing to fold - back), and a re-grade that returns ``FinalStatus.ERROR``, which - ``Orchestrator.run()`` produces INSTEAD of raising and which would otherwise - make the row permanently un-regradeable. - - Returns the graded rows; the caller's exit gate fails the command whenever - any row is still ungraded, so a resume that graded nothing never exits 0. + ungraded result, so one bad row neither aborts the resume nor vanishes from + run.json. Returns the graded rows; the caller's exit gate fails the command + whenever any row is still ungraded. + + Rationale: .claude/notes/orchestration.md § When a resumed grade crashes """ from ..orchestration.regrade import ( RegradeError, @@ -829,10 +785,9 @@ async def _grade_resumed_tasks( graded: list[tuple[ResolvedTask, TaskResult]] = [] for rt in to_grade: - # Inside the try: an unreadable row must skip like any other grading - # failure. Outside it, one bad task.json propagates out of the loop and - # aborts the whole resume BEFORE run_batch, so none of the `to_run` - # tasks execute either — the opposite of "one bad row never aborts". + # Inside the try: outside it, one bad task.json aborts the whole resume + # before run_batch, so none of the `to_run` tasks execute either. + # Rationale: .claude/notes/orchestration.md § When a resumed grade crashes prior: EvaluationResult | None = None try: prior = load_prior_result(rt.run_dir) @@ -856,26 +811,13 @@ async def _grade_resumed_tasks( except (RegradeError, OSError, RuntimeError, ValueError) as e: console.print(f"[yellow]⚠[/] Could not grade {rt.task.task_id}: {e}") if prior is None: - # The row could not even be read, so there is no recorded result - # to fold back — but dropping it entirely removes it from - # run.json AND from `tasks_not_graded`, which is what the exit - # gate counts, so a resume whose rows were all unreadable would - # report success. Stand in a minimal ungraded row instead: it - # keeps the task visible and keeps the command non-zero. + # Dropping it removes it from run.json AND from tasks_not_graded, + # which is what the exit gate counts. Stand in a placeholder. result = _unreadable_row_placeholder(rt, e) else: - # Stamp the reason onto the row. Without it the failure survives - # only in this console line: the folded-back result keeps the - # execute phase's empty error_message, so run.json, the reports - # and CI show an ungraded row with no explanation. - # - # APPEND, don't replace. "Keeps the execute phase's empty - # error_message" holds for a NOT_GRADED row and not for one that - # already carries an execution fact — a container-death row - # arrives here with "Container exited with code 137 without - # producing task.json", and overwriting it published a message - # naming the wrong cause while the on-disk record still named - # the right one. + # APPEND, don't replace: a row carrying an execution fact already + # names its own cause, and overwriting published the wrong one + # while the on-disk record still named the right one. result = prior grading_note = f"Grading failed during --resume: {e}" result.error_message = ( @@ -883,18 +825,10 @@ async def _grade_resumed_tasks( ) else: if result.final_status is FinalStatus.ERROR: - # An orchestrator-level grading crash is not a verdict about the - # run. Orchestrator.run() converts internal failures into a - # populated ERROR result rather than raising, so without this the - # `except` above never sees them and the ERROR row replaces a - # perfectly re-gradeable NOT_GRADED one — and ERROR is "complete" - # for both commands, so the row could never be graded again. - # - # Fixing the in-memory result is only half of it: _finalize_result - # already wrote the ERROR task.json into this same directory - # before returning, so run.json would say NOT_GRADED while the - # row on disk says ERROR — and the on-disk one is what a later - # --resume reads. Put the pre-grade record back. + # A grading crash is not a verdict about the run, and + # _finalize_result has already written the ERROR task.json into + # this directory -- which is what a later --resume reads. + # Rationale: .claude/notes/orchestration.md § When a resumed grade crashes restore_pre_grade_record(rt.run_dir) console.print( f"[yellow]⚠[/] Grading {rt.task.task_id} errored ({result.error_message}); " @@ -936,18 +870,13 @@ async def _apply_resume( part = partition_for_resume(resolved, grade=grade) prior_results = list(part.prior_results) prior_resolved = list(part.prior_resolved) - # A re-run task re-executes from scratch, so any leftover artifacts (only - # DIRECT_WRITE writes them live; a container killed mid-run leaves partials) - # are stale and could let a file-based criterion pass on the old output. - # to_grade is deliberately NOT cleared: its artifacts are the run's output - # and the very thing being graded. + # Leftover artifacts from a partial run could let a file-based criterion pass on + # the old output. to_grade is deliberately NOT cleared: its artifacts are what is + # being graded. cleared = clear_rerun_artifacts(part.to_run) - # `to_grade` rows are about to be graded by `_grade_resumed_tasks` below - # (which delegates to `regrade_in_place`), so the same refusal `to_run` - # gets via `_reject_empty_criteria_under_grade` applies here too -- checked - # explicitly rather than relying solely on `regrade_in_place`'s own guard - # so the whole batch is refused up front (exit 2) instead of one row at a - # time turning into a per-task "could not grade" warning mid-resume. + # Checked explicitly rather than left to regrade_in_place's own per-row guard, + # so the whole batch is refused up front instead of one row at a time. + # Rationale: .claude/notes/orchestration.md § Refusing a criteria-free task under grade _reject_empty_criteria_under_grade(part.to_grade, grade=grade) console.print( f"[cyan]↻ Resume:[/] {len(prior_results)} task(s) already complete, " @@ -955,9 +884,8 @@ async def _apply_resume( + (f", grading {len(part.to_grade)} executed-but-ungraded" if part.to_grade else "") + (f" (cleared {cleared} stale artifact dir(s))" if cleared else "") ) - # Grade the rows `execute` left behind, reusing the trajectory and workspace - # already on disk rather than paying for the agent twice. Folded in as - # prior_results so the summary covers them like any other. + # Reusing the trajectory and workspace already on disk rather than paying for + # the agent twice. Folded in as prior_results so the summary covers them. for rt, tr in await _grade_resumed_tasks(part.to_grade, allow_host_grading=allow_host_grading): prior_results.append(tr) prior_resolved.append(rt) @@ -990,24 +918,15 @@ def _reject_simulation_under_execute(resolved: list[ResolvedTask], *, grade: boo def _reject_empty_criteria_under_grade(resolved: list[ResolvedTask], *, grade: bool) -> None: """Refuse a task with zero ``success_criteria`` under ``run``/``evaluate`` rather than scoring it. - ``TaskDefinition.success_criteria`` accepts an empty list at the model level - (needed so the Harbor agent-phase ``task.yaml`` -- criteria-free by design, - see ``harbor/packager.py::_write_agent_phase_task_yaml`` -- can round-trip - through ``coder-eval execute``, which never grades). But `EvaluationResult`'s - scoring is vacuous over an empty list: `all_criteria_passed` returns `True` - and `calculate_weighted_score` returns `0.0`, so a criteria-free task graded - under `run` would silently finalize as `FinalStatus.SUCCESS` with - `weighted_score: 0.0` -- an internally contradictory "successful" result for - what is actually a misconfigured task (a typo, a bad merge, a `-D` override - that cleared the list). `execute` (`grade=False`) is exactly the case this - is legal for, so the check is scoped to `grade` the same way - ``_reject_simulation_under_execute`` scopes its own check. + Scoring is vacuous over an empty list -- `all_criteria_passed` returns True and + `calculate_weighted_score` returns 0.0 -- so such a task would finalize as + SUCCESS at `weighted_score: 0.0`. `execute` (`grade=False`) is exactly the case + this is legal for, so the check is scoped to `grade`. Callers must pass the POST-`--resume` set (``to_run``, not the full - ``resolved``): a resumed, already-finalized row is folded back from - ``prior_results`` and never re-executed or re-graded, so its own - (possibly empty) criteria are moot to this run and must not block one - that is not actually going to grade it. + ``resolved``). + + Rationale: .claude/notes/orchestration.md § Refusing a criteria-free task under grade """ if not grade: return @@ -1079,14 +998,9 @@ async def _run_with_experiment( else: default_experiment = experiment # fall back to custom as its own baseline - # Resolve tasks through experiment layer (applies all 5 config layers). - # Global failures raise ValueError here — duplicate task IDs, early-stop - # arming, or an invocation error that trips every task identically (bad - # --type / -D value, repeats over the cap) — and we surface them as a clean - # CLI error instead of a traceback. Per-task config-resolution failures - # (e.g. sdk_options on a non-claude agent) among otherwise-resolvable tasks - # are NOT raised: resolve_all_tasks isolates them into `skipped` so one - # incompatible task can't abort the whole suite. + # Applies all 5 config layers. GLOBAL failures raise here and surface as a clean + # CLI error; per-task resolution failures are isolated into `skipped`. + # Rationale: .claude/notes/orchestration.md § How a resolution failure reaches the operator try: resolved, skipped = resolve_all_tasks( task_files=all_task_files, @@ -1106,11 +1020,8 @@ async def _run_with_experiment( + "(load errors or `skip: true` — see run.json `skipped_tasks` for reasons)" ) - # Warn (don't refuse) when a --resume config differs from the original run. The - # per-task path key (variant/task_id/NN) doesn't encode the run config, so resumed - # tasks keep their original-config results — surfacing the mismatch makes the - # resulting mixed-config run.json visible instead of silent. Best-effort and - # informational: a missing stamp (run predates this feature) is tolerated. + # Warn, don't refuse: resumed tasks keep their original-config results, so + # surfacing the mismatch makes a mixed-config run.json visible instead of silent. current_fingerprint = compute_run_fingerprint( config, experiment.experiment_id, settings.api_backend.value, settings.bedrock_model ) @@ -1127,10 +1038,8 @@ async def _run_with_experiment( ) write_run_fingerprint(config.run_dir, current_fingerprint) - # On --resume, peel off tasks already finalized in the run dir. They are not - # re-executed but are folded back into run.json (and all downstream reports) - # via prior_results so the summary covers the whole run. `resolved` stays the - # full set — suite rollups below need every task, run or not. + # Peeled off, not re-executed, but folded back via prior_results so the summary + # covers the whole run. `resolved` stays the full set -- suite rollups need it. to_run: list[ResolvedTask] = resolved prior_results: list[TaskResult] = [] prior_resolved: list[ResolvedTask] = [] @@ -1139,11 +1048,8 @@ async def _run_with_experiment( resolved, grade=grade, allow_host_grading=allow_host_grading ) - # Checked against `to_run`, not `resolved`: a `--resume` peels off tasks - # already finalized (folded back from `prior_results`, never re-executed - # or re-graded), so an already-finalized row with empty success_criteria - # (e.g. it was originally run via `execute`) must not block a `run - # --resume` that isn't actually going to grade it. + # Against `to_run`, not `resolved`: an already-finalized row is never re-graded, + # so its own empty criteria must not block this run. _reject_empty_criteria_under_grade(to_run, grade=grade) # Print execution mode @@ -1163,10 +1069,8 @@ async def _run_with_experiment( stream_mode=stream_mode, ) except ValueError as e: - # run_batch's own resolution-time guards (e.g. --workspace-dir requiring - # exactly one non-docker task) raise a plain ValueError -- convert it to - # the same clean CLI error every other resolution-time refusal in this - # function gets, instead of an unhandled traceback. + # run_batch's own resolution-time guards raise plain ValueError; convert to + # the same clean CLI error every other refusal here gets. raise typer.BadParameter(str(e)) from e # Generate experiment reports @@ -1180,11 +1084,10 @@ async def _run_with_experiment( # Reports are written at run root level (no experiment_id subfolder) ExperimentReportGenerator.write_reports(experiment_result, config.run_dir, experiment=experiment) - # Per-suite pass-rate rollups for dataset-backed tasks (no-op when none were used). - # Pass `resolved` through so suite_thresholds on each criterion can be evaluated. - # Skipped entirely under `execute`: a rollup aggregates per-criterion results, - # and there are none — running it would gate a suite on an empty aggregate and - # report a threshold failure for a run that was never measured. + # No-op when no dataset-backed tasks were used. `resolved` is passed through so + # per-criterion suite_thresholds can be evaluated. Skipped entirely under + # `execute`, which produces no per-criterion results to aggregate. + # Rationale: .claude/notes/orchestration.md § What the exit code counts if not grade: return summary, 0 diff --git a/src/coder_eval/cli/run_helpers.py b/src/coder_eval/cli/run_helpers.py index 3c5b74bcd..5b96f7213 100644 --- a/src/coder_eval/cli/run_helpers.py +++ b/src/coder_eval/cli/run_helpers.py @@ -79,11 +79,10 @@ def expand_task_files(task_files: list[Path]) -> list[Path]: typer.Exit: If any pattern matches no task file """ all_task_files = [] - # Per-pattern, not just on the union. Accumulating and checking only the - # total meant one stale entry among several (a renamed or moved suite) - # silently ran the surviving subset and exited 0, so a CI gate reported - # green over tasks it never measured. A pattern the caller wrote is a - # pattern the caller expects to match something. + # Per-pattern, not just on the union: one stale entry among several silently + # ran the surviving subset and exited 0, so a CI gate reported green over tasks + # it never measured. + # Rationale: .claude/notes/orchestration.md § How a resolution failure reaches the operator unmatched = [] for pattern in task_files: if pattern.is_file(): @@ -132,22 +131,18 @@ def print_execution_summary(run_dir: Path, summary: RunSummary) -> None: summary: Run execution summary """ console.print(f"\n[bold green]Run complete:[/bold green] {run_dir}") - # An ungraded run has no pass rate to report — printing "0/N succeeded" for a - # clean `coder-eval execute` reads as a total failure. Report what actually - # happened instead, and keep the graded line for whatever WAS graded. + # An ungraded run has no pass rate: "0/N succeeded" for a clean `execute` reads + # as a total failure. Report what happened instead. + # Rationale: .claude/notes/orchestration.md § What the exit code counts if summary.tasks_not_graded: console.print(f"[bold]Results:[/bold] {summary.tasks_not_graded}/{summary.tasks_run} executed, not graded") - # Point at the run-dir form, not `evaluate `: the - # two-argument shape grades a bare directory with NO trajectory, so - # command_executed / skill_triggered / trajectory-reading judges score - # differently from what `run` would have produced. The run-dir form - # restores the trajectory AND the resolved config. + # The run-dir form, not `evaluate `: the + # two-argument shape grades a bare directory with NO trajectory. console.print(f"[dim]Grade later: uv run coder-eval run --run-dir {run_dir} --resume[/dim]") console.print("[dim] or: uv run coder-eval evaluate ///00[/dim]") if summary.tasks_graded or not summary.tasks_not_graded: - # The `or not ...` keeps the pre-existing "0/0 succeeded" line for an - # empty run: without it a run with no tasks at all prints no Results - # line whatsoever, since both counters are falsy. + # The `or not ...` keeps the "0/0 succeeded" line for an empty run, whose + # counters are both falsy. console.print(f"[bold]Results:[/bold] {summary.tasks_succeeded}/{summary.tasks_graded} succeeded") console.print(f"[dim]View report: open {run_dir / 'experiment.md'}[/dim]") console.print(f"[dim]View report: uv run coder-eval report {run_dir}[/dim]") diff --git a/src/coder_eval/cli/run_task_internal_command.py b/src/coder_eval/cli/run_task_internal_command.py index 786ddba96..85e938d89 100644 --- a/src/coder_eval/cli/run_task_internal_command.py +++ b/src/coder_eval/cli/run_task_internal_command.py @@ -63,34 +63,18 @@ def _arm_host_heartbeat_watchdog(output_dir: Path) -> None: process-lethal code in this command sits behind one named, testable seam instead of being a side effect of the command body. """ - # Start the host-heartbeat watchdog: if the host process dies - # ungracefully (SIGKILL, Claude-Code Escape, crash) before it can - # `docker kill` us, the heartbeat file in output_dir goes stale and - # we self-exit -- otherwise the container would keep burning LLM - # budget orphaned. Daemon thread so it doesn't block normal shutdown. + # Daemon thread so it doesn't block normal shutdown. import os as _os import threading import time - # ARMED ONLY INSIDE THE CONTAINER, and not even defined outside one. The - # watchdog's whole authority is `os._exit(137)` on the process it runs in, - # and the only process that may be reaped that way is the container's own - # disposable main -- there is no container to orphan anywhere else, so - # outside one the thread can do nothing but harm. It did: a test invoked - # this command in-process (legitimately -- the command must refuse a - # malformed context.json, and proving that means calling it) and the pytest - # worker inherited the thread, which found no heartbeat and 40s later exited - # the worker mid-way through an unrelated test file. It named a different - # test on each run and on each platform, carried no traceback, and took that - # worker's coverage data with it -- so the gate reported "65.13 < 80.00", - # naming neither the test nor the cause. - # - # Gated on CODER_EVAL_IN_CONTAINER (set by docker_runner on the container's - # argv), NOT on `driver`, for the same reason the reference-permission - # window is: this command rewrites `driver: docker` -> `tempdir` before - # building the in-container Orchestrator, so a driver-based gate would - # disarm itself on exactly the path that needs it. See - # `Sandbox.enforces_permission_windows`. + # HAZARD: armed ONLY inside the container. The thread's whole authority is + # `os._exit(137)` on the process it runs in, so anywhere else it can only harm + # -- it once killed a pytest worker mid-test-file and took its coverage with it. + # Gated on CODER_EVAL_IN_CONTAINER, NOT on `driver`, for the same reason + # `Sandbox.enforces_permission_windows` is: this command rewrites + # `driver: docker` -> `tempdir` before building the Orchestrator. + # Rationale: .claude/notes/isolation.md § The heartbeat watchdog is armed only inside a container if _os.environ.get(IN_CONTAINER_ENV) == "1": def _watch_host_heartbeat() -> None: @@ -119,11 +103,8 @@ def _watch_host_heartbeat() -> None: "Host heartbeat stale (>%ss); exiting to reap orphan container.", HEARTBEAT_STALE_SECONDS, ) - # os._exit skips atexit and IO flushing, so the error line - # above would routinely be lost -- making a genuine - # stale-heartbeat suicide indistinguishable from an external - # SIGKILL in the archived logs. Flush best-effort first; - # never let a flush failure stop the exit. + # os._exit skips atexit and IO flushing, so this line would + # routinely be lost. Best-effort; never block the exit. import sys as _sys for _handler in logging.getLogger().handlers: @@ -165,10 +146,8 @@ def run_task_internal_command( ), ) -> None: """Run a single staged task inside the container.""" - # Use the same logging path as the host CLI so LOG_LEVEL from the - # forwarded env is honoured. Without this, root stays at INFO and the - # DEBUG-level task_log_handler attached by Orchestrator never sees the - # agent's per-tool-call DEBUG records. + # Same logging path as the host CLI, so the forwarded LOG_LEVEL is honoured and + # Orchestrator's DEBUG-level task_log_handler sees the agent's records. log_level = "DEBUG" if verbose else settings.log_level setup_logging(level=log_level) @@ -184,12 +163,9 @@ def run_task_internal_command( raise typer.Exit(2) context = json.loads(context_json.read_text(encoding="utf-8")) - # Checked, not just annotated. `json.loads` returns `Any`, so pyright accepts - # `variant_id: str = context["variant_id"]` for a value that may be anything - # at all — the annotation reads like a guarantee and enforces nothing. A - # `"replicate_index": "00"` then reached `build_task_run_dir` typed as `int`. - # This is the host→container boundary; the comment below (about `grade` - # being the one raw value) was only true because these two looked checked. + # CHECKED, not just annotated: `json.loads` returns `Any`, so an annotation + # here reads like a guarantee and enforces nothing. + # Rationale: .claude/notes/isolation.md § The context payload is untrusted input variant_id = context["variant_id"] if not isinstance(variant_id, str): typer.echo(f"FATAL: context.json 'variant_id' must be a string, got {variant_id!r}", err=True) @@ -198,55 +174,41 @@ def run_task_internal_command( if not isinstance(replicate_index, int) or isinstance(replicate_index, bool): typer.echo(f"FATAL: context.json 'replicate_index' must be an integer, got {replicate_index!r}", err=True) raise typer.Exit(2) - # The host resolves the driver-derived default before dispatch; the container - # obeys it verbatim. This command only ever runs inside the docker driver, so - # a missing key falls back to the docker default (DIRECT_WRITE) — a deliberate - # default, not version back-compat. + # The host resolves the driver-derived default; the container obeys it. A + # missing key falls back to the docker default -- deliberate, not back-compat. preservation_mode = PreservationMode(context.get("preservation_mode", PreservationMode.DIRECT_WRITE.value)) - # `coder-eval run` vs `coder-eval execute`, decided host-side. Defaults to - # True (grade) so a host that predates `execute` — which never writes the - # key — keeps its exact behavior. - # Coerced, not annotated, like every other value crossing this boundary — - # `grade` was once the only raw one, so a hand-edited or older-format - # `"grade": "false"` arrived as a truthy str typed as bool and silently - # graded a run that asked not to be graded. + # `run` vs `execute`, decided host-side. Defaults to True so a host predating + # `execute` keeps its behaviour. COERCED, like every value crossing this + # boundary: a `"grade": "false"` is a truthy str typed as bool. grade_raw = context.get("grade", True) if not isinstance(grade_raw, bool): typer.echo(f"FATAL: context.json 'grade' must be a boolean, got {grade_raw!r}", err=True) raise typer.Exit(2) grade: bool = grade_raw # A DETACHED GRADE, not a run: seed from the staged prior.json and adopt the - # already-executed workspace instead of starting an agent. Coerced for the - # same reason `grade` is — a hand-edited `"regrade": "false"` is a truthy - # str, and getting this one wrong would re-RUN the agent against a workspace - # the operator asked only to grade, destroying the trajectory being graded. + # already-executed workspace. Getting this one wrong re-RUNS the agent against + # the workspace it was asked only to grade. regrade_raw = context.get("regrade", False) if not isinstance(regrade_raw, bool): typer.echo(f"FATAL: context.json 'regrade' must be a boolean, got {regrade_raw!r}", err=True) raise typer.Exit(2) regrade: bool = regrade_raw - # What task.json RECORDS as the task's source path, as distinct from the - # path this process resolves TASK_DIR against (see Orchestrator's - # `recorded_task_file`). Absent on an older host -> None -> the container - # path is recorded, which is the pre-existing behaviour. + # What task.json RECORDS, as distinct from the path this process resolves + # TASK_DIR against. Absent on an older host -> the container path is recorded. + # Rationale: .claude/notes/orchestration.md § Recording the task as authored host_task_file_raw = context.get("host_task_file") recorded_task_file = Path(host_task_file_raw) if host_task_file_raw else None - # Docker WORKDIR alignment: the host resolves the concrete WORKDIR - # (config value / "auto" -> `docker inspect` / fallback) and forwards it here. - # Absent -> None -> standard run_dir/artifacts workspace. + # Docker WORKDIR alignment, resolved host-side. Absent -> the standard + # run_dir/artifacts workspace. workspace_dir_raw = context.get("workspace_dir") workspace_dir = Path(workspace_dir_raw) if workspace_dir_raw else None config_lineage = {k: ConfigLineageEntry.model_validate(v) for k, v in (context.get("config_lineage") or {}).items()} - # Prefer the host's raw source_yaml so task.json's audit trail matches - # the in-process driver. Fall back to the staged (post-override) YAML - # for older host versions that didn't forward it. + # The host's RAW source_yaml, so task.json's audit trail matches the in-process + # driver. Falls back to the staged post-override YAML on an older host. host_source_yaml: str | None = context.get("source_yaml") - # Load the post-override spec from the staged YAML. We then point - # `task_file` at a path *under the symmetric task_dir mount* so the - # Orchestrator's `task_file.parent` reasoning -- specifically the - # `TASK_DIR` env exposed to `run_command` criteria -- resolves to the - # original host task directory rather than `/work/input/`. + # `task_file` is then pointed under the task_dir mount, so the `TASK_DIR` the + # Orchestrator exposes to `run_command` criteria resolves there, not /work/input. task, source_yaml = load_task(task_yaml) if host_source_yaml is not None: source_yaml = host_source_yaml @@ -254,26 +216,16 @@ def run_task_internal_command( runtime_task_file = task_dir / "task.yaml" if task_dir.is_dir() else task_yaml # Captured BEFORE the rewrite below: this is what `task.json` records. - # Recording the rewritten copy made a docker run's own record claim - # `driver: tempdir`, so `evaluate ` skipped the host-grading - # refusal and the `graded_on_host` stamp entirely. See Orchestrator's - # `recorded_task`. + # Rationale: .claude/notes/orchestration.md § Recording the task as authored authored_task = task # Force driver back to tempdir for the actual in-container run. - # We're already inside the container; another nested docker would be - # both wrong and impossible (no docker CLI in image). if task.sandbox.driver == "docker": # noqa: CE051 — the ONE legitimate rewrite. We are already inside the - # container the docker driver asked for, so the isolation the driver - # names is present, not bypassed; a nested docker would be both wrong - # and impossible (no docker CLI in the image). - # Re-validated rather than `model_copy(update=...)`, matching its sibling - # `regrade.grading_sandbox_config`: `update` skips BOTH pydantic and - # pyright, so a typo would produce a SandboxConfig violating its own - # `Literal` and only surface far downstream. Two driver-rewrite sites - # landing in one change with two different levels of type safety is how - # the weaker one becomes the pattern people copy. + # container the docker driver asked for, so the isolation it names is + # present, not bypassed. Re-validated rather than `model_copy(update=...)`, + # which skips both pydantic and pyright. + # Rationale: .claude/notes/orchestration.md § The in-container driver rewrite rewritten = SandboxConfig.model_validate({**task.sandbox.model_dump(), "driver": "tempdir"}) # noqa: CE051 task = task.model_copy(update={"sandbox": rewritten}) @@ -312,9 +264,7 @@ def run_task_internal_command( recorded_task=authored_task, ) - # Install the stdout-NDJSON stream callback so per-tool-call events - # reach the host. Late import keeps the streaming module out of the - # default --help path. + # Late import keeps the streaming module out of the default --help path. from coder_eval.streaming.wire import StdoutNDJsonCallback orchestrator.stream_callback = StdoutNDJsonCallback() @@ -337,33 +287,19 @@ def _grade_recorded_run( ) -> None: """Grade an already-executed row INSIDE the container that produced it. - This is the container half of `evaluate ` / `run --resume` over a - `driver: docker` task. The host stages `prior.json` next to `task.yaml` and - bind-mounts the executed workspace at ``CONTAINER_GRADE_WORKSPACE``; here we - seed from that row and run its criteria against that workspace. - - Why it must happen here at all: a container task's criteria address the - image's paths and toolchain, so grading them on the host scores a FAILURE for - a run that passed. The host path therefore REFUSES by default and demands - `--allow-host-grading`. Running them back inside the same image is the only - place the verdict means what it meant during the run — so a container-graded - detached row carries no `graded_on_host` stamp, exactly like a `run` row. - - ``task`` is the driver-rewritten copy (docker -> tempdir, done above because - we are already inside the container the driver asked for), which is also what - keeps ``regrade_in_place`` from trying to dispatch a container from within - one. ``authored_task`` is what gets RECORDED, so the row keeps saying - `driver: docker`. ``recorded_task_file`` is the path half of that same - distinction and travels with it: without it the row re-records - ``/work/task_dir/task.yaml`` as its ``source_file``, a path on no host, and a - later ``evaluate `` over the row refuses or mounts the wrong tree. - The ordinary run branch above has always forwarded it; this one is the - second consumer and must not be the one that forgets. - - Delegates to the same ``regrade_in_place`` the host uses rather than - restating it. The two implementations that already drifted apart once — - `evaluate`'s run-dir mode hardcoding `replicate_index=0` and relabelling - every replicate but the first — are the reason that function exists. + The container half of `evaluate ` / `run --resume` over a + `driver: docker` task: the host stages `prior.json` next to `task.yaml` and + bind-mounts the executed workspace at ``CONTAINER_GRADE_WORKSPACE``. + + ``task`` is the driver-rewritten copy (docker -> tempdir), which is also what + keeps ``regrade_in_place`` from dispatching a container from within one. + ``authored_task`` is what gets RECORDED, and ``recorded_task_file`` is the path + half of that same distinction and travels with it. + + Delegates to the same ``regrade_in_place`` the host uses rather than restating + it. + + Rationale: .claude/notes/isolation.md § Grading a docker row inside a container """ from coder_eval.models import CONTAINER_GRADE_WORKSPACE from coder_eval.orchestration.regrade import RegradeError, regrade_in_place @@ -375,10 +311,8 @@ def _grade_recorded_run( try: prior = EvaluationResult.model_validate_json(prior_path.read_text(encoding="utf-8")) except (OSError, ValueError) as e: - # Degrade to a clean message rather than a traceback: the host parses - # this container's task.json, so a crash here surfaces as the opaque - # "container exited without producing task.json" rather than naming the - # staged file that could not be read. + # A clean message, not a traceback: the host parses this container's + # task.json, so a crash here surfaces as the opaque "no task.json". typer.echo(f"FATAL: {prior_path} is not a readable EvaluationResult: {e}", err=True) raise typer.Exit(2) from e @@ -403,8 +337,7 @@ def _grade_recorded_run( ) ) except RegradeError as e: - # Surfaced as a clean message, not a traceback: the host parses this - # container's task.json, and a RegradeError means none was written. Exit - # 2 keeps it distinguishable from an agent failure. + # Clean message, not a traceback. Exit 2 keeps it distinguishable from an + # agent failure. typer.echo(f"FATAL: {e}", err=True) raise typer.Exit(2) from e diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 76dd0f76f..e259dcb4d 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -68,14 +68,12 @@ # RESERVED_CONTAINER_DIRS) are imported above from models.container_paths and # kept in lockstep with docker/coder_eval_entrypoint.sh. -# In-image path of the framework entrypoint, pinned by the host via -# `docker run --entrypoint` (the image bakes no ENTRYPOINT). MUST equal the -# `COPY` destination in docker/Dockerfile -- a drift guard test enforces that. +# MUST equal the `COPY` destination in docker/Dockerfile (drift-guarded by a test). +# Rationale: .claude/notes/isolation.md § The entrypoint and the image contract CONTAINER_ENTRYPOINT = "/usr/local/bin/coder_eval_entrypoint.sh" -# Docker Desktop's stable alias for the host, from inside a bridge-network -# container. Auto-resolves on macOS/Windows; on Linux it must be published -# explicitly via `--add-host host.docker.internal:host-gateway`. +# Docker Desktop's stable host alias from a bridge-network container. Auto-resolves +# on macOS/Windows; on Linux it must be published via `--add-host`. _DOCKER_HOST_ALIAS = "host.docker.internal" _LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) @@ -96,25 +94,10 @@ def _rewrite_loopback_for_container(url: str) -> str | None: return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) -# Top-level entries under ~/.claude that the per-task RW copy SKIPS. We copy -# the host's ~/.claude into a throwaway tmp dir and mount that copy read-WRITE -# so the in-container CLI can write anywhere it needs without ever touching the -# host's real ~/.claude. The container needs only auth + settings + plugins; -# everything else under ~/.claude is heavy, transient, or host-local state it -# never reads, so we drop it to keep the per-task copy cheap. On a real host -# this is the difference between a ~300 MB copy and a few MB: `security/` (the -# security plugin's data) alone is often hundreds of MB, and `projects/` -# (transcripts), `cache/`, `file-history/`, `backups/`, `sessions/`, -# `telemetry/`, `downloads/`, and `shell-snapshots/` all accumulate without -# bound. `session-env/` (per-Bash ephemera) is recreated fresh in the copy by -# the container. The last group is volatile per-session churn the *running* CLI -# rewrites continuously (this harness itself runs inside Claude Code, so the live -# host ~/.claude is mutating while we copy): dropping it both keeps the copy lean -# AND shrinks the window for a mid-walk vanish/rewrite race under --max-parallel -# (the residual race is covered by the bounded retry in `_copy_claude_home`). -# Patterns match by basename at every level (shutil.ignore_patterns semantics), so -# this is a denylist: anything NOT listed here (settings.json, .credentials.json, +# DENYLIST of top-level entries the per-task RW copy of ~/.claude skips. Matched by +# basename at every level, so anything unlisted (settings.json, .credentials.json, # plugins/) is copied through. +# Rationale: .claude/notes/isolation.md § The lean ~/.claude copy CLAUDE_COPY_IGNORE = ( "projects", "shell-snapshots", @@ -136,30 +119,20 @@ def _rewrite_loopback_for_container(url: str) -> str | None: "tasks", ) -# Bounded retries for the lean ~/.claude copy. The live host dir is rewritten by -# the running CLI while we walk it, so a file can vanish mid-copy and raise; a -# couple of retries clears the transient case before we give up (see -# `_copy_claude_home`). +# The live host dir is rewritten while we walk it, so a file can vanish mid-copy. +# Rationale: .claude/notes/isolation.md § The lean ~/.claude copy CLAUDE_COPY_MAX_ATTEMPTS = 3 -# Host-side heartbeat: the runner touches this file every HEARTBEAT_INTERVAL -# seconds while alive. The in-container watchdog exits if the file is stale -# (older than HEARTBEAT_STALE_SECONDS) -- our only defence against the host -# being SIGKILL'd (e.g. Claude Code's Escape) before the asyncio cleanup -# runs. Lives in the output dir, which is bind-mounted into the container. +# The runner touches this file while alive; the in-container watchdog exits if it +# goes stale. Lives in the output dir, which is bind-mounted into the container. +# Rationale: .claude/notes/isolation.md § The heartbeat watchdog is armed only inside a container HEARTBEAT_FILENAME = ".coder_eval_host_heartbeat" HEARTBEAT_INTERVAL_SECONDS = 2.0 HEARTBEAT_STALE_SECONDS = 20 -# asyncio's StreamReader caps a single line at 64 KiB by default. The -# container streams stream events as one NDJSON line each (wire.py), and a -# single event carrying a large tool input -- e.g. an agent Write of a whole -# .flow/.json file -- serialises well past 64 KiB. The default-limit reader -# then raises ValueError mid-stream, which tore the container down before it -# wrote task.json: the entire task was lost and the host recorded a bare -# ERROR with no per-task report. Give the line reader generous headroom (run() -# also degrades gracefully past it). Mirrors Orchestrator._POST_RUN_STREAM_LIMIT, -# the same guard on the orchestrator's post-run subprocesses. +# asyncio's StreamReader caps a line at 64 KiB by default, which a single stream +# event can exceed. Mirrors Orchestrator._POST_RUN_STREAM_LIMIT. +# Rationale: .claude/notes/isolation.md § The stdout line limit STDOUT_LINE_LIMIT_BYTES = 64 * 1024 * 1024 # 64 MiB @@ -249,9 +222,9 @@ def _preflight_image_version(image: str) -> None: timeout=10, ) except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as exc: - # Image absent locally or inspect failed. Let `docker run` raise the - # canonical error; suppress here so we don't double-fail in argv - # logging paths that hit this even when the image is fine. + # Image absent locally or inspect failed: let `docker run` raise the canonical + # error. Suppressed rather than raised because argv-logging paths reach here + # even when the image is fine, and would then double-fail. logger.debug("Pre-flight image inspect failed for %s: %s", image, exc) return image_version = result.stdout.strip() @@ -272,10 +245,8 @@ def _preflight_image_version(image: str) -> None: _CONTAINER_NAME_INVALID = re.compile(r"[^a-zA-Z0-9_.-]") -# A leading Windows drive letter (``C:\foo`` / ``c:/foo``). Used so the colon -# in ``C:\foo`` is not misread as the ``src:dst`` separator when a Windows -# task author writes an extra_mounts entry. Bare ``C:`` (no path body) is -# intentionally not matched — that is malformed and should fail downstream. +# A leading Windows drive letter. Bare ``C:`` is deliberately not matched. +# Rationale: .claude/notes/isolation.md § Extra mounts and reserved destinations _DRIVE_PREFIX = re.compile(r"^[A-Za-z]:[\\/]") @@ -288,10 +259,8 @@ def _sanitize_container_name_component(s: str) -> str: return _CONTAINER_NAME_INVALID.sub("_", s) -# Destinations that would shadow framework-owned mounts inside the container. -# Letting a user spec collide with these silently breaks input/output staging. -# Same reserved set the workspace-dir validator uses (single source of truth in -# models.container_paths). Extra-mount destinations and WORKDIR both reject these. +# Single source of truth in models.container_paths; extra-mount destinations and +# WORKDIR both reject these. _RESERVED_MOUNT_DESTS = RESERVED_CONTAINER_DIRS @@ -314,10 +283,7 @@ def _validate_extra_mount(spec: str) -> str: - Destinations colliding with framework mounts (``/work``, ``/``, etc.) are rejected outright. """ - # Split off an optional leading Windows drive letter so the colon in - # ``C:\foo`` is not misread as the ``src:dst`` separator. The container - # side is always POSIX (Docker containers are Linux), so only the source - # side can carry a drive letter. + # The container side is always POSIX, so only the source can carry a drive letter. if _DRIVE_PREFIX.match(spec): head, body = spec[:2], spec[2:] else: @@ -326,9 +292,7 @@ def _validate_extra_mount(spec: str) -> str: if len(parts) < 2 or len(parts) > 3: raise ValueError(f"Invalid extra_mounts entry {spec!r}: expected `src:dst[:ro|rw]`.") src, raw_dst = head + parts[0], parts[1] - # Default to read-only when mode is omitted. Mounting host paths RW - # by default is the wrong sandbox stance: the few RW use-cases are - # better stated explicitly than implied by silence. + # Default read-only: mounting host paths RW by default is the wrong sandbox stance. mode = parts[2] if len(parts) == 3 else "ro" if not src: raise ValueError(f"Invalid extra_mounts entry {spec!r}: empty source path.") @@ -337,9 +301,8 @@ def _validate_extra_mount(spec: str) -> str: # Expanded before the absolute-path check: that is the point. expanded_src = os.path.expandvars(os.path.expanduser(src)) dst = os.path.expandvars(os.path.expanduser(raw_dst)) - # A variable whose value carries a ':' would add fields to the spec rebuilt - # at the bottom, silently moving the destination or widening the mode. - # The drive prefix is excluded: its colon is legitimate and already split off. + # HAZARD: a variable expanding to a ':' would add fields to the rebuilt spec, + # silently moving the destination or widening the mode. if ":" in dst or ":" in expanded_src[len(head) :]: raise ValueError(f"Invalid extra_mounts entry {spec!r}: expansion introduced a ':' into a path.") if not dst.startswith("/"): @@ -350,10 +313,8 @@ def _validate_extra_mount(spec: str) -> str: raise ValueError(f"Invalid extra_mounts entry {spec!r}: mode must be 'ro' or 'rw'.") if not Path(expanded_src).exists(): raise ValueError(f"Invalid extra_mounts entry {spec!r}: source path does not exist on host.") - # Reject destinations that shadow framework-owned mounts inside the - # container. ``/work`` substrings are caught too -- /work/foo would - # land underneath our staging dir and shadow the input/output tree. - # Expanded form: a var could itself expand to a reserved path. + # Checked in EXPANDED form: a var could itself expand to a reserved path, and a + # ``/work/...`` destination shadows the input/output tree. dst_norm = dst.rstrip("/") or "/" if dst_norm in _RESERVED_MOUNT_DESTS or dst_norm.startswith(CONTAINER_WORK_DIR + "/"): raise ValueError( @@ -458,12 +419,9 @@ def _copy_claude_home(host_claude_dir: Path, claude_copy: Path) -> None: host_claude_dir, claude_copy, ignore=shutil.ignore_patterns(*CLAUDE_COPY_IGNORE), - # Copy symlinks AS symlinks (do not follow): a plugin marketplace - # cache can contain a self-referential symlink (e.g. uipath-marketplace - # `plugins/uipath -> ..`) that makes a symlink-following walk recurse - # infinitely ("too many levels of symbolic links") and abort the copy. - # Copying them verbatim is correct and loop-proof. Dangling ones are - # skipped via ignore_dangling_symlinks. + # AS symlinks, not followed: a self-referential marketplace link + # makes a following walk recurse infinitely. + # Rationale: .claude/notes/isolation.md § The lean ~/.claude copy symlinks=True, ignore_dangling_symlinks=True, dirs_exist_ok=True, @@ -489,47 +447,26 @@ def _copy_claude_home(host_claude_dir: Path, claude_copy: Path) -> None: def grant_container_access(root: Path, *, writable: bool) -> list[tuple[Path, int]]: """Widen ``root`` (recursively) so the container can reach it without DAC caps. - Paired with the ``--cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH`` in - :meth:`DockerRunner._build_argv`. The container runs as **root but is not - the owner** of any framework-owned bind mount: on native Linux the mount - preserves the uid that ran ``coder-eval`` (uid 1000/1001), so every access - root makes to those paths is an "other" access. It only ever succeeded via - ``CAP_DAC_OVERRIDE``. Dropping that capability to make the reference's - mode-000 window real therefore also revoked the container's ability to write - its own output -- the in-container orchestrator died on the very first - ``open('/work/output/task.log', 'w')`` with EACCES, taking every - ``driver: docker`` task with it (regression-guarded by - ``TestContainerAccessWidening``). - - Widening the *host* side restores that access through the ``other`` bits - instead of through a capability, which is what keeps the drop affordable. - Semantics match ``chmod -R o+rwX`` (``o+rX`` when ``writable=False``): the - ``X`` form adds execute only to directories and to files that are already - executable, so a copied hook script stays runnable and a data file does not - silently become one. - - ``writable=False`` is not cosmetic -- it is what keeps ``/work/references`` - off the list of things the agent can overwrite. The container only ever - *reads* and ``chmod``s that copy (``chmod`` is gated on owner-or-CAP_FOWNER, - and FOWNER is deliberately retained), so it needs no write bit, and - withholding it keeps ``_verify_reference_integrity`` from being the sole - guard against tampering. - - Returns ``(path, original_mode)`` for every entry it actually changed, so a - caller that widened a tree it does not own can put it back (see - :func:`restore_modes`). The framework-created staging dirs are disposable and - ignore it; the graded workspace is not. - - No-op on Windows, where POSIX mode bits are not the access-control mechanism. + COUNTERPART to the ``--cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH`` in + :meth:`DockerRunner._build_argv`: the container is root but owns no + framework-owned mount, so every access it makes is an "other" access. Semantics + match ``chmod -R o+rwX`` (``o+rX`` when ``writable=False``). ``writable=False`` + is load-bearing, not cosmetic -- it keeps ``/work/references`` off the list of + things the agent can overwrite. + + Returns ``(path, original_mode)`` for every entry it changed, so a caller that + widened a tree it does not own can put it back (see :func:`restore_modes`). + No-op on Windows. + + Rationale: .claude/notes/isolation.md § grant_container_access """ widened_paths: list[tuple[Path, int]] = [] if os.name == "nt": # pragma: no cover - POSIX mode bits are meaningless here return widened_paths extra = 0o006 if writable else 0o004 for path in (root, *root.rglob("*")): - # lstat + skip: chmod follows symlinks, so widening one would silently - # re-mode its target -- which for the ~/.claude copy can be an arbitrary - # path outside the staging tree (it is copied with symlinks=True). + # HAZARD: chmod follows symlinks, so widening one would re-mode its target -- + # for the ~/.claude copy, an arbitrary path outside the staging tree. if path.is_symlink(): continue try: @@ -613,39 +550,28 @@ def __init__( self.stream_callback = stream_callback self.verbose = verbose # DETACHED GRADE. Both set together or neither: `prior_result` is the - # already-executed row (trajectory + execution facts) the in-container - # Orchestrator seeds from, and `grade_workspace` is the host directory - # that run left behind, mounted at CONTAINER_GRADE_WORKSPACE and ADOPTED - # rather than recreated. - # - # This is what makes `evaluate` over a `driver: docker` row honest. The - # criteria of such a task address container paths and the image's - # toolchain, so grading them on the host scores a FAILURE for a run that - # passed. Running them back inside the same image is not a workaround for - # that — it is the only place the verdict means what it meant during the - # run. + # already-executed row the in-container Orchestrator seeds from, and + # `grade_workspace` is the host directory that run left behind, mounted at + # CONTAINER_GRADE_WORKSPACE and ADOPTED rather than recreated. + # Rationale: .claude/notes/isolation.md § Grading a docker row inside a container self.prior_result = prior_result self.grade_workspace = grade_workspace if (prior_result is None) != (grade_workspace is None): raise ValueError("prior_result and grade_workspace must be passed together") - # Forwarded to the in-container orchestrator via context.json. It is a - # run-level decision made by the CLI, so it cannot be recovered from the - # staged task.yaml on the other side. + # Forwarded via context.json: a run-level decision by the CLI, not + # recoverable from the staged task.yaml on the other side. self.grade = grade - # Set by _prepare_host_mounts: the tmp lean copy of ~/.claude that - # _build_argv mounts read-write. None when there is no ~/.claude to - # forward or the mount is opted out (CODER_EVAL_NO_CLAUDE_MOUNT). + # Set by _prepare_host_mounts: the lean RW copy of ~/.claude. None when + # there is none to forward or CODER_EVAL_NO_CLAUDE_MOUNT is set. self._claude_mount_src: Path | None = None - # Set by _prepare_host_mounts: a throwaway copy of the reference - # directory, mounted read-WRITE at CONTAINER_REFERENCE_DIR. It must be a - # copy, and it must be writable -- see _prepare_host_mounts. + # A throwaway COPY of the reference, mounted read-WRITE. Both are + # load-bearing -- see _prepare_reference_mount. self._reference_mount_src: Path | None = None # Host path the copy came from, cached by _prepare_reference_mount so the # argv builder doesn't re-stat it (and re-emit its warning). self._reference_source_dir: Path | None = None - # Set by _prepare_task_dir_mount: a throwaway copy of the task directory, - # mounted read-WRITE at CONTAINER_TASK_DIR so the agent-turn window can - # chmod it. None when the task has no task_file. + # A throwaway COPY of the task dir, mounted read-WRITE so the agent-turn + # window can chmod it. None when the task has no task_file. self._task_dir_mount_src: Path | None = None # Resolved in run() (needs the built image for "auto"). Concrete WORKDIR the # agent runs at + copies out from; None = standard artifacts workspace. @@ -668,18 +594,13 @@ async def run(self) -> EvaluationResult: dispatcher converts that to an ERROR-status EvaluationResult. """ _preflight() - # Resolve the run image: build from a Dockerfile if configured (which - # overrides `image`), else use the configured image. The build is - # side-effecting, so it runs in a worker thread like the other docker - # calls in this method. + # Side-effecting, so it runs in a worker thread like the other docker calls. try: image = await asyncio.to_thread(self._build_image) except DockerBuildError as exc: - # The build happens before run_dir/docker.log/task.json exist, so a - # build failure would otherwise leave an empty result dir with no - # trace. Persist the build log to docker.log and a BUILD_FAILED - # synthetic task.json so the failure is visible per-task, then - # re-raise for the batch dispatcher to record run-level. + # The build precedes run_dir/docker.log and task.json, so persist the + # log and a BUILD_FAILED record before re-raising. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json await self._record_build_failure(exc) raise # The version-label preflight only makes sense for the framework image; @@ -688,18 +609,13 @@ async def run(self) -> EvaluationResult: await asyncio.to_thread(_preflight_image_version, image) await asyncio.to_thread(self.rt.run_dir.mkdir, parents=True, exist_ok=True) - # Docker WORKDIR alignment: resolve the concrete workspace path - # once, host-side (config value / "auto" -> inspect the built image / fallback - # /root). Forwarded to the in-container orchestrator via the staged context - # and rendered as `docker run -w`. None keeps the standard artifacts workspace. + # Docker WORKDIR alignment: config value / "auto" -> inspect / fallback /root. + # None keeps the standard artifacts workspace. self._workspace_dir = await asyncio.to_thread(_resolve_workspace_dir, self._docker_config.working_dir, image) - # Stage only the inputs (task YAML + context). The *output* dir is - # the host's run_dir itself, bind-mounted at the same path inside - # the container so the in-container Orchestrator writes - # task.json/task.log/task.html/artifacts/ straight into the host - # filesystem -- no copy step, paths are symmetric inside and out. - # Sanitize task_id: dataset ids are ``suite_id/row_id`` and the ``/`` breaks mkdtemp (missing parent dir). + # Stage only the inputs. The OUTPUT dir is the host's run_dir itself, + # bind-mounted at the same path inside the container, so paths are symmetric. + # Sanitize task_id: dataset ids are ``suite_id/row_id`` and ``/`` breaks mkdtemp. safe_staging_id = _sanitize_container_name_component(self.rt.task.task_id) staging = Path(await asyncio.to_thread(tempfile.mkdtemp, prefix=f"coder_eval_docker_{safe_staging_id}_")) input_dir = staging / "input" @@ -712,49 +628,26 @@ async def run(self) -> EvaluationResult: try: await self._stage_inputs(input_dir) - # Give the container a stable, *unique* name so cancellation can - # target it. PID alone collides under --max-parallel >1 (same - # host process spawns N concurrent containers); the uuid suffix - # and replicate_index disambiguate. Sanitize+truncate task_id - # so dataset row ids like ``suite/row`` don't break docker name - # validation. + # Stable and UNIQUE so cancellation can target it: PID alone collides + # under --max-parallel >1. Sanitized and truncated -- dataset row ids break + # docker name validation, and a 30-char cap collided on shared prefixes. short_uuid = uuid.uuid4().hex[:8] - # Docker name limit is 253 chars; keep generous task_id headroom - # so `docker ps` rows stay readable. Earlier 30-char cap collided - # visibly on long shared prefixes; 80 covers all realistic ids - # while leaving room for the suffix. safe_task_id = _sanitize_container_name_component(self.rt.task.task_id)[:80] container_name = f"coder-eval-{safe_task_id}-r{self.rt.replicate_index}-{os.getpid()}-{short_uuid}" - # Side-effecting prep that _build_argv must NOT do (argv rendering - # stays pure for testability). Makes a lean RW copy of ~/.claude - # under `staging` and records it on self._claude_mount_src for - # _build_argv to mount. Cleaned up with `staging` in the finally. + # Side-effecting prep _build_argv must NOT do: argv rendering stays pure + # so it is testable without a docker daemon. Cleaned up with `staging`. await asyncio.to_thread(self._prepare_host_mounts, staging) await asyncio.to_thread(self._prepare_reference_mount, staging) await asyncio.to_thread(self._prepare_task_dir_mount, staging) - # AFTER staging, BEFORE the container starts: the DAC caps are - # dropped, so every framework-owned mount must be reachable through - # its `other` bits. Read-only for the inputs the container merely - # consumes; writable only for the run dir it must produce into. + # AFTER staging, BEFORE the container starts: the DAC caps are dropped, so + # every framework-owned mount must be reachable through its `other` bits. + # Rationale: .claude/notes/isolation.md § grant_container_access await asyncio.to_thread(grant_container_access, input_dir, writable=False) await asyncio.to_thread(grant_container_access, output_dir, writable=True) if self.grade_workspace is not None: - # The graded workspace is a framework-owned mount like any other, - # so it needs the same widening -- and it is the one mount whose - # files the harness did NOT create, so the owner bits cannot be - # assumed. It happens to work when container #1 (running as root) - # wrote the tree, which is exactly what makes the broken case - # expensive: an operator-supplied `--workspace`, or artifacts - # re-created host-side, are owned by the host uid, and container - # root without DAC_OVERRIDE reaches them only through `other`. - # Criteria then fail EACCES and book a gating 0.0 that reads as - # an agent failure -- the CE039 shape this feature exists to end. - # - # Recorded and restored in the `finally` below, unlike the two - # staging dirs above: those are disposable and deleted with the - # dispatch, while this tree survives it. An operator-supplied - # `--workspace` left world-writable forever is a real, permanent - # exposure on a shared host. + # The one mount whose files the harness did NOT create, so the owner + # bits cannot be assumed -- and the one that SURVIVES the dispatch, + # which is why it is recorded and restored in the `finally` below. widened_workspace = await asyncio.to_thread(grant_container_access, self.grade_workspace, writable=True) argv = self._build_argv(input_dir, output_dir, container_name=container_name, image=image) logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) @@ -771,35 +664,28 @@ async def run(self) -> EvaluationResult: ) log_path = self.rt.run_dir / DOCKER_LOG_FILENAME log_fh = await asyncio.to_thread(log_path.open, "w", encoding="utf-8") - # Cancellation guard: `docker run --rm` does NOT propagate kill - # to the container daemon-side. Without this `finally`, Ctrl-C - # on the host leaves the container running and burning LLM - # budget. Covers CancelledError, KeyboardInterrupt, and any - # other exit-by-exception path uniformly. + # HAZARD: `docker run --rm` does NOT propagate a kill daemon-side, so + # without this `finally` Ctrl-C leaves the container burning budget. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json try: returncode = await self._stream_container_output(proc, log_fh) finally: heartbeat_task.cancel() - # await the cancellation so the task doesn't outlive us; - # narrow to CancelledError so genuine KeyboardInterrupt / - # SystemExit from a parallel sibling still propagates. + # Narrowed so a genuine KeyboardInterrupt / SystemExit from a + # parallel sibling still propagates. with contextlib.suppress(asyncio.CancelledError): await heartbeat_task await asyncio.to_thread(log_fh.close) - # If proc is still alive we got cancelled mid-flight. Kill - # the container *and* the docker CLI subprocess. Best-effort, - # no exception leak from cleanup. + # Cancelled mid-flight: kill the container AND the docker CLI + # subprocess, best-effort. if proc.returncode is None: await self._kill_container(proc, container_name) return await self._parse_result_or_raise(output_dir, returncode, log_path) finally: - # rmtree_restrictive, not rmtree(ignore_errors=True): `staging` - # holds the /work/references copy, which the in-container - # orchestrator keeps at mode 000 for the whole of every turn. A - # container killed mid-turn never restores it, and scandir on a 000 - # directory raises PermissionError -- which ignore_errors swallows, - # orphaning a tempdir that holds the reference solution. + # rmtree_restrictive, not ignore_errors: `staging` holds the references + # copy, which a container killed mid-turn leaves at mode 000. + # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies await asyncio.to_thread(rmtree_restrictive, staging) # The graded workspace is the caller's tree, not ours; give it back # the modes it had. See `restore_modes`. @@ -810,10 +696,9 @@ async def _stage_inputs(self, input_dir: Path) -> None: staging ``input_dir`` (``task.yaml`` + ``context.json``). Pure I/O off the event loop; no control-flow change. """ - # Always serialise the *post-override* TaskDefinition. We can't use - # rt.source_yaml because that's the raw on-disk text -- _apply_cli_overrides - # has since mutated rt.task in-memory (e.g. --model, -D run_limits.max_turns), and the - # container needs to see those mutations. + # POST-override, not rt.source_yaml: _apply_cli_overrides has since mutated + # rt.task in-memory and the container must see those mutations. + # Rationale: .claude/notes/isolation.md § The context payload is untrusted input task_yaml_in = input_dir / "task.yaml" def _dump_task_yaml() -> str: @@ -821,11 +706,8 @@ def _dump_task_yaml() -> str: task_yaml_text = await asyncio.to_thread(_dump_task_yaml) await asyncio.to_thread(task_yaml_in.write_text, task_yaml_text, encoding="utf-8") - # Lineage + variant metadata so the in-container Orchestrator - # reconstructs the same context (variant_id is load-bearing for - # report grouping). source_yaml carries the *raw* on-disk text - # so the in-container Orchestrator records the same audit trail - # as the in-process driver (task.json.task_config.source_yaml). + # Lineage + variant metadata so the in-container Orchestrator reconstructs + # the same context (variant_id is load-bearing for report grouping). context_payload = json.dumps( { "variant_id": self.rt.variant_id, @@ -835,18 +717,13 @@ def _dump_task_yaml() -> str: # `coder-eval run` vs `coder-eval execute`. Not derivable from # task.yaml on the container side (deliberately not a task field). "grade": self.grade, - # A detached grade: seed from prior.json (staged beside this - # file) and adopt CONTAINER_GRADE_WORKSPACE instead of running - # an agent. Absent/False on every ordinary run. + # A detached grade: seed from prior.json and adopt + # CONTAINER_GRADE_WORKSPACE instead of running an agent. "regrade": self.prior_result is not None, "source_yaml": self.rt.source_yaml, - # The HOST's task-file path, recorded verbatim into task.json's - # audit trail. The container resolves TASK_DIR against - # /work/task_dir/task.yaml, which is right in there and exists on - # no host -- recording THAT made a detached grade of this row - # rebuild the task around an unresolvable path and silently mount - # no task dir. Absent -> the container falls back to its own path, - # so an older host keeps today's behaviour. + # The HOST's path, recorded verbatim into task.json's audit trail -- + # distinct from the container path TASK_DIR resolves against. + # Rationale: .claude/notes/orchestration.md § Recording the task as authored "host_task_file": str(self.rt.task_file) if self.rt.task_file else None, # Docker WORKDIR alignment: concrete path the in-container # orchestrator runs at + captures out (None = standard workspace). @@ -855,9 +732,8 @@ def _dump_task_yaml() -> str: ) await asyncio.to_thread((input_dir / "context.json").write_text, context_payload, encoding="utf-8") if self.prior_result is not None: - # The row being graded, carried in whole. The container seeds from - # it exactly as the host path does, so the trajectory an `llm_judge` - # or `command_executed` criterion reads is the ORIGINAL run's. + # Carried in whole, so the trajectory an `llm_judge` or + # `command_executed` criterion reads is the ORIGINAL run's. await asyncio.to_thread( (input_dir / PRIOR_RESULT_FILENAME).write_text, self.prior_result.model_dump_json(indent=2), @@ -874,20 +750,15 @@ async def _stream_container_output(self, proc: asyncio.subprocess.Process, log_f touches the heartbeat/log-fh/container teardown. """ assert proc.stdout is not None - # Explicit readline loop (not `async for`) so a single - # over-limit line degrades to a dropped line instead of a - # ValueError that tears the whole task down -- see below. + # Explicit readline loop (not `async for`) so a single over-limit line + # degrades to a dropped line instead of tearing the task down. + # Rationale: .claude/notes/isolation.md § The stdout line limit while True: try: raw_line = await proc.stdout.readline() except ValueError: - # A single line exceeded STDOUT_LINE_LIMIT_BYTES. - # readline() drains the offending bytes and resyncs at - # the next newline, so we keep streaming. The dropped - # line is a STREAM_EVENT (host-side live render) or a - # log line; task.json crosses via the bind mount, not - # stdout, so the task result is unaffected. Degrade, - # don't die. + # readline() drains the offending bytes and resyncs at the next + # newline. task.json crosses via the bind mount, not stdout. logger.warning( "Dropped a stdout line over %d bytes from task %r's container; continuing to stream.", STDOUT_LINE_LIMIT_BYTES, @@ -897,14 +768,10 @@ async def _stream_container_output(self, proc: asyncio.subprocess.Process, log_f if not raw_line: break line = raw_line.decode("utf-8", errors="replace").rstrip("\n") - # Three-way split: - # - Has the wire-format prefix AND parses cleanly -> emit - # to the host StreamCallback; do not echo to docker.log - # (the StreamCallback is the canonical destination). - # - Has the prefix but parses badly -> wire bug; - # deserialize_event already logged a WARN. Preserve - # the raw line in docker.log so it isn't lost. - # - No prefix -> plain log line. + # Three-way split: wire-prefixed and parses -> the host StreamCallback + # (canonical destination, not echoed to docker.log); prefixed but + # unparseable -> wire bug, preserved raw so it is not lost; no prefix -> + # plain log line. if has_prefix(line): event = deserialize_event(line) if event is not None: @@ -938,10 +805,8 @@ async def _kill_container(self, proc: asyncio.subprocess.Process, container_name if kill_result.returncode == 0: logger.info("Container %s killed cleanly.", container_name) else: - # Non-zero from `docker kill` typically means the - # container was already gone (race with --rm) OR - # the daemon refused. Surface stderr so the - # ambiguity is debuggable. + # Usually the container was already gone (race with --rm) or the + # daemon refused. Surface stderr so the ambiguity is debuggable. logger.warning( "docker kill %s returned %s; container may already be gone or daemon refused: %s", container_name, @@ -959,9 +824,8 @@ async def _kill_container(self, proc: asyncio.subprocess.Process, container_name logger.warning("docker kill failed: %s", kill_exc) with contextlib.suppress(ProcessLookupError): proc.kill() - # Narrow to CancelledError -- a generic BaseException - # catch here would silently eat KeyboardInterrupt / - # SystemExit propagation from parallel tasks. + # HAZARD: narrow to CancelledError -- a generic BaseException catch here + # eats KeyboardInterrupt / SystemExit propagation from parallel tasks. with contextlib.suppress(asyncio.CancelledError): await proc.wait() @@ -974,13 +838,9 @@ async def _parse_result_or_raise(self, output_dir: Path, returncode: int, log_pa """ task_json = output_dir / TASK_JSON_FILENAME if not await asyncio.to_thread(task_json.exists): - # The container died before its orchestrator's `finally` could - # write task.json (e.g. it was torn down by the cleanup above - # after a host-side stream failure, or killed externally). - # Persist a synthetic ERROR task.json so the test stays - # visible on dashboards/timelines instead of silently - # vanishing -- the batch layer's in-memory skeleton never - # reaches the per-task dir. + # Persist a synthetic ERROR task.json so the row stays visible instead of + # vanishing -- the batch layer's skeleton never reaches the per-task dir. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json error = DockerRunError( f"Container exited with code {returncode} without producing task.json. " + f"See {log_path} for container output." @@ -993,9 +853,8 @@ async def _parse_result_or_raise(self, output_dir: Path, returncode: int, log_pa try: result = EvaluationResult.model_validate_json(task_json_text) except ValueError as exc: - # Present but unparseable (schema skew from a stale image, or a - # truncated/torn write). Degrade like the missing-file branch - # rather than crashing with an uncaught ValidationError/JSONDecodeError. + # Present but unparseable (schema skew from a stale image, a torn + # write): degrade like the missing-file branch. raise await self._handle_malformed_task_json(task_json, log_path, exc) from exc self._warn_on_version_mismatch(result) self._assert_grade_honored(result, task_json) @@ -1005,28 +864,17 @@ async def _parse_result_or_raise(self, output_dir: Path, returncode: int, log_pa def _assert_regrade_honored(self, result: EvaluationResult, task_json: Path | None = None) -> None: """Fail loudly when a detached GRADE came back as a fresh agent run. - Exactly the sibling of :meth:`_assert_grade_honored`, for exactly the - same reason one release later. ``regrade`` crosses the boundary only - through ``context.json``; an image that predates container-side grading - ignores the unknown key, ignores the staged ``prior.json``, ignores the - ``/work/workspace`` mount, and falls through to the ordinary - ``Orchestrator`` branch -- which **starts an agent** from - ``initial_prompt``. - - Nothing else catches it. ``_warn_on_version_mismatch`` only warns (and is - skipped entirely for ``dockerfile_path`` tasks), and - ``_assert_grade_honored`` early-returns because a grading container is - dispatched with ``grade=True``. So the host would fold a fabricated - trajectory back over the recorded row as its "grade" -- publishing a - verdict for work it never looked at, and billing the model for it. - - Keyed on EVIDENCE, like its sibling: a container that honored the request - seeds from ``prior`` and never runs the agent, so the trajectory it - returns is the one we sent in. A DIFFERENT ``started_at`` is the tell -- - ``_seed_from_prior_result`` restores the agent run's ``started_at`` - verbatim (deliberately, so a re-graded row does not report the grading - pass's 2 seconds into ``average_duration``), so a fresh run is the only - way that field can move. + ``regrade`` crosses the boundary only through ``context.json``; an image + that predates container-side grading ignores it and falls through to the + ordinary ``Orchestrator`` branch -- which **starts an agent**. Nothing else + catches it: ``_assert_grade_honored`` early-returns because a grading + container is dispatched with ``grade=True``. + + Keyed on EVIDENCE: a container that honored the request seeds from + ``prior`` and never runs the agent, so a DIFFERENT ``started_at`` is the + tell. + + Rationale: .claude/notes/isolation.md § The two honored-request guards """ if self.prior_result is None: return @@ -1044,30 +892,20 @@ def _assert_regrade_honored(self, result: EvaluationResult, task_json: Path | No def _assert_grade_honored(self, result: EvaluationResult, task_json: Path | None = None) -> None: """Fail loudly when `execute` came back with a graded verdict. - ``grade`` crosses the boundary only through ``context.json``. An image - that predates ``execute`` ignores the unknown key and grades anyway, and - the image-version preflight only warns — so ``execute --driver docker`` - against a stale image would silently produce SUCCESS/FAILURE rows that - look like a normal graded run. Version skew must not change what a - command MEANS, so refuse the row rather than publish it. - - ``task_json`` is the on-disk record, quarantined before the raise. The - refusal used to be in-memory only, which left the graded ``task.json`` - sitting in the bind-mounted host run dir: a later - ``execute --resume`` read it back as a completed row (its category is - ``succeeded``, so the resume partition files it under prior results) and - plain ``aggregate`` folded it straight into ``run.json`` — publishing - exactly the row this guard declined to publish. Refusing in memory while - leaving contradictory bytes on disk is not a refusal. + ``grade`` crosses the boundary only through ``context.json``. An image that + predates ``execute`` ignores the unknown key and grades anyway, and the + image-version preflight only warns -- so version skew would change what a + command MEANS. + + ``task_json`` is the on-disk record, quarantined before the raise: refusing + in memory while leaving contradictory bytes on disk is not a refusal. + + Rationale: .claude/notes/isolation.md § The two honored-request guards """ if self.grade: return - # Keyed on EVIDENCE, not on the label. Exempting every execution-fact - # status let a stale image return a fully graded MAX_TURNS_EXHAUSTED row - # — criteria vector, weighted score and all — unchallenged, because the - # exemption exists for statuses a *fresh* image also produces, and a - # fresh one produces them with neither. The question is not "what status - # is this" but "did it grade". + # Keyed on EVIDENCE, not on the label: the question is not "what status is + # this" but "did it grade". graded_anyway = bool(result.success_criteria_results) or result.weighted_score is not None if not graded_anyway and ( result.final_status.is_execution_fact or result.final_status is FinalStatus.NOT_GRADED @@ -1157,14 +995,10 @@ async def _write_synthetic_task_json( def _write() -> None: if target.exists(): return - # Through `write_text_atomic` like every other writer of this file. - # The hand-rolled tmp+replace here used `Path.write_text`, which - # FOLLOWS symlinks — so a pre-planted `task.json.synthetic.tmp` in a - # run directory (a shareable artifact, bind-mounted writable into the - # agent's own container) redirected this harness-privileged write to - # any path the grading user could reach. It also falsified the - # helper's "one writer, so the crash semantics cannot differ" claim, - # which is the property future readers rely on. + # HAZARD: through `write_text_atomic` like every other writer of this + # file. A hand-rolled `Path.write_text` FOLLOWS symlinks, and a run + # directory is bind-mounted writable into the agent's own container. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json write_text_atomic(target, result.model_dump_json(indent=2)) try: @@ -1248,43 +1082,22 @@ def _prepare_host_mounts(self, staging: Path) -> None: return claude_copy = staging / "claude-home" _copy_claude_home(host_claude_dir, claude_copy) - # Writable: the CLI rewrites settings/state in place. copytree preserves - # the host modes, and ~/.claude is routinely 0700 with 0600 files -- with - # DAC_OVERRIDE dropped that is unreadable to the container, so the agent - # cannot authenticate. + # Writable: the CLI rewrites settings and state in place, and ~/.claude is + # routinely 0700/0600 -- unreadable to the container without DAC_OVERRIDE. + # Rationale: .claude/notes/isolation.md § grant_container_access grant_container_access(claude_copy, writable=True) self._claude_mount_src = claude_copy def _prepare_task_dir_mount(self, staging: Path) -> None: """Copy the task directory under ``staging`` for a read-WRITE mount. - Replaces the old *symmetric* ``-v ::ro`` - mount, and for the same reason ``_prepare_reference_mount`` copies: the - in-container orchestrator holds this path at mode 000 for the duration of - every agent turn, and neither alternative works. - - * ``:ro`` rejects the chmod outright -- verified: ``chmod: /ro: - Read-only file system``. No window is expressible at all. - * Read-write *without* a copy chmods the operator's REAL ``tasks/`` tree. - Verified: the host directory came back 0600 and even the harness's own - cleanup then failed with ``Permission denied``. A crashed run would - strand a checkout at 000. - - Shielding the whole tree (rather than masking just - ``reference.directory`` with a tmpfs, as the symmetric mount required) - also closes a leak that mask could not: a task at ``tasks/foo.yaml`` has - parent ``tasks/``, so the old mount exposed every SIBLING task's - directory -- including their reference solutions, which the - single-subdir mask never covered. - - Symmetry was never load-bearing. The container is told where the task - dir is via ``--task-dir``, and ``run_task_internal_command`` uses that - path only to seed ``TASK_DIR`` -- it is never re-read. ``TASK_DIR`` is - exposed solely in ``_build_run_command_env`` (criterion subprocesses), so - the agent has no legitimate need for this tree mid-turn. - - Lives under ``staging``, which ``run()`` removes in its ``finally``; one - container per task means no cross-task interference. + The in-container orchestrator holds this path at mode 000 for the duration + of every agent turn, which neither a ``:ro`` mount (EROFS) nor an uncopied + read-write mount (it would chmod the operator's real ``tasks/`` tree) + survives. Lives under ``staging``, which ``run()`` removes in its + ``finally``; one container per task means no cross-task interference. + + Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies """ if not self.rt.task_file: return @@ -1293,9 +1106,8 @@ def _prepare_task_dir_mount(self, staging: Path) -> None: return task_dir_copy = staging / "task_dir" shutil.copytree(source, task_dir_copy, ignore=ignore_patterns_and_symlinks(REFERENCE_COPY_IGNORE)) - # Read-only for the same reason as the reference copy: criteria read - # fixtures here, nothing legitimately writes them, and withholding `o+w` - # keeps an agent from rewriting the expectations it is graded against. + # Read-only like the reference copy: criteria read fixtures here, nothing + # legitimately writes them. grant_container_access(task_dir_copy, writable=False) self._task_dir_mount_src = task_dir_copy @@ -1320,12 +1132,10 @@ def _prepare_reference_mount(self, staging: Path) -> None: return reference_copy = staging / "reference" shutil.copytree(source, reference_copy, ignore=ignore_patterns_and_symlinks(REFERENCE_COPY_IGNORE)) - # Read-only on purpose: the harness reads this copy for grading and - # chmods it (owner-or-CAP_FOWNER, and FOWNER is retained), but nothing - # legitimately writes it. Withholding `o+w` keeps the agent from being - # able to overwrite the solution during the gaps between windows, so - # _verify_reference_integrity is not the only thing standing between an - # agent and a forged reference_comparison score. + # HAZARD: read-only on purpose -- withholding `o+w` keeps + # _verify_reference_integrity from being the only thing between an agent + # and a forged reference_comparison score. + # Rationale: .claude/notes/isolation.md § grant_container_access grant_container_access(reference_copy, writable=False) self._reference_mount_src = reference_copy self._reference_source_dir = source @@ -1333,36 +1143,24 @@ def _prepare_reference_mount(self, staging: Path) -> None: def _build_image(self) -> str: """Resolve the image to run, building from a Dockerfile when configured. - When ``docker.dockerfile_path`` is set it overrides ``docker.image``: - we shell out to ``docker build`` using the Dockerfile's parent directory - as the build context (so relative ``COPY`` paths resolve) and tag the - result with a deterministic, per-task name so Docker's layer cache is - reused across runs. ``docker.build`` (:class:`DockerBuildConfig`) adds - ``--build-arg`` / ``--secret`` / extra flags; the build runs with - BuildKit enabled. Otherwise the configured ``image`` is returned - unchanged. - - **Contract:** the container runs the coder-eval orchestrator. The image - bakes no ``ENTRYPOINT``; the host pins it at run time via - ``docker run --entrypoint`` (see :meth:`_build_argv`). A task Dockerfile - must therefore start ``FROM coder-eval-agent:`` and only ADD - task-specific layers, so the runtime (the ``coder_eval_entrypoint.sh`` - script + the ``coder-eval`` CLI + the ``org.coder-eval.version`` label) - is present. After building we assert that label is present and fail with - an actionable error otherwise -- without this, a bare ``FROM ubuntu`` - image builds fine, then dies at ``docker run`` with a cryptic - ``exec: "/usr/local/bin/coder_eval_entrypoint.sh": no such file``. + ``docker.dockerfile_path`` overrides ``docker.image``: the Dockerfile's + parent directory is the build context (so relative ``COPY`` paths resolve) + and the result is tagged deterministically per task so Docker's layer cache + is reused. ``docker.build`` adds ``--build-arg`` / ``--secret`` / extra + flags; BuildKit is enabled. Side-effecting (network + docker daemon state); call via ``asyncio.to_thread`` from :meth:`run`, never from :meth:`_build_argv`, which must stay pure. + Rationale: .claude/notes/isolation.md § The entrypoint and the image contract + Returns: The image reference to pass to ``docker run``. Raises: - DockerRunError: If ``docker build`` exits non-zero, or the built - image is not a coder-eval runtime image (missing the + DockerRunError: If ``docker build`` exits non-zero, or the built image + is not a coder-eval runtime image (missing the ``org.coder-eval.version`` label). """ cfg = self._docker_config @@ -1370,9 +1168,8 @@ def _build_image(self) -> str: return cfg.image dockerfile = Path(cfg.dockerfile_path) context = dockerfile.parent - # Image repository names must be lowercase; task ids are typically - # already kebab-case, but lowercase defensively. Deterministic tag -> - # Docker layer cache is reused across runs of the same task. + # Lowercase: image repository names must be. Deterministic tag -> Docker + # layer cache is reused across runs of the same task. safe_id = _sanitize_container_name_component(self.rt.task.task_id).lower() image = f"coder-eval-task-{safe_id}:built" @@ -1507,66 +1304,31 @@ def _reference_mount_args(self) -> list[str]: """ if self._reference_mount_src is None: return [] - # Read-WRITE, and of a COPY: the in-container orchestrator chmods this - # path to 000 for every agent turn, which a `:ro` mount would reject with - # EROFS. See _prepare_reference_mount. + # Read-WRITE, and of a COPY: the anti-cheat window chmods this path to 000 + # every turn, which a `:ro` mount rejects with EROFS. return ["-v", f"{self._reference_mount_src}:{CONTAINER_REFERENCE_DIR}"] def _build_argv( self, input_dir: Path, output_dir: Path, *, container_name: str, image: str | None = None ) -> list[str]: cfg = self._docker_config - # `image` is resolved by run() via _build_image() (which may shell out to - # `docker build`). _build_argv stays pure -- no side effects -- so it - # remains testable without a docker daemon. Fall back to the configured - # image when called directly (e.g. unit tests of mount rendering). + # _build_argv stays PURE -- no side effects -- so it remains testable without + # a docker daemon. Fall back to the configured image when called directly. if image is None: image = cfg.image argv: list[str] = ["docker", "run", "--rm", "--name", container_name] - # Pin the framework entrypoint at run time rather than trusting whatever - # the task image baked into ENTRYPOINT. This makes the orchestrator launch - # robust to a task Dockerfile that sets its own ENTRYPOINT/CMD (or clears - # it via `ENTRYPOINT []`). `--entrypoint` resets the image CMD, which is - # fine -- the run command (`--output`/`--task-dir`, appended after the - # image) is passed explicitly below and is forwarded to the entrypoint. + # Pinned at run time, not trusted from the image: this survives a task + # Dockerfile that sets or clears its own ENTRYPOINT/CMD. + # Rationale: .claude/notes/isolation.md § The entrypoint and the image contract argv += ["--entrypoint", CONTAINER_ENTRYPOINT] - # ANTI-CHEAT (load-bearing, not hardening boilerplate). The container runs - # as root, and root bypasses ordinary file permissions via CAP_DAC_OVERRIDE - # / CAP_DAC_READ_SEARCH. Without dropping both, the mode-000 window that - # fs_permissions.py puts around every agent turn is a NO-OP on - # native Linux -- verified: a `chmod 000` dir is still readable by root in a - # default container, and Permission denied once these two caps are dropped. - # (It appears to work on macOS Docker Desktop even without this, because - # virtiofs enforces host-side; that is a platform accident, not the rule.) - # Nothing in a sandbox legitimately needs to override discretionary access - # control, so dropping these costs the task nothing. - # - # FOWNER/CHOWN are deliberately NOT dropped, though an earlier revision - # did. chmod(2) is gated on owner-OR-CAP_FOWNER, so dropping FOWNER does - # stop a root agent from restoring the mode — but it stops the HARNESS - # from applying it in the first place, because the in-container - # orchestrator that opens the window is the same root process with the - # same capability set. On native Linux the bind mount preserves the host - # uid that ran coder-eval, so `chmod 000 /work/references` then fails - # with EPERM (verified: container root, uid-1000-owned dir, FOWNER - # dropped -> "Operation not permitted") and the run completes UNPROTECTED - # while still looking protected. The drop therefore only ever bites on - # the hosts where it also disables the control it is meant to enforce. - # Keeping the caps means the mode-000 window works on every host; a - # deliberate re-chmod by a root agent stays the documented KNOWN GAP, - # closed by running the agent as a non-root uid (see - # docs/DOCKER_ISOLATION.md). - # # COUNTERPART, do not remove one without the other: dropping DAC_OVERRIDE - # revokes root's bypass on EVERY framework-owned mount, not just the - # reference -- including the run dir it must write task.json/task.log - # into. `grant_container_access` widens those host-side so the container - # reaches them through `other` instead of through the capability. Drop - # the caps without that widening and every docker task dies on its first - # log write; widen without the drop and the anti-cheat window is a no-op. + # revokes root's bypass on every framework-owned mount, and + # `grant_container_access` widens those host-side to compensate. Drop without + # widening and every docker task dies on its first log write. + # Rationale: .claude/notes/isolation.md § Capability drops and the anti-cheat window argv += [ "--cap-drop", "DAC_OVERRIDE", @@ -1586,15 +1348,9 @@ def _build_argv( if self._limits.max_pids is not None: argv += ["--pids-limit", str(self._limits.max_pids)] - # Forward environment variables: explicit allowlist (optionally extended via env_passthrough_extra). - # `--env VAR` (name-only) tells docker to copy the value from our current env at - # run time, so secrets stay out of the rendered argv list that we log. - # - # The run's backend rides this same path: API_BACKEND is in the default allowlist, - # and `--backend` syncs it into os.environ at the CLI (run_command), so it forwards - # here exactly like every other allowlisted var. A flag that only mutated in-process - # Settings would be dropped at the container boundary and the in-container Settings - # would silently default to DIRECT — downgrading the judge (and agent) route. + # Explicit allowlist. `--env VAR` (name-only) tells docker to copy the value + # from our env at run time, so secrets stay out of the argv we log. + # Rationale: .claude/notes/isolation.md § Environment forwarding merged_allowlist = set(cfg.env_passthrough) | set(cfg.env_passthrough_extra) for env_var in merged_allowlist: # LITELLM_BASE_URL / LITELLM_COST_LOG are forwarded below with a value @@ -1604,13 +1360,9 @@ def _build_argv( if env_var in os.environ: argv += ["--env", env_var] - # LITELLM_BASE_URL points at a proxy on the HOST. A bridge-network container - # can't reach the host's loopback, so rewrite localhost/127.0.0.1 to the - # docker host alias and publish that alias (`--add-host`) for Linux parity - # (it's automatic on macOS/Windows Docker Desktop). It's only a URL, so an - # explicit `--env VAR=value` is safe to render in the logged argv — unlike - # the auth token, which stays name-only above. Skipped when the container - # has no network (the proxy is unreachable anyway → validation errors). + # A bridge-network container cannot reach the host's loopback, so rewrite it + # to the docker host alias and publish that alias. Only a URL, so an + # explicit `--env VAR=value` is safe in the logged argv -- unlike the token. litellm_base_url = os.environ.get("LITELLM_BASE_URL") if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist and cfg.network != "none": rewritten = _rewrite_loopback_for_container(litellm_base_url) @@ -1619,104 +1371,62 @@ def _build_argv( else: argv += ["--env", "LITELLM_BASE_URL"] - # LITELLM_COST_LOG is the proxy's per-call cost log, written on the HOST by - # the proxy; the in-container Orchestrator's actual-cost join READS it. So - # bind-mount its directory at the SAME host path (read-only — the container - # only reads; the host proxy is the sole writer) and forward the resolved - # ABSOLUTE path so it points at the mount regardless of a relative/env value. - # Skipped when the dir is absent → the join no-ops and the run keeps static - # pricing, exactly as a local run does when the log is missing. + # The proxy's per-call cost log: written on the HOST, READ by the + # in-container cost join, so bind-mount its dir at the SAME host path + # read-only and forward the resolved ABSOLUTE path. litellm_cost_log = os.environ.get("LITELLM_COST_LOG") if litellm_cost_log and "LITELLM_COST_LOG" in merged_allowlist and cfg.network != "none": abs_log = Path(litellm_cost_log).expanduser().resolve() if abs_log.parent.is_dir(): argv += ["-v", f"{abs_log.parent}:{abs_log.parent}:ro", "--env", f"LITELLM_COST_LOG={abs_log}"] - # Signal to in-container agents that the harness already provides OS-level - # isolation. The Codex agent reads this to fall back to its full-access - # sandbox: Codex's Landlock-backed read-only / workspace-write sandboxes - # can't initialize inside a container and otherwise fail writes silently. + # Tells in-container agents the harness already provides OS-level isolation; + # Codex reads it to fall back to its full-access sandbox. argv += ["--env", f"{IN_CONTAINER_ENV}=1"] - # Hard-disable telemetry INSIDE the container. The app ships a baked-in - # default connection string, so without this the in-container orchestrator - # would emit CoderEval.Task.End — and the host re-emits the same event after - # the container result is parsed (orchestration/batch.py), double-counting - # every docker-driver task. The invariant is "container silent, host emits - # once"; this restores it regardless of the host's own telemetry setting. - # Explicit value (not name-only) so it overrides any inherited/baked value. + # The invariant is "container silent, host emits once": the host re-emits + # CoderEval.Task.End after parsing the result. Explicit value, not + # name-only, so it overrides any inherited or baked-in value. + # Rationale: .claude/notes/isolation.md § Environment forwarding argv += ["--env", "TELEMETRY_ENABLED=false"] argv += ["-v", f"{input_dir.resolve()}:{CONTAINER_INPUT_DIR}:ro"] - # Mount the host run_dir to the container's standard output location - # so the in-container Orchestrator writes task.json/task.log/etc. - # directly to the host filesystem via bind-mount. + # The host run_dir at the container's standard output location, so the + # in-container Orchestrator writes straight to the host filesystem. argv += ["-v", f"{output_dir}:{CONTAINER_OUTPUT_DIR}"] - # Mount a COPY of the task dir at a fixed container path. Read-WRITE and - # a copy for the same reason as the reference (see - # _prepare_task_dir_mount): the agent-turn window chmods it to 000, which - # `:ro` rejects with EROFS and which -- applied to the real tree -- - # would chmod the operator's own `tasks/`. + # A COPY at a fixed container path, read-WRITE: see _prepare_task_dir_mount. if self._task_dir_mount_src is not None: argv += ["-v", f"{self._task_dir_mount_src}:{CONTAINER_TASK_DIR}"] - # DETACHED GRADE: the already-executed workspace, read-WRITE and NOT a - # copy. Read-write because criteria legitimately mutate what they grade - # (a `run_command` that compiles, a post_run that cleans up), and the - # real tree because copying is what the host path proved wrong — - # `_setup_template` filters out node_modules / dist / build / .venv, so a - # criterion reading those would fail as a copying artifact rather than as - # a verdict. + # DETACHED GRADE: the already-executed workspace, read-WRITE and NOT a copy + # -- criteria legitimately mutate what they grade, and _setup_template's + # filtering would drop node_modules / dist / .venv from a copy. + # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies if self.grade_workspace is not None: argv += ["-v", f"{self.grade_workspace.resolve()}:{CONTAINER_GRADE_WORKSPACE}"] - # ANTI-CHEAT: the reference solution normally lives INSIDE the task dir, - # so the symmetric mount above would hand the agent the answer via - # `$TASK_DIR/`. Two things close that: - # - # 1. An empty tmpfs is layered over the reference's path inside the - # task-dir mount, masking it. The agent sees an empty directory there. - # 2. A throwaway COPY of the reference is mounted read-WRITE at - # /work/references, and the in-container orchestrator shields THAT - # path directly rather than re-copying it. Writable is load-bearing, - # not an oversight: the window chmods this exact path to 000 every - # turn, and chmod on a `:ro` bind mount fails with EROFS. - # - # Ordering matters: docker applies mounts by target-path depth, so the - # tmpfs at the deeper path wins over the task-dir bind regardless of argv - # order, but we emit it after for readability. + # ANTI-CHEAT: the reference normally lives INSIDE the task dir, so an empty + # tmpfs masks its path there and a writable COPY is mounted at + # /work/references instead. Docker applies mounts by target-path depth, so + # the deeper tmpfs wins regardless of argv order. + # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies argv += self._reference_mount_args() - # Forward the host's Claude Code OAuth state so the in-container CLI - # inherits the same login as the host. We mount a *throwaway lean copy* - # of ~/.claude (made by _prepare_host_mounts) read-WRITE at the host's - # ~/.claude path — HOME is forwarded, so the path is symmetric inside - # the container. The container can therefore write anywhere under - # ~/.claude (settings, session ephemera, cache) without ever mutating - # the host's real ~/.claude. _claude_mount_src is None when ~/.claude - # doesn't exist or the mount is opted out (CODER_EVAL_NO_CLAUDE_MOUNT=1). + # A throwaway lean COPY of ~/.claude, read-WRITE at the host's own path + # (HOME is forwarded, so the path is symmetric), so the container never + # mutates the host's real one. + # Rationale: .claude/notes/isolation.md § The lean ~/.claude copy if self._claude_mount_src is not None: host_claude_dir = Path.home() / ".claude" argv += ["-v", f"{self._claude_mount_src}:{host_claude_dir}"] - # Auto-mount host paths the task references so they resolve inside - # the container at the *same* path they have on the host. - # Includes: - # - Claude Code plugin dirs (`agent.plugins[].path`) - # - Template directories (`sandbox.template_sources[].path` for - # TemplateDirSource entries -- already absolute after - # resolve_template_paths runs on the host). - # `run_command` criteria that use `$TASK_DIR/...` are covered by the - # symmetric task_dir mount above. The reference is deliberately NOT here: - # it gets its own mount at CONTAINER_REFERENCE_DIR and is masked out of - # the task_dir mount (see _reference_mount_args). ``mounted`` dedupes overlapping entries. + # Host paths the task references (plugin dirs, resolved template dirs), at + # the SAME path inside the container. The reference is deliberately NOT + # here -- it has its own mount and is masked out of the task_dir mount. + # ``mounted`` dedupes overlapping entries. mounted: set[Path] = set() - # Auto-mount sources that look like credential / secret dirs get a - # loud warning. Task YAMLs typically come from in-house suite authors, - # but the `plugin.path` / `reference.directory` / `template_sources` - # fields are user-controlled strings, and a typo (or a hostile suite) - # can silently expose `~/.ssh` etc. Warning, not hard fail, because - # legitimate uses exist (a task that does in fact want to read - # `~/.aws/config`). The warning surfaces the surprise. + # Warned, not refused: `plugin.path` / `reference.directory` / + # `template_sources` are user-controlled strings, and legitimate uses exist. + # Rationale: .claude/notes/isolation.md § Extra mounts and reserved destinations sensitive_sources = self._sensitive_source_paths() def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: @@ -1749,28 +1459,22 @@ def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: if isinstance(source, TemplateDirSource): _auto_mount(source.path) - # Defensive: system_prompt_file is normally inlined into - # system_prompt by load_task / experiment resolution, but a variant - # could conceivably inject an absolute path that survives. Cover - # that path so the in-container Orchestrator can read it. + # Defensive: normally inlined into system_prompt by load_task / experiment + # resolution, but a variant could inject an absolute path that survives. agent_cfg = self.rt.task.agent if agent_cfg and agent_cfg.system_prompt_file: _auto_mount(agent_cfg.system_prompt_file, dir_only=False) - # NOTE: task.reference.directory is deliberately NOT auto-mounted at its - # host path here. A copy of it gets a single dedicated read-write mount - # at CONTAINER_REFERENCE_DIR (above; writable so the anti-cheat window - # can chmod it), and mounting the original at its host path too would - # re-expose it to the agent through $TASK_DIR — the exact hole the tmpfs - # mask above closes. + # HAZARD: task.reference.directory is deliberately NOT auto-mounted at its + # host path -- that would re-expose it through $TASK_DIR, the exact hole + # the tmpfs mask closes. for mount in cfg.extra_mounts: normalized = _validate_extra_mount(mount) argv += ["-v", normalized] - # Docker WORKDIR alignment: run the agent at the image's own WORKDIR. Set - # the container's initial cwd via `-w` (the in-container orchestrator also - # runs the agent there). NO bind mount targets it -- capture is a copy-out - # (see Orchestrator._cleanup), not a mount, so baked inputs/HOME survive. + # Docker WORKDIR alignment: `-w` only. NO bind mount targets it -- capture + # is a copy-out (see Orchestrator._cleanup), so baked inputs and HOME + # survive. if self._workspace_dir is not None: _assert_workspace_not_reserved(self._workspace_dir) argv += ["-w", self._workspace_dir] @@ -1782,10 +1486,8 @@ def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: argv += ["-v"] argv += ["--output", str(CONTAINER_OUTPUT_DIR)] if self._task_dir_mount_src is not None: - # The container-side path, not the host's. run_task_internal_command - # uses this only to seed TASK_DIR for run_command criteria; it never - # re-reads the path, which is why the mount no longer has to be - # symmetric. + # The container-side path: it only ever seeds TASK_DIR and is never + # re-read, which is why the mount no longer has to be symmetric. argv += ["--task-dir", CONTAINER_TASK_DIR] return argv diff --git a/src/coder_eval/sandbox.py b/src/coder_eval/sandbox.py index e95277138..dcf2d00ea 100644 --- a/src/coder_eval/sandbox.py +++ b/src/coder_eval/sandbox.py @@ -34,19 +34,11 @@ logger = logging.getLogger(__name__) -# Entries excluded from Sandbox.capture_to (docker WORKDIR-alignment). -# Two classes of exclusion: -# -# 1. SECURITY denylist: credential files/dirs that must never leak into -# captured artifacts (which get uploaded). Defense-in-depth -- the eval -# images don't bake credentials, but any future image that does should -# not silently expose them. -# -# 2. NOISE suppression: sandbox-created bulk and home-dir infrastructure -# written by tools (uv, pip, npm, shell) when WORKDIR overlaps HOME -# (e.g. /root). These are never task deliverables. -# -# Matched by basename at every level via shutil.ignore_patterns. +# Entries excluded from Sandbox.capture_to (docker WORKDIR-alignment): a SECURITY +# denylist of credential stores that must never reach uploaded artifacts, plus NOISE +# written by uv/pip/npm/shell when WORKDIR overlaps HOME. Matched by basename at +# every level via shutil.ignore_patterns. +# Rationale: .claude/notes/isolation.md § preserve_to, capture_to, and the capture denylist _WORKSPACE_CAPTURE_IGNORE = ( # --- Security: credential stores --- ".claude", # RW lean copy of host ~/.claude (carries .credentials.json) @@ -78,9 +70,8 @@ # not automatic: `Sandbox.resolve_files` tries the literal path first. _GLOB_METACHARACTERS = "*?[" -# Cap on how many matches an ambiguity error enumerates. The message is -# persisted to task.json and injected into judge prompts, so an unbounded -# listing over a wide pattern is a real payload. +# HAZARD: the message is persisted to task.json and injected into judge prompts, +# so an unbounded listing over a wide pattern is a real payload. _MAX_LISTED_MATCHES = 10 @@ -169,9 +160,8 @@ def __init__( self._cleanup_on_exit = True # True once adopt() takes over an existing workspace. Read by the # Orchestrator to suppress every step that would MUTATE the tree it was - # asked to grade (pre_run/post_run above all — several in-tree tasks - # copy fixtures over the workspace there) and to keep sandbox_path in - # the result, since an adopted directory outlives cleanup(). + # asked to grade, and to keep sandbox_path in the result. + # Rationale: .claude/notes/isolation.md § Why pre_run and post_run each run exactly once self.was_adopted = False self.installed_tool_versions: dict[str, str] = {} self._command_base_path: str | None = None @@ -184,22 +174,15 @@ def enforces_permission_windows(self) -> bool: """Whether a chmod window is a real, safe control in this sandbox. True only inside a ``driver: docker`` container, where the filesystem is - private to this one task: chmod-ing the reference and task directories - there affects nothing else, and the container drops ``DAC_OVERRIDE`` / - ``DAC_READ_SEARCH`` so the mode actually binds against its root user. - - On the host (``driver: tempdir``) it is a deliberate no-op. Parallel - tasks in one batch share the checked-out ``tasks//`` tree, so - chmod-ing it is a cross-task side effect on the user's own working copy - for no isolation benefit -- there is no boundary to enforce when the - agent is just another process with the same uid. + private to this one task. On the host (``driver: tempdir``) it is a + deliberate no-op. NOTE the predicate is the ``CODER_EVAL_IN_CONTAINER`` env var, NOT - ``config.driver``. The in-container entry point rewrites - ``driver: docker`` to ``tempdir`` before constructing the Orchestrator - (nested docker is impossible in the image), so keying on the driver - would read "tempdir" inside the container and silently disable the - anti-cheat window on exactly the path that needs it. + ``config.driver``: the in-container entry point rewrites ``driver: docker`` + to ``tempdir`` before constructing the Orchestrator, so keying on the + driver would silently disable the window on the path that needs it. + + Rationale: .claude/notes/isolation.md § Capability drops and the anti-cheat window """ return os.environ.get(IN_CONTAINER_ENV) == "1" @@ -266,10 +249,9 @@ def setup(self, target_dir: Path | None = None) -> Path: if self.config.driver == "tempdir": return self._setup_tempdir(target_dir=target_dir) if self.config.driver == "docker": - # Docker isolation is dispatched at the orchestrator-entry boundary - # (coder_eval.isolation.docker_runner). Inside the container, the - # task is re-run with driver=tempdir, so this branch is never - # reached on a correctly routed call. + # Dispatched at the orchestrator-entry boundary; inside the container + # the task is re-run with driver=tempdir, so this is unreachable on a + # correctly routed call. raise RuntimeError( "Sandbox.setup() called with driver='docker' -- Docker tasks must be " + "dispatched via DockerRunner from the host. This indicates a routing bug." @@ -279,38 +261,20 @@ def setup(self, target_dir: Path | None = None) -> Path: def adopt(self, workspace: Path) -> Path: """Use ``workspace`` **as** the sandbox, materializing nothing into it. - The grade-in-place counterpart to :meth:`setup`. ``setup`` builds a - workspace: it copies template sources in, generates ``record_cli`` shims, - creates a venv, installs packages. ``adopt`` takes a workspace that - already exists — an ``execute`` run's artifacts, or a verifier's ``/app`` - — and only derives the *environment* the criteria need to run against it - (mock-dir ``+x``, venv discovery, the plugin-tools pin). - - Why not ``setup(target_dir=workspace)``: that already adopts a - caller-supplied directory and sets ``_cleanup_on_exit=False``, but it - then runs ``_setup_template()``, which would write over the very files - it was asked to grade. - - In-place is more CORRECT here, not merely faster: - - * ``_setup_template`` filters what it copies through - ``_should_ignore_template_file`` — ``node_modules``, ``dist``, - ``build``, ``venv``, ``.git`` and friends are dropped. A criterion like - ``test -f dist/bundle.js`` therefore fails as a *copying artifact* - rather than as a verdict on the agent's work. - * ``run_command`` criteria execute with ``cwd = sandbox_dir``, so on the - copy path they see the copy's paths, not the ones the agent worked at. - * Copying a real workspace costs minutes. - - "Materializing nothing" means it writes no FILES. It does still chmod - ``+x`` over the task's declared mock-PATH directories inside the tree — - a mode change the criteria need in order to resolve the same shimmed - binaries the agent did. + The grade-in-place counterpart to :meth:`setup`: it takes a workspace that + already exists -- an ``execute`` run's artifacts, or a verifier's ``/app`` + -- and derives only the *environment* the criteria need (mock-dir ``+x``, + venv discovery, the plugin-tools pin). In-place is more CORRECT here, not + merely faster. + + "Materializing nothing" means it writes no FILES; it does still chmod + ``+x`` over the task's declared mock-PATH directories, a mode change the + criteria need to resolve the same shimmed binaries the agent did. The caller keeps ownership: ``_cleanup_on_exit`` stays False, so - ``cleanup()`` never deletes an adopted directory. Criteria CAN still - mutate it (a ``run_command`` that writes), which is why the copy path - remains the default for a bare user-supplied work dir. + ``cleanup()`` never deletes an adopted directory. + + Rationale: .claude/notes/isolation.md § Detached grading and `Sandbox.adopt` Args: workspace: An existing directory to grade in place. @@ -335,28 +299,17 @@ def adopt(self, workspace: Path) -> Path: self._cleanup_on_exit = False self.was_adopted = True - # Only NON-materializing steps below. Deliberately skipped, and why: - # _setup_template would overwrite the workspace being graded - # _generate_cli_recorders writes shims into it - # _setup_virtualenv / - # _install_*_packages the execute phase already provisioned these; - # re-running mutates the graded tree - # _maybe_remediate_home_plugins_pollution - # destructive on $HOME, and it is remediation - # rather than derivation — the execute phase - # already ran it if it was enabled + # Only NON-materializing steps below. Deliberately skipped: + # _setup_template (overwrites the tree being graded), + # _generate_cli_recorders (writes shims into it), _setup_virtualenv / + # _install_*_packages (the execute phase provisioned these), and + # _maybe_remediate_home_plugins_pollution (destructive on $HOME, and + # remediation rather than derivation). self._prepare_mock_path_dirs() - # Discover an existing venv instead of creating one, so `run_command` - # criteria get the same VIRTUAL_ENV/PATH the agent had. Absent venv -> - # None, exactly as for a task with no python config. - # - # Gated on `config.python` for the same reason `setup` is: venv_dir - # prepends the venv's bin/ to PATH and exports VIRTUAL_ENV for every - # criterion subprocess, so discovering one a task never asked for grades - # it under a PATH it never ran under — the exact divergence the - # command_base_path round trip exists to close. It would also let an - # agent shadow binaries by writing `.venv/bin/` into its own workspace. + # DISCOVER rather than create, so criteria get the same VIRTUAL_ENV/PATH + # the agent had. Gated on `config.python` for the same reason `setup` is. + # Rationale: .claude/notes/isolation.md § Why the venv gets system site packages if self.config.python: candidate = self.sandbox_dir / VENV_DIRNAME if candidate.is_dir(): @@ -387,10 +340,9 @@ def _setup_tempdir(self, target_dir: Path | None = None) -> Path: # "parent/row" -- flatten path separators so they don't become subdirectories # under /tmp (mkdtemp does not auto-create parent dirs). safe_task_id = self.task_id.replace("/", "_").replace("\\", "_") - # On Windows root off the home dir, not the user temp tree: the agent's Git Bash - # mounts /tmp onto the base temp dir while Python's mkdtemp honors %TEMP% (a CI-set - # subdir), so a temp-rooted sandbox gets a divergent /tmp twin the grader never reads. - # POSIX has one namespace (dir=None keeps the system temp, unchanged for driver:docker). + # On Windows, root off the home dir: Git Bash mounts /tmp onto the base + # temp dir while mkdtemp honors %TEMP%, giving the sandbox a divergent + # /tmp twin the grader never reads. POSIX has one namespace. self.sandbox_dir = Path( tempfile.mkdtemp(prefix=f"coder_eval_{safe_task_id}_", dir=Path.home() if os.name == "nt" else None) ) @@ -406,9 +358,8 @@ def _setup_tempdir(self, target_dir: Path | None = None) -> Path: # Mark mock binaries executable so the agent's PATH can shadow real CLIs self._prepare_mock_path_dirs() - # Set up Python virtual environment (only if python config is provided). - # The venv is created with system site packages -- see _setup_virtualenv - # for why an isolated one was actively harmful. + # With system site packages -- see _setup_virtualenv for why an + # isolated venv was actively harmful. if self.config.python: self._setup_virtualenv() @@ -428,11 +379,9 @@ def _setup_tempdir(self, target_dir: Path | None = None) -> Path: # Cache canonical @uipath dir for PLUGIN_TOOLS_DIR pin; no-op if `uip` absent. self._refresh_plugin_tools_dir() except Exception: - # Clean up on failure -- but ONLY a temp dir we created ourselves. - # For a caller-supplied target_dir (DIRECT_WRITE persistent mode) we - # must not rmtree it: it may be a pre-existing artifacts dir, and the - # mode's contract is to never clear it. A self-created tempdir always - # has _cleanup_on_exit=True at this point; target_dir flips it False. + # ONLY a temp dir we created ourselves: a caller-supplied target_dir + # (DIRECT_WRITE) may be a pre-existing artifacts dir whose contract is + # never to be cleared. if self._cleanup_on_exit: shutil.rmtree(self.sandbox_dir, ignore_errors=True) self.sandbox_dir = None @@ -456,12 +405,10 @@ def _apply_repo_source(self, source: RepoSource) -> None: assert self.sandbox_dir is not None, "Sandbox directory not initialized" repo_dir = self.sandbox_dir / "repo" - # `--` before the URL: it is argv position 2, so without the separator a - # value beginning with `-` is parsed by git as an OPTION rather than a - # repository (`--upload-pack=…` runs a command of the caller's choosing). - # That URL is task-authored, and since `evaluate ` rebuilds the - # task from a shareable run directory it is no longer necessarily the - # operator's own string. + # HAZARD: `--` before the URL. Without it a value beginning with `-` parses + # as an OPTION (`--upload-pack=...` runs a command of the caller's + # choosing), and that URL is task-authored. + # Rationale: .claude/notes/isolation.md § Materializing a template into the sandbox cmd = ["git", "clone", "--", source.url, str(repo_dir)] try: @@ -554,10 +501,8 @@ def _apply_template_dir_source(self, source: TemplateDirSource) -> None: for item in template_path.rglob("*"): # Calculate relative path rel_path = item.relative_to(template_path) - # Match ignore patterns against the template-relative path only — - # checking the absolute path would let an ancestor directory named - # `dist`, `build`, `env`, `venv`, or `node_modules` filter out the - # entire template (e.g. if the repo is cloned under ~/build/…). + # Template-RELATIVE, not absolute: an ancestor named `dist`, `build`, + # `env`, `venv` or `node_modules` would filter out the whole template. if self._should_ignore_template_file(rel_path) and not self._matches_template_include_pattern( rel_path, source.include_patterns ): @@ -565,38 +510,25 @@ def _apply_template_dir_source(self, source: TemplateDirSource) -> None: dest_path = mount_root / rel_path - # is_symlink() must come first — is_dir() / is_file() follow - # symlinks, so a `tools/node_modules/fil-compiler -> ../fil` - # link would look like a directory and we'd create an empty - # dir at the destination, breaking npm workspace resolution. + # is_symlink() first -- is_dir()/is_file() follow symlinks, so a link + # to a dir would produce an empty dir at the destination. + # Rationale: .claude/notes/isolation.md § Materializing a template into the sandbox if item.is_symlink(): - # `is_symlink()` before `exists()` because `exists()` follows - # the link; a *broken* symlink at dest is still an overwrite - # we need to clear. + # `is_symlink()` before `exists()`, which follows the link: a + # BROKEN symlink at dest is still an overwrite to clear. if dest_path.is_symlink() or dest_path.exists(): - # Only a real directory needs rmtree; symlinks-to-dir, - # symlinks-to-file, and regular files all clear with - # unlink() (which removes the link, not its target). + # Only a real directory needs rmtree; unlink() removes the + # link rather than its target. if dest_path.is_dir() and not dest_path.is_symlink(): shutil.rmtree(dest_path) else: dest_path.unlink() overwrites.add(str(rel_path)) dest_path.parent.mkdir(parents=True, exist_ok=True) - # `item.is_dir()` follows the symlink, so it tells us - # whether the target is a directory. On Windows - # `os.symlink` needs `target_is_directory=True` for - # directory targets — without it Windows creates a - # file-symlink that can't be traversed. POSIX ignores - # the flag. - # - # We preserve `os.readlink(item)` verbatim — both - # relative (npm workspaces, e.g. `node_modules/foo - # -> ../foo`) and absolute targets. Absolute targets - # remain live links into the host filesystem inside - # the sandbox; template authors are trusted infra - # (see `templates/` in this repo), so this is the - # intended behavior, not a defense boundary. + # Windows `os.symlink` needs `target_is_directory=True` for + # directory targets; POSIX ignores it. Targets are preserved + # verbatim, absolute ones included -- template authors are trusted + # infra, so this is intended, not a defense boundary. os.symlink( os.readlink(item), dest_path, @@ -654,10 +586,9 @@ def resolved_mock_path_dirs(self) -> list[Path]: if self.sandbox_dir is None: return [] resolved: list[Path] = [] - # Generated recorders go FIRST: `_generate_cli_recorders` refuses to - # generate a shim whose name a user mock dir already provides, so this - # order can never silently shadow a task's own mock — it only fixes which - # directory wins for names the harness itself owns. + # Generated recorders go FIRST, and refuse to generate a shim whose name a + # user mock dir already provides, so this can never shadow a task's own mock. + # Rationale: .claude/notes/isolation.md § The criterion environment, layer by layer if self.config.record_cli: generated = self._resolve_within_sandbox(RECORD_CLI_DIR, field="record_cli directory") if generated.is_dir(): @@ -696,9 +627,8 @@ def _generate_cli_recorders(self) -> None: if not user_dir.is_dir(): continue for spec in self.config.record_cli: - # Every name this feature generates, not just the bare one: on - # Windows PATHEXT resolves `uip` to the generated `uip.cmd` ahead of - # the task's own `mocks/uip.cmd`, silently changing what runs. + # Every generated name, not just the bare one: Windows PATHEXT + # resolves `uip` to `uip.cmd` ahead of the task's own mock. clash = next( ( user_dir / name @@ -717,17 +647,14 @@ def _generate_cli_recorders(self) -> None: raise RuntimeError(msg) recorder_dir = self._resolve_within_sandbox(RECORD_CLI_DIR, field="record_cli directory") - # Wipe rather than reuse: DIRECT_WRITE (the docker default) does not clear the - # target dir, so a reused --run-dir would leave a previous run's log to be - # scored as this run's, and stale shims for tools no longer declared on PATH. + # Wipe rather than reuse: DIRECT_WRITE does not clear the target dir, so a + # reused --run-dir would leave a previous run's log to be scored as this one's. if recorder_dir.exists(): shutil.rmtree(recorder_dir, ignore_errors=True) recorder_dir.mkdir(parents=True, exist_ok=True) - # Seed the log so it always exists: `cli_called` treats a MISSING log as a - # harness fault (score 0 even for a negative guard), which is right when a - # mock never ran, but wrong for a correct run that legitimately called - # nothing. An empty file distinguishes the two. + # Seeded so it always exists: `cli_called` reads a MISSING log as a harness + # fault, which is wrong for a run that legitimately called nothing. log_path = self.sandbox_dir / RECORD_CLI_LOG log_path.write_text("", encoding="utf-8") @@ -741,9 +668,8 @@ def _generate_cli_recorders(self) -> None: ) raise RuntimeError(msg) shim.write_text(render_recorder(spec, interpreter), encoding="utf-8", newline="\n") - # +x here rather than relying on _prepare_mock_path_dirs: that pass is - # what makes the bit real for PATH lookup, but the shim must be - # executable even if the recorder dir is consumed some other way. + # Not left to _prepare_mock_path_dirs: the shim must be executable + # even if the recorder dir is consumed some other way. shim.chmod(shim.stat().st_mode | 0o111) # `python "%~dp0" %*` — the extensionless script beside this file. cmd_lines = [ @@ -758,10 +684,8 @@ def _generate_cli_recorders(self) -> None: newline="", ) - # Once for the whole directory, not once per entry: every rules-bearing shim - # imports the same sidecar, so writing it inside the loop above just rewrote - # identical bytes N times. Skipped entirely when no entry declares rules -- - # such a shim never consults the matcher and needs no sibling file. + # Once for the directory, not per entry: every rules-bearing shim imports + # the same sidecar. Skipped when no entry declares rules. sidecars = sorted(SIDECAR_MODULES) if any(spec.responses for spec in self.config.record_cli) else [] for module in sidecars: (recorder_dir / module).write_text(sidecar_source(module), encoding="utf-8", newline="\n") @@ -790,9 +714,9 @@ def _apply_starter_files_source(self, source: StarterFilesSource) -> None: overwrites: set[str] = set() for starter_file in source.files: - # Reject path traversal before any filesystem write; the helper allows - # the resolved path to equal sandbox_root, which is harmless for files - # because subsequent mkdir/write_text would fail on an empty path anyway. + # HAZARD: reject path traversal before any filesystem write. The helper + # allows the resolved path to EQUAL sandbox_root, harmless for a file + # because the mkdir/write_text below fails on an empty name anyway. file_path = self._resolve_within_sandbox(starter_file.path, field="starter_files path") # Track overwrites @@ -839,26 +763,17 @@ def _matches_template_include_pattern(self, rel_path: Path, include_patterns: li def _setup_virtualenv(self) -> None: """Create a Python virtual environment in the sandbox, with system site packages. - ``--system-site-packages`` is load-bearing, not a convenience. An ISOLATED - venv here shadows the interpreter while providing nothing: the sandbox venv - goes on the criterion PATH (``_build_run_command_env``, which governs every - ``run_command`` criterion plus ``pre_run``/``post_run``), so inside a task - image that provisions packages globally, ``python`` resolved to the empty - venv and could not import them while ``pip`` -- which ``uv venv`` does not - place in the venv at all -- fell through to the image's global pip and - reported them present. Measured in a task image: ``import langchain`` raised - ``ModuleNotFoundError`` while ``pip list`` showed ``langchain 1.3.14``. An - agent that tried to verify its own work chased that contradiction for ten - turns and ran out of budget before finishing. - - Note the venv is NOT on the agent's own PATH -- the orchestrator prepends - only ``resolved_mock_path_dirs`` there -- so the contradiction above is a + ``--system-site-packages`` is load-bearing, not a convenience: an ISOLATED + venv on the criterion PATH shadows the interpreter while providing nothing, + so inside a task image that provisions packages globally ``python`` could + not import what ``pip`` reported present. System site packages keeps both + halves -- the image's globals stay importable and installs still land in + the venv, so a task's ``env_packages`` cannot leak into the image. + + Note the venv is NOT on the agent's own PATH, so the contradiction is a property of criterion and pre/post-run subprocesses. - System site packages fixes it in the direction that keeps both halves: the - image's globals stay importable, ``python`` and ``pip`` agree, and installs - still land in the venv (``sys.prefix`` remains the sandbox), so a task's - ``env_packages`` cannot leak into the image. + Rationale: .claude/notes/isolation.md § Why the venv gets system site packages """ if not self.sandbox_dir: raise RuntimeError("Sandbox directory not initialized") @@ -873,9 +788,8 @@ def _setup_virtualenv(self) -> None: cmd = ["uv", "venv", "--system-site-packages", str(self.venv_dir)] subprocess.run(cmd, check=True, capture_output=True, text=True, encoding="utf-8", timeout=60) except (subprocess.CalledProcessError, FileNotFoundError): - # Fallback to standard venv if uv is not available. The two paths do not - # produce the same artifact -- this one seeds pip, `uv venv` does not -- - # so say which shape this host got rather than leaving it to be inferred. + # The two paths do not produce the same artifact -- this one seeds pip, + # `uv venv` does not -- so say which shape this host got. import venv logger.warning("uv unavailable; created %s with stdlib venv (pip seeded)", self.venv_dir) @@ -1115,10 +1029,8 @@ def _maybe_remediate_home_plugins_pollution(self) -> Path | None: target = Path(home) / "node_modules" / "@uipath" if not target.is_dir(): return None - # Refuse to touch anything outside the configured HOME — if HOME - # somehow points at root or a system dir, bail out loudly rather - # than rm-rf'ing it. The check is belt-and-suspenders: the path - # construction above already anchors at $HOME. + # HAZARD: refuse to touch anything outside the configured HOME. The path + # construction above already anchors there; this is belt-and-suspenders. try: resolved_target = target.resolve(strict=True) resolved_home = Path(home).resolve(strict=True) @@ -1160,34 +1072,13 @@ def _maybe_remediate_home_plugins_pollution(self) -> Path | None: def _build_run_command_env(self) -> dict[str, str]: """Build the environment for ``run_command``. - Each layer is independent — none breaks if another is absent: - - 1. Inherit parent env (so agent tools / credentials remain reachable). - 2. (MST-9265) If the orchestrator has captured the agent's SDK PATH - via :meth:`set_command_base_path`, **prepend** it ahead of the - host PATH (not replace) — the agent's PATH only needs to win - the lookup race for its bundled toolchain, but system binaries - (``python``, ``node``, ``/usr/bin/*``) must remain reachable to - criteria. Prepend semantics also stay symmetric with the venv / - node_bin prepends below. - 3. Activate the sandbox virtualenv (if present). First-hit-wins: - if the agent's PATH already contains the venv scripts dir - (likely, since the agent inherits this process's env), this - prepend duplicates the entry. Harmless on every OS we target; - left explicit so the order stays independent of what the agent - SDK happens to inject. - 4. Prepend ``/node_modules/.bin`` to PATH (if present). - 5. (MST-9674) Pin ``NODE_PATH=""`` so Node's fallback search paths - cannot pick up contaminated parent-dir installs. Note: this - does NOT disable parent-walking from cwd — that is hard-wired - in Node — but it eliminates ``NODE_PATH``-mediated leaks. - 6. (MST-9674) Pin ``NPM_CONFIG_PREFIX`` to a sandbox-scoped - directory so any ``npm install`` / ``bun add`` from inside the - sandbox writes into the sandbox, not into - ``$HOME/node_modules`` where concurrent sandboxes would shadow - each other. - 7. Expose ``TASK_DIR`` for criterion scripts. - 8. Expose ``REFERENCE_DIR`` (staged reference copy) for criterion scripts. + Eight layers, each independent -- none breaks if another is absent: the + parent env, the agent's captured SDK PATH (PREPENDED, so system binaries + stay reachable), the sandbox venv, ``/node_modules/.bin``, + ``NODE_PATH=""``, a sandbox-scoped ``NPM_CONFIG_PREFIX``, ``TASK_DIR``, + and ``REFERENCE_DIR``. + + Rationale: .claude/notes/isolation.md § The criterion environment, layer by layer """ assert self.sandbox_dir is not None env = os.environ.copy() @@ -1209,11 +1100,9 @@ def _build_run_command_env(self) -> dict[str, str]: env["PLUGIN_TOOLS_DIR"] = self._plugin_tools_dir if self.task_dir: env["TASK_DIR"] = str(self.task_dir) - # 8. Expose ``REFERENCE_DIR`` (the per-run staged copy of the reference - # solution) for criterion scripts. Set by the orchestrator once the - # reference is staged; absent for tasks with no `reference:` block. - # Safe to expose here because `run_command` criteria execute AFTER the - # agent's turn, outside the mode-000 anti-cheat window. + # Safe to expose: `run_command` criteria execute AFTER the agent's turn, + # outside the mode-000 anti-cheat window. Absent for tasks with no + # `reference:` block. if self.reference_dir: env["REFERENCE_DIR"] = str(self.reference_dir) return env @@ -1222,23 +1111,12 @@ def _check_parent_node_modules_contamination(self) -> list[Path]: """Walk up from ``sandbox_dir`` and report any ancestor that has a populated ``node_modules/`` directory. - Concurrent tasks (or anything else on the host that runs - ``cd && npm install ... --save``) drop packages into - shared parent dirs. Node's parent-walking module resolver finds - those before the sandbox-local install, which is the proximate - cause of MST-9674's ``unknown command 'run'`` failure — but the - failure mode is generic to Node module resolution, not specific - to any one npm scope. The check therefore stays - scope-agnostic: ``coder_eval`` is a generic evaluation framework - and should not single out one ecosystem's namespace. Operators - read the logged entry list to decide whether the contamination - actually matters for their agent's toolchain. - - This is a *detection-only* helper. It returns the list of - ancestor ``node_modules`` dirs found and logs a single warning - per dir. Auto-remediation is intentionally avoided — those dirs - may legitimately belong to the user and silently deleting them - would be destructive. + Detection only: it logs one warning per directory and returns the list. + Auto-remediation is intentionally avoided -- those dirs may legitimately + belong to the user. The check stays scope-agnostic because the failure mode + is generic to Node module resolution. + + Rationale: .claude/notes/isolation.md § The criterion environment, layer by layer """ if self.sandbox_dir is None: return [] @@ -1254,9 +1132,8 @@ def _check_parent_node_modules_contamination(self) -> list[Path]: if not node_modules_dir.is_dir(): continue try: - # Skip dot-entries (``.bin``, ``.cache``, …) — they are - # package-manager bookkeeping, not installed packages - # that would shadow a sandbox-local install. + # Dot-entries are package-manager bookkeeping, not installed + # packages that would shadow a sandbox-local install. entries = sorted(p.name for p in node_modules_dir.iterdir() if not p.name.startswith(".")) except OSError: # Permission denied / race-with-delete — skip silently. @@ -1300,11 +1177,9 @@ def run_command(self, command: str, timeout: float | int | None = None) -> tuple env = self._build_run_command_env() try: - # Shell execution is intentional for sandbox - allows pipes, redirects, and complex commands. - # Decode stdout/stderr as UTF-8 with replacement on bad bytes so an agent that emits - # non-UTF-8 output (e.g. raw binary, locale-encoded compiler errors on Windows) does not - # kill the run with UnicodeDecodeError. Downstream callers (e.g. json_check) only need - # JSON-parseable strings; a replacement char is preferable to a crash. + # Shell execution is intentional here: pipes, redirects and compound + # commands. Decoded with `errors="replace"` so an agent emitting + # non-UTF-8 output does not kill the run with UnicodeDecodeError. result = subprocess.run( command, shell=True, # nosec B602 - Required for sandbox command execution @@ -1337,36 +1212,23 @@ def run_command(self, command: str, timeout: float | int | None = None) -> tuple logger.warning(error_msg) return -1, "", error_msg - # NOTE: get_file_content, file_exists, and list_files intentionally do NOT validate - # path traversal. The sandbox is a trusted execution environment where the agent - # needs filesystem access beyond the sandbox root (e.g., reading installed packages, - # system headers). Path traversal protection is handled at the agent permission level. + # NOTE: get_file_content, file_exists and list_files intentionally do NOT + # validate path traversal -- the agent legitimately reads installed packages and + # system headers. That protection lives at the agent permission level. def _within_sandbox(self, candidate: Path) -> bool: """Whether a resolved criterion path stays inside the sandbox. The read-side twin of :meth:`_resolve_within_sandbox`, which every OTHER - task-authored path already goes through. Criterion paths were the one - consumer that skipped it, and ``Path('/tmp/sandbox') / '/etc/passwd'`` is - ``/etc/passwd`` — pathlib discards the prefix on an absolute right - operand — so ``file_contains`` / ``file_check`` / ``file_matches_regex`` - were a pass-fail oracle over any file the grading user could read, and - ``json_check`` could surface parsed values in ``details``. - - That was defensible while a task YAML was operator-supplied. It stopped - being so when ``evaluate `` began rebuilding the criteria list - from a shareable run directory. + task-authored path already goes through. Returns False rather than raising: an out-of-sandbox path is - indistinguishable to the criterion from a file that is not there, which - is the same answer the template and mock-dir paths give, and raising - here would book a config error as an agent crash (CE039). - - Silent by design — :meth:`resolve_files` reports the escape ONCE per - criterion, naming the pattern the task author actually wrote. Logging - here instead named a resolved absolute path (uninformative: it is the - author's own string joined onto a tempdir) once per rejected glob - match, so a wide pattern produced a burst of near-identical warnings. + indistinguishable to the criterion from a file that is not there, and + raising would book a config error as an agent crash (CE039). Silent by + design -- :meth:`resolve_files` reports the escape ONCE per criterion, + naming the pattern the task author actually wrote. + + Rationale: .claude/notes/isolation.md § Criterion paths are contained, quietly """ assert self.sandbox_dir is not None root = self.sandbox_dir.resolve() @@ -1393,21 +1255,14 @@ def _warn_escaped(self, path: str) -> None: def _reject_escaped(self, path: str, candidate: Path) -> None: """Refuse a criterion path that names an existing file OUTSIDE the sandbox. - Returning ``[]`` here booked an eval-CONFIG error as an agent failure: - the criterion scored a gating 0.0 with "file does not exist" for a file - that plainly does exist, and the only other signal was a WARNING in the - task log. `tasks/byod_smoke_test.yaml` was broken exactly that way — it - checks `/opt/byod_marker`, baked into the BYOD image, and joining an - absolute path discards the sandbox prefix, so containment dropped it and - the suite reported a 0.0 nobody could explain from the score alone. - No agent behaviour can ever satisfy such a path, so it is not a verdict - about the agent. That is precisely the distinction CE039 exists to - enforce, and `CheckerMisuseError` is its prescribed signal. + about the agent -- precisely the distinction CE039 enforces, with + ``CheckerMisuseError`` as its prescribed signal. - Note the guard fires only when the escaping path EXISTS. A criterion - naming a merely-absent absolute path still resolves to "no match", which - is an ordinary failing verdict, not a misconfiguration. + Note the guard fires only when the escaping path EXISTS. A merely-absent + absolute path still resolves to "no match", an ordinary failing verdict. + + Rationale: .claude/notes/isolation.md § Criterion paths are contained, quietly """ raise CheckerMisuseError( f"Criterion path {path!r} resolves to {candidate}, outside the sandbox ({self.sandbox_dir}). " @@ -1419,24 +1274,14 @@ def _reject_escaped(self, path: str, candidate: Path) -> None: def resolve_files(self, path: str) -> list[Path]: """Resolve a criterion ``path`` to the sandbox files it addresses. - A path that names an existing file or directory resolves to itself, - **even when it contains a glob metacharacter** — a real file called - ``report[2024].json`` is graded as itself rather than reinterpreted as - a character class that would silently match ``report2.json``. Only when - the literal does not exist is a path containing ``*``, ``?`` or ``[`` - expanded against the sandbox root, so a criterion can address a file - whose exact location the task prompt does not pin — e.g. ``**/*.flow`` - matches a scaffolded wrapper directory the agent was free to name. - - Glob matches are filtered through the sandbox's ignore patterns - (``.venv``, ``node_modules``, ``dist``, … — see - :func:`~coder_eval.resources.get_ignore_patterns`), because the sandbox - root holds harness-created content the agent never authored and - grading off it is neither fair nor deterministic. Only path segments - the glob *discovered* are filtered: a segment the pattern names - literally (``dist/**/*.js``) is an explicit opt-in and survives. - Matches are sorted so grading is deterministic, and directories are - dropped so a glob cannot resolve to something unreadable. + A path that names an existing file or directory resolves to itself, **even + when it contains a glob metacharacter**. Only when the literal does not + exist is it expanded against the sandbox root, filtered through the + sandbox's ignore patterns -- and only for segments the glob *discovered*, + so ``dist/**/*.js`` is an explicit opt-in that survives. Matches are sorted + for determinism and directories dropped. + + Rationale: .claude/notes/isolation.md § Criterion paths are contained, quietly Args: path: Relative path or glob pattern @@ -1610,28 +1455,22 @@ def preserve_to(self, artifact_dir: Path) -> Path: old_sandbox_dir = self.sandbox_dir shutil.move(str(old_sandbox_dir), str(preserve_path)) - # mkdtemp creates the sandbox root at 0700. Under driver:docker the - # container runs as root, so the preserved tree lands on the host - # bind-mount owned by root with that 0700 top dir -- the host user - # (a different uid) then can't traverse it, so the blob upload and any - # `ls` see an empty dir and silently skip the artifacts. Grant a+rX on - # the preserved tree so artifacts are readable across the uid boundary. - # No-op-ish on the host path, where the sandbox is already owner-readable. + # mkdtemp creates the sandbox root at 0700, and under driver:docker the + # tree lands owned by container root -- the host user (a different uid) + # then cannot traverse it and silently sees no artifacts. + # Rationale: .claude/notes/isolation.md § preserve_to, capture_to, and the capture denylist _grant_read_traverse(preserve_path) - # Sandbox now lives at the artifact path -- redirect pointers so that a - # subsequent cleanup() is a no-op. Venv absolute paths inside the venv - # are not rewritten (same behaviour as the prior copy-based code). + # Repoint so a subsequent cleanup() is a no-op. Absolute paths inside the + # venv are not rewritten (same as the prior copy-based code). self.sandbox_dir = preserve_path if self.venv_dir is not None: try: rel = self.venv_dir.relative_to(old_sandbox_dir) self.venv_dir = preserve_path / rel except ValueError: - # Defensive: venv_dir is currently always created under - # sandbox_dir (see _setup_virtualenv), so relative_to should - # always succeed. If a future code path places it elsewhere, - # leave the pointer untouched -- the move did not relocate it. + # Defensive: venv_dir is always created under sandbox_dir today. + # If that changes, leave the pointer untouched. pass self._cleanup_on_exit = False return preserve_path @@ -1640,27 +1479,15 @@ def capture_to(self, artifact_dir: Path) -> Path: """Copy an in-place workspace out to ``artifact_dir/`` (docker WORKDIR mode). Sibling to :meth:`preserve_to`, but COPIES instead of ``shutil.move``: the - sandbox here is the container's own WORKDIR (e.g. ``/root``), which is - discarded with ``--rm``, and the orchestrator's own cwd may sit under it -- - so a copy is safe and non-destructive. ``symlinks=True`` + - ``ignore_dangling_symlinks=True`` makes a dangling symlink a no-op rather - than a failure (the exact breakage the old ``cp -a "$PWD/." "/root/"`` - reconciliation prelude hit). Grants cross-uid read on the COPY, since that - is the artifact the host reads (mirrors preserve_to's grant on its dest). - - Because the WORKDIR can be HOME (``/root``) or otherwise overlap - framework mounts, we exclude framework/sensitive entries via - :data:`_WORKSPACE_CAPTURE_IGNORE` -- most importantly ``.claude`` (the - RW lean copy of the host ``~/.claude`` carries ``.credentials.json``; - without this a ``/root`` WORKDIR would leak it into artifacts), plus - ``.venv``/``node_modules``/``.npm-prefix`` (sandbox-created bulk), and - Linux home-directory noise (``.cache``, ``.config``, ``.npm``, - ``.local``, shell dotfiles) written by tools like uv/pip/npm when - HOME == WORKDIR. + sandbox here is the container's own WORKDIR, discarded with ``--rm``, and + the orchestrator's own cwd may sit under it. Excludes the credential and + noise entries in :data:`_WORKSPACE_CAPTURE_IGNORE`, because the WORKDIR can + BE ``$HOME``. Returns the destination path; unlike preserve_to it does NOT repoint - ``self.sandbox_dir`` -- the workspace persists in-container and is reaped - with the container, and ``_cleanup_on_exit`` is already False (run-in-place). + ``self.sandbox_dir``. + + Rationale: .claude/notes/isolation.md § preserve_to, capture_to, and the capture denylist """ if not self.sandbox_dir: raise RuntimeError("Sandbox not set up") diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 3d65f466f..3f16b659f 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 46_898 +_ESSAY_BASELINE_WORDS = 36_569 _SRC = Path("src/coder_eval") From 0feacac25d1ed417a59458916528c58c37b657f0 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 20:28:01 -0700 Subject: [PATCH 06/19] =?UTF-8?q?docs:=206/7=20=E2=80=94=20move=20criteria?= =?UTF-8?q?,=20routing=20and=20judging=20rationale=20into=20.claude/notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 14,406 words across 34 files down to about 3,300, with no code change and no generated surface touched. Two constraints made this phase different from the others, and both were verified by parsing rather than by trusting a downstream check: the first non-blank line of every criterion class docstring, which CE033 copies into the plugin reference, is byte-identical; and no Field(description=...) string under models/ changed, which is what CE030 ripples into the task guide. The LiveVerdict determinism and monotonicity properties stay at the definition site in criteria/base.py, because they are the contract an author has to satisfy. What moved is the derivation: why CE036 replays trajectories, and what CE025 cannot see. contracts.md gains the live_verdict contract, the checker base class, route resolution, judge context and untrusted text, sub-agent judging, and recording a CLI invocation. persistence.md gains judge persistence; agents.md the sdk_options pass-through; orchestration.md the rates, the early-stop guardrail placement, and the armed gate. Docstrings that restated a Field description, or the task guide's own YAML examples, were deleted rather than moved — the guide is the source of truth. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/agents.md | 36 ++ .claude/notes/contracts.md | 339 ++++++++++++++++++ .claude/notes/orchestration.md | 65 ++++ .claude/notes/permissions.md | 5 +- .claude/notes/persistence.md | 43 +++ src/coder_eval/criteria/agent_judge.py | 168 +++------ src/coder_eval/criteria/base.py | 149 +++----- .../criteria/classification_match.py | 4 +- src/coder_eval/criteria/cli_called.py | 56 +-- src/coder_eval/criteria/command_executed.py | 99 ++--- src/coder_eval/criteria/llm_judge.py | 49 +-- .../criteria/reference_comparison.py | 24 +- src/coder_eval/criteria/run_command.py | 5 +- src/coder_eval/criteria/skill_triggered.py | 70 ++-- src/coder_eval/criteria/uipath_eval.py | 16 +- src/coder_eval/evaluation/checker.py | 64 +--- src/coder_eval/evaluation/judge_anthropic.py | 5 +- src/coder_eval/evaluation/judge_bedrock.py | 9 +- src/coder_eval/evaluation/judge_context.py | 148 +++----- src/coder_eval/evaluation/judge_litellm.py | 33 +- .../evaluation/judge_persistence.py | 125 ++----- src/coder_eval/evaluation/sub_agent.py | 128 +++---- src/coder_eval/evaluation/verdict_tool.py | 8 +- src/coder_eval/models/__init__.py | 9 +- src/coder_eval/models/agent_config.py | 116 ++---- src/coder_eval/models/cli_match.py | 28 +- src/coder_eval/models/container_paths.py | 49 +-- src/coder_eval/models/criteria.py | 184 +++------- src/coder_eval/models/enums.py | 46 +-- src/coder_eval/models/experiment.py | 18 +- src/coder_eval/models/judge.py | 21 +- src/coder_eval/models/limits.py | 25 +- src/coder_eval/models/merge_strategy.py | 15 +- src/coder_eval/models/results.py | 127 ++----- src/coder_eval/models/routing.py | 163 +++------ src/coder_eval/models/sandbox.py | 127 +++---- src/coder_eval/models/tasks.py | 40 +-- src/coder_eval/models/telemetry.py | 61 ++-- src/coder_eval/models/templates.py | 15 +- tests/lint/prose_budget.py | 2 +- 40 files changed, 1190 insertions(+), 1504 deletions(-) diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index 656ccce74..382b1185c 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -8,6 +8,20 @@ - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. The residual is almost always positive; a NEGATIVE one means the captured generations over-report some bucket, which is why the note's wording is branched — a `-512` entry must not read as "billed but not surfaced". +### The result_tokens measure and CE043 + +`result_tokens` approximates the size of the tool result the model received, derived from the +UNTRUNCATED summary content. It is deliberately cache-independent — available identically +whether prompt caching was on or off — because the alternative, inferring result size from +prompt-cache growth, is unavailable when caching is disabled. Approximate rather than the +API's exact count, but deterministic and always present. + +The measure is only meaningful while the summary stays whole. An agent that truncates a +command's output before recording it, as one once did, silently under-reports that command's +result — which is what CE043 forbids. One-line summaries of non-command tool items are +intentionally brief and out of scope; trimming for DISPLAY belongs in the renderers. + + ## Harness run-limit parity - **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode and Pi each keep a native unit too, because their CLIs stream a real multi-step loop per `communicate()` (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. @@ -623,3 +637,25 @@ The session id is sanitized because dataset-row tasks have path-shaped ids (`suite/row_3`, set in `task_loader`) and Pi derives its session file from the id under `--session-dir` — so a raw `/` resolves to a non-existent subdir and fails the row before any work is done. + +## The sdk_options pass-through + +`sdk_options` forwards SDK fields the framework does not model. Validation is an ALLOW rule — +a key must be a real SDK field AND not framework-owned — so the user-visible set is the +difference of the two. The denylist is explicit rather than derived, so the reason each key is +withheld stays next to the code. + +What is withheld: anything `coder_eval` already owns as a typed field (setting it here would +silently shadow the typed one), anything transport- or lifecycle-critical, and anything +security-critical. Hooks, MCP servers, the permission-prompt tool, the tool callback and +sub-agent definitions all run BEFORE any allowed-tools gate — `agent_judge` forces +`setting_sources=[]` for exactly that reason, and letting `hooks` through would re-open the +hole. Session lifecycle is owned by the orchestrator's "advance the session id only on a clean +turn" logic. Budgeting overlaps the run limits the orchestrator enforces with explicit final +statuses, and two independent budget guards would disagree on counts. Telemetry is required to +recover per-emission output tokens around an upstream bug, so turning it off would silently +drop per-message accounting. + +The classification is kept from failing open as the SDK grows: a test asserts EVERY field on +the SDK's options type is classified, either typed-mirrored or framework-owned, so a new SDK +release adding an unclassified field fails loudly instead of silently passing through. diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index 5a3a55a8f..edbf2dca4 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -7,3 +7,342 @@ - **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.) - **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure. + +### Criterion aggregation + +The observed-label sentinels are their own classes in the confusion matrix, so "wrote +nothing" and "wrote something unrecognisable" are visible failure modes rather than rows +that vanish from the rollup. + +## The live_verdict contract + +`LiveVerdict` is a criterion's verdict from a PARTIAL, mid-run trajectory. Every override +must be deterministic (a pure function of the prefix it is handed) and monotonic (once it +answers pass or fail, it answers the same for every longer prefix); `undecided` is the only +verdict allowed to change on a later call. Both properties are stated at the definition +site in `criteria/base.py`, because they are the contract an author has to satisfy. + +`EarlyStopWatcher`'s deferred fail-stop and its pass/fail flip-attribution are correct ONLY +because the two shipped implementations honor them. A non-monotonic or non-deterministic +override compiles, passes CE025, and silently corrupts the stop logic. + +### Why replay is the only sound check + +CE025 checks the SHAPE — that a `LiveSuccessCriterion` subclass pairs with a `live_verdict` +override — and can check nothing else. Monotonicity over arbitrary Python is undecidable, +so CE036 replays every live criterion against every prefix of recorded trajectories and +asserts both properties directly. That is why adding a live criterion REQUIRES adding +`ContractCase` fixtures in the same change: CE036 fails on a live type with no cases, and +on a polarity an instance claims decidable that no fixture reaches. + +The limit is worth naming: replay proves the contract on the trajectories an author +supplied, not in general. Honoring it is still on the author. + +`live_decidable_polarities` is typed as the narrower CAPABILITY type rather than a bare +`frozenset[str]`, so an override returning a typo — or `undecided`, which is never a +decidable polarity — is a pyright error rather than a runtime-only lint gap. + +### Any-engagement, and why order does not matter + +`skill_triggered` scores a row on whether the skill was engaged AT ALL. A positive +criterion passes if its skill was engaged anywhere in the run, so a wrong skill engaged +first does not fail it — that is recall. A distractor criterion fails on ANY engagement of +its skill — that is precision. `live_verdict` latches the instant its own skill is engaged, +which is the same policy read forward. + +Engagement is detected agent-agnostically so the score does not depend on the harness. +Claude emits an explicit `Skill` tool call; a harness that names that argument something +else renames it at the AGENT boundary rather than growing an alternative here. Every other +agent engages a skill by reading its files, so both the repo layout and the sandbox symlink +contain `skills//`, matched in any string parameter. The trailing separator is +required, so `uipath-agents` does not collide with `uipath-agents-foo`. The function returns +the full SET rather than a yes/no, which is what lets a caller detect a competing +engagement. + +### Normalizing a shell command before matching it + +`command_pattern` regexes are written against the logical command, but telemetry records +the raw `bash -lc "..."` wrapper — so whichever way the agent happened to quote an argument +leaks into the pattern. Authors then hand-model that escaping and get it subtly wrong, +silently under-counting correct calls. Unwrapping the wrapper and resolving quoting with +`shlex` lets a pattern match argv semantics instead. Shell operators survive as their own +tokens, so patterns that reference them keep working. + +Adding the normalized haystack is NOT purely additive. A `command_pattern` can only gain +matches, but the same haystacks feed `exclude_pattern` and the `max_count` gate, so a +normalized form can newly satisfy an exclusion or trip a cap — a command that counted on +the raw text alone can stop counting. + +It is memoized because the early-stop watcher re-scans the whole accumulated trajectory on +every tool-call event, normalizing the same command many times per run. + +The regex search window is capped to bound ReDoS on a large command string, and +normalization runs over that same truncated window, so `shlex` never sees more than the cap +and needs no separate guard. + +## Recording a CLI invocation + +### Evidence, not attestation + +The invocation log is an ordinary file in the sandbox the agent writes to, so an agent that +wants to can append a record for a call it never made, or delete one it did. `cli_called` is +built to keep an HONEST run honest — a missing or unreadable log FAILS rather than passing a +`max_count: 0` guard vacuously — not to withstand an adversary. An anti-cheat control needs +what `docs/DOCKER_ISOLATION.md` describes, not this. + +A flat log line cannot express "verb X was called AND flag Y had value Z" without stacked +lookaheads, cannot tell a quoted argument containing spaces from two arguments, and cannot +stop a match running across a shell operator. That is why the criterion matches +field-by-field over a structured record rather than regexing a flattened string. + +### The five refuse-to-score paths are uniform at a gating 0.0 + +A missing log, a write sentinel, a sidecar import error, unusable records and a rule error +all score 0.0 and none of them raises. Every one is agent-REACHABLE, because the whole +recorder directory lives inside the sandbox the agent writes to: an escalation would be a +`FinalStatus.ERROR`, normally read as "harness broken, discard this data point", which is a +strictly better outcome for a failing agent than FAILED. An earlier revision raised +`CheckerMisuseError` on the rule error believing only a task author could cause it; +appending one line to the log disproved that. + +The legitimate concern behind that escalation — a task author's unevaluable response spec +must not be booked as an agent failure — is handled where the agent cannot reach it instead. +`RecordedCli` proves every response rule is evaluable at LOAD time, so an authoring mistake +is a validation error before the sandbox exists. For the same reason the pattern is compiled +at validation rather than in a checker: a response rule evaluates its pattern inside the +sandbox, where a `PatternError` is swallowed and the tool serves its fallback — a log line +indistinguishable from a legitimate no-match. + +Both fault checks are scoped to the records the criterion is about. One log serves every +shadowed tool, so a `uip` shim that could not import its matcher must not fail a +`tool: curl` guard that has nothing to do with response dispatch. + +A verb is compared against the NON-FLAG arguments, so a flag written into one can never +match — silently: the criterion scores 0 against a log holding the very call it describes. +The check mirrors the splitter's own number rule, so it cannot forbid a token the matcher +would in fact have seen. + +### What the shim is, and is not + +The shim records the invocation, writes the configured output, and exits. Nothing is +executed, so there is no network, no auth and no side effect. It stubs a tool; it does not +proxy one — a test that needs a REAL executable's behaviour recorded on the way through +supplies its own wrapper under `mock_path_dirs`, which depends on the tool being installed, +on PATH order, and usually on live credentials. + +The recorder directory is deliberately NOT dot-prefixed: CI artifact upload skips hidden +files, and the log is primary evidence for every `cli_called` criterion. Names are +case-folded when checked against the reserved set, because on a case-insensitive filesystem +`CALLS.JSONL` is the seeded log and `ARGV_MATCH.PY` is the sidecar. Shadowing an interpreter +breaks the harness rather than the tool under test: the shim is a script run by an +interpreter and its directory goes FIRST on a PATH the orchestrator also reuses for +`run_command`, so `tool: python3` made the shim re-resolve its own interpreter to itself — +an exec loop that spins to the task timeout. + +## The checker base class + +### Exactly one of _check_impl or _check_impl_async + +Async is the strictly more general shape — it covers a CPU-bound check and a genuine I/O +one — so the PRIMARY surface is async and a checker implements whichever form is natural. +The base derives the other: a sync `_check_impl` is offloaded with `asyncio.to_thread`, and +a native-async checker's sync entry point runs the coroutine to completion. + +`__init_subclass__` enforces EXACTLY one at class-definition time, so the check cannot be +escaped by registering through the registry directly instead of the decorator. Overriding +neither would recurse forever between the two defaults the first time either is called, and +overriding both would let two live implementations drift into different scores for identical +agent output depending on which entry point ran — the class of bug the derivation exists to +eliminate. `abstract=True` opts a shared base out; its own subclasses are still checked. + +### What escalates instead of scoring 0.0 + +A judge-infrastructure outage and a checker-contract misuse are not agent failures, so they +propagate to `FinalStatus.ERROR` rather than being captured into a scored 0.0. This is the +CE039 distinction, and it is why several sites raise rather than assert: the call runs inside +a wrapper that catches plain `Exception` (`AssertionError` included) and downgrades it, which +is the opposite of the intended behaviour — and an `assert` is stripped under `-O` anyway. + +`reference_comparison` is the worked example on the task-definition side. A typo in +`reference_file` scored as 0.0 counts against the agent's pass rate, and on a dataset-fanned +suite it silently zeroes every row and drags down the aggregate mean, the `suite_thresholds` +gate, the JUnit report and the evalboard alike. A missing AGENT file is the opposite case and +is genuinely a gating 0.0. `reference_file` is confined to the reference directory, unlike a +judge's author-written `files:` entry, because it names one file of the solution being +compared against and traversal out of the staged copy is always a mistake. + +Grading time is accumulated at the checker, not at the four orchestrator call sites, so a +fifth site cannot be added without it — the same reason the tool subtraction lives at one +collector seam. It is monotonic, booked in a `finally` so a grade that raises still records +what it spent, and `None` until something is checked so an ungraded row reports "never +measured" rather than an instant 0.0 (CE058). + +## Route resolution + +The agent's route and the evaluation side's route are resolved separately. +`resolve_evaluation_route` decides the `llm_judge` / `agent_judge` transport, and a +`model` lands on `checker_context.api_route.model` ONLY when a real override was given — +never the agent's own model. `criterion.model` is `None` rather than a materialized +default when unset, so the precedence (explicit per-criterion model, then the route's, then +the default judge model) survives a `model_dump(mode="json")` and reload, which a +`model_fields_set` check would not. + +Route resolution raises rather than asserts, because it is reached on the evaluate-only path +with no preceding key validation and must survive `-O`. The exhaustive final arm of each +match is unreachable but present, so every path returns explicitly. + +Under `DirectRoute` the judge transport is resolved at startup: `anthropic` when a key is +present, `None` otherwise, in which case an enabled `llm_judge` fails at dispatch. The +Bedrock backend routes the judge through the run's own backend and never reaches that +selection. The transport-unconfigured arm short-circuits BEFORE backend dispatch. + +Small-model fallback matters more than it looks: Claude Code routes page-summarization and +other small, fast steps through the small-fast model env var, which on Bedrock is exported +only when `small_model` is set. Leaving it unset made every WebFetch fail with "model +issues", so the main model is the fallback. + +### LiteLLM params and env_params + +Neither route object carries a base URL or a credential. They flow through orchestrator +state — `environment_info` recording, logging — that has no business handling config which +should be read live from the environment, so the agent path reads settings itself at the +point of use. + +The CHECKER side is not sourced from settings AT ALL. A gateway-routed judge model rarely +reuses the proxy or credential the agent's own backend points at, so there is no implicit +fallback: the task author owns the call shape. `params` is passed to `litellm.acompletion` +verbatim as extra kwargs, and `env_params` maps a kwarg name to the ENV VAR NAME to resolve +it from at call time — so an arbitrary provider's config, secrets included, is representable +without a secret landing in the task YAML. `env_params` values are env var names, never +secrets, so recording them verbatim is safe; `params` is not, because an author could put a +raw secret in one. + +Calling through the library rather than assuming one wire protocol lets `model` carry its own +provider hint and get that provider's real request and response shape handled, including +per-provider quirks. `drop_params` covers parameters the library's own static cost map knows a +model rejects; a custom or gateway-routed model id usually is not in that map, so it alone +does not protect a `params`-supplied kwarg the target model live-rejects. + +## Judge context and untrusted text + +A judge reads agent output and tool-call summaries, which are UNTRUSTED text. Everything +below exists so that what the judge was shown, and what is persisted afterwards, stay +under the harness's control rather than the agent's. + +`$TASK_DIR` and `$REFERENCE_DIR` in a judge's `files:` are resolved against host directories +and read from the host filesystem, mirroring the same-named env vars `run_command` exposes so +judges and shell criteria address the same places by the same name. `$REFERENCE_DIR` is how a +task attaches SPECIFIC grading assets instead of the whole tree, and it is readable here only +because judges run outside the agent's turn — the directory sits at mode 000 for the whole of +`agent.communicate`. + +### Scrub before truncate + +`scrub_reference` redacts by exact substring, so ordering is load-bearing everywhere it is +used. Clip first and a multi-KB reference cut by a per-field budget leaves a partial fragment +that no longer matches the full secret: `replace` finds nothing and the prefix is persisted +unsanitized. Scrubbing first replaces the secret with a short marker before any clipping, so +on-disk fields can never carry partial reference content. The same rule is why a truncating +renderer must pass its per-file cap into the key collection — a key built only from +untruncated text would never match what the judge was actually shown. + +The secret list is `list[str]`, not `str | Iterable[str]` and not `Sequence[str]`: `str` +satisfies both of those, so a caller passing a bare string type-checked clean and then had its +CHARACTERS iterated as individual secrets, each under the length floor, silently redacting +nothing. Secrets shorter than the floor are skipped, because redacting a tiny common substring +would produce gibberish and is not a realistic leak vector — with the caveat that in directory +mode every file becomes its own entry, so a one-line `__init__.py` is left unscrubbed. The +empty input is a no-op, guarding the `"".replace("", ...)` pathology that ballooned strings. + +### What counts as reference-derived + +The scrub gate keys on the recorded reference-derived text being non-empty, NOT on +`include_reference`. A `$REFERENCE_DIR/...` entry in `files:` attaches reference bytes with +`include_reference=false`, which is the documented way to show a judge one rubric without +inlining the tree — keying on the flag left exactly that combination persisting the solution +verbatim into the archived transcript. + +Scrub keys are the per-FILE contents, not the single rendered block: a model is far more +likely to echo one file back than to reproduce the whole concatenation verbatim, and a +whole-block key would never match. `agent_judge` has TWO routes reference bytes take to the +judge — inlined `files:` entries, and the `_reference/` mount it browses directly — so both +are collected, and both the truncated and untruncated forms are emitted. + +### The reference walk's budget and symlink rules + +One walk backs both consumers, so the budget and symlink rules cannot diverge between what +the judge is shown and what is redacted from its output — a divergence there leaks reference +content into a persisted transcript. + +Symlinks are NOT followed: a reference bundle shipping `secrets -> /etc/passwd` would +otherwise read a host file into the scrub-key list, and a symlinked subdir back to the root +would loop forever. Binary and unreadable files are skipped silently. File count and total +bytes are capped, sized well above any realistic reference skeleton, and the size is +pre-checked with `stat` BEFORE reading, so a single huge file is not pulled into memory in +full before the per-iteration check fires. A directory at mode 000 — meaning this was called +during an agent turn, which should never happen — yields nothing rather than raising. + +Rendering drops TRAILING files rather than truncating mid-block, and says so explicitly, so +the judge does not read the omission as "the reference doesn't implement that". The same +shape applies to a dialog transcript, where the per-message cap is applied first so one huge +message cannot crowd out later turns. Remaining transcript budget is split by importance +rather than evenly — an even split clipped the verdict, the most important field, for tasks +with long rubrics. + +When both routes are in play the judge is told what the `$REFERENCE_DIR/x` label maps to: its +shell has no such variable and its workspace exposes the tree at `_reference/`, so a judge +asked to re-Read a file it was shown could not otherwise resolve the path it was given. + +## Sub-agent judging + +### The judge's identity is its system prompt + +A sub-agent's system prompt is its ENTIRE identity — judge instructions, simulator persona — +so the coding-agent preset must never prefix it. The judge must not carry an engineering +persona ahead of its grading role, and its verdicts must not shift when the preset does. +That takes both halves: an omitted prompt gets the bare preset, which is the same failure, +so neither is accepted. The runner RAISES on a misconfigured caller rather than mutating the +config, because the field is part of the type contract and callers own theirs; not `assert`, +so the check survives `-O`. + +When YAML supplies a partial `agent:` block, pydantic constructs a fresh config from those +keys and the judge defaults never apply — so user-set fields are overlaid on top of a fresh +judge default instead, keeping the hardened defaults for every key the user did not set. +`sdk_options` gets a deep merge to match the experiment layer, which is a no-op today and +prevents a foot-gun when judge defaults grow. `system_prompt_file` must be cleared in the +SAME update as `system_prompt`: they are mutually exclusive and assignment is validated, so +a sequential assignment would raise on the intermediate state. + +### The security floor + +The judge runs with the evaluator's credentials and can execute arbitrary Bash by default, +against four surfaces: artifacts it executes, prompt injection from included agent output, +credential exfiltration through any network-capable tool, and hooks or MCP servers the main +agent planted. `setting_sources=[]` is forced regardless of user YAML so the SDK does not +load settings or MCP config from the judge's cwd — those can install pre-LLM lifecycle hooks +or MCP subprocesses that run with the evaluator's credentials BEFORE the allowed-tools gate. +The ignore-patterns floor is set-unioned in so it is present even when the user supplied +their own list, and the verdict tool is forced into `allowed_tools` because the judge must be +able to report. Author convenience does not override the floor. `llm_judge` is the answer for +adversarial generation. + +The sandbox is copied into an isolated temp dir, so the sub-agent never touches the original +and later criteria are unaffected. Symlinks are SKIPPED rather than preserved, so a planted +`creds -> /root/.aws/credentials` cannot leak host files to a Bash-enabled sub-agent. The +reference mount uses a SEPARATE ignore list defaulting to empty, because the sandbox-side list +contains `_reference` as defense-in-depth against an agent-planted collision and reusing it +would silently drop a user's own nested `_reference/` subdir. The mount point is rmtree'd +first and the copy deliberately does NOT pass `dirs_exist_ok`: if any file survives, a loud +failure beats silently merging the reference into agent-planted content under the same path. + +### Cancellation safety + +`run_async` is awaited on the orchestrator's own loop, so it is reachable by cancellation at +any await — including mid-copy. `asyncio.to_thread` is NOT itself cancellable: the worker +keeps running after the awaiting coroutine raises. Every such call is therefore shielded and +tracked, and the `finally` awaits any still-in-flight one BEFORE the cleanup, so an orphan +thread can never recreate files after cleanup already ran. The temp directory itself is bound +with a plain synchronous call, because offloading a single fast syscall only widens the +cancellation window. The cleanup itself is deliberately synchronous: a bare `await` inside +`finally` is cancellable, and cancelling as that line is reached would skip cleanup and leak +the copy with no reaper. diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index 0a79fe32d..fd203e57b 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -57,6 +57,38 @@ VERDICT, never the facts — the seeding cannot restore a fact the execute phase captured. The budget gate runs AFTER the criteria on the graded path purely for partial-credit visibility, and there is no partial credit under `execute`. +### Rates need verdict evidence, not bucket counts + +A published rate divides by rows that actually carry a verdict, not by a bucket count. The +four category buckets cannot tell a graded FAILURE from a TIMEOUT no criterion ever saw, so +`tasks_measured` is counted evidence and sits beside them without being part of the +sum-to-`tasks_run` invariant. + +`nothing_was_measured` is the ONE definition of "this rate has no numerator to be a fraction +of", shared by the run summary, the variant aggregate and the suite rollup — three copies of +a published rate is how one surface reports `n/a` and another reports `0.0%` for the same +run, which already happened when the guard shipped on only one of them. Its first version +tested `succeeded + failed == 0` and was wrong for the same reason the bug it fixed was +wrong: TIMEOUT and the budget stops are category `failed` and reachable under `execute`, so +ONE timed-out row in a 100-task ungraded night read as "something was measured" and published +a real 0% point on the evalboard trend for a run that graded nothing. + +An ungraded score stays `None` everywhere it is published. A plain float would launder it +into 0.000, which renders as — and is picked as a best variant against — a real score of +zero. + +Every status maps to exactly one reporting category EXPLICITLY, with no catch-all, so a new +status fails the classification assert until someone decides where it belongs rather than +silently collapsing into `failed` and skewing both the reports and the telemetry dimension. +A failed image build is grouped with ERROR, being an environment fault rather than a task +outcome the agent could have avoided. The ungraded bucket is a FOURTH category, not a fold: +folding into `failed` would depress every pass rate, into `succeeded` would invent verdicts, +and into `error` would report a healthy run as broken. + +`is_execution_fact` is explicit for the same reason — each status is either "the agent phase +ended this way" (preserved by a detached grade) or "grading decided this" (replaced). +Defaulting either way silently is how an ERROR row becomes a SUCCESS. + ### Refusing a criteria-free task under grade `TaskDefinition.success_criteria` accepts an empty list at the model level, because the @@ -210,6 +242,24 @@ At the default threshold both bounds collapse exactly to "any single armed crite effective fail stops the run" and "every `on_pass: stop` criterion has live-passed". Lowering it lets a low-weight armed criterion's failure be absorbed without truncation. +### The armed gate is not the watcher's bounds + +The ceiling and floor above decide WHETHER to stop. `armed_criteria_passed` is the +separate, post-hoc question of how a run that was cut gets graded, and it runs over the +ARMED subset only — an unarmed criterion took no part in the decision to truncate, so +gating on it would judge the run against evidence the truncation guaranteed would be +missing. + +Each armed criterion is BINARISED against its own `pass_threshold` before weighting, so +the gate asks "did this criterion pass?" rather than averaging raw scores — which is what +makes `gate_threshold=1.0` an EXACT equivalence with the strict-AND rule, not an +approximation of it. That exactness is what the fired-only rule above rests on. + +It raises rather than returning False on a criteria/results length mismatch or an empty +armed set. Neither is a verdict — the first is a caller bug, and the second means the +caller reached a fired-only gate for a run the watcher never fired on, which +`early_stop is not None` is there to rule out. + ### Precision is traded, recall is not A pass-stop cuts the run the instant the floor locks in, so a fail-armed criterion (a @@ -251,6 +301,21 @@ trajectory. to a full run. Because live verdicts are triggers and not truth, this can never produce a FALSE early stop — it only ever errs toward running more. +### Why the guardrails are not model validators + +A degenerate gate threshold on an ARMED task, and the removed master arm, are both rejected — +but in `orchestration/early_stop.py`, not on the model. Whether a task is armed lives on the +CRITERIA, which the run-limits model cannot see, and that model is field-merged across five +layers, so a model-level validator cannot tell a real mistake from a value merged forward +from a sibling layer. + +The placement also decides what the failure DOES. A dedicated error gets the same hard-stop +CLI treatment as every other early-stop guardrail — it flips the plan exit code and aborts +the run — whereas a plain pydantic error would land in the plan command's generic per-variant +"resolution failed" branch, which prints red text but deliberately does not flip the exit +code, so a model-level raise would silently pass CI. Cross-field semantics that are warnings +rather than errors live in the run-limits validator for the same post-merge visibility. + ## Recording the task as authored `task_config.resolved` and `source_file` describe the task as AUTHORED, which is NOT diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md index 49591654e..0e32a4d8d 100644 --- a/.claude/notes/permissions.md +++ b/.claude/notes/permissions.md @@ -42,8 +42,9 @@ has to be able to read it exactly then, while the agent still cannot. That is wh Only the REFERENCE is shielded, never the sandbox. A live criterion reading the agent's own output files needs no window change at all — and should not get one. Reading the -static reference mid-turn cannot break the `LiveVerdict` monotonicity contract; reading -the half-written sandbox can, and is the "end-state peeking" `live_verdict` rules out. +static reference mid-turn cannot break the `LiveVerdict` monotonicity contract +(contracts.md § The live_verdict contract); reading the half-written sandbox can, and is +the "end-state peeking" `live_verdict` rules out. ### Not wired up yet diff --git a/.claude/notes/persistence.md b/.claude/notes/persistence.md index 37aeea65d..cacd4b413 100644 --- a/.claude/notes/persistence.md +++ b/.claude/notes/persistence.md @@ -68,3 +68,46 @@ A regrade writes to `grade.log`, not `task.log`, because the log handler opens i trajectory log the run had already paid for. `grade.docker.log` exists for the same reason one layer down: on the `run --resume` path `docker.log` is already the executed container's log. + +## Judge persistence + +A judge transcript — tool calls, raw verdict, rendered prompt, system prompt — runs 10-100 +KB. Inlining it into every `task.json` inflates the row record for the consumers that never +need it (suite rollups, report renderers), so it spills to a sibling file and the row keeps +only a path. The inline value is left in place in memory so the orchestrator's own HTML +render still sees it; the JSON dump excludes it. + +The sibling is YAML rather than JSON because the transcript carries multi-line text, which +YAML's literal block scalar renders as readable paragraphs instead of one line full of `\n` +escapes. Its consumers are humans. The reader accepts `.json` too, so previously-spilled runs +keep rendering, and a row with no path at all is a no-op, so old inline records keep working. + +FILE ORDER IS LOAD-BEARING. Each filename is keyed off the criterion's position in its result +list, and the reader resolves the stored path, so each list must retain its order through +persistence. Fields inside the file lead with the human-readable summary and put the bulkiest +last. + +### transcript_path is untrusted input + +`task.json` travels across trust boundaries — CI artifacts, shared eval bundles — so a path +read back out of one is attacker-controlled. The writer only ever emits a generated +basename, so the reader ALLOWLISTS that shape directly rather than joining first and hoping +`is_relative_to` catches the result: `/etc/passwd` and `../../secrets` are refused at the +door. + +The shape is checked under BOTH POSIX and Windows path semantics. `subdir\judge-0.yaml` +passes a POSIX check on Linux, where a backslash is an ordinary character, and resolves to a +nested file on Windows; rejecting under either interpretation enforces the policy regardless +of which platform the record travels to next. Windows reserved device basenames are rejected +for the same reason: `CON.yaml`, `NUL` and `COM1` open the console, the null device or a +serial port wherever they sit in the tree, the extension is ignored by Win32, and the check +runs platform-independently so a record minted on Linux is refused before it travels. +Containment is then re-verified after resolution, because a symlink inside the directory +could still redirect outside it. + +A scalar, list or `None` payload is rejected early: it would land on the result and crash the +renderer with an `AttributeError` on its first `.get()`. The typed model is preferred so +isinstance checks see the same shape they get during the original run, with a fallback to the +raw dict so an older sibling or a forward-compatible key does not break re-render. The +assignment bypasses pydantic's setter, since a loaded subclass's config might validate or +reject it, and the renderer accepts both shapes. diff --git a/src/coder_eval/criteria/agent_judge.py b/src/coder_eval/criteria/agent_judge.py index 739ad8b01..032cd2a75 100644 --- a/src/coder_eval/criteria/agent_judge.py +++ b/src/coder_eval/criteria/agent_judge.py @@ -44,8 +44,8 @@ path_uses_token, ) -# Private helper + shared security-floor constant — not part of the public -# coder_eval.models surface, but the single source of truth for both files. +# Not part of the public coder_eval.models surface, but the single source of truth +# for both files. from coder_eval.models.criteria import ( # noqa: CE001 JUDGE_SECURITY_IGNORE_FLOOR, _default_judge_agent_config, @@ -65,11 +65,10 @@ logger = logging.getLogger(__name__) -# This is the judge's ENTIRE identity: _build_agent_config forces -# system_prompt_mode="replace" so the Claude Code coding-agent preset never -# reaches the scoring instrument — the judge must not carry an engineering -# persona (terse, proactively edits files) ahead of its grading role, and its -# verdicts must not shift when the preset does. +# The judge's ENTIRE identity: _build_agent_config forces +# system_prompt_mode="replace" so the coding-agent preset never reaches the +# scoring instrument. +# Rationale: .claude/notes/contracts.md § The judge's identity is its system prompt _SYSTEM_PROMPT = """\ You are a strict code reviewer evaluating a project generated by a coding agent. @@ -116,10 +115,9 @@ async def _check_impl_async( route = ctx.route reference_dir = ctx.reference_dir - # Master enablement gate. Skipped criteria don't spawn the sub-agent and - # don't affect cost; weighted score includes them as 1.0 so they don't penalize. - # The route precondition below is intentionally NOT checked when skipped — - # disabled criteria should be free to declare in tasks where the route isn't set. + # Master enablement gate. A skipped criterion spawns nothing and scores 1.0, + # and the route precondition below is deliberately NOT checked -- a disabled + # criterion should be declarable in a task where the route is unset. if not criterion.enabled: return JudgeCriterionResult( criterion_type=criterion.type, @@ -139,19 +137,12 @@ async def _check_impl_async( ) raise ValueError(msg) - # ``criterion.files`` is optional: when set, the named paths are pre-attached - # to the prompt envelope (fast verdict, narrow tool surface); when empty, the - # judge inspects the sandbox copy via its tools. Both modes compose — the judge - # can ``Read`` anything else even when files are pre-attached. - # .build() does synchronous file I/O — offload to a worker thread so it - # doesn't stall the event loop (see llm_judge.py's identical comment). - # - # ``include_reference=False`` is passed to the BUILDER on purpose, even when - # the criterion opted in: agent_judge attaches the reference by MOUNTING it at - # ``_reference/`` (below) for the judge to Glob/Read, so also inlining the whole - # tree into the prompt would duplicate it and blow the context budget on large - # references. ``$REFERENCE_DIR/...`` entries in ``files:`` still resolve — that - # is the supported way to pre-attach specific reference assets here. + # .build() does synchronous file I/O -- offload it so it does not stall the + # event loop. ``include_reference=False`` is passed to the BUILDER on purpose + # even when the criterion opted in: agent_judge attaches the reference by + # MOUNTING it, so inlining the tree too would duplicate it. + # ``$REFERENCE_DIR/...`` entries in ``files:`` still resolve. + # Rationale: .claude/notes/contracts.md § What counts as reference-derived judge_ctx = await asyncio.to_thread( JudgeContextBuilder( files=criterion.files, @@ -163,19 +154,15 @@ async def _check_impl_async( max_file_chars=criterion.max_file_chars, ).build, sandbox, - # Passed UNCONDITIONALLY: the builder uses this to resolve - # `$REFERENCE_DIR/...` entries in `files:`, which are documented as - # working regardless of include_reference. Gating it here made such an - # entry silently render "". Whether the WHOLE tree is - # attached is controlled by include_reference=False above (inlining) - # and ref_dir_for_runner below (the _reference/ mount). + # UNCONDITIONAL: the builder needs it to resolve `$REFERENCE_DIR/...` + # entries, which work regardless of include_reference. Gating it here + # made such an entry silently render "". reference_dir, turn_records, ) - # Mount the reference directory only when the criterion opted into seeing it. - # When include_reference=False the judge MUST NOT see the grading material, so - # zero out reference_dir before passing to the runner regardless of what came in. + # HAZARD: when include_reference=False the judge MUST NOT see the grading + # material, so zero this out regardless of what came in. ref_dir_for_runner = reference_dir if criterion.include_reference else None user_msg = _render_user_message( criterion.prompt, @@ -190,10 +177,8 @@ async def _check_impl_async( runner = SubAgentRunner( sandbox=sandbox, agent_config=agent_config, - # Use the floor-enforced patterns from the built config, not the - # user's raw ``criterion.agent.ignore_patterns`` — _build_agent_config - # injects the required security entries (.claude / .mcp.json / - # _reference) unconditionally. + # The floor-enforced patterns from the BUILT config, not the user's raw + # list -- _build_agent_config injects the security entries. ignore_patterns=agent_config.ignore_patterns, route=route, reference_dir=ref_dir_for_runner, @@ -208,12 +193,9 @@ async def _check_impl_async( turn_timeout=float(criterion.turn_timeout), ) except (TurnTimeoutError, AgentCrashError) as e: - # Two pre-output failure modes share one return path: - # - TurnTimeoutError: sub-agent's per-turn watchdog fired. - # - AgentCrashError: SDK/CLI emitted an is_error ResultMessage - # (e.g. Bedrock's ``error_during_execution / end_turn`` flake - # when the model returns an empty assistant turn). - # No turn was produced, so no token usage is attributable to the judge. + # Two pre-output failure modes share one return path (watchdog timeout, + # SDK error result). No turn was produced, so no token usage is + # attributable to the judge. is_timeout = isinstance(e, TurnTimeoutError) if is_timeout: logger.warning("agent_judge: turn timeout after %ds", criterion.turn_timeout) @@ -222,11 +204,9 @@ async def _check_impl_async( logger.warning("agent_judge: sub-agent crashed: %s", str(e)[:200]) details = f"Judge agent crashed before producing a verdict: {str(e)[:200]}" - # No turn was produced, so there's no transcript to capture — but - # return a JudgeCriterionResult anyway so renderers / aggregators - # that switch on ``isinstance(cr, JudgeCriterionResult)`` see the - # uniform shape (findings=[], transcript=None) instead of having to - # special-case a base CriterionResult with criterion_type='agent_judge'. + # Still a JudgeCriterionResult, so renderers and aggregators that switch + # on isinstance see the uniform shape instead of special-casing a base + # CriterionResult with criterion_type='agent_judge'. return JudgeCriterionResult( criterion_type=criterion.type, description=criterion.description, @@ -238,19 +218,11 @@ async def _check_impl_async( token_usage=None, ) - # The scrub set must cover every route reference bytes took to the judge, - # and this criterion has two: - # - # 1. `$REFERENCE_DIR/...` entries in `files:`, inlined by the builder - # regardless of include_reference — recorded on the context as it - # attached them, already in the truncated shape the judge saw. - # 2. include_reference=true, which MOUNTS the tree at _reference/ for - # the judge to Read directly. Nothing inlines it, so the builder - # never sees it; collect it here. - # - # max_file_chars is passed because route 1 truncates: scrub_reference - # redacts by exact substring, so an untruncated-only key cannot match the - # text the judge was actually shown. Both forms are emitted. + # The scrub set must cover BOTH routes reference bytes took to this judge: + # inlined `$REFERENCE_DIR/...` entries (recorded on the context) and the + # `_reference/` MOUNT (which nothing inlines, so it is collected here). + # max_file_chars is passed because the first route truncates. + # Rationale: .claude/notes/contracts.md § What counts as reference-derived scrub_secrets: list[str] = list(judge_ctx.reference_secrets) if criterion.include_reference and reference_dir is not None: scrub_secrets.extend(collect_reference_secrets(reference_dir, criterion.max_file_chars)) @@ -271,17 +243,11 @@ def _build_agent_config( *, system_prompt: str, ) -> ClaudeCodeAgentConfig: - # When YAML supplies a partial `agent:` block (e.g. only ``model:``), - # Pydantic constructs a fresh AgentConfig from those keys and the judge - # defaults from _default_judge_agent_config never apply. Overlay the - # user-set fields on top of a fresh judge default so missing keys keep - # the hardened defaults (read-only toolkit, bypassPermissions, etc.). - # - # ``sdk_options`` gets a special deep-merge to match the experiment-layer - # behavior: if defaults ever supply pass-through keys, a partial user - # override should add to / override individual keys without wiping the - # rest. Today defaults['sdk_options'] is empty so this is a no-op, but - # the symmetry prevents a future-foot-gun when judge defaults grow. + # A partial `agent:` block makes pydantic build a fresh config from those keys, + # so the judge defaults never apply -- overlay the user-set fields on a fresh + # judge default instead. ``sdk_options`` deep-merges to match the experiment + # layer (a no-op today, symmetry for when judge defaults grow). + # Rationale: .claude/notes/contracts.md § The judge's identity is its system prompt defaults = _default_judge_agent_config() user_overrides = criterion.agent.model_dump(exclude_unset=True) if "sdk_options" in user_overrides: @@ -289,31 +255,25 @@ def _build_agent_config( config = defaults.model_copy( update={ **user_overrides, - # Force the judge's own prompt in replace mode regardless of user YAML - # (see the note on _SYSTEM_PROMPT for why, and SubAgentRunner.__init__ - # for the enforcement point). system_prompt_file must be cleared in the - # SAME update: it is mutually exclusive with system_prompt, and - # BaseAgentConfig has validate_assignment=True, so a user YAML that set - # it would make a sequential assignment raise on the intermediate state. + # HAZARD: system_prompt_file must be cleared in the SAME update -- the + # two are mutually exclusive and assignment is validated, so a + # sequential one would raise on the intermediate state. "system_prompt": system_prompt, "system_prompt_file": None, "system_prompt_mode": "replace", - # SECURITY: force setting_sources=[] regardless of user YAML so the SDK - # does NOT load .claude/settings.json or .mcp.json from the judge's cwd. - # Those files can install pre-LLM lifecycle hooks (SessionStart / - # PreToolUse) or MCP subprocesses that run with the evaluator's - # credentials BEFORE the allowed_tools gate kicks in. + # SECURITY: forced regardless of user YAML. Settings and MCP config in + # the judge's cwd can install pre-LLM lifecycle hooks or MCP + # subprocesses that run BEFORE the allowed_tools gate. + # Rationale: .claude/notes/contracts.md § The security floor "setting_sources": [], }, deep=True, ) - # SECURITY: ensure the ignore_patterns floor is present even if the - # user supplied their own list. Set-union guarantees idempotence and - # doesn't depend on order. + # SECURITY: the floor is present even when the user supplied their own list. + # Set-union is idempotent and order-independent. config.ignore_patterns = list({*config.ignore_patterns, *JUDGE_SECURITY_IGNORE_FLOOR}) - # SECURITY/contract: the judge MUST be able to call its verdict tool. - # Force the MCP tool name into ``allowed_tools`` regardless of the user's - # override (mirrors the ignore_patterns floor above). + # SECURITY/contract: the judge MUST be able to call its verdict tool, so the + # tool name is forced in regardless of the user's override. config.allowed_tools = list({*(config.allowed_tools or []), SUBMIT_VERDICT_MCP_TOOL_NAME}) return config @@ -328,13 +288,11 @@ def _build_result( user_msg: str, capture: VerdictCapture, ) -> CriterionResult: - # Iterable scrub set: covers both code/file references (single string) and - # directory references (every file's content). Empty list when the criterion - # didn't opt into seeing the reference — no scrubbing needed. + # Covers both a single-string reference and a directory (every file's content). + # Empty when the criterion did not opt into seeing it. scrub_key: list[str] | None = scrub_secrets if scrub_secrets else None - # Persist the structured verdict (when present) as the ``raw_verdict`` so - # HTML / task.json auditors see the actual scoring payload instead of the + # The structured verdict, so auditors see the scoring payload rather than the # agent's post-call "Verdict submitted." filler. raw_verdict_for_transcript = capture.verdict.model_dump_json() if capture.verdict is not None else turn.agent_output @@ -357,13 +315,9 @@ def _build_result( if parse_err is not None: logger.debug("agent_judge: parse error — %s", parse_err) - # Scrub BEFORE the 500-char slice. ``scrub_reference`` uses ``str.replace`` - # and only matches the secret as a contiguous whole string — if we sliced - # first, a reference longer than 500 chars would leave its leading prefix - # in the slice with no full secret left for replace to match, persisting - # an unsanitized fragment in ``details``. Scrub-before-slice replaces the - # full secret with the short ```` marker before any - # truncation, so on-disk details can never carry partial reference content. + # HAZARD: scrub BEFORE the slice. Slicing first leaves a prefix with no + # full secret left for `str.replace` to match. + # Rationale: .claude/notes/contracts.md § Scrub before truncate scrubbed_output = scrub_reference(turn.agent_output, scrub_key) return JudgeCriterionResult( criterion_type=criterion.type, @@ -416,11 +370,8 @@ def _render_user_message( points the judge there instead of inlining a single file's content. """ reference_block = "" - # `$REFERENCE_DIR/x` entries in `files:` are inlined under that literal - # label, but the judge's shell has no REFERENCE_DIR variable and its - # workspace exposes the tree at `_reference/` — so a judge asked to re-Read - # or Glob a file it was shown could not resolve the path it was given. Say - # what the label maps to whenever both routes are in play. + # The judge's shell has no REFERENCE_DIR variable and its workspace exposes the + # tree at `_reference/`, so say what the inlined label maps to. if reference_dir_mounted and any(path_uses_token(b.path, REFERENCE_DIR_TOKEN) for b in context.files): reference_block = ( f"NOTE: FILE blocks labelled `{REFERENCE_DIR_TOKEN}/` below come from the reference " @@ -460,9 +411,8 @@ def _render_user_message( ) if context.files: - # Mirror llm_judge's FILE-block format so authors get consistent rendering. - # Missing files surface as ```` rather than being silently - # dropped — the judge can then penalize per the rubric. + # Mirrors llm_judge's FILE-block format. A missing file surfaces as + # ```` rather than being dropped, so the judge can penalize. file_blocks = "\n".join( f"--- FILE: {f.path} ---\n{f.content if f.content is not None else ''}" for f in context.files diff --git a/src/coder_eval/criteria/base.py b/src/coder_eval/criteria/base.py index 4f82ee470..03f3d76dc 100644 --- a/src/coder_eval/criteria/base.py +++ b/src/coder_eval/criteria/base.py @@ -33,20 +33,10 @@ # - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST # return that SAME verdict for every longer prefix (i.e. every later call in the # same run). "undecided" is the only verdict allowed to change on a later call. -# EarlyStopWatcher's deferred fail-stop and pass/fail flip-attribution -# (early_stop.py::_prev_verdicts) are correct only because both existing -# implementations (skill_triggered, command_executed) honor this. A non-monotonic or -# non-deterministic override compiles and passes CE025 (which only checks -# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts -# the stop logic. # -# ENFORCEMENT: lint rule CE036 (tests/lint/live_verdict_contract.py) replays every live -# criterion against every prefix of recorded trajectories and asserts both properties — -# monotonicity over arbitrary Python is undecidable, so replay is the only sound check. -# Adding a LiveSuccessCriterion REQUIRES adding ContractCase fixtures for it in the same -# change (CE036 fails on a live type with no cases, and on a polarity its instances claim -# decidable but no fixture reaches). Note the limit: CE036 proves the contract on the -# trajectories an author supplied, not in general — honoring it is still on the author. +# Enforced by lint rule CE036, which replays every live criterion against every prefix +# of recorded trajectories; CE025 checks only the subclassing/pairing shape. +# Rationale: .claude/notes/contracts.md § The live_verdict contract LiveVerdict = Literal["pass", "fail", "undecided"] @@ -75,15 +65,13 @@ class CheckContext: reference_dir: "Path | None" = None -# Module-level ParamSpec (rather than the PEP 695 `def f[**P](...)` form ruff's -# UP047 prefers) — CodeQL's Python extractor doesn't yet parse PEP 695 type -# parameters referenced via `P.args`/`P.kwargs` and flags `P` as a potentially -# uninitialized local; a plain `typing.ParamSpec` is unambiguous to both tools. +# Module-level ParamSpec, not the PEP 695 form ruff's UP047 prefers: CodeQL's Python +# extractor flags `P` as a possibly-uninitialized local there. P = ParamSpec("P") -# Exceptions that must escalate rather than be captured into a scored-0.0 -# CriterionResult — a judge-infra outage or a checker-contract misuse is not an -# agent failure. Shared by both handle_criterion_errors(_async) wrappers below. +# Exceptions that must ESCALATE rather than be captured into a scored-0.0 +# CriterionResult. Shared by both handle_criterion_errors(_async) wrappers below. +# Rationale: .claude/notes/contracts.md § What escalates instead of scoring 0.0 _ESCALATING_EXCEPTIONS: tuple[type[Exception], ...] = (JudgeInfrastructureError, CheckerMisuseError) @@ -138,9 +126,8 @@ def wrapper( try: return func(self, criterion, *args, **kwargs) except _ESCALATING_EXCEPTIONS: - # Judge infra failure / checker-contract misuse is NOT an agent - # failure — do not score it 0.0. Propagates to Orchestrator.run()'s - # broad except → FinalStatus.ERROR. + # NOT an agent failure -- do not score it 0.0. Propagates to + # Orchestrator.run()'s broad except -> FinalStatus.ERROR. raise except Exception as e: return _failed_result(self, criterion, e, "check") @@ -178,38 +165,17 @@ async def wrapper( class BaseCriterion[C: BaseSuccessCriterion](ABC): """Abstract base class for all criterion checkers. - The checking logic's PRIMARY surface is async (``_check_impl_async``) — - every criterion, at bottom, is "read some inputs, produce a score," and - async is the strictly more general shape: it covers both a criterion that - never awaits anything (a CPU/file-bound check) and one that awaits genuine - I/O (an LLM judge call). A checker implements exactly ONE of the two - ``_check_impl*`` methods — whichever is its natural form — and the base - class derives the other automatically: - - - CPU/file-bound criteria (file_exists, command_executed, ...) override - ``_check_impl`` (plain sync code, no event loop to think about). The - base's default ``_check_impl_async`` offloads it to a worker thread via - ``asyncio.to_thread`` so it never blocks the event loop. - - Criteria that make genuine async I/O (llm_judge, agent_judge) override - ONLY ``_check_impl_async`` (using an async HTTP client / subprocess - bridge) — there is no reason to hand-maintain a second, sync-client - implementation just for the rarely-used direct-sync-call path. The - base's default ``_check_impl`` derives a sync call by running the async - one to completion on a fresh event loop (``asyncio.run``). - - ``__init_subclass__`` enforces that a checker overrides EXACTLY ONE of - the two, at class-definition time — overriding neither would recurse - forever between the defaults (``asyncio.run`` <-> ``asyncio.to_thread``) - the first time either is called, and overriding both would let the two - implementations silently drift into different scores depending on which - entry point (``check`` vs ``check_async``) ran. - - ``check()`` / ``check_async()`` are FINAL — they apply centralized error - handling and must not be overridden; implement ``_check_impl`` / - ``_check_impl_async`` instead. - - Type parameter C binds the checker to its specific criterion model for - better IDE support and static type checking. + A checker implements exactly ONE of ``_check_impl`` (plain sync) or + ``_check_impl_async`` (native async I/O) -- whichever is its natural form -- and + the base class derives the other. ``__init_subclass__`` enforces that at + class-definition time. + + ``check()`` / ``check_async()`` are FINAL: they apply centralized error handling + and must not be overridden. + + Type parameter C binds the checker to its specific criterion model. + + Rationale: .claude/notes/contracts.md § Exactly one of _check_impl or _check_impl_async Example: @register_criterion @@ -244,25 +210,15 @@ def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]": def __init_subclass__(cls, *, abstract: bool = False, **kwargs: Any) -> None: """Enforce the ``_check_impl`` / ``_check_impl_async`` override contract - at class-definition time (module import), regardless of which entry - point later registers the class — closing the gap where a subclass - registered via ``CriterionRegistry.register`` directly (bypassing the - ``register_criterion`` decorator) escaped the check, and turning the - mutual-recursion failure mode (``asyncio.run`` <-> ``asyncio.to_thread`` - exhausting OS threads) into an immediate, clearly-named ``TypeError``. - - Enforces "exactly one", not just "at least one": overriding BOTH is - also rejected — a checker with two live implementations (sync-path - `_check_impl` and async-path `_check_impl_async`) is free to have them - drift into different scores for identical agent output depending on - which entry point (``check`` vs ``check_async``) happened to run it, - which is exactly the class of bug this derivation design exists to - eliminate. - - Pass ``abstract=True`` on a class that intentionally implements - neither (e.g. a shared abstract base for a family of related - checkers) to opt out of the check for that one class; every one of - ITS subclasses is still checked normally. + at class-definition time (module import), regardless of which entry point + later registers the class. + + Enforces "exactly one", not just "at least one". Pass ``abstract=True`` on a + class that intentionally implements neither (e.g. a shared abstract base for + a family of checkers) to opt out for that one class; every one of its + subclasses is still checked normally. + + Rationale: .claude/notes/contracts.md § Exactly one of _check_impl or _check_impl_async """ super().__init_subclass__(**kwargs) if abstract: @@ -333,10 +289,9 @@ def _check_impl( ) -> CriterionResult: """Sync checking logic. Override this OR ``_check_impl_async`` (not both). - Base default: runs ``_check_impl_async`` to completion on a fresh event - loop (``asyncio.run``) — the bridge for checkers whose natural form is - async (they override ``_check_impl_async`` only). Override THIS instead - when the checker's natural form is plain sync CPU/file-bound code. + Base default: runs ``_check_impl_async`` to completion on a fresh event loop + — the bridge for checkers whose natural form is async. Override THIS when + the checker's natural form is plain sync CPU/file-bound code. Args: criterion: The specific criterion definition (Pydantic model) @@ -350,11 +305,9 @@ def _check_impl( CriterionResult with score (0.0-1.0), details, and error info Raises: - CheckerMisuseError: this bridge is called from inside a running - event loop (``asyncio.run`` cannot start a nested loop) — this - is a caller mistake (the async-primary surface should have - been awaited instead), not an agent failure, so it escalates - rather than silently scoring 0.0. + CheckerMisuseError: this bridge is called from inside a running event + loop (``asyncio.run`` cannot start a nested one) — a caller + mistake, not an agent failure, so it escalates. Any other exception - will be caught by @handle_criterion_errors """ try: @@ -430,25 +383,21 @@ def live_verdict( ) -> LiveVerdict: """Decide this criterion from a PARTIAL, mid-run trajectory (early-stop). - Reads ONLY ``turn_records`` — a live verdict, by definition, may not peek - at the finished sandbox (that would invite end-state peeking), so there is - no ``sandbox`` parameter. Returns ``"pass"``/``"fail"`` only when the - outcome is already knowable from the events seen so far, else - ``"undecided"``. + Reads ONLY ``turn_records``: a live verdict may not peek at the finished + sandbox, so there is no ``sandbox`` parameter. Returns ``"pass"``/``"fail"`` + only when the outcome is already knowable, else ``"undecided"``. This only *triggers* an early stop; the authoritative scores always come - from ``check()``/``_check_impl`` run on the frozen trajectory after the - stop, so a live/final divergence can never corrupt scoring. - - Base default: ``"undecided"`` (not observable mid-run). A checker - overrides this iff its criterion model is a ``LiveSuccessCriterion`` - subclass (``models/criteria.py``) — that subclassing is the single - source of truth for "is this criterion type live-observable", checked - by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint - rule CE025. An override MUST also satisfy the deterministic + monotonic - contract documented on the ``LiveVerdict`` type above, enforced by lint - rule CE036 — which requires this criterion type to supply replay fixtures - (``tests/lint/live_verdict_contract.py::CASES``) in the same change. + from ``check()`` on the frozen trajectory, so a live/final divergence can + never corrupt scoring. + + Base default: ``"undecided"``. Override iff the criterion model is a + ``LiveSuccessCriterion`` subclass — that subclassing is the single source of + truth for "is this type live-observable" (CE025). An override MUST satisfy + the deterministic + monotonic contract on the ``LiveVerdict`` type above, + and must supply CE036 replay fixtures in the same change. + + Rationale: .claude/notes/contracts.md § The live_verdict contract """ return "undecided" diff --git a/src/coder_eval/criteria/classification_match.py b/src/coder_eval/criteria/classification_match.py index 8102a6a34..52261a26a 100644 --- a/src/coder_eval/criteria/classification_match.py +++ b/src/coder_eval/criteria/classification_match.py @@ -22,9 +22,9 @@ logger = logging.getLogger(__name__) -# Sentinels for the observed label — surfaced as their own classes in the -# suite rollup's confusion matrix so "didn't write anything" and "wrote +# Their own classes in the confusion matrix, so "wrote nothing" and "wrote # something unrecognisable" are visible failure modes rather than vanishing. +# Rationale: .claude/notes/contracts.md § Criterion aggregation _SENTINEL_NONE = "(none)" _SENTINEL_OTHER = "(other)" diff --git a/src/coder_eval/criteria/cli_called.py b/src/coder_eval/criteria/cli_called.py index 1f2516026..f9a8054a0 100644 --- a/src/coder_eval/criteria/cli_called.py +++ b/src/coder_eval/criteria/cli_called.py @@ -53,10 +53,8 @@ def _check_impl( Result with binary score (1.0 when the match count is within [min_count, max_count], 0.0 otherwise) """ - # No pre-flight re.compile here: `FlagMatch` compiles the pattern at - # validation, so an uncompilable one never reaches a checker -- and it has - # to be caught there, because the response-rule surface that shares this - # model cannot report an error at all. + # No pre-flight re.compile: `FlagMatch` compiles at validation, which is + # where it has to happen -- the response-rule surface cannot report. if not sandbox.file_exists(criterion.log): # Harness fault, not agent behaviour. Failing stops a max_count: 0 # guard passing vacuously against a log that never existed. @@ -86,20 +84,14 @@ def _check_impl( usable, unusable = parse_log(content) - # Both fault checks below are scoped to the records this criterion is about. - # One log serves every shadowed tool, so a `uip` shim that could not import - # its matcher must not fail a `tool: curl` guard that has nothing to do with - # response dispatch -- and whose error message would not explain why. + # Scoped to the records this criterion is about: one log serves every + # shadowed tool. mine = [record for _, record in usable if criterion.tool is None or record.get("tool") == criterion.tool] - # Booked on every record when the shim could not IMPORT its matcher, so - # no rule was ever tried and the agent saw the entry defaults throughout. - # Scored 0.0 rather than raised, unlike `rule_error` below: the sidecar - # lives in the agent-writable recorder directory, so an agent can cause - # this, and escalating would hand it a way to turn a failing run into an - # ERROR. The records themselves are still trustworthy -- the shim keeps - # logging -- which is what stops a `max_count: 0` guard passing on a - # forbidden call that would otherwise have gone unrecorded entirely. + # The shim could not IMPORT its matcher, so no rule was ever tried and the + # agent saw the entry defaults throughout. Scored 0.0, never raised -- the + # RECORDS stay trustworthy (the shim keeps logging), which is what stops a + # `max_count: 0` guard passing on a call that went unrecorded. broken = [record for record in mine if record.get("sidecar_error") is not None] if broken: return CriterionResult( @@ -113,32 +105,14 @@ def _check_impl( ), ) - # The shim books this when its own rule evaluation RAISED. If it fires, the - # responses the agent saw were not the ones the task described, so no verdict - # over this log means anything. + # Booked when the shim's own rule evaluation RAISED: the responses the agent + # saw were not the ones the task described, so no verdict over this log + # means anything. # - # ALL FIVE of this checker's refuse-to-score paths are uniform at a gating - # 0.0, and that uniformity is the point: - # - # missing log an agent can `rm` it - # write sentinel an agent can fill the disk or chmod the dir - # sidecar_error an agent can delete the matcher beside the shim - # unusable records an agent can append garbage to the log - # rule_error an agent can append a crafted record, or edit the shim - # - # EVERY one is agent-REACHABLE, because the whole recorder directory lives - # inside the sandbox the agent writes to. So none of them may raise: an - # escalation here is a `FinalStatus.ERROR`, which is normally read as "harness - # broken, discard this data point", and that is a strictly better outcome for - # a failing agent than FAILED. An earlier revision raised `CheckerMisuseError` - # on `rule_error` believing only a task author could cause it; appending one - # line to `calls.jsonl` disproved that. - # - # The legitimate concern that motivated the escalation -- a task author's - # unevaluable response spec must not be booked as an agent failure -- is - # handled where the agent cannot reach it instead: `RecordedCli` proves every - # rule is evaluable at LOAD time (see `_validate_responses_are_evaluable`), so - # an authoring mistake is a validation error before the sandbox even exists. + # HAZARD: all five of this checker's refuse-to-score paths are uniform at a + # gating 0.0 and NONE may raise -- every one is agent-reachable, and an + # escalation to ERROR is a better outcome for a failing agent than FAILED. + # Rationale: .claude/notes/contracts.md § The five refuse-to-score paths are uniform at a gating 0.0 faults = [record for record in mine if record.get("rule_error") is not None] if faults: return CriterionResult( diff --git a/src/coder_eval/criteria/command_executed.py b/src/coder_eval/criteria/command_executed.py index e21497495..80cedc7e1 100644 --- a/src/coder_eval/criteria/command_executed.py +++ b/src/coder_eval/criteria/command_executed.py @@ -18,9 +18,8 @@ logger = logging.getLogger(__name__) -# Limit regex search input length to mitigate ReDoS on large command strings. -# Normalization runs over this same truncated window (see _match_haystacks), so -# shlex never sees more than this many chars and needs no separate size guard. +# HAZARD: bounds ReDoS on a large command string. Normalization runs over this +# same truncated window, so shlex needs no separate size guard. _MAX_PATTERN_SEARCH_LEN = 2000 @@ -53,32 +52,20 @@ def _is_command_flag(tok: str) -> bool: def _normalize_shell(cmd_text: str) -> str | None: """Quote-resolved, wrapper-stripped form of a shell command, or None. - ``command_pattern`` regexes are written against the *logical* command - (``uip is resources run list ``), but telemetry records the - raw ``bash -lc "..."`` wrapper — so whichever way the agent happened to quote - an argument (bare, ``"double"``, ``'single'``, ``\\"escaped\\"``) leaks into - the pattern. Authors then hand-model that escaping and get it subtly wrong - (e.g. allowing ``"`` but not ``'``), silently under-counting correct calls. - - This unwraps a ``bash``/``sh``/``zsh -c`` wrapper and resolves shell quoting - with ``shlex`` so a pattern can match argv semantics regardless of quoting. - Shell operators (``&&``, ``|``, ``>``) survive as their own tokens, so - patterns that reference them keep working, and embedded newlines collapse to - single spaces. Returns ``None`` when the text can't be parsed (an odd quote - count — NOT heredocs, which tokenize fine); the caller then keeps only the - raw text as a haystack. - - Adding a second (normalized) haystack is *not* purely additive: a - ``command_pattern`` can only gain matches, but the same haystacks feed - ``exclude_pattern`` and the ``max_count`` gate, so a normalized form can - newly satisfy an exclusion or trip a ``max_count`` cap — i.e. a command that - counted on the raw text alone can stop counting. See - ``CommandExecutedChecker._matching_commands``. - - Memoized (pure function of ``cmd_text``): the early-stop watcher re-scans the - whole accumulated trajectory on every tool-call event, so the same command is - normalized many times per run — the cache collapses that to once per distinct - (already-truncated) command string. + Unwraps a ``bash``/``sh``/``zsh -c`` wrapper and resolves shell quoting with + ``shlex``, so a ``command_pattern`` can match argv semantics regardless of how + the agent happened to quote an argument. Shell operators survive as their own + tokens; embedded newlines collapse to single spaces. Returns ``None`` when the + text cannot be parsed (an odd quote count -- NOT heredocs, which tokenize + fine); the caller then keeps only the raw text as a haystack. + + NOT purely additive: the same haystacks feed ``exclude_pattern`` and the + ``max_count`` gate, so a command that counted on the raw text alone can stop + counting. + + Memoized -- a pure function of ``cmd_text``. + + Rationale: .claude/notes/contracts.md § Normalizing a shell command before matching it """ try: tokens = shlex.split(cmd_text, posix=True) @@ -94,18 +81,15 @@ def _normalize_shell(cmd_text: str) -> str | None: if _is_command_flag(tok): rest = tokens[i + 1 :] if len(rest) == 1: - # Quoted whole-script form (`bash -lc "uip ... 'arg' ..."`): - # the script is a single token that may still hold inner - # quotes — re-split to resolve them. + # Quoted whole-script form: one token that may hold inner + # quotes -- re-split to resolve them. try: tokens = shlex.split(rest[0], posix=True) except ValueError: return None else: - # Argv-joined form: Codex rollout recovery joins argv WITHOUT - # re-quoting (codex_agent.py), so `bash -lc uip is resources ...` - # already arrives split — keep every token instead of - # collapsing to the first word. + # Argv-joined form: Codex rollout recovery joins argv without + # re-quoting, so it already arrives split. tokens = rest break if not tok.startswith("-"): @@ -173,12 +157,9 @@ def _matching_commands( if criterion.require_success and cmd.result_status != "success": continue - # Extract text for pattern matching (Bash: command param; others: JSON-serialized params). - # ``parameters`` is ``dict[str, Any]``, and a ``command`` value is not - # guaranteed to be a ``str`` — Codex sub-agent rollout recovery can carry - # it as an argv *list* (codex_agent.py). Narrow with ``isinstance`` so a - # non-``str`` value never reaches ``shlex.split``/slicing (which would raise - # ``AttributeError`` and zero the whole criterion); fall back to the JSON blob. + # A ``command`` value is not guaranteed to be a ``str`` -- Codex rollout + # recovery can carry it as an argv LIST. Narrow with ``isinstance`` so a + # non-str never reaches ``shlex.split`` and zeroes the whole criterion. raw_command = cmd.parameters.get("command") if cmd.tool_name == "Bash" and isinstance(raw_command, str) and raw_command: cmd_text = raw_command @@ -187,11 +168,9 @@ def _matching_commands( cmd_text = json.dumps(cmd.parameters) is_shell = False - # Match the pattern against the raw command AND its quote-resolved - # form, so authors need not encode shell quoting/escaping (which they - # do inconsistently, silently under-counting correctly-quoted calls). - # ``is_shell`` (decided once, above) tells the helper whether cmd_text - # is a shell command — it must not re-derive that from tool_name alone. + # Raw command AND its quote-resolved form. ``is_shell`` is decided once + # above; the helper must not re-derive it from tool_name alone. + # Rationale: .claude/notes/contracts.md § Normalizing a shell command before matching it haystacks = _match_haystacks(cmd_text, is_shell=is_shell) # Filter by command pattern @@ -279,16 +258,12 @@ def _check_impl( all_commands = [cmd for turn in turn_records for cmd in turn.commands] - # Note: do NOT short-circuit when ``all_commands`` is empty. A - # negative-assertion criterion (``min_count: 0`` + ``max_count: 0``) - # SHOULD pass here — zero commands trivially satisfies "must not - # call X". Falling through into the matching loop sets - # ``match_count = 0`` and the scoring branch handles both shapes. + # Do NOT short-circuit on an empty list: a negative assertion + # (``min_count: 0`` + ``max_count: 0``) SHOULD pass, and falling through + # sets ``match_count = 0`` so the scoring branch handles both shapes. - # Compile regex patterns if provided. - # re.DOTALL so `.` matches newlines — agents commonly write multi-line bash - # commands using backslash line-continuation (e.g. `uip ... \\\n --body ...`), - # and patterns like `foo.*--body` must span those line breaks. + # re.DOTALL so `.` matches newlines: agents write multi-line bash with + # backslash continuations, and `foo.*--body` must span them. pattern: re.Pattern[str] | None = None if criterion.command_pattern is not None: try: @@ -321,15 +296,9 @@ def _check_impl( match_count = len(matching_commands) - # Score model: - # max_count is None → fractional towards min_count (legacy behavior). - # When min_count == 0, the criterion is trivially - # satisfied (no minimum to hit) and scores 1.0. - # max_count is set → binary in-range. Pass iff - # min_count <= match_count <= max_count. - # The "negative assertion" pattern (min_count: 0, max_count: 0) drops - # naturally out of the binary branch — it now expresses "must NOT match" - # exactly. The model validator already rejects max_count < min_count. + # Score model: no max_count -> fractional towards min_count (min_count 0 is + # trivially satisfied, scoring 1.0); max_count set -> binary in-range. The + # negative assertion falls out of the binary branch. if criterion.max_count is None: score = 1.0 if criterion.min_count == 0 else min(1.0, match_count / criterion.min_count) else: diff --git a/src/coder_eval/criteria/llm_judge.py b/src/coder_eval/criteria/llm_judge.py index 42edad250..a8d1fbae8 100644 --- a/src/coder_eval/criteria/llm_judge.py +++ b/src/coder_eval/criteria/llm_judge.py @@ -74,19 +74,14 @@ async def _check_impl_async( ctx = context or CheckContext() route = ctx.route reference_dir = ctx.reference_dir - # Precedence: an explicit per-criterion `model:` always wins; otherwise fall - # back to `checker_context.api_route.model` (baked into route.model by - # resolve_evaluation_route — set only when a real override was given, never - # the agent's own model); otherwise DEFAULT_JUDGE_MODEL. `criterion.model` is - # `None` (not a materialized default) when unset, so this precedence survives - # a `model_dump(mode="json")` / reload round trip (e.g. the docker driver's - # task-serialization step) unlike a `model_fields_set` check would. + # Precedence: per-criterion `model:`, then the route's, then the default. + # `criterion.model` is None rather than a materialized default when unset, so + # this survives a model_dump/reload round trip as a model_fields_set check + # would not. Rationale: .claude/notes/contracts.md § Route resolution judge_model = criterion.model or (route.model if route is not None else None) or DEFAULT_JUDGE_MODEL - # Master enablement gate. Skipped criteria don't make an LLM call and don't - # affect cost; weighted score includes them as 1.0 so they don't penalize. - # Authors who want them excluded from weighted score should remove the - # criterion from the YAML or use experiment variants to override. + # Master enablement gate. A skipped criterion makes no LLM call and scores + # 1.0; to exclude it from the weighted score, remove it or use a variant. if not criterion.enabled: return JudgeCriterionResult( criterion_type=criterion.type, @@ -95,9 +90,8 @@ async def _check_impl_async( details="(skipped: enabled=false)", ) - # .build() does synchronous file I/O (reading sandbox/reference files) — offload - # to a worker thread so it doesn't stall the event loop this checker otherwise - # never blocks (that's the whole point of it being native-async). + # .build() does synchronous file I/O -- offload it so it does not stall the + # event loop this native-async checker otherwise never blocks. judge_ctx = await asyncio.to_thread( JudgeContextBuilder( files=criterion.files, @@ -115,9 +109,8 @@ async def _check_impl_async( user_msg = _render_user_message(criterion.prompt, judge_ctx) - # Transport-unconfigured arm needs to short-circuit BEFORE backend dispatch. - # Hit when the run uses the Direct backend with no ANTHROPIC_API_KEY (or no - # route at all). The Bedrock backend always has a usable judge transport. + # Short-circuits BEFORE backend dispatch: Direct with no ANTHROPIC_API_KEY, + # or no route at all. Bedrock always has a usable judge transport. if route is None or (isinstance(route, DirectRoute) and route.judge_transport is None): logger.error("llm_judge unreachable: no usable judge transport for the current backend") return JudgeCriterionResult( @@ -133,17 +126,10 @@ async def _check_impl_async( ), ) - # Scrub keys are the per-FILE contents of the reference directory, not the - # single rendered block: the model is far more likely to echo one file back - # than to reproduce the whole concatenation verbatim, and a whole-block key - # would never match. - # - # Taken from the CONTEXT, not recomputed from `criterion.include_reference`: - # the builder records every reference-derived byte it actually attached, - # which includes `$REFERENCE_DIR/...` entries in `files:` — the documented - # way to show a judge one reference asset with include_reference=false. - # Gating on the flag left exactly that combination unscrubbed, persisting - # the solution verbatim into the archived judge transcript. + # HAZARD: per-FILE contents, and taken from the CONTEXT rather than + # recomputed from `include_reference` -- gating on the flag left a + # `$REFERENCE_DIR/...` entry unscrubbed in the archived transcript. + # Rationale: .claude/notes/contracts.md § What counts as reference-derived scrub_key = judge_ctx.reference_secrets or None # Attribute the judge's API call to ``JudgeCriterionResult.token_usage`` @@ -244,11 +230,8 @@ async def _invoke_tool_channel( verdict, err = extract_verdict_from_anthropic_response(anthropic_response) response_usage = token_usage_from_anthropic_dict(anthropic_response, model=model) case LiteLLMRoute(): - # Reachable via an explicit `checker_context.api_route.route: litellm` - # override (see resolve_evaluation_route). Dispatches through the - # `litellm` library (see invoke_litellm_judge_async's module docstring) - # rather than assuming one wire protocol — task authors point this at - # whatever gateway their judge model actually lives behind. + # Reached via an explicit `route: litellm` override. Dispatches through + # the `litellm` library rather than assuming one wire protocol. litellm_response = await invoke_litellm_judge_async( route=route, model=model, diff --git a/src/coder_eval/criteria/reference_comparison.py b/src/coder_eval/criteria/reference_comparison.py index 4045f554e..54aa1347f 100644 --- a/src/coder_eval/criteria/reference_comparison.py +++ b/src/coder_eval/criteria/reference_comparison.py @@ -56,19 +56,11 @@ def _check_impl( error="No reference directory provided (task.reference not set)", ) - # Confined to the reference dir on purpose: unlike a judge's `files:` - # entry (author-written, trusted, and deliberately allowed to escape via - # `$REFERENCE_DIR/../shared/...`), this field names one file *of the - # solution being compared against*, so traversal out of the staged copy - # is always a mistake. - # Every failure below is a TASK-DEFINITION error, not an agent failure, so - # they raise CheckerMisuseError (-> FinalStatus.ERROR) instead of returning - # a gating score=0.0 (-> FinalStatus.FAILURE). A typo in `reference_file` - # scored as 0.0 is counted against the agent's pass rate, and on a - # dataset-fanned suite it silently zeroes every row and drags down the - # CriterionAggregate mean, the suite_thresholds gate, the JUnit report and - # the evalboard alike. The pre-directory-only equivalent raised out of - # `load_reference`, so this restores the loud behaviour. + # HAZARD: confined to the reference dir, unlike a judge's author-written + # `files:` entry -- this names one file OF the solution, so traversal out of + # the staged copy is always a mistake. Every failure below is a + # TASK-DEFINITION error and raises rather than scoring a gating 0.0. + # Rationale: .claude/notes/contracts.md § What escalates instead of scoring 0.0 ref_path = (reference_dir / criterion.reference_file).resolve() if not ref_path.is_relative_to(reference_dir.resolve()): raise CheckerMisuseError( @@ -93,15 +85,13 @@ def _check_impl( error="Sandbox not initialized", ) - # Load agent code through the shared path seam, so `agent_file` resolves - # (glob expansion, ignore filtering, exactly-one) like every other + # Through the shared path seam, so `agent_file` resolves like every other # sandbox-relative criterion path. try: agent_code = sandbox.get_file_content(criterion.agent_file) except FileNotFoundError: # CE039 exemption: genuinely the AGENT's failure, unlike reference_file - # above: the task asked for this file and the agent did not produce - # it, which is exactly what a gating 0.0 means. + # above -- the task asked for this file and the agent did not produce it. return CriterionResult( # noqa: CE039 criterion_type="reference_comparison", description=criterion.description, diff --git a/src/coder_eval/criteria/run_command.py b/src/coder_eval/criteria/run_command.py index 7a50471c7..7cdf880e6 100644 --- a/src/coder_eval/criteria/run_command.py +++ b/src/coder_eval/criteria/run_command.py @@ -16,9 +16,8 @@ logger = logging.getLogger(__name__) -# Per-stream output budget included in criterion details. Large enough for -# typical tool diagnostics (e.g. uip --output json, pytest tracebacks) while -# preventing a runaway process from blowing up the JSON trace or HTML report. +# Per-stream output budget in criterion details: enough for typical diagnostics, +# bounded so a runaway process cannot blow up the JSON trace or HTML report. _OUTPUT_BUDGET_CHARS = 4000 diff --git a/src/coder_eval/criteria/skill_triggered.py b/src/coder_eval/criteria/skill_triggered.py index 6151fe33f..35b8e2b6a 100644 --- a/src/coder_eval/criteria/skill_triggered.py +++ b/src/coder_eval/criteria/skill_triggered.py @@ -33,11 +33,9 @@ _YES = "yes" _NO = "no" -# Extracts the skill name between ``skills//`` path segments (the Codex -# file-read signal). Accept both POSIX and Windows separators because telemetry -# records the command exactly as the agent emitted it. One-or-more separators -# also handles JSON-escaped commands that retain doubled backslashes. The -# lookahead permits overlapping matches such as ``.../skills/skills//...``. +# Both separator styles, because telemetry records the command exactly as the agent +# emitted it; one-or-more also handles JSON-escaped doubled backslashes. The +# lookahead permits overlapping matches like ``.../skills/skills//...``. _SKILL_PATH_RE = re.compile(r"(?=skills[\\/]+([A-Za-z0-9][A-Za-z0-9_-]*)[\\/]+)") @@ -45,25 +43,13 @@ def _engaged_skill_names(cmd: CommandTelemetry) -> set[str]: """All skill names engaged by ONE command, agent-agnostically (any-skill). Detects both engagement signals so the criterion scores identically across - agents, and returns the (possibly empty) set of engaged skill names: - - - Claude: an explicit ``Skill`` tool call carries the skill in - ``parameters['skill']``, optionally namespaced (e.g. - ``plugin:uipath-agents``); the namespace is stripped via ``.split(":")[-1]``. - A harness whose skill tool names that argument something else renames it at - the AGENT boundary (OpenCode's ``name`` -> ``skill``, via - ``_OPENCODE_ARG_RENAME``), so this stays keyed on one canonical name rather - than growing an alternative per harness. - - Codex (and any non-Claude agent): no ``Skill`` tool exists, so a skill is - engaged by reading its files off disk via shell. Both the repo layout - (``.../skills//...``) and the sandbox symlink - (``.agents/skills//...``) contain the substring ``skills//``, - matched here in any string parameter (Bash ``parameters['command']`` or a - file-path parameter). The trailing separator required by ``_SKILL_PATH_RE`` - prevents prefix collisions (``uipath-agents`` vs ``uipath-agents-foo``). - - Returning the full set (rather than a single-skill yes/no) lets callers detect - a *competing* skill engagement. + agents: Claude's explicit ``Skill`` tool call (namespace stripped), and, for + every other agent, the ``skills//`` substring in any string parameter. + + Returning the full SET rather than a single-skill yes/no lets callers detect a + *competing* skill engagement. + + Rationale: .claude/notes/contracts.md § Any-engagement, and why order does not matter """ names: set[str] = set() if cmd.tool_name == "Skill": @@ -130,12 +116,9 @@ def _check_impl( error="turn_records not provided to checker", ) - # Any-engagement policy (mirrors ``live_verdict``): the row is scored on - # whether this skill was engaged AT ALL, regardless of order. A positive - # criterion (skill_name == expected_skill) passes iff the expected skill - # was engaged somewhere in the run — a wrong skill engaged first does not - # fail it (recall). A distractor/negative criterion fails on ANY - # engagement of its skill (precision). + # Any-engagement policy, mirroring ``live_verdict``: scored on whether this + # skill was engaged AT ALL, regardless of order. + # Rationale: .claude/notes/contracts.md § Any-engagement, and why order does not matter triggered: bool = criterion.skill_name in _all_engaged_skill_names(turn_records) expected_yes: bool = criterion.expected_skill == criterion.skill_name score = 1.0 if triggered == expected_yes else 0.0 @@ -159,25 +142,14 @@ def live_verdict( ) -> LiveVerdict: """Any-engagement latch: decide the instant THIS skill is engaged. - Mirrors ``_check_impl``'s any-engagement policy, latched monotonically - over the growing partial trajectory: - - - this skill not engaged yet -> ``"undecided"`` (the final outcome still - depends on the rest of the run — the expected skill may load later, or - a distractor may yet fire); - - this skill engaged -> ``expected_skill == skill_name`` decides: - ``"pass"`` for a positive criterion (the expected skill loaded), - ``"fail"`` for a distractor/negative one (a wrong skill loaded). - - Because engagement is monotonic (a skill, once engaged, stays engaged), a - latched verdict never flips, so it agrees with ``_check_impl`` on the - frozen trajectory by construction — whether or not the run stopped early. - A positive criterion can therefore only ever live-``pass`` and a - distractor/negative one only ever live-``fail``; their *absence* is never - decidable mid-run (see ``SkillTriggeredCriterion.live_decidable_polarities`` - in models/criteria.py). This is the change from first-engagement: a wrong - skill engaged first no longer live-fails a positive row — the run keeps - going so the expected skill can still load. + Monotonic and deterministic, as the ``LiveVerdict`` contract requires: a + decided verdict is a latch over a prefix that only grows. + + A positive criterion can therefore only ever live-``pass`` and a distractor + only ever live-``fail``; the ABSENCE of an engagement is never decidable + mid-run (see ``live_decidable_polarities``). + + Rationale: .claude/notes/contracts.md § Any-engagement, and why order does not matter """ if criterion.skill_name not in _all_engaged_skill_names(turn_records): return "undecided" diff --git a/src/coder_eval/criteria/uipath_eval.py b/src/coder_eval/criteria/uipath_eval.py index 0b564c886..6aecbb21e 100644 --- a/src/coder_eval/criteria/uipath_eval.py +++ b/src/coder_eval/criteria/uipath_eval.py @@ -11,10 +11,9 @@ from coder_eval.models import CriterionResult, UiPathEvalCriterion -# Detector for "the `uipath` CLI is not available in the sandbox". Sandboxes run -# on Linux (/bin/sh — dash on Ubuntu), so exit code 127 + "command not found" is -# the only shell signal we need to recognize. The regex also matches the python -# ModuleNotFoundError format for tasks that run `python -m uipath`. +# Detects "the `uipath` CLI is not available in the sandbox". Sandboxes run on +# Linux, so exit 127 + "command not found" is the only shell signal needed; the +# regex also matches ModuleNotFoundError for `python -m uipath`. _UIPATH_MISSING_PATTERN = re.compile( r"\buipath\b.*command not found|no module named ['\"]?uipath['\"]?(?:\s|$)", re.IGNORECASE, @@ -72,15 +71,12 @@ def _check_impl( if exit_code != 0: stderr_text = stderr or "" - # Exit code 127 is the POSIX "command not found" signal; the regex - # also catches `ModuleNotFoundError` for tasks that invoke - # `python -m uipath` instead of the CLI. + # 127 is the POSIX "command not found" signal. cli_missing = exit_code == 127 or bool(_UIPATH_MISSING_PATTERN.search(stderr_text)) hint = "" if cli_missing: - # The host's `[uipath]` extra installs `uipath` into the host - # venv; the sandbox runs in its own `uv` environment and must - # resolve `uipath` from the task's own deps. + # The host extra installs into the HOST venv; the sandbox runs in + # its own environment and resolves from the task's own deps. hint = ( " — the sandbox could not resolve the `uipath` CLI. " "Ensure the task's Python deps include `uipath` (the in-sandbox " diff --git a/src/coder_eval/evaluation/checker.py b/src/coder_eval/evaluation/checker.py index 8ec436a88..bb07690c8 100644 --- a/src/coder_eval/evaluation/checker.py +++ b/src/coder_eval/evaluation/checker.py @@ -87,21 +87,16 @@ def __init__( """ self.sandbox = sandbox self._checker_instances: dict[str, BaseCriterion[Any]] = {} - # Cached reference directory (the per-run staged copy of - # task.reference.directory). Set by check()/check_all() when provided and - # reused by subsequent calls that don't pass it explicitly. + # The per-run staged copy, set by check()/check_all() and reused by calls + # that do not pass it explicitly. self._reference_dir: Path | None = None # Cached turn records - set by check()/check_all() when provided self._turn_records: TurnRecords | None = None self.route = route - # Cumulative wall ms spent grading, across every `check_all_async` call - # this checker serves. Accumulated HERE rather than at the four - # orchestrator call sites (single-shot, evaluate-only, the per-dialog-turn - # check, the post-failure diagnostics) so a fifth call site cannot be - # added without it — the same reason the tool subtraction lives at the - # one collector seam. `None` until something is actually checked, so an - # ungraded row reports "never measured" rather than an instant 0.0 - # (CE058). + # Accumulated HERE, not at the four orchestrator call sites, so a fifth + # cannot be added without it. `None` until something is checked, so an + # ungraded row reports "never measured" rather than 0.0 (CE058). + # Rationale: .claude/notes/contracts.md § What escalates instead of scoring 0.0 self.grading_ms: float | None = None # V3: Lazy initialization - registry loaded here, not at import @@ -181,43 +176,26 @@ async def check_all_async( ) -> CriteriaResults: """Async twin of ``check_all`` — the orchestrator's entry point. - Runs every criterion SEQUENTIALLY, strictly in declaration order — - the same order/isolation guarantee ``check_all`` provides. A checker - is "native async" when it overrides ``_check_impl_async`` itself (see - ``BaseCriterion``) — that's the criteria making genuine async I/O - (``llm_judge``, ``agent_judge``); those are awaited directly on the - event loop instead of pinning a thread-pool thread for the network - wait. Everything else (CPU/file-bound criteria that only override the - sync ``_check_impl``) is offloaded to a worker thread via - ``asyncio.to_thread`` so it doesn't block the loop either. Neither of - those is about concurrency between criteria — each criterion is fully - awaited before the next one starts, exactly like ``check_all``. - - Concurrent dispatch of adjacent judge criteria (the actual GH #55 - motivation — firing multiple judges' LLM calls at once instead of - serializing them) is DELIBERATELY NOT done here; that scheduling - change is scoped to a follow-up PR so it can be reviewed (and its - sandbox-mutation-ordering implications tested) on its own. This - method exists as the async entry point the orchestrator now calls - unconditionally, with sequential semantics identical to ``check_all`` - in the meantime. + Runs every criterion SEQUENTIALLY, strictly in declaration order, with the + same isolation guarantee ``check_all`` provides. A natively-async checker is + awaited directly on the event loop; every other one is offloaded to a worker + thread. Neither is concurrency BETWEEN criteria — each is fully awaited + before the next starts. Args: criteria: List of criterion definitions. turn_records: Optional turn records for command inspection. reference_dir: Optional resolved path to a reference directory. - Consumed by ``reference_comparison`` (which scores 0.0 without - it), ``llm_judge`` and ``agent_judge``; other criteria accept the - uniform signature and ignore it. + Consumed by ``reference_comparison`` (which scores 0.0 without it), + ``llm_judge`` and ``agent_judge``; the rest ignore it. Returns: List of criterion results with scores, in the same order as ``criteria``. """ records, ref_dir = self._resolve_refs(turn_records, reference_dir) - # Monotonic, like every other duration in this codebase: a wall-clock - # delta would move if the clock stepped mid-grade, and an `agent_judge` - # criterion can run for minutes. + # Monotonic: a wall-clock delta would move if the clock stepped mid-grade, + # and an `agent_judge` criterion can run for minutes. started = time.monotonic() results: list[CriterionResult] = [] try: @@ -227,9 +205,7 @@ async def check_all_async( else: results.append(await asyncio.to_thread(self._check_single, criterion, records, ref_dir)) finally: - # In `finally` so a grade that raises still books the time it spent. - # Its cost is what the caller is trying to account for, and a crash - # does not un-spend it. + # In `finally` so a grade that raises still books what it spent. self.grading_ms = (self.grading_ms or 0.0) + (time.monotonic() - started) * 1000.0 return results @@ -401,12 +377,8 @@ async def _check_single_async( ) return self._finalize_result(criterion, result) except KeyError: - # Deliberate dead-code defence: an unregistered type resolves to - # `_is_native_async(...) is False` (see that method), so - # `_check_single_async` is only ever invoked for already-registered - # types — this arm exists only to keep the two `_check_single*` - # methods' exception shape identical, in case that invariant ever - # changes. + # Deliberate dead-code defence: unreachable today, kept so the two + # `_check_single*` methods' exception shape stays identical. return self._missing_checker_result(criterion) except _ESCALATING_EXCEPTIONS: raise diff --git a/src/coder_eval/evaluation/judge_anthropic.py b/src/coder_eval/evaluation/judge_anthropic.py index 4bddcc0fa..7e1f99837 100644 --- a/src/coder_eval/evaluation/judge_anthropic.py +++ b/src/coder_eval/evaluation/judge_anthropic.py @@ -67,8 +67,7 @@ async def invoke_anthropic_judge_async( # by default) — do not add another retry loop here. raise JudgeInfrastructureError(f"Anthropic judge API error: {e}") from e except Exception as e: - # A signature/contract break (e.g. a removed or renamed kwarg after an - # SDK bump) must not be scored as an agent failure — see CLAUDE.md's - # CE039 rationale: an eval-infra fault is not the agent's fault. + # A signature break after an SDK bump is an eval-infra fault, not the + # agent's (CE039). raise JudgeInfrastructureError(f"Anthropic judge call failed: {e}") from e return response.model_dump() diff --git a/src/coder_eval/evaluation/judge_bedrock.py b/src/coder_eval/evaluation/judge_bedrock.py index e9f5c474f..974ddbc92 100644 --- a/src/coder_eval/evaluation/judge_bedrock.py +++ b/src/coder_eval/evaluation/judge_bedrock.py @@ -71,12 +71,9 @@ async def invoke_bedrock_judge_async( JudgeInfrastructureError: no bearer token configured; retries exhausted; non-retryable HTTP failure (e.g. 400/401/403); or a non-dict JSON body. """ - # Raise (not assert): this call runs inside LLMJudgeChecker's - # handle_criterion_errors(_async) wrapper, which catches plain Exception - # (including AssertionError) and downgrades it to a scored 0.0 — the - # opposite of the intended "internal-contract violation escalates to - # FinalStatus.ERROR" behavior. JudgeInfrastructureError is in - # _ESCALATING_EXCEPTIONS, so it propagates instead of being scored. + # Raise, not assert: the wrapper around this call catches plain Exception + # (AssertionError included) and downgrades it to a scored 0.0. + # Rationale: .claude/notes/contracts.md § What escalates instead of scoring 0.0 if settings.aws_bearer_token_bedrock is None: raise JudgeInfrastructureError("Bedrock requires aws_bearer_token_bedrock") qualified = to_bedrock_model(model, route.region) diff --git a/src/coder_eval/evaluation/judge_context.py b/src/coder_eval/evaluation/judge_context.py index aa0dcc7bc..4787a1c56 100644 --- a/src/coder_eval/evaluation/judge_context.py +++ b/src/coder_eval/evaluation/judge_context.py @@ -28,17 +28,11 @@ ) -# Paths in `llm_judge.files` / `agent_judge.files` that begin with one of these -# tokens are resolved against a host directory and read from the host filesystem -# instead of the sandbox: `$TASK_DIR` against the task YAML's parent directory, -# `$REFERENCE_DIR` against the per-run staged copy of `task.reference.directory`. -# Both mirror the same-named env vars `run_command` exposes, so judges and shell -# criteria address the same places by the same name. -# -# `$REFERENCE_DIR` is how a task attaches *specific* grading assets from the -# reference (`$REFERENCE_DIR/rubric.md`) instead of the whole tree, and it is -# readable here because judges run outside the agent's turn — the directory sits -# at mode 000 for the whole of `agent.communicate`. +# Paths beginning with one of these tokens resolve against a HOST directory rather +# than the sandbox, mirroring the same-named env vars `run_command` exposes. +# `$REFERENCE_DIR` is readable here only because judges run outside the agent's +# turn -- the directory sits at mode 000 for all of `agent.communicate`. +# Rationale: .claude/notes/contracts.md § Judge context and untrusted text if TYPE_CHECKING: @@ -100,22 +94,14 @@ def scrub_reference(content: str, secrets: list[str] | None) -> str: """Redact any occurrence of each secret in ``content``. Takes the per-file contents of a reference directory, or ``None`` (no-op). - Deliberately ``list[str]``, not the old ``str | Iterable[str]`` and not - ``Sequence[str]``: ``str`` satisfies both of those, so a caller passing a - bare string type-checked clean and then had its characters iterated as - individual "secrets" (each under the 8-char floor, so silently redacting - nothing). ``list[str]`` is the one spelling that makes that a type error. - - No-op for ``None`` or empty inputs — guards against the - ``"".replace("", "")`` pathology that ballooned strings. - Secrets shorter than 8 characters are skipped: redacting a tiny common - substring (e.g. ``" = 1"``) would produce gibberish output and isn't a - realistic leak vector — references at that scale carry no proprietary - information. Directory-mode caveat: every file in the reference directory - becomes its own secret entry, so a reference solution that includes very - short files (a one-liner ``__init__.py``, a tiny config blob) will leave - those files unscrubbed — mention this explicitly because the per-file - threshold is invisible at the call site. + Deliberately ``list[str]``: ``str`` satisfies ``str | Iterable[str]`` and + ``Sequence[str]`` alike, so a caller passing a bare string type-checked clean + and had its CHARACTERS iterated as individual secrets. + + Secrets shorter than 8 characters are skipped, so a reference that includes + very short files leaves those files unscrubbed. + + Rationale: .claude/notes/contracts.md § Scrub before truncate """ if secrets is None: return content @@ -129,12 +115,8 @@ def scrub_reference(content: str, secrets: list[str] | None) -> str: return out -# Budget for ``collect_reference_secrets``. Reference directories are expected to -# be small project skeletons (a handful of source files); a runaway tree (vendored -# node_modules, generated XML, embedded assets) must not pull the host into an OOM -# or hang the walk. When a budget triggers we log + stop reading more files — -# remaining files are left unscrubbed, which is no worse than the pre-budget world -# would have been if the user had pointed at a code-form reference instead. +# A runaway tree must not pull the host into an OOM or hang the walk. When the +# budget trips we log and stop; remaining files are left unscrubbed. _MAX_REFERENCE_FILES = 200 _MAX_REFERENCE_BYTES = 2 * 1024 * 1024 # 2 MB total content cap @@ -142,28 +124,15 @@ def scrub_reference(content: str, secrets: list[str] | None) -> str: def iter_reference_files(reference_dir: Path) -> Iterator[tuple[Path, str]]: """Yield ``(path, text)`` for every readable text file under ``reference_dir``. - The single walk behind both reference consumers: ``collect_reference_secrets`` - (scrub keys) and ``render_reference_dir`` (judge prompt content). Sharing it - means the budget and symlink rules can't diverge between "what we show the - judge" and "what we redact from the judge's output" — a divergence there - would leak reference content into a persisted transcript. - - Binary and unreadable files are skipped silently — they're not realistic - leak vectors and reading them would raise UnicodeDecodeError. - - Symlinks are NOT followed: a reference bundle that ships - ``secrets -> /etc/passwd`` would otherwise read the host file into the - scrub-key list (and quietly grow it), and a symlinked subdir back to the - root would loop ``rglob`` forever. + The single walk behind both reference consumers, so the budget and symlink + rules cannot diverge between what the judge is shown and what is redacted from + its output. - File count and total content are capped (``_MAX_REFERENCE_FILES`` / - ``_MAX_REFERENCE_BYTES``) — when either trips we log + stop. The cap is - sized well above any realistic reference skeleton, so this only fires for - misconfigured trees. + Symlinks are NOT followed. Binary and unreadable files are skipped silently. + File count and total content are capped; when either trips we log and stop. + Yields nothing when the directory is missing, empty, or sitting at mode 000. - Yields nothing when the directory is missing or empty. A reference - directory sitting at mode 000 (i.e. this was called during an agent turn, - which should never happen) also yields nothing rather than raising. + Rationale: .claude/notes/contracts.md § The reference walk's budget and symlink rules """ if not reference_dir.is_dir(): return @@ -191,12 +160,9 @@ def iter_reference_files(reference_dir: Path) -> Iterator[tuple[Path, str]]: continue if not path.is_file(): continue - # Size pre-check BEFORE read_text: a single 100 MB file in an otherwise - # small reference dir would otherwise be pulled into memory in full before - # the per-iteration budget check fires. Using stat() bytes (rather than - # char count post-read) also keeps the accounting unit consistent with the - # ``_MAX_REFERENCE_BYTES`` constant name. OSError on stat (race with delete, - # permission edge cases) → skip silently. + # BEFORE read_text: one huge file would otherwise be pulled into memory in + # full before the per-iteration budget check fires. stat() bytes also keep + # the unit consistent with the constant's name. try: file_size = path.stat().st_size except OSError: @@ -249,9 +215,8 @@ def collect_reference_secrets(reference_dir: Path, max_file_chars: int | None) - return keys -# Total budget for the rendered reference block. `max_file_chars` bounds each -# file, but a tree of many small files could still blow the judge's context — -# which surfaces as a failed judge call scored 0.0, not as graceful degradation. +# `max_file_chars` bounds each file, but many small files could still blow the +# judge's context -- which surfaces as a failed call scored 0.0. _MAX_RENDERED_REFERENCE_CHARS = 200_000 @@ -275,9 +240,8 @@ def render_reference_dir(reference_dir: Path, max_file_chars: int) -> str | None except ValueError: # pragma: no cover - rglob results are always relative label = path.name block = f"--- {label} ---\n{truncate(text, max_file_chars)}" - # Drop trailing files rather than truncating mid-block, mirroring how the - # trajectory degrades, and tell the judge explicitly so it doesn't read - # the omission as "the reference doesn't implement that". + # Drop TRAILING files rather than truncating mid-block, and say so, or the + # judge reads the omission as "the reference doesn't implement that". if used + len(block) > _MAX_RENDERED_REFERENCE_CHARS and blocks: dropped += 1 continue @@ -315,13 +279,10 @@ class JudgeContext: files: list[FileBlock] = field(default_factory=list) reference: str | None = None - # Every piece of reference-derived text that reached the prompt, in the exact - # shape the judge saw it (post-truncation). The scrub gate keys on THIS being - # non-empty, not on ``include_reference``: a `$REFERENCE_DIR/...` entry in - # ``files:`` attaches reference bytes with include_reference=false, which is - # the documented way to show a judge one rubric without inlining the tree. - # Keying on the flag left that combination persisting the solution verbatim - # into the archived judge transcript. + # HAZARD: every reference-derived byte that reached the prompt, in the exact + # shape the judge saw. The scrub gate keys on THIS being non-empty, never on + # ``include_reference``. + # Rationale: .claude/notes/contracts.md § What counts as reference-derived reference_secrets: list[str] = field(default_factory=list) agent_output: str | None = None tool_calls_summary: str | None = None @@ -439,10 +400,8 @@ def _collect_reference(self, reference_dir: Path | None, ctx: JudgeContext) -> N rendered = render_reference_dir(reference_dir, self.max_file_chars) if rendered: ctx.reference = rendered - # max_file_chars is mandatory here: render_reference_dir truncated - # each file, and scrub_reference redacts by exact substring, so a - # key built only from the untruncated text would never match what - # the judge was actually shown. + # Mandatory: the renderer truncated each file, and a key built only + # from untruncated text would never match what the judge was shown. ctx.reference_secrets.extend(collect_reference_secrets(reference_dir, self.max_file_chars)) return # Silent omission matches legacy behavior — some tasks deliberately run without a reference. @@ -473,12 +432,9 @@ def _collect_trajectory(self, turn_records: list[TurnRecord] | None, ctx: JudgeC if not turn_records: ctx.degraded_notes.append("include_dialog requested but no turn records available") else: - # Aggregate budget cap (max_dialog_chars) prevents an N-turn simulation from - # blowing out the judge's context window. Per-message cap (max_file_chars) is - # applied first so a single huge message can't crowd out later turns. When the - # aggregate budget is exhausted we drop *trailing* turns and record a note — - # TODO: a smarter strategy (keep first+last K, or middle-ellipsis) better matches - # what graders want, but the naïve cap is enough for the common case. + # The per-message cap is applied FIRST so one huge message cannot + # crowd out later turns; the aggregate cap then drops trailing turns + # and records a note. TODO: keep first+last K instead. total = 0 dropped = 0 for turn in turn_records: @@ -508,10 +464,8 @@ def format_details(score: float, rationale: str, missing_files: list[str], degra return "\n".join(lines) -# Per-tool detail/result_preview cap used when capturing an agent_judge transcript. -# Generous enough to preserve audit value (a typical Bash command, a grep pattern, -# a tool result blurb) but small enough that 100+ tool calls fit under the default -# max_transcript_chars=100_000 cap before truncation kicks in. +# Generous enough to preserve audit value, small enough that 100+ tool calls fit +# under the default max_transcript_chars before truncation. _TRANSCRIPT_DETAIL_CAP = 200 _TRANSCRIPT_RESULT_CAP = 200 @@ -563,10 +517,8 @@ def build_judge_transcript( cmds = commands or [] tool_calls = [_summarize_command_for_transcript(c) for c in cmds] - # Budget pass: keep tool calls until we exhaust the cap, then start clipping. - # The only way ``len(kept) < len(tool_calls)`` is to break out of the loop, - # and that branch already sets ``truncated = True`` — no post-loop redundant - # assignment needed. + # The only way ``len(kept) < len(tool_calls)`` is to break out of the loop, and + # that branch already sets ``truncated``. used = 0 kept: list[JudgeTranscriptToolCall] = [] truncated = False @@ -578,14 +530,9 @@ def build_judge_transcript( kept.append(tc) used += cost - # SECURITY: scrub BEFORE clipping. ``scrub_reference`` uses ``str.replace``, - # which only matches the secret as a contiguous whole string. If we clipped - # first, a multi-KB reference cut by the per-field budget would leave a - # partial fragment in the field that no longer matches the full secret — - # ``replace`` finds nothing, the prefix gets persisted unsanitized. Scrubbing - # first guarantees the secret is replaced with the short ```` - # marker before any clipping, so on-disk fields can never carry partial - # reference content. + # SECURITY: scrub BEFORE clipping. Clipping first leaves a partial fragment that + # no longer matches the full secret, so `str.replace` finds nothing. + # Rationale: .claude/notes/contracts.md § Scrub before truncate if scrub_key: raw_verdict = scrub_reference(raw_verdict, scrub_key) judge_prompt = scrub_reference(judge_prompt, scrub_key) @@ -600,9 +547,8 @@ def build_judge_transcript( for tc in kept ] - # Distribute the remaining budget across raw_verdict / judge_prompt / system_prompt. - # A naive even split would clip the verdict (the most important field) for tasks - # with long rubrics; weight verdict at 60%, user prompt at 30%, system at 10%. + # Weighted, not an even split: an even one clipped the verdict -- the most + # important field -- for tasks with long rubrics. remaining = max(0, max_chars - used) verdict_budget = int(remaining * 0.6) prompt_budget = int(remaining * 0.3) diff --git a/src/coder_eval/evaluation/judge_litellm.py b/src/coder_eval/evaluation/judge_litellm.py index 705c1e04f..8519f1363 100644 --- a/src/coder_eval/evaluation/judge_litellm.py +++ b/src/coder_eval/evaluation/judge_litellm.py @@ -1,30 +1,15 @@ """Single-completion invoker for the LiteLLM judge backend, via the ``litellm`` library (the ``coder-eval[litellm]`` extra) rather than a hand-rolled HTTP call. -Unlike the AGENT's own LiteLLM backend (which points the Claude Code SDK at -``settings.litellm_base_url``/``settings.litellm_auth_token``), this module -reads NOTHING from ``coder_eval.config.settings`` — the task author fully owns -the call shape via ``LiteLLMRoute.params``/``LiteLLMRoute.env_params`` (see -that class's docstring). A gateway-routed judge model rarely reuses the same -proxy/credential the agent's own LiteLLM backend points at, so there is no -implicit fallback here; if the provider needs ``api_base``/``api_key``/ -whatever else, the task author names it via ``params``/``env_params`` like any -other kwarg. - -Calling through ``litellm.acompletion`` — rather than assuming one specific -wire protocol — lets ``model`` carry its own provider hint (e.g. -``azure_ai/gpt-5.6-luna``) and get that provider's actual request/response -shape handled by the library, including per-provider quirks (``max_tokens`` -vs ``max_completion_tokens`` naming, unsupported-parameter drops via -``drop_params``) instead of this module hand-coding them. +Reads NOTHING from ``coder_eval.config.settings``: the task author fully owns the +call shape via ``LiteLLMRoute.params``/``env_params``, with no implicit fallback to +the AGENT's own LiteLLM proxy settings. ``litellm.acompletion`` always returns an OpenAI-shaped ``ModelResponse`` -regardless of the underlying provider, so the caller reuses -``extract_verdict_from_openai_response``/``token_usage_from_openai_dict`` -unchanged. +regardless of provider, so the caller reuses the OpenAI extractors unchanged. +Async on purpose, mirroring the other two judge invokers. -Async on purpose: mirrors ``invoke_anthropic_judge_async`` / -``invoke_bedrock_judge_async`` — the judge's only network call, no sync twin. +Rationale: .claude/notes/contracts.md § LiteLLM params and env_params """ from __future__ import annotations @@ -123,10 +108,8 @@ def _resolve_env_params() -> dict[str, str]: "tool_choice": {"type": "function", "function": {"name": tool_spec["name"]}}, "max_completion_tokens": max_tokens, "timeout": timeout_seconds, - # `drop_params` covers params litellm's own static model-cost map KNOWS a - # model rejects; a custom/gateway-routed model id (e.g. one behind an - # Azure AI deployment) usually isn't in that map, so this alone doesn't - # protect a `params`-supplied kwarg the target model live-rejects. + # HAZARD: `drop_params` only covers params litellm's static cost map knows a + # model rejects; a gateway-routed model id usually is not in that map. "drop_params": True, } # `params` (literal passthrough) applies first; `env_params` (resolved from diff --git a/src/coder_eval/evaluation/judge_persistence.py b/src/coder_eval/evaluation/judge_persistence.py index aad1fbf7c..dcd8fbfe6 100644 --- a/src/coder_eval/evaluation/judge_persistence.py +++ b/src/coder_eval/evaluation/judge_persistence.py @@ -1,38 +1,18 @@ """Sibling-file persistence for ``JudgeCriterionResult.transcript``. -The full judge transcript (tool calls, raw verdict, rendered prompt and -system prompt) can run 10-100 KB. Inlining it into every ``task.json`` -inflates the row record for consumers (suite rollups, report renderers) -that don't need it. Spilling each transcript to a sibling YAML file next to -``task.json`` keeps the row record lean and lets reviewers grep transcripts -independently. - -YAML (over JSON) for the sibling: the transcript carries multi-line text -(``judge_prompt``, ``judge_system_prompt``, ``raw_verdict``) which YAML's -literal block scalar (``|``) renders as readable paragraphs instead of -single-line strings with ``\\n`` escapes. Sibling consumers are humans; -YAML wins. - -Two functions: - -- ``spill_judge_transcripts``: called by the orchestrator just before it - writes ``task.json``. For each judge result with an inline ``transcript``, - writes a sibling file and sets ``transcript_path`` on the result. The - inline ``transcript`` is left in place so HTML rendering against the - in-memory ``EvaluationResult`` still sees it; ``model_dump_json`` - callers strip it via ``exclude={...}``. - -- ``load_judge_transcripts``: called by re-render paths - (``coder-eval report``, stats loaders) after - ``EvaluationResult.model_validate_json``. For each result whose - ``transcript_path`` points at an existing sibling file, reads it back - and attaches it as a dict on ``transcript`` so renderers see the same - shape they get during the original run. Accepts both ``.yaml`` (the - current format) and ``.json`` (the previous format) so previously-spilled - runs keep rendering. - -Backward compatibility: old ``task.json`` files with inline ``transcript`` -keep working — the loader treats ``transcript_path is None`` as a no-op. +A judge transcript runs 10-100 KB, so it spills to a sibling YAML file next to +``task.json`` and the row keeps only a ``transcript_path``. The inline value is +left in place so in-memory HTML rendering still sees it; ``model_dump_json`` +callers strip it via ``exclude={...}``. + +- ``spill_judge_transcripts``: called by the orchestrator just before it writes + ``task.json``. +- ``load_judge_transcripts``: called by re-render paths after + ``EvaluationResult.model_validate_json``. Accepts both ``.yaml`` (current) and + ``.json`` (previous), and treats ``transcript_path is None`` as a no-op, so old + inline records keep working. + +Rationale: .claude/notes/persistence.md § Judge persistence """ from __future__ import annotations @@ -63,20 +43,15 @@ } -# Windows reserved device basenames. The Win32 API maps these to character -# devices regardless of the directory they sit in — opening ``CON`` or ``NUL.yaml`` -# inside ``task_dir`` resolves to the console or the null device, not a file. -# Case-insensitive; the trailing extension (if any) is ignored by Win32 too, -# so ``con``, ``CON``, ``CON.yaml``, ``nul.txt`` all map to devices. -# Only COM1-9 / LPT1-9 are device names; COM10 and beyond are regular files. +# Win32 maps these to character devices wherever they sit, extension ignored, so +# ``con``, ``CON.yaml`` and ``nul.txt`` all open a device. Only COM1-9 / LPT1-9 are +# device names. Rationale: .claude/notes/persistence.md § transcript_path is untrusted input _WINDOWS_RESERVED_BASENAMES = frozenset( {"CON", "PRN", "AUX", "NUL"} | {f"COM{i}" for i in range(1, 10)} | {f"LPT{i}" for i in range(1, 10)} ) -# Field order in the YAML output: lead with the human-readable summary fields -# (durations, token counts, prompts), then the long body fields. Verdict-shape -# (raw_verdict) goes last because it's the bulkiest. +# Human-readable summary fields first, bulkiest (raw_verdict) last. _TRANSCRIPT_FIELD_ORDER = ( "duration_seconds", "truncated", @@ -141,9 +116,8 @@ def spill_judge_transcripts(result: EvaluationResult, output_dir: Path) -> int: """ output_dir.mkdir(parents=True, exist_ok=True) spilled = 0 - # ORDER IS LOAD-BEARING. Each filename is keyed off the criterion's - # position in its result list; ``load_judge_transcripts`` reads the stored - # path, so each list must retain its order through persistence. + # ORDER IS LOAD-BEARING: each filename is keyed off the criterion's position in + # its result list, so each list must retain its order through persistence. result_groups = ( ("judge", result.success_criteria_results), ("post-failure-judge", result.post_failure_criteria_results), @@ -196,39 +170,27 @@ def load_judge_transcripts(result: EvaluationResult, task_dir: Path) -> int: path = getattr(cr, "transcript_path", None) if not path: continue - # Skip results that already have an inline transcript — typical for - # the orchestrator's own first HTML render, which runs against the - # in-memory result before it gets dumped + reloaded. + # Already inline -- typical for the orchestrator's own first HTML render, + # which runs against the in-memory result. if getattr(cr, "transcript", None): continue - # SECURITY: transcript_path comes from task.json, which may travel across - # trust boundaries (CI artifacts, shared eval bundles). spill_judge_transcripts - # only ever writes generated basename-only paths, with no separators or - # ``..``. Allowlist that shape directly so a tampered - # ``transcript_path: '/etc/passwd'`` or ``../../secrets`` is refused at the - # door rather than relying on ``is_relative_to`` to catch it after a join. - # Check BOTH PurePosixPath (forward-slash separator) AND PureWindowsPath - # (forward and back-slash separators, drive-letter prefixes): a path like - # ``subdir\judge-0.yaml`` passes the POSIX check on Linux (backslash is a - # regular char) but resolves to a nested file on Windows. Rejecting under - # either interpretation enforces the basename-only policy regardless of - # which platform the task.json travels to next. + # SECURITY: transcript_path comes from task.json, which travels across trust + # boundaries. The writer only ever emits a generated basename, so ALLOWLIST + # that shape -- under BOTH POSIX and Windows semantics, since + # ``subdir\judge-0.yaml`` passes a POSIX check and nests on Windows. + # Rationale: .claude/notes/persistence.md § transcript_path is untrusted input if path in {".", ".."} or PurePosixPath(path).name != path or PureWindowsPath(path).name != path: logger.warning("Refusing to load judge transcript with non-basename path: %s", path) continue - # Reject Windows reserved device basenames. On Windows, ``CON.yaml`` / - # ``NUL`` / ``COM1`` open the console / null device / serial port - # regardless of where they sit in the directory tree. The check is - # platform-independent so a task.json minted on Linux that ships such a - # transcript_path is rejected before it travels to Windows. + # Platform-INDEPENDENT, so a task.json minted on Linux carrying such a path + # is rejected before it travels to Windows. stem_upper = path.split(".", 1)[0].upper() if stem_upper in _WINDOWS_RESERVED_BASENAMES: logger.warning("Refusing to load judge transcript with reserved Windows device name: %s", path) continue sibling = task_dir / path - # Defense-in-depth: even with the basename guard above, resolve and verify - # containment before reading — symlinks inside ``task_dir`` could redirect - # outside it (a malicious bundle could ship one). + # Defense-in-depth: a symlink inside ``task_dir`` could still redirect + # outside it, so verify containment after resolving. try: resolved_sibling = sibling.resolve() resolved_root = task_dir.resolve() @@ -249,42 +211,31 @@ def load_judge_transcripts(result: EvaluationResult, task_dir: Path) -> int: sibling = resolved_sibling try: text = sibling.read_text(encoding="utf-8") - # Parse as JSON for legacy ``.json`` siblings; anything else (today's - # ``.yaml`` and any future format yaml.safe_load handles) goes through - # PyYAML, which also accepts JSON as a subset. + # JSON for legacy siblings; everything else through PyYAML, which + # accepts JSON as a subset anyway. data = json.loads(text) if path.endswith(".json") else yaml.safe_load(text) except Exception as e: logger.warning("Failed to read judge transcript %s: %s", sibling, e) continue if not isinstance(data, dict): - # A scalar / list / None payload would silently land on the result and - # crash the HTML renderer (which assumes dict-or-typed) with an - # AttributeError on the first ``.get()``. Reject early. + # A scalar / list / None payload would land on the result and crash the + # renderer on its first ``.get()``. logger.warning( "Judge transcript %s is %s, expected mapping — skipping", sibling, type(data).__name__, ) continue - # Prefer typed JudgeTranscript so renderer / aggregator code that does - # isinstance checks sees the same shape it gets during the original run. - # Fall back to the raw dict on ValidationError — older spilled siblings - # (pre-schema-change) or forward-compat keys shouldn't break re-render. + # Typed, so isinstance checks see the same shape as during the original + # run; the raw-dict fallback keeps an older sibling rendering. attached: JudgeTranscript | dict[str, Any] try: attached = JudgeTranscript.model_validate(data) except ValidationError as e: logger.debug("Judge transcript %s did not match JudgeTranscript schema, attaching as dict: %s", sibling, e) attached = data - # Use object.__setattr__ so we don't go through pydantic's setter, - # which (depending on model_config of the loaded subclass) might - # validate or reject. The HTML renderer accepts both typed - # JudgeTranscript and dict-shape so either shape works downstream. - # NOTE: With the ``CriterionResultUnion`` discriminator on both - # ``EvaluationResult`` criterion-result lists, ``cr`` is now a - # properly-typed ``JudgeCriterionResult`` after reload (not a base - # ``CriterionResult`` with the field in ``__pydantic_extra__``), so - # the assignment lands on the declared field directly. + # Bypasses pydantic's setter, which a loaded subclass's config might + # validate or reject. The renderer accepts both shapes. try: object.__setattr__(cr, "transcript", attached) except Exception as e: diff --git a/src/coder_eval/evaluation/sub_agent.py b/src/coder_eval/evaluation/sub_agent.py index 89021c551..536370c71 100644 --- a/src/coder_eval/evaluation/sub_agent.py +++ b/src/coder_eval/evaluation/sub_agent.py @@ -55,23 +55,19 @@ def __init__( extra_mcp_servers: dict[str, Any] | None = None, capture: VerdictCapture | None = None, ) -> None: - # SECURITY: setting_sources=[] is enforced by the caller — it's the caller's - # responsibility to build the AgentConfig correctly because the field is part of - # the type contract. We raise rather than mutate so a misconfigured caller fails - # loudly instead of silently having its config changed. Not `assert` because the - # check must survive `python -O`. + # SECURITY: raise rather than mutate, so a misconfigured caller fails loudly + # instead of silently having its config changed. Not `assert`: the check + # must survive `python -O`. + # Rationale: .claude/notes/contracts.md § The security floor if agent_config.setting_sources != []: raise ValueError( "SubAgentRunner requires agent_config.setting_sources=[] so the SDK does not " + "load .claude/settings.json or .mcp.json from the sub-agent's working directory." ) - # A sub-agent's system_prompt is its entire identity (judge instructions, - # simulator persona) — the claude_code coding-agent preset must never - # prefix it. That takes BOTH halves: an omitted prompt gets the bare - # preset, which is the same failure reached via the other branch, so - # neither is accepted. Same fail-loud contract as setting_sources above: - # callers own their config, so a misconfigured one raises instead of - # being silently mutated. + # A sub-agent's system_prompt is its ENTIRE identity, so the coding-agent + # preset must never prefix it -- and an omitted prompt gets the bare preset, + # which is the same failure, so neither is accepted. + # Rationale: .claude/notes/contracts.md § The judge's identity is its system prompt if agent_config.system_prompt is None or agent_config.system_prompt_mode != "replace": raise ValueError( "SubAgentRunner requires agent_config.system_prompt to be set with " @@ -83,55 +79,34 @@ def __init__( self._agent_config = agent_config self._ignore_patterns = ignore_patterns self._route = route - # When set, copied into ``judge_dir / "_reference"`` after the main sandbox - # copy. The judge can browse it via Read/Glob. ``None`` skips the copy — - # callers MUST set this to None when the criterion has ``include_reference=False`` - # so the judge can't see grading material it was opted out of. + # HAZARD: callers MUST set this to None when the criterion has + # ``include_reference=False``, or the judge sees grading material it was + # opted out of. self._reference_dir = reference_dir - # Separate ignore set for the reference-side copytree. Defaults to ``[]`` - # so we DON'T silently strip a nested ``_reference/`` inside the user's - # reference dir — the sandbox-side ignore set legitimately contains - # ``_reference`` (defense-in-depth against agent-planted collisions at - # the mount point), but reusing it here would silently drop a customer - # subdir of the same name. Symlinks are stripped unconditionally by - # ``ignore_patterns_and_symlinks([])``. + # SEPARATE from the sandbox-side set, defaulting to ``[]``: that one + # contains ``_reference`` as defense-in-depth, and reusing it here would + # drop a user's own nested subdir of the same name. + # Rationale: .claude/notes/contracts.md § The security floor self._reference_ignore_patterns = reference_ignore_patterns or [] - # Runtime-only in-process MCP server injection (e.g. the judge - # submit_verdict tool). NOT routed through ``sdk_options`` — - # ``mcp_servers`` is in ``_FRAMEWORK_OWNED_SDK_FIELDS``. + # Runtime-only in-process MCP injection. NOT routed through ``sdk_options`` + # -- ``mcp_servers`` is framework-owned. self._extra_mcp_servers = extra_mcp_servers or {} - # Public attribute so the criterion can read it after ``run_async()`` returns. - # When the caller passes ``capture=None`` the runner doesn't expose one, - # matching the opt-in-per-construction contract for the verdict channel. + # Public so the criterion can read it after ``run_async()`` returns; absent + # when the caller passed ``capture=None``. self.capture = capture async def run_async(self, user_msg: str, *, max_turns: int | None, turn_timeout: float) -> TurnRecord: """Copy sandbox → start agent → communicate → stop. Kill on any exception. - Async so a genuine network/subprocess wait yields the event loop instead - of pinning a thread-pool thread — lets ``SuccessChecker.check_all_async`` - await this directly without pinning a thread. (``check_all_async`` - currently runs criteria sequentially; running this concurrently with - other judge-type criteria is deferred to a follow-up PR.) The blocking - filesystem work (``copytree``/``rmtree``) is pushed to a worker thread - via ``asyncio.to_thread`` so it doesn't block the loop either. - - Cancellation safety: ``run_async`` is awaited directly on the - orchestrator's own loop (not under its own ``asyncio.run`` on a worker - thread), so it is reachable by cancellation (e.g. the ``task_timeout`` - watchdog cancelling the orchestrator task) at any ``await`` — including - mid-``copytree``. ``asyncio.to_thread`` is NOT itself cancellable (the - worker thread keeps running after the awaiting coroutine raises - ``CancelledError``), so every such call here is wrapped in - ``asyncio.shield`` and tracked in ``self._pending`` — the ``finally`` - below awaits any still-in-flight one BEFORE ``rmtree``, so an orphan - thread can never recreate files in ``judge_dir`` after cleanup already - ran. ``judge_dir`` itself is bound with a plain synchronous - ``tempfile.mkdtemp`` (a single fast syscall — offloading it via - ``to_thread`` only widens the cancellation window for no benefit, since - a bare syscall isn't itself an await point cancellation can land on). + Async so a genuine network/subprocess wait yields the event loop instead of + pinning a thread-pool thread; the blocking filesystem work is pushed to a + worker thread. Every such call is SHIELDED and awaited in the ``finally`` + before cleanup, so an orphan thread cannot recreate files in ``judge_dir`` + after cleanup already ran. Raises ``TurnTimeoutError`` when the agent exceeds ``turn_timeout``. + + Rationale: .claude/notes/contracts.md § Cancellation safety """ # Narrow via local var — checked in __init__ but pyright doesn't track that. src_dir = self._sandbox.sandbox_dir @@ -140,10 +115,8 @@ async def run_async(self, user_msg: str, *, max_turns: int | None, turn_timeout: judge_dir = Path(tempfile.mkdtemp(prefix="sub_agent_")) pending: list[asyncio.Task[Any]] = [] try: - # Copy the sandbox into an isolated temp dir. The sub-agent never touches - # the original sandbox, so later criteria are unaffected by whatever it - # does. Symlinks are skipped (vs preserved) so a malicious - # `creds -> /root/.aws/credentials` plant can't leak host files to a + # HAZARD: symlinks are SKIPPED, not preserved, so a planted + # `creds -> /root/.aws/credentials` cannot leak host files to a # Bash-enabled sub-agent. await self._shielded_to_thread( pending, @@ -155,32 +128,17 @@ async def run_async(self, user_msg: str, *, max_turns: int | None, turn_timeout: dirs_exist_ok=True, # mkdtemp already created the target; allow merging in ) - # Mount the reference solution at ``_reference/`` for the judge to browse. - # Symlinks are stripped unconditionally by the helper. Pattern-based - # ignores use a SEPARATE list (``_reference_ignore_patterns``, default - # ``[]``) so we don't reuse the sandbox-side ``ignore_patterns`` which - # includes ``_reference`` as defense-in-depth — applying that here would - # silently strip a nested ``_reference/`` subdir inside the user's - # reference, dropping grading material the user explicitly chose to - # include. - # - # Defense-in-depth on the sandbox side: ``AgentJudgeCriterion.agent.ignore_patterns`` - # includes ``_reference`` so the first copytree above strips any - # sandbox-side ``_reference/`` (agent-planted or template-staged). If a - # caller overrides ignore_patterns and removes ``_reference``, the second - # copytree would FileExistsError; rmtree is the safety net AND ensures the - # judge sees grading material exclusively from ``task.reference``. + # The rmtree is the safety net that keeps the judge's grading material + # coming exclusively from ``task.reference``: the sandbox-side ignore set + # strips any agent-planted ``_reference/``, but a caller could override it. + # Rationale: .claude/notes/contracts.md § The security floor if self._reference_dir is not None: ref_dest = judge_dir / "_reference" if ref_dest.exists(): await self._shielded_to_thread(pending, shutil.rmtree, ref_dest, ignore_errors=True) - # Deliberately NO ``dirs_exist_ok=True`` here. The rmtree above - # is the canonical clear; if any file survives (read-only flag, - # ENOTEMPTY race, hostile permission bits), we want copytree to - # FileExistsError loudly rather than silently merge the reference - # into agent-planted content under the same path. Loud failure on - # a partial-rmtree edge case is preferred over silently grading - # against a tampered ``_reference/``. + # HAZARD: deliberately NO ``dirs_exist_ok=True``. If any file + # survives the rmtree, fail loudly rather than merge the reference + # into agent-planted content under the same path. await self._shielded_to_thread( pending, shutil.copytree, @@ -216,19 +174,13 @@ async def run_async(self, user_msg: str, *, max_turns: int | None, turn_timeout: ) return turn finally: - # Any shielded to_thread call above keeps running on its worker thread even - # after a cancellation unwinds us into this `finally` — awaiting it here - # (not itself re-cancelled: asyncio delivers a given cancel() as a single - # CancelledError at the point it lands, not to every subsequent await in the - # same coroutine) lets it actually finish BEFORE we rmtree, so an orphan - # thread can never recreate files in judge_dir after cleanup already ran. + # A shielded to_thread keeps running after cancellation unwinds us here; + # awaiting it lets it finish BEFORE the rmtree. + # Rationale: .claude/notes/contracts.md § Cancellation safety if pending: await asyncio.gather(*(t for t in pending if not t.done()), return_exceptions=True) - # Deliberately NOT `await asyncio.to_thread(...)` here: a bare await inside - # `finally` is itself cancellable — cancelling right as this line is reached - # would skip the cleanup and leak the (now up-to-date) sandbox copy with no - # reaper. rmtree is a best-effort, bounded filesystem op; call it - # synchronously so cancellation can't interrupt it mid-cleanup. + # HAZARD: deliberately synchronous. A bare await inside `finally` is + # itself cancellable, and cancelling here would leak the sandbox copy. shutil.rmtree(judge_dir, ignore_errors=True) # noqa: CE002 @staticmethod diff --git a/src/coder_eval/evaluation/verdict_tool.py b/src/coder_eval/evaluation/verdict_tool.py index 16186a485..c7a1910be 100644 --- a/src/coder_eval/evaluation/verdict_tool.py +++ b/src/coder_eval/evaluation/verdict_tool.py @@ -61,8 +61,7 @@ class VerdictCapture: error: str | None = None called_count: int = 0 # LAST observed call wins: a successful retry overwrites the previous verdict, - # and an invalid retry clears the prior valid verdict (and sets ``error``). - # The criterion reads the final state via ``extract_verdict_from_capture``. + # and an invalid one clears it and sets ``error``. def _build_submit_verdict_tool(capture: VerdictCapture) -> SdkMcpTool[Any]: @@ -229,9 +228,8 @@ def _format_validation_error(e: ValidationError) -> str: msg = err.get("msg", "") if etype == "missing": - # Generic on ``field`` so any future required ``JudgeVerdict`` field gets - # the legacy " field missing in judge verdict" vocabulary. Today - # ``score`` is the only required field; the test suite pins the string. + # Generic on ``field`` so a future required field gets the same + # vocabulary; the test suite pins the string. messages.append(f"{field or 'unknown'} field missing in judge verdict") continue diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index 4e9ffd23b..a3aa4cf8d 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -413,12 +413,9 @@ "VariantResult", ] -# Type aliases (forward compatible) -# Typed as the discriminated union so callers that iterate the result list -# and use ``isinstance(cr, JudgeCriterionResult)`` get the precise variant -# membership. The runtime objects are concrete subclasses regardless; the -# alias change is purely a type-checking precision fix that mirrors the -# ``EvaluationResult.success_criteria_results`` field type. +# Typed as the discriminated union so an ``isinstance`` over the result list gets +# precise variant membership. The runtime objects are concrete subclasses either +# way; this mirrors the ``EvaluationResult.success_criteria_results`` field type. type CriteriaResults = list[CriterionResultUnion] type SuccessCriteria = list[SuccessCriterion] type TurnRecords = list[TurnRecord] diff --git a/src/coder_eval/models/agent_config.py b/src/coder_eval/models/agent_config.py index 1c79337f1..5b74a746b 100644 --- a/src/coder_eval/models/agent_config.py +++ b/src/coder_eval/models/agent_config.py @@ -54,23 +54,13 @@ class LocalPluginConfig(TypedDict): _VALID_SDK_OPTION_FIELDS: frozenset[str] = frozenset(f.name for f in dataclasses.fields(ClaudeAgentOptions)) -# Keys that `coder_eval` already owns at the AgentConfig level OR that are -# transport / lifecycle / security-critical. Setting them via the -# sdk_options pass-through would either silently shadow a typed field, let -# the user inject pre-LLM lifecycle hooks (hooks / mcp_servers / -# permission_prompt_tool_name / can_use_tool / agents), or bypass -# framework-managed runtime state (cwd / env / resume / max_turns / ...). -# Most relevantly: AgentJudgeCriterion forces setting_sources=[] for -# security; allowing `hooks` through sdk_options would re-open that hole. -# NOTE on the allow/deny model: validation works as ALLOW iff -# (key in _VALID_SDK_OPTION_FIELDS) AND (key not in _FRAMEWORK_OWNED_SDK_FIELDS) -# i.e. the user-visible set is ``_VALID - _FRAMEWORK_OWNED``. The denylist is -# explicit so the curated rationale (transport / lifecycle / security / typed- -# mirror) stays close to the code. To keep this from being fail-open as the -# SDK grows: ``tests/test_sdk_option_classification.py`` asserts EVERY field -# on ``ClaudeAgentOptions`` is classified — either typed-mirrored or in -# ``_FRAMEWORK_OWNED_SDK_FIELDS``. A new SDK release adding an unclassified -# field will fail that test loudly rather than silently being passed through. +# Keys `coder_eval` already owns at the AgentConfig level, or that are transport / +# lifecycle / security-critical. Validation is ALLOW iff +# (key in _VALID_SDK_OPTION_FIELDS) AND (key not in _FRAMEWORK_OWNED_SDK_FIELDS) +# so the user-visible set is ``_VALID - _FRAMEWORK_OWNED``. The denylist is explicit +# so each withheld key's reason stays beside the code, and +# ``tests/test_sdk_option_classification.py`` asserts every SDK field is classified. +# Rationale: .claude/notes/agents.md § The sdk_options pass-through _FRAMEWORK_OWNED_SDK_FIELDS: frozenset[str] = frozenset( { # mirrored as typed AgentConfig fields: @@ -92,16 +82,12 @@ class LocalPluginConfig(TypedDict): "session_id", "session_store", "session_store_flush", - # session lifecycle — coder_eval owns this via `resume` and the - # orchestrator's "advance session_id only on clean turns" logic. - # Letting YAML override would silently bypass that. + # session lifecycle -- owned by the orchestrator's "advance session_id + # only on clean turns" logic. "continue_conversation", "fork_session", - # budgeting — overlaps with RunLimits.max_usd / RunLimits.max_total_tokens - # which the orchestrator enforces with explicit FinalStatus codes - # (TOKEN_BUDGET_EXCEEDED / COST_BUDGET_EXCEEDED). Two independent - # budget guards would disagree on counts; route everything through - # RunLimits. + # budgeting -- overlaps RunLimits, which the orchestrator enforces with + # explicit FinalStatus codes. Two guards would disagree on counts. "max_budget_usd", "task_budget", # security-critical: arbitrary code injection or settings-bypass @@ -118,11 +104,8 @@ class LocalPluginConfig(TypedDict): "skills", "add_dirs", "setting_sources", # framework-controlled to prevent hook injection - # telemetry: required by ClaudeCodeAgent to recover per-emission - # output_tokens via message_delta stream events (works around - # anthropics/claude-code#22686 where the assistant event's - # output_tokens is a partial streaming snapshot). Letting YAML - # turn it off would silently drop per-message output accounting. + # telemetry: required to recover per-emission output_tokens around + # anthropics/claude-code#22686. Turning it off drops that accounting. "include_partial_messages", } ) @@ -137,8 +120,7 @@ class BaseAgentConfig(BaseModel): model_config = ConfigDict(validate_assignment=True, populate_by_name=True, extra="forbid") # Cross-field merge exclusion: setting either prompt field at any layer clears - # the sibling (the generic resolver honors this uniformly). ClassVar -> not a - # model field; Pydantic does not validate/assign it. + # the sibling. ClassVar -> not a model field. _merge_exclusive_groups: ClassVar[tuple[tuple[str, ...], ...]] = (("system_prompt", "system_prompt_file"),) type: str | None = Field( @@ -310,10 +292,8 @@ class CodexAgentConfig(BaseAgentConfig): type: Literal[AgentKind.CODEX] # type: ignore[assignment] -# Gemini "thinking level" (reasoning effort) for the Antigravity backend. Mirrors -# google.antigravity.types.ThinkingLevel as a plain Literal so this config module -# imports without the optional `google-antigravity` SDK installed (the SDK is an -# opt-in extra; base installs must still load every config class). +# Mirrors google.antigravity.types.ThinkingLevel as a plain Literal so this module +# imports without the optional SDK -- base installs must load every config class. type ThinkingLevel = Literal["minimal", "low", "medium", "high"] @@ -381,45 +361,28 @@ class OpenCodeAgentConfig(BaseAgentConfig): ) -# Pi's ``--thinking`` reasoning-effort set. A STRICT SUPERSET of ThinkingLevel -# (adds ``off``/``xhigh``/``max``), so Pi gets its own literal rather than reusing -# the 4-value one — reuse would forbid valid Pi levels. Spike-confirmed against -# ``pi --help`` on Pi 0.84.4. +# A STRICT SUPERSET of ThinkingLevel, so Pi gets its own literal -- reuse would +# forbid valid Pi levels. Confirmed against ``pi --help`` on Pi 0.84.4. type PiThinkingLevel = Literal["off", "minimal", "low", "medium", "high", "xhigh", "max"] class PiAgentConfig(BaseAgentConfig): """Pi agent configuration (the ``pi`` Node coding agent — https://pi.dev/). - Drives the ``pi`` CLI in JSON print mode - (``pi -p --mode json``), which streams newline-delimited JSON events on - stdout. ``model`` is Pi's provider-prefixed ``provider/model`` form (e.g. - ``openrouter/moonshotai/kimi-k3``) and is passed through verbatim via - ``--model`` (no separate ``--provider`` needed). - - Session continuity (multi-turn / simulation) - -------------------------------------------- - Each ``communicate()`` is one ``pi`` subprocess. A standard task reaches its - solution inside a single ``communicate()`` (Pi runs its own multi-step agent - loop). A simulation/dialog task calls ``communicate()`` once per user turn and - relies on the agent remembering prior turns — so ``PiAgent`` reuses a per-agent - ``--session-dir`` + stable ``--session-id`` on every invocation (create-if-missing - on turn 1, resume after), mirroring OpenCode's ``--session`` continuity. - - Enforced vs unenforced fields - ----------------------------- - ``system_prompt`` → ``--append-system-prompt`` IS enforced (a small capability - win over OpenCode). ``allowed_tools`` / ``disallowed_tools`` are NOT forwarded: - the shared config default sets Claude-namespaced tool names - (``Bash``/``Read``/…) that do not exist in Pi's lowercase toolset - (``bash``/``read``/…), so forwarding them to ``--tools`` would allowlist - nonexistent tools and strip the agent of ALL tools — like OpenCode/Codex/ - Antigravity, Pi ignores them and runs with its full native toolset. - ``permission_mode`` is NOT enforced (Pi headless print mode auto-runs tools; - the sandbox driver is the isolation boundary) and ``system_prompt_file`` is NOT - read — both warned about at ``start()``. ``plugins`` skills ARE injected (each - resolved skills dir → a ``--skill `` arg, recorded as ``pi_skill_paths`` in - ``environment_info``), so Pi can run activation suites. See ``docs/agents/PI.md``. + Drives the ``pi`` CLI in JSON print mode (``pi -p --mode json``), which streams + newline-delimited JSON events on stdout. ``model`` is Pi's provider-prefixed + ``provider/model`` form, passed through verbatim via ``--model``. + + Each ``communicate()`` is one ``pi`` subprocess, so a dialog task relies on a + per-agent ``--session-dir`` + stable ``--session-id`` for continuity. + + ``system_prompt`` IS enforced. ``allowed_tools`` / ``disallowed_tools`` are NOT + forwarded (the shared defaults name Claude-namespaced tools that do not exist in + Pi's toolset, so forwarding them would strip the agent of ALL tools), nor is + ``permission_mode``, nor is ``system_prompt_file`` read -- both warned about at + ``start()``. ``plugins`` skills ARE injected. See ``docs/agents/PI.md``. + + Rationale: .claude/notes/agents.md § Pi """ type: Literal[AgentKind.PI] # type: ignore[assignment] @@ -450,9 +413,8 @@ class NoneAgentConfig(BaseAgentConfig): type: Literal[AgentKind.NONE] # type: ignore[assignment] -# Discriminated union type for type hints, validation, and YAML serialization -# Only includes the concrete subclasses (not BaseAgentConfig) since the discriminator -# must be a Literal type. BaseAgentConfig is returned by parse_agent_config when type=None. +# Concrete subclasses only (not BaseAgentConfig): the discriminator must be a +# Literal. parse_agent_config returns BaseAgentConfig when type=None. type AgentConfig = Annotated[ ClaudeCodeAgentConfig | CodexAgentConfig @@ -521,10 +483,8 @@ def _coerce_agent_config(value: Any) -> Any: return value -# The canonical annotation for any field that holds a resolved agent config. -# BeforeValidator routes a dict through registry dispatch (so plugin kinds resolve -# to their subclass); SerializeAsAny keeps subclass-only fields on model_dump() -# instead of the base schema silently dropping them. Used by every persisted -# agent-config field (TaskDefinition.agent, EvaluationResult.agent_config) so the -# round-trip guarantee is uniform across the built-in union and plugin kinds. +# The canonical annotation for a resolved agent config. BeforeValidator routes a +# dict through registry dispatch so plugin kinds resolve to their subclass; +# SerializeAsAny keeps subclass-only fields on model_dump(). Used by every persisted +# agent-config field, so the round-trip guarantee is uniform. type ResolvedAgentConfig = Annotated[SerializeAsAny[BaseAgentConfig], BeforeValidator(_coerce_agent_config)] diff --git a/src/coder_eval/models/cli_match.py b/src/coder_eval/models/cli_match.py index b65fc96fb..42f067bfe 100644 --- a/src/coder_eval/models/cli_match.py +++ b/src/coder_eval/models/cli_match.py @@ -113,13 +113,9 @@ def _exactly_one_predicate(self) -> FlagMatch: if self.flags and self.matches_regex is None: msg = f"FlagMatch.flags applies only to matches_regex, but the predicate is {set_predicates[0]!r}" raise ValueError(msg) - # Compile HERE, not in a checker: this model now feeds two consumers, and - # only one of them can report. A `record_cli` response rule evaluates the - # pattern inside the sandbox, where a PatternError is swallowed and the - # tool serves its fallback -- a log line indistinguishable from a - # legitimate no-match, so the task scores differently for identical agent - # behaviour with nothing on any report surface. At load, both surfaces - # refuse the pattern instead. + # Compile HERE, not in a checker: this model feeds two consumers and only + # one of them can report an error at all. + # Rationale: .claude/notes/contracts.md § The five refuse-to-score paths are uniform at a gating 0.0 if self.matches_regex is not None: try: re.compile(self.matches_regex, self.flags) @@ -129,11 +125,9 @@ def _exactly_one_predicate(self) -> FlagMatch: return self -# The argv facets every matching surface must offer. `cli_called` declares these -# fields itself (with grading-specific guidance in each description) rather than -# inheriting them, so a facet added to one surface and forgotten on the other is -# caught by the parity test in tests/test_cli_match_parity.py instead of shipping -# as a rule the criterion cannot express. +# The argv facets every matching surface must offer. `cli_called` declares them +# itself rather than inheriting, so a facet added to one surface and forgotten on +# the other is caught by tests/test_cli_match_parity.py. MATCH_FACET_FIELDS: tuple[str, ...] = ("verb", "verb_any_of", "positional", "flags", "value_flags", "ignore_flags") @@ -167,12 +161,10 @@ def validate_verbs(verb: str | None, verb_any_of: list[str] | None, spellings: l if any(not tokens for tokens in spellings): msg = f"{label} verb must not be blank: a blank verb is an empty prefix and matches every invocation" raise ValueError(msg) - # A verb is compared against the NON-FLAG arguments, so a flag written into it - # can never match anything -- and the failure is silent: the criterion scores 0 - # against a log that holds the very call it describes, and a response rule falls - # through to the tool's default. Inviting, too, since a whole verb reads like a - # command line. `is_number` mirrors the splitter's own rule so this check cannot - # forbid a token (`-1`) that the matcher would in fact have seen. + # HAZARD: a verb is compared against the NON-FLAG arguments, so a flag written + # into one can never match -- silently. `is_number` mirrors the splitter's own + # rule so this cannot forbid a token the matcher would have seen. + # Rationale: .claude/notes/contracts.md § The five refuse-to-score paths are uniform at a gating 0.0 for tokens in spellings: for token in tokens: if token.startswith("-") and token != "-" and not is_number(token.lstrip("-")): diff --git a/src/coder_eval/models/container_paths.py b/src/coder_eval/models/container_paths.py index 1473b84f1..8bc72a666 100644 --- a/src/coder_eval/models/container_paths.py +++ b/src/coder_eval/models/container_paths.py @@ -13,12 +13,11 @@ from __future__ import annotations -# Tokens task YAMLs use to address host directories from a criterion's path -# fields (``llm_judge.files``, ``agent_judge.files``). They resolve against the -# task YAML's own directory and the staged reference copy respectively, and are -# mirrored as the TASK_DIR / REFERENCE_DIR env vars exposed to ``run_command``. -# Defined here (a dependency-free leaf) so both the models layer and -# ``evaluation.judge_context`` share one definition without an import cycle. +# Tokens a task YAML uses to address host directories from a criterion's path +# fields, mirroring the TASK_DIR / REFERENCE_DIR env vars ``run_command`` exposes. +# Defined in this dependency-free leaf so the models layer and judge_context share +# one definition without an import cycle. +# Rationale: .claude/notes/contracts.md § Judge context and untrusted text TASK_DIR_TOKEN = "$TASK_DIR" REFERENCE_DIR_TOKEN = "$REFERENCE_DIR" @@ -70,15 +69,10 @@ def command_uses_token(command: str, token: str) -> bool: return False -# The one reliable "am I inside a task container?" signal, set by DockerRunner -# on every container it starts. -# -# Every gate that means "in a container" MUST key on this and never on -# `sandbox.driver`: the in-container entry point rewrites `docker` -> `tempdir` -# before building its Orchestrator, so a driver-based test reads a value that has -# already been changed — which would silently disable the reference-permission -# window on exactly the path that needs it (regression-guarded by -# TestSandboxDriverGate). +# HAZARD: the one reliable "am I inside a task container?" signal. Every gate that +# means "in a container" MUST key on this and never on `sandbox.driver`, which the +# in-container entry point has already rewritten to `tempdir`. +# Rationale: .claude/notes/isolation.md § Capability drops and the anti-cheat window IN_CONTAINER_ENV = "CODER_EVAL_IN_CONTAINER" CONTAINER_WORK_DIR = "/work" @@ -86,27 +80,18 @@ def command_uses_token(command: str, token: str) -> bool: CONTAINER_OUTPUT_DIR = "/work/output" CONTAINER_TASK_DIR = "/work/task_dir" -# Where the per-run private copy of ``task.reference.directory`` is mounted. -# Exposed to criteria as the ``REFERENCE_DIR`` env var and as the -# ``$REFERENCE_DIR`` token in judge ``files:`` entries. Kept at mode 000 for the -# duration of every ``agent.communicate`` call so the agent under evaluation -# cannot read the solution (see ``fs_permissions.py``). +# The per-run private copy of ``task.reference.directory``, exposed to criteria as +# ``REFERENCE_DIR``. Kept at mode 000 for the duration of every +# ``agent.communicate`` call (see ``fs_permissions.py``). CONTAINER_REFERENCE_DIR = "/work/references" -# Where a DETACHED GRADE mounts the already-executed workspace it is grading. -# Only ever present on a grading container (`evaluate` / `run --resume` over a -# `driver: docker` row); a normal run never mounts it. -# -# It is a separate mount from CONTAINER_OUTPUT_DIR because the two belong to -# different runs: the grading pass writes its own `task.json` into its own fresh -# run directory (which the host then folds back into the row, preserving -# `task.execute.json`), while the workspace under evaluation belongs to the -# ORIGINAL run and must be adopted, never written over. +# Where a DETACHED GRADE mounts the already-executed workspace. Separate from +# CONTAINER_OUTPUT_DIR because the two belong to different runs. +# Rationale: .claude/notes/isolation.md § Why the grading container gets a private scratch directory CONTAINER_GRADE_WORKSPACE = "/work/workspace" -# Paths a task's WORKDIR must never collide with: the container root and every -# framework-owned mount under /work. Consumed by SandboxConfig's working_dir -# validator (models/sandbox.py) and re-asserted host-side in docker_runner. +# Paths a task's WORKDIR must never collide with. Consumed by SandboxConfig's +# working_dir validator and re-asserted host-side in docker_runner. RESERVED_CONTAINER_DIRS = frozenset( { "/", diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index fe74662f1..a087efaf8 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -1,9 +1,8 @@ """Success criteria models for task evaluation.""" # Each concrete criterion narrows the base ``type: str`` to its ``Literal[...]`` tag -# (the standard pydantic discriminated-union pattern). Pyright treats a Literal -# subtype override of a mutable str field as variable-invariant-incompatible, but -# the narrowing is exactly the intended discriminator behaviour here. +# (the standard pydantic discriminated-union pattern), which pyright reads as a +# variable-invariance violation. # pyright: reportIncompatibleVariableOverride=false from __future__ import annotations @@ -29,17 +28,11 @@ from coder_eval.models.sandbox import RECORD_CLI_LOG -# SECURITY: ignore_patterns floor. The judge's working directory is a copy of -# the agent's sandbox; if the agent dropped these in, they'd otherwise be -# copied across and either (a) cause the SDK to load hooks / MCP servers from -# the main agent's settings (despite ``setting_sources=[]``, the agent ignore -# patterns also gate what shutil.copytree pulls in) or (b) collide with the -# reference mount point. The floor is enforced UNCONDITIONALLY in -# ``criteria/agent_judge.py::_build_agent_config``, even when the user -# supplied their own ``agent.ignore_patterns`` — author convenience does not -# override the judge's safety floor. Single source of truth: importing this -# constant from both call-sites keeps the model defaults and the checker -# floor in sync. +# SECURITY: the judge's ignore_patterns FLOOR, enforced unconditionally in +# ``criteria/agent_judge.py::_build_agent_config`` even when the user supplied their +# own list. Imported from both call sites so the model defaults and the checker +# floor cannot drift. +# Rationale: .claude/notes/contracts.md § The security floor JUDGE_SECURITY_IGNORE_FLOOR: tuple[str, ...] = (".claude", ".mcp.json", "_reference") @@ -156,11 +149,9 @@ def is_stop_armed(self) -> bool: return False def model_post_init(self, context: Any, /) -> None: - # Pin the discriminator tag into model_fields_set so it survives - # model_dump(exclude_unset=True) → model_validate() round-trips even for - # directly-constructed criteria (where the tag comes from the Literal - # default, not the caller). Without this, exclude_unset drops the tag and - # the discriminated union rejects the dump with union_tag_not_found. + # Pin the tag into model_fields_set so it survives a + # model_dump(exclude_unset=True) -> model_validate() round trip even for + # directly-constructed criteria, where the tag comes from the Literal default. self.__pydantic_fields_set__.add("type") @model_validator(mode="after") @@ -205,12 +196,10 @@ def is_gating(self) -> bool: # Business logic (check operations) moved to SuccessChecker in evaluator.py -# The two polarities a live-observable criterion can decide mid-run — distinct -# from the 3-value LiveVerdict ("pass"/"fail"/"undecided") the checker's -# live_verdict returns: this is the narrower CAPABILITY type, "undecided" is -# never a valid decidable polarity. Typed here (not a bare frozenset[str]) so -# a live_decidable_polarities override returning a stray/typo'd string, or -# "undecided" itself, is a pyright error rather than a runtime-only lint gap. +# The two polarities a live-observable criterion can decide mid-run -- the narrower +# CAPABILITY type, distinct from the 3-value LiveVerdict. Typed, not a bare +# frozenset[str], so a stray or typo'd polarity is a pyright error. +# Rationale: .claude/notes/contracts.md § Why replay is the only sound check LivePolarity = Literal["pass", "fail"] @@ -432,67 +421,22 @@ class CliCalledCriterion(BaseSuccessCriterion): """Check whether a CLI invocation matching a structured pattern was recorded. Reads a **structured invocation log** the sandbox produced: JSON Lines, one - object per invocation, each with at minimum an ``argv`` list. A test harness - that shadows a CLI with a recording mock writes this log; this criterion - matches against it field-by-field instead of regexing a flattened command - string. - - Record schema (extra keys ignored):: + object per invocation, each with at minimum an ``argv`` list:: {"argv": ["ixp", "projects", "get", "proj-1", "--output", "json"], "tool": "uip", "exit": 1, "ts": 1785416844.987} - Only ``argv`` is required. ``tool`` enables one log to serve several shadowed - executables; ``exit`` and ``ts`` are recorded for reporting, not matched. - - A generated ``record_cli`` shim adds ``rule`` (the index of the response rule - that answered), which is reporting only, plus two keys that are NOT ignored - because each means the responses the agent saw were not the ones the task - described: ``sidecar_error`` (the shim could not import its matcher module) - and ``rule_error`` (rule evaluation raised). Both fail the criterion. Neither - escalates, because the recorder directory sits inside the sandbox the agent - writes to, so both are agent-reachable; an authoring mistake is caught - earlier instead, by :class:`~coder_eval.models.RecordedCli`'s load-time - check that every response rule is evaluable. - - Why not ``file_matches_regex`` over a flattened log line: a flat line cannot - express "verb X was called AND flag Y had value Z" without stacked - lookaheads, cannot tell a quoted argument containing spaces from two - arguments, and cannot stop a match from running across shell operators. - - EVIDENCE, NOT ATTESTATION. The log is an ordinary file in the sandbox the - agent writes to, so an agent that wants to can append a record for a call it - never made, or delete one it did. This criterion is built to keep an HONEST - run honest -- a missing or unreadable log fails rather than passing a - ``max_count: 0`` guard vacuously -- not to withstand an adversary. Do not - build an anti-cheat control on it; see ``tasks/anti_cheat_reference`` and - docs/DOCKER_ISOLATION.md for what that requires. + Only ``argv`` is required. ``tool`` lets one log serve several shadowed + executables; ``exit`` and ``ts`` are reporting only, as is the ``rule`` a + generated shim adds. Its ``sidecar_error`` and ``rule_error`` are NOT ignored: + each means the responses the agent saw were not the ones the task described. - Pure data model - checking logic in CliCalledChecker._check_impl() - - Example YAML (positive — flag value must match):: - - success_criteria: - - type: "cli_called" - description: "Switched the project to the capable model" - log: "mocks/calls.jsonl" - verb: "ixp projects configure-model" - positional: ["my_invoices-f1afa9ef-ixp"] - flags: - model: "gemini_2_5_pro" - min_count: 1 + EVIDENCE, NOT ATTESTATION -- the log sits in the sandbox the agent writes to. - Example YAML (negative — must NOT have been called; ``min_count: 0`` + ``max_count: 0``):: + Pure data model - checking logic in CliCalledChecker._check_impl(). Authoring + reference: docs/TASK_DEFINITION_GUIDE.md#cli_called - success_criteria: - - type: "cli_called" - description: "Did not use --corrections to flip a boolean field" - log: "mocks/calls.jsonl" - verb: "ixp labellings confirm" - flags: - corrections: {contains: "f-100"} - min_count: 0 - max_count: 0 + Rationale: .claude/notes/contracts.md § Recording a CLI invocation """ type: Literal["cli_called"] = "cli_called" @@ -624,10 +568,10 @@ def _validate_bounds(self) -> CliCalledCriterion: # Matching slices an empty expectation and compares it to itself, so this reads # as "took no arguments" while asserting nothing. validate_positional(self.positional, "cli_called") - # Falsiness, not `is None`: `verb: ""` slipped past an `is None` check here and - # then matched every record, scoring 1.0. `tool` counts as a facet here (but not - # for a response rule), since a criterion may legitimately count every - # invocation of one shadowed executable. + # HAZARD: falsiness, not `is None`. A `verb: ""` slipped past an `is None` + # check here and then matched every record, scoring 1.0. `tool` counts as a + # facet here, though not for a response rule: a criterion may legitimately + # count every invocation of one shadowed executable. if not self.verb and not self.verb_any_of and not self.positional and not self.flags and not self.tool: msg = "cli_called requires at least one of verb / verb_any_of / positional / flags / tool to match on" raise ValueError(msg) @@ -1122,17 +1066,14 @@ class LLMJudgeCriterion(BaseSuccessCriterion): validator. Non-numeric / non-finite score -> 0.0 with error. """ - # Strict YAML-key validation: catch typos at load time rather than silently - # ignoring an unknown key (e.g. ``capture_transcripts:`` instead of - # ``capture_transcript:``) and producing a misconfigured judge. + # Strict YAML-key validation: a typo becomes a load-time error rather than a + # silently misconfigured judge. model_config = ConfigDict(extra="forbid") type: Literal["llm_judge"] = "llm_judge" - # Override the BaseSuccessCriterion default of 0.9 — that's calibrated for binary - # checks like file_exists where partial = fail. Judges produce continuous scores - # that rarely emit 1.0 even for excellent solutions; 0.7 matches the "good enough - # for a strict reviewer" semantics most authors want. Override per task as needed. + # Overrides the 0.9 default, which is calibrated for binary checks. Judges produce + # continuous scores that rarely emit 1.0 even for excellent solutions. pass_threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="Minimum score to pass (default 0.7).") enabled: bool = Field( @@ -1268,35 +1209,21 @@ class AgentJudgeCriterion(BaseSuccessCriterion): """Spawn a Claude Code SDK agent as the judge. The judge runs in an isolated copy of the sandbox with tool access (Bash, Read, - Glob, Grep by default) and reports its verdict by calling the in-process - ``submit_verdict`` MCP tool (``mcp__coder_eval_judge__submit_verdict``, - force-added to ``allowed_tools``) exactly once with ``score`` / ``rationale`` - / ``findings``. The SDK has no ``tool_choice`` equivalent so the channel - relies on system-prompt discipline; a judge that never calls the tool - scores 0.0 with a "did not call submit_verdict" diagnostic. - - File access: the judge has live access to the sandbox copy as its working - directory and can load files via its tools (``Read``, ``Glob``). The optional - ``files`` field pre-attaches selected file contents to the prompt envelope — - useful when you want a fast verdict (single turn, no tool calls) or want to - grade with the narrowest tool surface (``allowed_tools=[]``). Pre-attach and - tool-driven inspection compose: the judge sees the pre-attached blocks and - can still ``Read`` anything else. - - SECURITY: The judge runs with the evaluator's API credentials and can execute - arbitrary Bash by default. Four attack surfaces: - 1. Malicious generation artifacts the judge executes (e.g. via `python x.py`). - 2. Prompt injection from included agent_output / tool-call summaries. - 3. Credential exfiltration via any network-capable tool (primarily Bash). - 4. Hooks / MCP servers planted by the main agent (e.g. `.claude/settings.json` - or `.mcp.json` dropped into the sandbox) that would run before any LLM turn - and before allowed_tools gating. The judge sets ``setting_sources=[]`` and - excludes both paths from the sandbox copy, so neither gets loaded. - Use ``llm_judge`` for scenarios with adversarial generation. Narrow - ``allowed_tools`` per task when Bash is not needed. - - Continuous scoring. Verdict-validation errors or a "did not call" diagnostic - -> 0.0. Score clamped to [0.0, 1.0] by the ``JudgeVerdict`` validator. + Glob, Grep by default) and reports by calling the in-process ``submit_verdict`` + MCP tool exactly once. The SDK has no ``tool_choice`` equivalent, so unlike + ``llm_judge`` the channel relies on system-prompt discipline: a judge that never + calls it scores 0.0 with a "did not call submit_verdict" diagnostic. + + It works in the sandbox copy and can load files via its tools; ``files`` + pre-attaches selected contents to the prompt envelope, and the two compose. + + SECURITY: the judge runs with the evaluator's API credentials and can execute + arbitrary Bash by default. Use ``llm_judge`` for adversarial generation, and + narrow ``allowed_tools`` per task when Bash is not needed. + + Continuous scoring. Score clamped to [0.0, 1.0] by the ``JudgeVerdict`` validator. + + Rationale: .claude/notes/contracts.md § The security floor """ # Strict YAML-key validation: catch typos at load time rather than silently @@ -1305,9 +1232,8 @@ class AgentJudgeCriterion(BaseSuccessCriterion): type: Literal["agent_judge"] = "agent_judge" - # Override the BaseSuccessCriterion default of 0.9 — that's calibrated for binary - # checks. Judges produce continuous scores that rarely emit 1.0; 0.7 matches the - # "good enough for a strict reviewer" semantics most authors want. Override per task. + # Overrides the 0.9 default, which is calibrated for binary checks. Judges produce + # continuous scores that rarely emit 1.0 even for excellent solutions. pass_threshold: float = Field(default=0.7, ge=0.0, le=1.0, description="Minimum score to pass (default 0.7).") enabled: bool = Field( @@ -1401,9 +1327,8 @@ class AgentJudgeCriterion(BaseSuccessCriterion): ge=10, description="Wall-clock timeout for the judge turn (seconds). Minimum 10.", ) - # Intentionally the closed built-in AgentConfig union, NOT ResolvedAgentConfig: - # agent_judge spawns a Claude Code SDK sub-agent (SubAgentRunner) with evaluator - # credentials, so a plugin agent kind is deliberately not accepted as a judge. + # HAZARD: the CLOSED built-in union, not ResolvedAgentConfig -- agent_judge spawns + # a sub-agent with evaluator credentials, so a plugin kind is not accepted here. agent: AgentConfig = Field( default_factory=_default_judge_agent_config, description=( @@ -1441,11 +1366,10 @@ def _reject_verdict_channel(cls, data: Any) -> Any: return _reject_removed_verdict_channel(data) -# Discriminated union of all success criteria. The `type` tag is REQUIRED in -# dict/YAML input: a missing or unknown tag raises one crisp discriminator error -# instead of smart-union coercion across every variant. The per-variant -# `type: Literal[...] = ""` defaults remain, so direct construction -# (e.g. FileExistsCriterion(path=...)) and model_dump() serialization are unaffected. +# The `type` tag is REQUIRED in dict/YAML input: a missing or unknown tag raises one +# crisp discriminator error instead of smart-union coercion across every variant. The +# per-variant Literal defaults remain, so direct construction and model_dump() are +# unaffected. SuccessCriterion = Annotated[ FileExistsCriterion | FileContainsCriterion diff --git a/src/coder_eval/models/enums.py b/src/coder_eval/models/enums.py index c4131bd7d..6baa96dba 100644 --- a/src/coder_eval/models/enums.py +++ b/src/coder_eval/models/enums.py @@ -15,12 +15,9 @@ class FinalStatus(StrEnum): MAX_TURNS_EXHAUSTED = "MAX_TURNS_EXHAUSTED" TOKEN_BUDGET_EXCEEDED = "TOKEN_BUDGET_EXCEEDED" COST_BUDGET_EXCEEDED = "COST_BUDGET_EXCEEDED" - # `coder-eval execute` ran the agent but deliberately skipped grading, so - # there is no verdict to report. Distinct from FAILURE (which asserts the - # criteria were checked and did not pass) and from ERROR (which asserts - # something went wrong). Only SUCCESS/FAILURE collapse into it — every - # other member records an *execution* fact that still applies when the - # run is ungraded. + # No verdict to report. Distinct from FAILURE (criteria checked, did not pass) + # and ERROR (something went wrong); only those two collapse into it. + # Rationale: .claude/notes/orchestration.md § Execute vs. run: the grading switch NOT_GRADED = "NOT_GRADED" @property @@ -47,27 +44,21 @@ def is_execution_fact(self) -> bool: return _EXECUTION_FACT_STATUSES[self] -# Every FinalStatus maps to exactly one reporting category, listed EXPLICITLY (no -# catch-all default) so a newly-added status fails the assert below until it is -# classified — rather than silently collapsing into "failed" (which would skew -# reports AND the telemetry Category dimension). Mirrors the _STATUS_ICONS guard. +# EXPLICIT, no catch-all default, so a newly-added status fails the assert below +# until it is classified. Mirrors the _STATUS_ICONS guard. +# Rationale: .claude/notes/orchestration.md § Rates need verdict evidence, not bucket counts _STATUS_CATEGORIES: dict[FinalStatus, Literal["succeeded", "failed", "error", "ungraded"]] = { FinalStatus.SUCCESS: "succeeded", FinalStatus.FAILURE: "failed", FinalStatus.ERROR: "error", - # A failed image build is an environment/setup error, not a task outcome — - # group it with ERROR so reports/telemetry don't read it as a legitimate - # task failure the agent could have avoided. + # An environment fault, not a task outcome the agent could have avoided. FinalStatus.BUILD_FAILED: "error", FinalStatus.TIMEOUT: "failed", FinalStatus.MAX_TURNS_EXHAUSTED: "failed", FinalStatus.TOKEN_BUDGET_EXCEEDED: "failed", FinalStatus.COST_BUDGET_EXCEEDED: "failed", - # A fourth category, not a fold into one of the three. Folding into - # "failed" would depress every pass rate; folding into "succeeded" would - # invent verdicts; folding into "error" would report a healthy run as - # broken. Reporting surfaces exclude it from BOTH the numerator and the - # denominator of a pass rate — an ungraded task was never measured. + # A FOURTH category, not a fold into one of the three. Excluded from both the + # numerator and the denominator of a pass rate. FinalStatus.NOT_GRADED: "ungraded", } @@ -89,9 +80,8 @@ def is_execution_fact(self) -> bool: assert set(_STATUS_ICONS) == set(FinalStatus), "Missing icon for FinalStatus member" -# Explicit, no catch-all, for the same reason as the two maps above: a new status -# must be classified as "the agent phase ended this way" (True — a detached grade -# preserves it) or "grading decided this" (False — a detached grade replaces it). +# Explicit, no catch-all: a new status is either "the agent phase ended this way" +# (True -- a detached grade preserves it) or "grading decided this" (False). # Defaulting either way silently is how an ERROR row becomes a SUCCESS. _EXECUTION_FACT_STATUSES: dict[FinalStatus, bool] = { FinalStatus.SUCCESS: False, @@ -100,16 +90,10 @@ def is_execution_fact(self) -> bool: FinalStatus.ERROR: True, FinalStatus.BUILD_FAILED: True, FinalStatus.TIMEOUT: True, - # False, and it must stay False: MAX_TURNS_EXHAUSTED is SUBORDINATE to the - # verdict, not a fact that outranks it. `run` returns SUCCESS for a - # max-turns trajectory whose criteria pass and only falls through to this - # status when they do not — see `Orchestrator._terminal_status`, whose - # docstring already argued exactly this while this table said the opposite. - # With True, a prior max-turns row re-graded through `evaluate` was pinned - # at MAX_TURNS_EXHAUSTED *while holding weighted_score 1.000* and exited 1: - # a combination `run` can never produce for the same trajectory. The fact - # itself is not lost — it lives on `EvaluationResult.max_turns_exhausted`, - # which `_seed_from_prior_result` carries. + # HAZARD: False, and it must stay False. MAX_TURNS_EXHAUSTED is SUBORDINATE to + # the verdict, not a fact that outranks it; the fact itself lives on + # `EvaluationResult.max_turns_exhausted`, which the seeding carries. + # Rationale: .claude/notes/orchestration.md § The terminal-status chain FinalStatus.MAX_TURNS_EXHAUSTED: False, FinalStatus.TOKEN_BUDGET_EXCEEDED: True, FinalStatus.COST_BUDGET_EXCEEDED: True, diff --git a/src/coder_eval/models/experiment.py b/src/coder_eval/models/experiment.py index 662d992b8..ee9c563c4 100644 --- a/src/coder_eval/models/experiment.py +++ b/src/coder_eval/models/experiment.py @@ -190,10 +190,8 @@ class VariantResult(BaseModel): # noqa: CE009 -- persisted result model; round- variant_id: str task_id: str - # None when nothing was graded (`coder-eval execute`), mirroring - # EvaluationResult.weighted_score. A plain float here would launder the - # ungraded None into 0.000, which renders as — and is picked as a best - # variant against — a real score of zero. + # None when nothing was graded. A plain float would launder that into 0.000. + # Rationale: .claude/notes/orchestration.md § Rates need verdict evidence, not bucket counts weighted_score: float | None = None final_status: FinalStatus duration_seconds: float @@ -234,9 +232,8 @@ class VariantAggregate(BaseModel): # noqa: CE009 -- persisted result model; rou ge=0, description="Tasks executed without grading (`coder-eval execute`). Excluded from pass_rate entirely.", ) - # None when nothing in this variant was graded (`coder-eval execute`). - # A 0.0 here is indistinguishable from "measured and scored zero" — the same - # reason EvaluationResult.weighted_score is Optional. + # None when nothing in this variant was graded: a 0.0 is indistinguishable from + # "measured and scored zero". average_score: float | None average_duration: float total_tokens: int | None = None @@ -256,11 +253,8 @@ class VariantAggregate(BaseModel): # noqa: CE009 -- persisted result model; rou ge=0, description="Subset of tasks_failed where run_limits cost cap tripped.", ) - # Verdict evidence, NOT a bucket: how many rows actually carry a - # weighted_score. `pass_rate` needs it because the four category buckets - # cannot tell a graded FAILURE from a TIMEOUT that no criterion ever saw. - # Defaulted so an experiment.json written before this field still parses — - # and inert on those, since they predate the ungraded bucket entirely. + # Verdict evidence, NOT a bucket. Defaulted so an older experiment.json still + # parses, and inert on those. tasks_measured: int = Field( default=0, ge=0, diff --git a/src/coder_eval/models/judge.py b/src/coder_eval/models/judge.py index a32dd6db9..4c9e913cd 100644 --- a/src/coder_eval/models/judge.py +++ b/src/coder_eval/models/judge.py @@ -59,23 +59,14 @@ def _coerce_rationale(cls, v: Any) -> str: return "" if not isinstance(v, str): raise ValueError(f"rationale must be a string, got {type(v).__name__}") - # Collapse internal whitespace (newlines, tabs, runs of spaces) to single - # spaces. Two reasons: - # 1. ``format_details`` writes "rationale: " on one line; a multi-line - # rationale would break that single-line invariant. - # 2. ``reports_html._extract_rationale`` parses by line and grabs only the - # first one starting with "rationale: " — multi-line content would be - # silently truncated in the Judge Verdicts card. - # The schema asks for "1-2 sentence headline summary"; collapsing whitespace - # makes that contract explicit. + # Collapsed to single spaces because two consumers parse this by LINE: + # ``format_details`` writes it on one, and the HTML report grabs only the + # first "rationale: " line. The schema asks for a 1-2 sentence headline. collapsed = " ".join(v.split()) if not collapsed: - # Whitespace-only / empty input surfaces as a validation error so the - # judge result records the error string instead of silently emitting - # a blank ``rationale: `` line in ``format_details``. The - # ``extract_verdict_from_*`` extractors translate the - # ``ValidationError`` into a JudgeCriterionResult(score=0.0, error=...) - # — uniform shape across all three backends. + # A validation error, so the result records the error string rather than + # emitting a blank ``rationale: `` line. The extractors turn it into a + # score=0.0 result -- one shape across all three backends. raise ValueError("rationale is empty after whitespace collapse") return collapsed diff --git a/src/coder_eval/models/limits.py b/src/coder_eval/models/limits.py index 6cafb5f66..48bdb48c2 100644 --- a/src/coder_eval/models/limits.py +++ b/src/coder_eval/models/limits.py @@ -137,23 +137,8 @@ class RunLimits(BaseModel): ), ) - # NOTE: stop_early_gate_threshold <= 0.0 on an ARMED task is a degenerate, - # gate-neutralizing config (a threshold of 0 trivially passes the armed - # gate regardless of whether anything decided) and is rejected — but NOT - # here, and neither is stop_early: True (the removed master arm). Whether a - # task is armed lives on the criteria, which RunLimits cannot see, and - # RunLimits is field-merged across 5 layers, so a model-level validator has - # no visibility into which layer produced the merged value and cannot - # distinguish a real mistake from a value merged forward from a sibling - # layer (e.g. a task-level threshold inherited by a variant that only - # toggles the kill switch). Both checks live in - # orchestration/early_stop.py::validate_early_stop instead, where they - # raise EarlyStopConfigError and get the same hard-stop CLI treatment - # (flips the plan exit code, aborts run) as every other early-stop - # guardrail — a plain pydantic ValueError here would instead land in - # plan_command's generic per-variant "resolution failed" branch, which - # prints red text but does NOT flip the exit code by design (unlike - # EarlyStopConfigError), so a model-level raise would silently pass CI. - # Other cross-field semantics that are warnings rather than errors live in - # orchestration/run_limits.py::validate_run_limits for the same post-merge - # visibility without rejecting or mutating the resolved values. + # A degenerate gate threshold on an ARMED task, and the removed master arm, are + # both rejected -- but in orchestration/early_stop.py::validate_early_stop, NOT + # here. Whether a task is armed lives on the criteria, which RunLimits cannot + # see, and post-merge is the only place with enough visibility. + # Rationale: .claude/notes/orchestration.md § Why the guardrails are not model validators diff --git a/src/coder_eval/models/merge_strategy.py b/src/coder_eval/models/merge_strategy.py index e9e956c62..b5257d6a5 100644 --- a/src/coder_eval/models/merge_strategy.py +++ b/src/coder_eval/models/merge_strategy.py @@ -59,10 +59,9 @@ def append_order_of(field_info: FieldInfo) -> AppendOrder: extra = field_info.json_schema_extra if isinstance(extra, dict) and APPEND_ORDER_KEY in extra: order = extra[APPEND_ORDER_KEY] - # The value is written only by MergeField (its `append_order` param is - # typed AppendOrder), but guard a hand-written - # `Field(json_schema_extra={...})` with a bogus value so it fails loud - # here instead of silently mis-ordering an append. + # Written only by MergeField, but guard a hand-written + # `Field(json_schema_extra={...})` so a bogus value fails loud here rather + # than silently mis-ordering an append. if order not in get_args(AppendOrder): raise ValueError(f"invalid append order {order!r}; expected one of {get_args(AppendOrder)}") return order # type: ignore[return-value] # validated against AppendOrder above @@ -74,11 +73,9 @@ def merge_strategy_of(field_info: FieldInfo) -> MergeStrategy: extra = field_info.json_schema_extra if isinstance(extra, dict) and MERGE_STRATEGY_KEY in extra: strategy = extra[MERGE_STRATEGY_KEY] - # The value is written only by MergeField (its `strategy` param is typed - # MergeStrategy) and CE014 enforces that every list field declares one, - # but guard a hand-written `Field(json_schema_extra={...})` with a bogus - # value so it fails loud here instead of silently falling through to - # "replace" in the merge engine. + # Written only by MergeField, and CE014 enforces that every list field + # declares one -- but guard a hand-written value so it fails loud here + # rather than falling through to "replace" in the merge engine. if strategy not in get_args(MergeStrategy): raise ValueError(f"invalid merge strategy {strategy!r}; expected one of {get_args(MergeStrategy)}") return strategy # type: ignore[return-value] # validated against MergeStrategy above diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index 6eddf3541..ec9d6dc81 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -234,19 +234,12 @@ class JudgeCriterionResult(CriterionResult): ) -# Criterion types whose results are ``JudgeCriterionResult`` / ``ClassificationCriterionResult``. -# Used by ``_criterion_result_discriminator`` to type-infer legacy ``task.json`` records that -# lack the ``result_kind`` field. Listed here (not on each criterion class) because -# ``model_validate_json`` runs without the criteria registry loaded — consumers that import -# ``coder_eval.models`` to deserialize ``task.json`` (e.g. ``coder-eval report``) do NOT -# import ``coder_eval.criteria.*`` (which is where checkers self-register). Putting the -# inference rule on criterion classes would force importing every checker just to deserialize -# a row record, and would create a circular dependency since ``criteria/*.py`` already imports -# from this module. +# Type-infers legacy ``task.json`` records that lack ``result_kind``. Listed HERE, not +# on each criterion class: deserializing a row must not require importing every checker +# (and ``criteria/*.py`` already imports from this module). # -# Convention: when a new criterion type needs a non-``"basic"`` result class, add its -# ``criterion_type`` value to the matching frozenset in the same PR that introduces the -# criterion. +# CONVENTION: a new criterion type needing a non-``"basic"`` result class adds its +# ``criterion_type`` to the matching frozenset in the same PR. _JUDGE_CRITERION_TYPES = frozenset({"llm_judge", "agent_judge"}) _CLASSIFICATION_CRITERION_TYPES = frozenset({"classification_match", "skill_triggered"}) @@ -660,9 +653,8 @@ class EvaluationResult(BaseModel): default_factory=dict, description="Version information and environment details" ) - # Agent configuration. ResolvedAgentConfig (base-typed + SerializeAsAny + registry - # coercion) so a plugin kind's subclass-only fields survive the dump to task.json - # and reload — the same round-trip guarantee TaskDefinition.agent has. + # ResolvedAgentConfig so a plugin kind's subclass-only fields survive the dump to + # task.json and reload -- the round-trip guarantee TaskDefinition.agent has. agent_config: ResolvedAgentConfig | None = Field( default=None, description="Agent configuration used for the evaluation (from task YAML)", @@ -790,32 +782,15 @@ def armed_criteria_passed( ) -> bool: """True iff the ARMED subset's weighted score meets ``gate_threshold``. - The early-stop gate: on an early-stopped run only the armed subset gates - ``final_status`` (non-armed criteria are advisory — recorded but never - decisive), so a smoke flavor is not dragged to FAILURE by criteria whose - work it deliberately skipped. Shares the same results/criteria length - pre-check as ``all_criteria_passed`` so the gate logic stays - single-sourced. Raises ``ValueError`` on an empty armed set — unreachable - when a stop actually fired (a stop requires an armed criterion), so this - is a defensive guard against misuse. No ``is_gating`` filter is needed - here: ``BaseSuccessCriterion`` rejects ``weight: 0`` together with any - early-stop trigger, so every armed criterion is gating by construction. - - Each armed criterion's OWN ``pass_threshold`` still decides whether it - individually passed — ``r.score`` is converted to a binary 1.0/0.0 via - ``r.score >= c.pass_threshold`` before weighting, exactly mirroring - ``all_criteria_passed``'s per-criterion comparison. Only the - combination rule changes: ``all_criteria_passed`` ANDs those binary - outcomes, this weights and averages them against ``gate_threshold``. - This is what makes the ``gate_threshold=1.0`` default an EXACT - equivalence with the pre-weighting ``all(...)`` rule, not merely an - approximation that happens to hold for binary-scoring criteria: a - weighted average of 1.0 requires every armed criterion's binary - outcome to be 1.0, i.e. every one to have individually passed its own - ``pass_threshold`` — identical to ``all(...)`` regardless of what - ``r.score`` itself was. Callers pass - ``run_limits.stop_early_gate_threshold`` to opt into a genuine - weighted average below 1.0. + The gate a run the watcher actually CUT is judged by, as opposed to the + strict-AND ``all_criteria_passed`` a naturally-completed run gets. + + Raises: + ValueError: ``criteria`` does not correspond 1:1 with + ``success_criteria_results``, or no criterion is armed — the + caller must check ``early_stop is not None`` first. + + Rationale: .claude/notes/orchestration.md § The armed gate is not the watcher's bounds """ if len(self.success_criteria_results) != len(criteria): raise ValueError( @@ -830,14 +805,9 @@ def armed_criteria_passed( ) total_weight = sum(c.weight for _, c in armed) if total_weight <= 0.0: - # Unreachable today (weight=0 + a stop trigger is rejected at the model - # layer, so every armed criterion carries weight > 0) — but a - # defensive guard on a pass/fail gate must fail CLOSED, not open, - # against a future criterion subclass that bypasses that - # validator. Mirrors EarlyStopWatcher._ceiling's lack of an - # equivalent guard: that one would raise ZeroDivisionError instead - # (fails by crashing, not by silently passing) rather than diverge - # toward a false pass. + # HAZARD: unreachable today, but a defensive guard on a pass/fail gate + # must fail CLOSED against a future subclass that bypasses the + # weight=0 validator. return False weighted_score = sum((1.0 if r.score >= c.pass_threshold else 0.0) * c.weight for r, c in armed) / total_weight return weighted_score >= gate_threshold @@ -968,9 +938,7 @@ class SuiteRollup(BaseModel): rows_passed: int rows_failed: int rows_error: int - # The fourth bucket, matching RunSummary.tasks_not_graded and - # VariantAggregate.tasks_not_graded. Defaulted so a suite.json written before - # `execute` existed still parses. + # The fourth bucket. Defaulted so a suite.json predating `execute` still parses. rows_not_graded: int = Field(default=0, ge=0) pass_rate: float | None = Field( default=None, @@ -1071,24 +1039,13 @@ def row_cost_incomplete(row: Mapping[str, Any]) -> bool: def nothing_was_measured(*, not_graded: int, measured: int) -> bool: """True when a run/variant/suite has ungraded rows and NO row produced a verdict. - The one definition of "this rate has no numerator to be a fraction of", - shared by ``RunSummary``, ``VariantAggregate`` and ``SuiteRollup`` because - three copies of a published rate is how one surface reports ``n/a`` and - another reports ``0.0%`` for the same run. That already happened: the guard - shipped on ``RunSummary`` only, so a 10-task ``execute`` run with one crash - rendered ``Pass Rate: n/a`` in ``run.md`` and ``Pass Rate: 0.0%`` in - ``experiment.md``. - - ``measured`` is COUNTED EVIDENCE — rows that actually carry a criteria - verdict — not a bucket count. The first version of this test used - ``tasks_succeeded + tasks_failed == 0`` and was wrong for the same reason - the bug it fixed was wrong: ``TIMEOUT`` and the two budget stops are - category ``failed`` and reachable under ``execute`` (``_check_run_limits`` - still runs on the ungraded branch), so ONE timed-out row in a 100-task - ungraded night read as "something was measured" and published - ``pass_rate: 0.0`` — a real 0% point on the evalboard trend for a run that - graded nothing. A bucket is where a row landed; only the verdict says - whether a criterion ever ran. + The one definition of "this rate has no numerator to be a fraction of", shared by + ``RunSummary``, ``VariantAggregate`` and ``SuiteRollup``. + + ``measured`` is COUNTED EVIDENCE -- rows that actually carry a criteria verdict -- + not a bucket count. + + Rationale: .claude/notes/orchestration.md § Rates need verdict evidence, not bucket counts """ return not_graded > 0 and measured == 0 @@ -1199,8 +1156,7 @@ class RunSummary(BaseModel): tasks_failed: int = Field(description="Number of tasks that failed") tasks_error: int = Field(description="Number of tasks that encountered errors") # Part of the task_count invariant (a fourth bucket, not a sub-counter), but - # defaulted so run.json written before `coder-eval execute` existed — where - # no task can be ungraded — still deserialises. + # defaulted so run.json predating `execute` still deserialises. tasks_not_graded: int = Field( default=0, ge=0, @@ -1210,13 +1166,9 @@ class RunSummary(BaseModel): ), ) - # Verdict evidence, NOT a bucket and NOT part of the invariant: how many - # rows actually carry a weighted_score. `pass_rate` / `error_share` need it - # because the four category buckets cannot tell a graded FAILURE from a - # TIMEOUT that no criterion ever saw. Symmetric with - # VariantAggregate.tasks_measured so the three published rates read the same - # input. Defaulted for old run.json, where tasks_not_graded is 0 and the - # gate is therefore inert. + # Verdict evidence, NOT a bucket and NOT part of the invariant. Symmetric with + # VariantAggregate.tasks_measured so the three published rates read one input. + # Rationale: .claude/notes/orchestration.md § Rates need verdict evidence, not bucket counts tasks_measured: int = Field( default=0, ge=0, @@ -1226,9 +1178,7 @@ class RunSummary(BaseModel): ), ) - # Informational sub-counters: subsets of tasks_failed (NOT part of the - # task_count invariant). Default 0 so old serialized RunSummary JSON - # without these fields deserialises cleanly. + # Informational sub-counters: subsets of tasks_failed, NOT part of the invariant. tasks_token_budget_exceeded: int = Field( default=0, ge=0, @@ -1240,10 +1190,8 @@ class RunSummary(BaseModel): description="Subset of tasks_failed where run_limits cost cap tripped.", ) - # Tasks excluded at resolution time — either load failures (YAML / schema - # errors) or intentional opt-outs (``skip: true``). Distinct from - # tasks_error: these never reached the orchestrator. Empty for runs - # where every task YAML loaded cleanly and none opted out. + # Excluded at resolution time (load failure or `skip: true`). Distinct from + # tasks_error: these never reached the orchestrator. skipped_tasks: list[SkippedTask] = Field( default_factory=list, description=( @@ -1312,10 +1260,9 @@ def _nothing_was_measured(self) -> bool: """ return nothing_was_measured(not_graded=self.tasks_not_graded, measured=self.tasks_measured) - # Derived run metrics: computed_fields over the stored counts and - # ``task_results``, so they serialize into run.json while staying impossible to - # set to something the rows disagree with. Consumers should read these rather - # than re-derive them. + # computed_fields over the stored counts, so they serialize into run.json while + # staying impossible to set to something the rows disagree with. Read these + # rather than re-deriving them. @computed_field # type: ignore[prop-decorator] @property diff --git a/src/coder_eval/models/routing.py b/src/coder_eval/models/routing.py index 4dc5e0a02..059543ed6 100644 --- a/src/coder_eval/models/routing.py +++ b/src/coder_eval/models/routing.py @@ -13,11 +13,10 @@ from coder_eval.config import Settings -# Resolved-at-startup transport for the `llm_judge` criterion under DirectRoute. -# - "anthropic": call api.anthropic.com via the Anthropic SDK (needs ANTHROPIC_API_KEY). -# - None: no ANTHROPIC_API_KEY; any enabled `llm_judge` under DirectRoute fails at -# dispatch. The Bedrock backend routes the judge through the run's own backend -# and never reaches this transport selection. +# Resolved-at-startup transport for the `llm_judge` criterion under DirectRoute: +# "anthropic" when ANTHROPIC_API_KEY is present, None otherwise (an enabled +# llm_judge then fails at dispatch). Bedrock never reaches this selection. +# Rationale: .claude/notes/contracts.md § Route resolution JudgeTransport = Literal["anthropic"] @@ -87,9 +86,8 @@ class DirectRoute: """ judge_transport: JudgeTransport | None = "anthropic" - # Unlike BedrockRoute/LiteLLMRoute, the AGENT never reads this — the Claude Agent - # SDK picks its own default when unset. It exists so ``checker_context.api_route.model`` - # (see resolve_evaluation_route) has somewhere to land when the eval side is on Direct. + # The AGENT never reads this -- the SDK picks its own default. It exists so + # ``checker_context.api_route.model`` has somewhere to land on Direct. model: str | None = None @@ -118,42 +116,22 @@ class BedrockRoute: @dataclass(frozen=True) class LiteLLMRoute: """Route through a custom endpoint — either the AGENT's own LiteLLM proxy - (an Anthropic-compatible gateway fronting Bedrock open-weight models), or, - on the CHECKER side (``checker_context.api_route.route: litellm``), an - arbitrary provider reached through the ``litellm`` library directly. - - AGENT side: the Claude Code SDK is pointed at the gateway via - ``ANTHROPIC_BASE_URL``/``ANTHROPIC_AUTH_TOKEN``. Deliberately carries NO - ``base_url``/credential field for this — same reasoning as ``BedrockRoute``'s - docstring: this route object flows through orchestrator state - (``environment_info`` recording, logging) that has no business handling - config that should always be read live from the environment. - ``ClaudeCodeAgent._build_sdk_env`` reads ``settings.litellm_base_url``/ - ``settings.litellm_auth_token`` itself, the same source ``resolve_route`` - validated before constructing this route. - - CHECKER side (``invoke_litellm_judge_async``): unlike the agent path, this - is NOT sourced from ``coder_eval.config.settings`` at all — the task author - fully owns it via ``params``/``env_params`` below (a gateway-routed judge - model rarely reuses the same proxy/credential the AGENT's own LiteLLM - backend points at). There is no implicit fallback to - ``settings.litellm_base_url``/``settings.litellm_auth_token``; if the - provider needs ``api_base``/``api_key``, the task author sets them via - ``params``/``env_params`` like any other kwarg. - - ``params``/``env_params`` (checker side only, from - ``checker_context.api_route.{params,env_params}``): ``litellm.acompletion`` - takes dozens of provider-specific kwargs (``api_base``, ``api_key``, - ``aws_access_key_id``, ``vertex_project``, ``api_version``, ...) that this - route has no dedicated field for. ``params`` is passed through verbatim as - extra kwargs. ``env_params`` maps a kwarg name to the ENV VAR NAME to - resolve it from at call time — e.g. ``{api_key: LITELLM_AUTH_TOKEN, - aws_access_key_id: AWS_ACCESS_KEY_ID}`` — so an arbitrary provider's config - (including secrets) is representable without a secret ever landing in the - task YAML. Both are ``None`` unless a task author set them; ``env_params``'s - values are env var *names*, never secrets, so it is safe to record verbatim - in ``environment_info`` (unlike ``params``, which a task author could — but - shouldn't — put a raw secret into). + (an Anthropic-compatible gateway fronting Bedrock open-weight models), or, on + the CHECKER side (``checker_context.api_route.route: litellm``), an arbitrary + provider reached through the ``litellm`` library directly. + + Carries NO ``base_url``/credential field: this route object flows through + orchestrator state that has no business handling config which should be read + live from the environment. + + ``params``/``env_params`` (checker side only) cover the provider-specific + kwargs this route has no dedicated field for. ``params`` is passed to + ``litellm.acompletion`` verbatim; ``env_params`` maps a kwarg name to the ENV + VAR NAME to resolve it from at call time, so a provider's config — secrets + included — is representable without a secret landing in the task YAML. Only + ``env_params`` is safe to record verbatim. + + Rationale: .claude/notes/contracts.md § LiteLLM params and env_params """ model: str | None = None @@ -197,17 +175,11 @@ def resolve_route(settings: Settings) -> ApiRoute: case ApiBackend.BEDROCK: assert settings.aws_bearer_token_bedrock is not None, "Bedrock requires aws_bearer_token_bedrock" assert settings.aws_region is not None, "Bedrock requires aws_region" - # BEDROCK_MODEL is the only route-level model source. CLI --model / - # -D agent.model and task-YAML agent.model are resolved later in the - # agent layer (via _resolve_effective_model), which also handles - # the anthropic.* + region prefix qualification on bare aliases. - # Fall back to the main model when no small/fast model is configured. - # Claude Code routes WebFetch's page-summarization (and other "small, - # fast" steps) through ANTHROPIC_SMALL_FAST_MODEL; on Bedrock that env - # var is only exported when small_model is set (see - # ClaudeCodeAgent._build_sdk_env). Leaving it unset made every - # WebFetch fail with "model issues" under the Bedrock backend. The main - # model is always a valid fallback, so default to it. + # BEDROCK_MODEL is the only route-level model source; agent.model is + # resolved later in the agent layer. Falling back to the main model is + # load-bearing: ANTHROPIC_SMALL_FAST_MODEL is exported only when + # small_model is set, and leaving it unset made every WebFetch fail. + # Rationale: .claude/notes/contracts.md § Route resolution model, small_model = _bedrock_model_pair( settings.bedrock_model, settings.bedrock_small_model, settings.aws_region ) @@ -215,10 +187,8 @@ def resolve_route(settings: Settings) -> ApiRoute: case ApiBackend.DIRECT: return DirectRoute(judge_transport=_resolve_direct_judge_transport(settings)) case ApiBackend.LITELLM: - # Validate here (raise, not assert): resolve_route is reached on the - # evaluate-only path WITHOUT a preceding validate_api_keys(), so this is - # the only guard there and must survive `python -O`. Checks presence + - # URL scheme, raising a field-named ValueError (review non-blocking #11). + # Raise, not assert: reached on the evaluate-only path without a + # preceding validate_api_keys(), so it must survive `python -O`. settings._validate_litellm_settings() # Narrowing for pyright only — _validate_litellm_settings guarantees these. assert settings.litellm_base_url is not None @@ -230,8 +200,7 @@ def resolve_route(settings: Settings) -> ApiRoute: small_model=small_model, ) case _: - # ApiBackend covers exactly BEDROCK/DIRECT/LITELLM above; this arm is - # unreachable but makes the match exhaustive so every path returns + # Unreachable, but keeps the match exhaustive so every path returns # explicitly (CodeQL: mixed explicit/implicit returns). raise AssertionError(f"unhandled ApiBackend: {settings.api_backend!r}") @@ -246,26 +215,10 @@ def _resolve_backend_route( ) -> ApiRoute: """Build the ``ApiRoute`` for an EXPLICITLY-requested backend. - Used only by the ``checker_context.api_route`` override path (see - ``resolve_evaluation_route``): raises ``ValueError`` naming the missing env - var when that backend isn't configured, rather than silently falling back - to a different backend — an explicit override that can't be honored must - fail loudly, not degrade to a backend the task author didn't ask for. - - ``model_override`` (``checker_context.api_route.model``) wins over the - backend's own env-configured default model when set. - - ``ApiBackend.LITELLM`` is the one exception to "env-sourced ``Settings`` - fields, credentials always come from the environment": unlike - BEDROCK/DIRECT (which reuse the agent's own env-configured credentials, since - grading still needs to reach the SAME Claude backend), a checker-side litellm - route is not assumed to share the agent's LiteLLM proxy/gateway at all — it - is built ENTIRELY from ``params_override``/``env_params_override`` - (``checker_context.api_route.{params,env_params}``), never from - ``settings.litellm_base_url``/``settings.litellm_auth_token``. Those two - settings fields are the AGENT's own LiteLLM-backend config (see - ``resolve_route``) — reusing them here would silently point the judge at - infrastructure the task author never named. + Raises rather than asserts on a missing credential: this is reached on the + evaluate-only path with no preceding key validation, so it must survive ``-O``. + + Rationale: .claude/notes/contracts.md § Route resolution """ match backend: case ApiBackend.BEDROCK: @@ -293,9 +246,8 @@ def _resolve_backend_route( env_params=env_params_override, ) case _: - # ApiBackend covers exactly BEDROCK/DIRECT/LITELLM above; this arm is - # unreachable but makes the match exhaustive so every path returns - # explicitly (CodeQL: mixed explicit/implicit returns, PR #137 review). + # Unreachable, but keeps the match exhaustive so every path returns + # explicitly (CodeQL: mixed explicit/implicit returns). raise AssertionError(f"unhandled ApiBackend: {backend!r}") @@ -309,38 +261,17 @@ def resolve_evaluation_route( env_params_override: dict[str, str] | None = None, ) -> ApiRoute: """Resolve the route used by the *evaluation* side — the ``llm_judge`` / - ``agent_judge`` criteria and the simulated user — which must stay on a - constant Claude backend regardless of the agent under test, so grading and - simulation stay comparable across models. - - All overrides come from the reserved ``checker_context.api_route`` namespace - (see ``TaskDefinition.checker_context``) — ``route`` (``backend_override``) - picks the backend, ``model`` (``model_override``) picks the model on - whichever route is resolved, and ``params``/``env_params`` (``params_override``/ - ``env_params_override``) only ever apply when ``backend_override`` resolves to - ``litellm`` (see ``_resolve_backend_route``). Criteria never read any of - these directly; they only ever see the resulting ``CheckContext.route``. - - - ``backend_override`` set: build that backend's route from env, regardless - of ``agent_route`` — an explicit task/variant choice always wins. Raises - ``ValueError`` if the string isn't a known ``ApiBackend`` or that backend - isn't configured (see ``_resolve_backend_route``). - - Agent on Bedrock/Direct (no ``backend_override``): the judge already runs - on Claude via that route, so reuse it — except its ``model`` is always - reset to ``model_override`` (``None`` when unset). The agent's own - env-sourced model (e.g. ``BEDROCK_MODEL``) must NOT leak into the judge's - default: ``route.model`` must mean "an explicit override was given", not - "whatever the agent happens to be using" — otherwise an unpinned judge - silently starts grading with a different model whenever the agent's - model changes, breaking before/after comparability (PR #137 review: - "the judge loses DEFAULT_JUDGE_MODEL as its floor"). - - Agent on LiteLLM (open-weight, no ``backend_override``): the agent route - cannot serve a Claude judge, so pin evaluation to Bedrock (preferred, from - the AWS bearer token) or Direct (``ANTHROPIC_API_KEY``), honoring - ``model_override`` there too — same "no override, no baked-in model" rule - as above. If neither backend is configured, fall back to a ``DirectRoute`` - with no judge transport so ``llm_judge`` fails with its clean - "unconfigured" error rather than silently scoring 0.0. + ``agent_judge`` criteria — which is resolved SEPARATELY from the agent's own. + + An explicit ``backend_override`` always wins. Otherwise a Bedrock or Direct + agent route is REUSED (with ``model_override`` applied), but a LiteLLM one is + NOT: evaluation is pinned to a constant Claude backend instead — Bedrock when + the credentials are present, else Direct. + + ``model_override`` lands on ``checker_context.api_route.model`` only when a + real override was given, never the agent's own model. + + Rationale: .claude/notes/contracts.md § Route resolution """ if backend_override is not None: try: diff --git a/src/coder_eval/models/sandbox.py b/src/coder_eval/models/sandbox.py index b3f3d569b..f795d0382 100644 --- a/src/coder_eval/models/sandbox.py +++ b/src/coder_eval/models/sandbox.py @@ -223,51 +223,38 @@ class DockerDriverConfig(BaseModel): "AWS_BEARER_TOKEN_BEDROCK", "AWS_REGION", "BEDROCK_MODEL", - # Claude Code SDK Bedrock toggle + optional model override; required - # alongside AWS_BEARER_TOKEN_BEDROCK to route the in-container SDK - # through Bedrock instead of falling back to ~/.claude OAuth. + # Required alongside AWS_BEARER_TOKEN_BEDROCK to route the in-container + # SDK through Bedrock instead of ~/.claude OAuth. "CLAUDE_CODE_USE_BEDROCK", "ANTHROPIC_MODEL", - # LiteLLM (Anthropic-compatible) open-weight backend. The proxy runs on - # the HOST, so LITELLM_BASE_URL is rewritten to host.docker.internal at - # the container boundary (see docker_runner); the rest forward verbatim. - # Without these the in-container Settings sees API_BACKEND=litellm with no - # creds and _validate_litellm_settings raises a hard ValueError. + # The proxy runs on the HOST, so LITELLM_BASE_URL is rewritten at the + # container boundary; the rest forward verbatim. + # Rationale: .claude/notes/isolation.md § Environment forwarding "LITELLM_BASE_URL", "LITELLM_AUTH_TOKEN", "LITELLM_MODEL", "LITELLM_SMALL_MODEL", - # Path to the proxy's per-call cost log for the actual-cost join. NOTE: - # forwarding the var is necessary but not sufficient under --driver docker - # — the log file itself must also be bind-mounted into the container for - # the join to see it (follow-up); without the mount, docker runs keep - # static pricing while local runs get real cost. + # HAZARD: forwarding the var is necessary but not sufficient under + # --driver docker -- the log file must also be bind-mounted. "LITELLM_COST_LOG", - # Codex agent auth/routing — without these the in-container codex - # binary falls back to a ChatGPT login that doesn't exist in the - # container and auth fails. CODEX_API_KEY drives login_api_key; - # CODEX_BASE_URL routes to a custom endpoint (e.g. gateway); - # CODEX_MODEL selects the model when agent.model is unset. + # Without these the in-container codex binary falls back to a ChatGPT + # login that does not exist there. CODEX_MODEL applies when agent.model + # is unset. "CODEX_API_KEY", "CODEX_BASE_URL", "CODEX_MODEL", - # Antigravity agent auth/routing — the google-antigravity local harness - # authenticates against the Gemini API with GEMINI_API_KEY; without it - # the in-container harness has no credential and fails. ANTIGRAVITY_MODEL - # selects the Gemini model when agent.model is unset. + # The Antigravity harness authenticates with GEMINI_API_KEY; without it + # the in-container harness has no credential. ANTIGRAVITY_MODEL applies + # when agent.model is unset. "GEMINI_API_KEY", "ANTIGRAVITY_MODEL", - # Pi agent provider credential — Pi addresses models as `provider/id` - # and reads OpenRouter's key from the env. The baked pi CLI + this - # passthrough make `--driver docker --type pi` work; without it the - # in-container pi has no credential and every turn fails auth. + # Pi addresses models as `provider/id` and reads OpenRouter's key from + # the env; without it every in-container turn fails auth. "OPENROUTER_API_KEY", - # User HOME used to keep ~/.claude resolution symmetric with the host. - # See docs/DOCKER_ISOLATION.md "HOME is forwarded by default" for the - # contract. tl;dr: Path.home() inside the container returns the - # host's HOME (the dir is auto-created by the ~/.claude bind mount); - # writes outside ~/.claude land in the container's ephemeral rootfs. - # Remove this entry if you don't want host HOME leakage. + # HAZARD: keeps ~/.claude resolution symmetric with the host, so + # Path.home() in the container returns the host's HOME. Remove this + # entry if you do not want that leakage. Contract: + # docs/DOCKER_ISOLATION.md § `HOME` is forwarded by default "HOME", ], description=( @@ -313,24 +300,20 @@ def _validate_working_dir(cls, v: str | None) -> str | None: return v -# Sandbox-relative location of the generated CLI recorders and their shared log. -# Not dot-prefixed on purpose: CI artifact upload (actions/upload-artifact) skips -# hidden files, and the log is primary evidence for every `cli_called` criterion, -# so it must survive into the run artifact. +# Not dot-prefixed on purpose: CI artifact upload skips hidden files, and the log is +# primary evidence for every `cli_called` criterion. +# Rationale: .claude/notes/contracts.md § What the shim is, and is not RECORD_CLI_DIR = "cli_mocks" RECORD_CLI_LOG_NAME = "calls.jsonl" RECORD_CLI_LOG = f"{RECORD_CLI_DIR}/{RECORD_CLI_LOG_NAME}" -# Modules copied into the recorder directory beside each shim that declares -# response rules. The shim imports them as siblings, so they must be -# stdlib-only (lint rule CE057) -- they run where coder_eval is not installed. +# Copied beside each shim that declares response rules and imported as siblings, so +# they must be stdlib-only (CE057) -- they run where coder_eval is not installed. SIDECAR_MODULES: tuple[str, ...] = ("argv_match.py",) -# Shadowing any of these breaks the harness rather than the tool under test: the -# shim is a script run by an interpreter, and its directory goes FIRST on a PATH -# the orchestrator also reuses for run_command criteria. `tool: python3` made the -# shim re-resolve its own interpreter to itself -- an exec loop that spins to the -# task timeout, since tempdir enforces no pid cap. +# HAZARD: shadowing an interpreter breaks the harness rather than the tool under +# test -- `tool: python3` made the shim re-resolve its interpreter to itself. +# Rationale: .claude/notes/contracts.md § What the shim is, and is not RECORD_CLI_RESERVED_TOOLS = frozenset( {"python", "python3", "py", "env", "sh", "bash", "zsh", "cmd", "node", "uv", "git"} ) @@ -373,23 +356,17 @@ class CliResponse(BaseModel): class RecordedCli(BaseModel): """One executable to shadow with a generated recording shim. - The shim records the invocation, writes the configured output, and exits — - nothing is executed, so there is no network, no auth, and no side effect. Each + The shim records the invocation, writes the configured output, and exits -- + nothing is executed, so there is no network, no auth and no side effect. Each invocation becomes a JSON Lines record in :data:`RECORD_CLI_LOG`, the log the - ``cli_called`` criterion reads by default, so a task asserts on what actually - ran without hand-rolling a mock and without the record shape being a contract - between two repositories. + ``cli_called`` criterion reads by default. The fields below are what every invocation gets; ``responses`` overrides them - per invocation, so one shadowed ``uip`` can answer ``ixp dummy1`` and - ``ixp dummy2`` differently — what an agent needs when its next step depends on - what the tool just told it. - - It stubs a tool; it does not proxy one. A test that needs a REAL executable's - behavior recorded on the way through still supplies its own wrapper under - ``mock_path_dirs`` — that depends on the tool being installed, on PATH order, - and usually on live credentials, which is a different problem with different - failure modes. + per invocation, so one shadowed ``uip`` can answer two verbs differently. + + It stubs a tool; it does not proxy one. + + Rationale: .claude/notes/contracts.md § What the shim is, and is not """ model_config = ConfigDict(extra="forbid") @@ -419,10 +396,8 @@ class RecordedCli(BaseModel): "would, so an agent reads a plausible error rather than silence" ), ) - # Plain Field, not MergeField: `RecordedCli` is never a merge root. The - # enclosing `SandboxConfig.record_cli` is a `replace` list, so a later layer - # substitutes the whole list of entries and no per-entry strategy is ever - # consulted. A strategy annotation here would read as a knob and be inert. + # Plain Field, not MergeField: `RecordedCli` is never a merge root, so a + # strategy annotation here would read as a knob and be inert. responses: list[CliResponse] = Field( default_factory=list, description=( @@ -456,13 +431,9 @@ def _validate_responses_are_reachable(self) -> RecordedCli: elif ( prior["positional"] is None and prior["flags"] is None - # BOTH sides free of flag predicates, not just the earlier - # one: a predicate makes its flag known and value-bearing in - # that rule's parse only. `--profile prod ixp projects get` - # leaves `prod` positional for a verb-only `ixp projects`, - # which therefore does NOT match, while a later - # `ixp projects get` + `flags: {profile: prod}` does -- so the - # later rule is reachable and rejecting it was wrong. + # BOTH sides free of flag predicates: a predicate makes its + # flag known and value-bearing in that rule's parse only, so a + # later rule declaring one is genuinely reachable. and spec["flags"] is None and prior["value_flags"] == spec["value_flags"] and prior["ignore_flags"] == spec["ignore_flags"] @@ -504,10 +475,9 @@ def _validate_responses_are_evaluable(self) -> RecordedCli: if not self.responses: return self - # Building the probe argvs reads the same spec the matcher will, so it sits - # INSIDE the guard: a spec malformed enough to break this loop is exactly - # the kind that must surface as a clean authoring error, not a TypeError - # escaping a validator. + # INSIDE the guard: building the probes reads the same spec the matcher + # will, and a spec malformed enough to break it must surface as a clean + # authoring error rather than a TypeError escaping a validator. try: rules = [ {"when": response.when.match_spec, "exit": response.exit_code, "stdout": "", "stderr": ""} @@ -550,15 +520,12 @@ def validate_tool_name(cls, v: str) -> str: + f"Reserved: {reserved}" ) raise ValueError(msg) - # Folded for the same reason as the reserved set: on a case-insensitive - # filesystem `CALLS.JSONL` is the seeded log, and the shim write would hit - # it -- reported as a confusing duplicate-filename error at setup instead. + # Case-folded: on a case-insensitive filesystem `CALLS.JSONL` is the + # seeded log. if v.lower() == RECORD_CLI_LOG_NAME: raise ValueError(f"record_cli tool {v!r} would overwrite the invocation log criteria read") - # Case-folded like the reserved check above: APFS and NTFS are - # case-insensitive, so `ARGV_MATCH.PY` names the same inode as the - # sidecar. The sidecar write would then clobber the agent's shim without - # `_generate_cli_recorders`' per-tool exists() guard ever firing. + # Case-folded: on APFS and NTFS `ARGV_MATCH.PY` names the sidecar's own + # inode, so the sidecar write would clobber the agent's shim. if v.lower() in {module.lower() for module in SIDECAR_MODULES}: names = ", ".join(sorted(SIDECAR_MODULES)) msg = ( diff --git a/src/coder_eval/models/tasks.py b/src/coder_eval/models/tasks.py index f1f36cf12..24b4f175a 100644 --- a/src/coder_eval/models/tasks.py +++ b/src/coder_eval/models/tasks.py @@ -150,9 +150,8 @@ class SimulationConfig(BaseModel): enabled: bool = Field(default=False, description="Master switch — when false, simulation is skipped entirely.") - # The simulator runs as a tools-disabled Claude Code agent sharing the coding - # agent's ApiRoute, so temperature/max_tokens are resolved at the route level and - # are not configured here. The MODEL is pinned below rather than inherited. + # The simulator shares the coding agent's ApiRoute, so temperature/max_tokens + # resolve at the route level. The MODEL is pinned below rather than inherited. model: str = Field( default=DEFAULT_SIMULATOR_MODEL, description=( @@ -471,10 +470,9 @@ class TaskDefinition(BaseModel): # noqa: CE009 -- soft-launch: see _warn_on_unk "the YAML — pair with a comment citing the blocker (e.g. Jira link, upstream dependency)." ), ) - # ResolvedAgentConfig = base-typed + registry-driven dict coercion + SerializeAsAny: - # the concrete subclass (built-in or plugin) is chosen by parse_agent_config, not a - # static discriminated union, so any registered plugin kind validates here and its - # subclass-only fields (e.g. sdk_options) survive model_dump(). + # ResolvedAgentConfig: the concrete subclass is chosen by parse_agent_config, not + # a static union, so a plugin kind validates and its subclass-only fields survive + # model_dump(). Rationale: .claude/notes/agents.md § The sdk_options pass-through agent: ResolvedAgentConfig | None = Field( default=None, description=( @@ -706,11 +704,8 @@ def check_reference_consumers_have_a_reference(self) -> Self: return self offenders: list[str] = [] for c in self.success_criteria: - # isinstance narrowing, NOT getattr(c, "files"/"command"): with an - # untyped string probe, renaming LLMJudgeCriterion.files or - # RunCommandCriterion.command turns this load-time guard into a - # silent no-op that pyright cannot see. The union members are - # imported here already. + # isinstance narrowing, NOT a getattr string probe: a rename would + # otherwise turn this load-time guard into a no-op pyright cannot see. if isinstance(c, ReferenceComparisonCriterion): offenders.append(f"{c.type} (needs a reference to compare against)") elif isinstance(c, LLMJudgeCriterion | AgentJudgeCriterion) and any( @@ -718,12 +713,10 @@ def check_reference_consumers_have_a_reference(self) -> Self: ): offenders.append(f"{c.type} (files: uses {REFERENCE_DIR_TOKEN})") elif isinstance(c, RunCommandCriterion) and command_uses_token(c.command, REFERENCE_DIR_TOKEN): - # run_command is the third documented consumer: with no reference - # the env var is simply absent, so `diff -r "$REFERENCE_DIR" out/` - # expands to an empty argument and misbehaves instead of failing. # command_uses_token, not a raw `in`: the brace form - # `${REFERENCE_DIR}` is standard shell and slipped straight past - # a substring test, while `$REFERENCE_DIRECTORY` false-positived. + # `${REFERENCE_DIR}` slipped past a substring test while + # `$REFERENCE_DIRECTORY` false-positived. Without the reference the + # var is simply absent, so the command misbehaves instead of failing. offenders.append(f"{c.type} (command: uses {REFERENCE_DIR_TOKEN})") if offenders: raise ValueError( @@ -741,9 +734,8 @@ def check_suite_thresholds_require_dataset(self) -> Self: """ if self.dataset is None and self.suite_id is None: for c in self.success_criteria: - # Direct attribute access, not getattr: suite_thresholds is - # declared on BaseSuccessCriterion, so every union member has it - # and pyright can see a rename. + # Direct attribute access, not getattr: declared on + # BaseSuccessCriterion, so pyright can see a rename. if c.suite_thresholds: raise ValueError( f"success_criteria[{c.type!r}].suite_thresholds requires a dataset: block " @@ -783,11 +775,9 @@ def check_removed_criteria_types(cls, v: Any) -> Any: if ctype in REMOVED_CRITERION_TYPES: raise ValueError(f"Criterion type '{ctype}' has been removed. {REMOVED_CRITERION_TYPES[ctype]}") if ctype in NORMALIZED_CRITERION_ALIASES: - # `{**item, **overlay}` — the overlay WINS, and the order is - # load-bearing. A legacy `command_not_executed` carrying an explicit - # `min_count: 2` is still asking for "this was NOT run"; letting that - # count survive would yield a criterion passing exactly when the - # original asked it to fail. + # HAZARD: the overlay WINS, and the order is load-bearing. A + # legacy `command_not_executed` with `min_count: 2` still + # means "was NOT run"; letting that count survive inverts it. item = {**item, **NORMALIZED_CRITERION_ALIASES[ctype]} normalized.append(item) return normalized diff --git a/src/coder_eval/models/telemetry.py b/src/coder_eval/models/telemetry.py index c48614be1..6015c9563 100644 --- a/src/coder_eval/models/telemetry.py +++ b/src/coder_eval/models/telemetry.py @@ -325,19 +325,12 @@ class ReconciliationMessage(BaseModel): """A synthetic transcript entry carrying tokens the agent billed but never surfaced as a streamed generation. - The authoritative turn total (Claude's ``ResultMessage.model_usage``, Codex's - thread total + folded sub-agent tokens) is consistently LARGER than the sum - of the per-``AssistantMessage`` token buckets, for reasons that are - architectural, not bugs: a fixed prompt slice (~512 input tokens on Claude) - is billed but rides on no SDK-emitted message, and sub-agent input/cache is - only partially bubbled into the parent stream. Rather than fabricate - per-generation numbers (which would match no real call), the residual is - booked once, explicitly, as this entry — so that summing the transcript's - token buckets EXACTLY reproduces the authoritative ``token_usage`` for the - turn. Consumers that sum the stream (the evalboard) therefore reconcile to - the bill without a separate aggregate; this entry is the visible "missing - tokens" line. Carries no cost (cost stays on the authoritative aggregate) and - is never an LLM generation — it is excluded from generation/turn counts. + Booking the residual once, explicitly, is what makes summing the transcript's + token buckets EXACTLY reproduce the authoritative ``token_usage`` for the turn. + Carries no cost and is never an LLM generation, so it is excluded from + generation and turn counts. + + Rationale: .claude/notes/agents.md § Token accounting and the reconciliation message """ role: Literal["reconciliation"] = "reconciliation" @@ -380,11 +373,9 @@ class CommandTelemetry(BaseModel): ), ) - # Timing: generation_completed_at is when Claude finished emitting the - # tool_use block; execution_* bracket the actual tool run. `timestamp` - # equals generation_completed_at and `duration_ms` equals - # execution_completed_at - execution_started_at; both are retained for - # downstream consumers that index on the original field names. + # generation_completed_at is when the tool_use block finished emitting; + # execution_* bracket the run. `timestamp` and `duration_ms` are retained for + # consumers that index on the original field names. timestamp: datetime = Field(description="Equal to generation_completed_at; when the tool_use block arrived.") duration_ms: float | None = Field( default=None, @@ -439,34 +430,24 @@ class CommandTelemetry(BaseModel): def result_tokens(self) -> int: """Approximate token size of the tool result the model received. - Derived from the untruncated ``result_summary`` content (≈4 chars/token, - matching the evalboard heuristic), so it is a DIRECT, cache-independent - measure of "tool output size" — available identically whether prompt - caching was on or off. The cost simulator uses this instead of inferring - result size from prompt-cache growth (which is unavailable when caching is - disabled). Approximate, not the API's exact tokenizer count, but - deterministic and always present. 0 when the tool returned no content. - - This measure is only meaningful while ``result_summary`` stays whole: an - agent that truncates a command's output before recording it (as the Codex - agent once did with ``output[:100]``) silently under-reports that command's - result. Lint rule CE043 forbids truncating captured command output - (stdout/stderr) in the agents, so the "untruncated" contract holds for - command results across agents. (One-line summaries of non-command tool - items — e.g. collab/MCP status lines built by ``_summarize_tool_item`` — - are intentionally brief and out of scope.) Trim for DISPLAY in the - renderers/reports instead. + Derived from the UNTRUNCATED ``result_summary`` (~4 chars/token), so it is a + direct, cache-independent measure of tool output size. Approximate, but + deterministic and always present; 0 when the tool returned no content. + + Only meaningful while ``result_summary`` stays whole -- lint rule CE043 + forbids truncating captured command output. Trim for DISPLAY in the + renderers instead. + + Rationale: .claude/notes/agents.md § The result_tokens measure and CE043 """ if not self.result_summary: return 0 return -(-len(self.result_summary) // 4) # ceil(len / 4) -#: One entry in a turn's per-message transcript, discriminated on ``role``. -#: ``ReconciliationMessage`` is the synthetic "missing tokens" line that makes the -#: transcript's token buckets sum to the authoritative turn total. Defined once -#: here and reused by ``TurnRecord.messages`` and ``AgentEndEvent.messages`` so the -#: list element type is identical everywhere (avoids list-invariance friction). +#: One entry in a turn's per-message transcript, discriminated on ``role``. Defined +#: once here and reused by ``TurnRecord.messages`` and ``AgentEndEvent.messages`` so +#: the element type is identical everywhere (avoids list-invariance friction). TranscriptMessage = Annotated[UserMessage | AssistantMessage | ReconciliationMessage, Discriminator("role")] diff --git a/src/coder_eval/models/templates.py b/src/coder_eval/models/templates.py index 1d9c741fa..d99dbbfb1 100644 --- a/src/coder_eval/models/templates.py +++ b/src/coder_eval/models/templates.py @@ -27,11 +27,9 @@ class BaseTemplateSource(BaseModel, ABC): model_config = ConfigDict(extra="forbid") def model_post_init(self, context: Any, /) -> None: - # Pin the discriminator tag into model_fields_set so it survives - # model_dump(exclude_unset=True) → model_validate() round-trips (the - # sandbox layer merge dumps with exclude_unset) even for - # directly-constructed sources, where the tag comes from the Literal - # default rather than the caller. + # Pin the tag into model_fields_set so it survives a + # model_dump(exclude_unset=True) -> model_validate() round trip even for + # directly-constructed sources. self.__pydantic_fields_set__.add("type") @@ -98,10 +96,9 @@ class RepoSource(BaseTemplateSource): commit: str | None = Field(default=None, description="Specific commit SHA to checkout") -# Discriminated union of template sources. The `type` tag is REQUIRED in -# dict/YAML input — a missing or unknown tag raises one crisp discriminator -# error instead of smart-union coercion. Per-variant Literal defaults remain -# for direct construction and serialization. +# The `type` tag is REQUIRED in dict/YAML input: a missing or unknown one raises a +# crisp discriminator error instead of smart-union coercion. Per-variant Literal +# defaults remain for direct construction and serialization. TemplateSource = Annotated[ TemplateDirSource | StarterFilesSource | RepoSource, Field(discriminator="type"), diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 3f16b659f..b6376a583 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 36_569 +_ESSAY_BASELINE_WORDS = 25_468 _SRC = Path("src/coder_eval") From 5aa3ded034355fff1f053df21e93235c3fd3e265 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 21:03:29 -0700 Subject: [PATCH 07/19] =?UTF-8?q?docs:=207/7=20=E2=80=94=20move=20reportin?= =?UTF-8?q?g,=20harbor=20and=20telemetry=20rationale=20into=20.claude/note?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 10,234 words across 30 files down to about 2,900, finishing the plan. The tree goes from 79,754 essay-shaped words to 18,163, and from 126 docstrings over 150 words to one. That one is agent.py::communicate, the plugin SPI contract every third-party agent author reads: the Args/Returns/Raises, the rule that a mid-turn failure sets pending_turn before raising and the caller rolls the counter back, and the streaming event protocol. It is the exemption the plan predicted, and the only one taken, so the 150-word bar held for the whole tree. reporting.md gains the Agent ABC contract, telemetry emission, cost joining, report rollups and the HTML twin, Harbor export, and the ATIF trajectory bridge. Two hazards the plan named had to be ADDED rather than kept: neither the pricing.py/pricing.ts mirror nor the "evalboard's static twin" parity claim was ever stated in its own source file — both lived only in CLAUDE.md. Editing one side of either pair without the other is exactly what they exist to prevent, so they now say so where the editor will be. errors/categories.py is untouched: its per-member retryability notes are the contract errors/categorization.py dispatches on, and they are one-line comments the plan puts out of scope. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/reporting.md | 365 ++++++++++++++++++- src/coder_eval/agent.py | 68 ++-- src/coder_eval/analysis.py | 17 +- src/coder_eval/argv_match.py | 57 ++- src/coder_eval/config.py | 91 ++--- src/coder_eval/errors/categorization.py | 34 +- src/coder_eval/errors/executor.py | 41 +-- src/coder_eval/formatting.py | 7 +- src/coder_eval/harbor/__init__.py | 22 +- src/coder_eval/harbor/agent.py | 80 ++-- src/coder_eval/harbor/atif_emit.py | 34 +- src/coder_eval/harbor/atif_hydrate.py | 37 +- src/coder_eval/harbor/atif_models.py | 35 +- src/coder_eval/harbor/experiment_packager.py | 9 +- src/coder_eval/harbor/packager.py | 221 ++++------- src/coder_eval/harbor/portability.py | 39 +- src/coder_eval/harbor/reward.py | 46 +-- src/coder_eval/invocation_log.py | 16 +- src/coder_eval/litellm_cost.py | 53 +-- src/coder_eval/logging_config.py | 55 +-- src/coder_eval/plugins.py | 10 +- src/coder_eval/pricing.py | 58 ++- src/coder_eval/reports.py | 107 ++---- src/coder_eval/reports_experiment.py | 63 ++-- src/coder_eval/reports_html.py | 57 ++- src/coder_eval/reports_junit.py | 29 +- src/coder_eval/reports_stats.py | 84 ++--- src/coder_eval/simulation/user_simulator.py | 47 +-- src/coder_eval/telemetry.py | 122 +++---- src/coder_eval/utils.py | 43 +-- tests/lint/prose_budget.py | 2 +- 31 files changed, 898 insertions(+), 1051 deletions(-) diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md index 3e1ea1895..791395a13 100644 --- a/.claude/notes/reporting.md +++ b/.claude/notes/reporting.md @@ -4,12 +4,373 @@ ## Published rates and run-time caps -- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)`. The guard originally shipped on `RunSummary` alone, so the same 10-task `execute` run with one crash rendered "Pass Rate: n/a" in `run.md` and "Pass Rate: 0.0%" in `experiment.md`. `measured` is **counted evidence** (`tasks_measured` / `rows_measured` — rows carrying a `weighted_score`), never a bucket count: the first version tested `tasks_succeeded + tasks_failed == 0`, but `TIMEOUT` and the two budget stops are category `failed` and reachable under `execute` (`_check_run_limits` still runs on the ungraded branch), so ONE timed-out row in a 100-task ungraded night read as "measured" and published `pass_rate: 0.0` — a real 0% point on the evalboard trend for a run that graded nothing. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender. +- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)` — see [orchestration.md](orchestration.md) § Rates need verdict evidence, not bucket counts for why, and why `measured` is counted evidence rather than a bucket count. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender. -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see [orchestration.md](orchestration.md) § Early stop on criterion) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. ## Plugin and GitHub Action layout plugins/coder-eval/ # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills//SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `-` (`lint-tasks`, `check-skill`). Never `-`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling. action.yml # Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the `v` tag + cuts the Release, so nothing consumer-visible moves before the wheel is on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing. + +## The Agent ABC contract + +`agent.py` is the plugin SPI: everything a third-party agent author must satisfy. The +authoring walkthrough is [docs/EXTENDING.md](../../docs/EXTENDING.md) and the five +numbered lifecycle requirements are in CLAUDE.md § Adding a New Agent; what follows is +why the seams are shaped the way they are. + +The turn-lifecycle bookkeeping lives on the BASE class as class-level defaults, so a +subclass gets the behaviour without re-declaring it. `_iteration_was_incremented` is set +right after the counter bump at the top of `communicate()` and consumed by +`discard_pending_turn()`, which rolls the counter back exactly once per failed turn — even +when partial-record assembly leaves `pending_turn` at None. That is why rollback is the +caller's move, not the agent's: only the caller knows a turn failed. + +Capability flags are declared rather than probed. `supports_cooperative_stop` gates +arming early-stop, so arming it on an agent that ignores `should_stop` is rejected at +resolution rather than silently never firing. `supports_cost_log_tags` and +`system_prompt_semantics` are declared for reasons of their own — see +[agents.md](agents.md) § Why the constructors declare every kwarg and § The +system_prompt_semantics marker. + +The shared mid-turn failure kernels are byte-identical fragments that recur across and +within the agent turn-loops. Each agent keeps its OWN outer try/except/finally bracket, +because the brackets genuinely differ (flat versus nested, `finally` or not), and calls +these from inside its existing branches. They take the agent's own per-turn `finalize` +callable, so the helper never needs to know how each agent assembles its end event. + +## Telemetry emission + +Telemetry is a self-contained, opt-out usage side-channel and is **never** part of the eval +data path. + +The Azure Monitor exporter routes an OpenTelemetry log record to the `customEvents` table +instead of the default `traces` table if and only if the record carries a particular +attribute; the event name is that attribute's value and every other attribute becomes a +custom dimension. That attribute is reachable through plain stdlib logging — an OTel +handler attached to a dedicated logger, and `track_event` calling `logger.info(name, +extra={...})`. So every OTel and Azure import lives inside `init_telemetry`, and +`track_event` is pure stdlib and a cheap no-op when telemetry is off. The attribute name is +hard-coded to match the exporter's internal constant for the same reason. + +### On by default, and what that obliges + +An ingestion-only connection string is baked into the app, so a fresh install reports usage +to the shared resource; an explicitly-set one takes precedence, and `TELEMETRY_ENABLED` is +the single canonical disable gate. It is INGESTION-ONLY — it can write telemetry to the +resource, never read, query or manage it — which is the same class of value embedded in +every distributed telemetry client, and it was approved for embedding. It is base64-wrapped +only to avoid tripping naive secret scanners and to mark it as an intentional, reviewed +default; base64 is trivially reversible and this is not secrecy. The residual risk is +telemetry spoofing and ingestion-cost abuse, bounded by the resource being dedicated to +coder-eval usage telemetry. + +No prompts, file contents or repo paths are ever captured — only enums, counts, durations, +an anonymous per-install id (a random UUID in the user config file: it identifies an +install, not a person) and non-PII platform identity. Because it is default-on, the first +run that initializes it prints a one-time stderr notice disclosing what is collected and +how to disable it. + +### Non-fatal, and what that costs + +Every public function wraps its body in `try/except Exception` and logs a warning rather +than raising: telemetry must never break a run. CE019 enforces it. Persisting the install +id is best-effort too — a missing HOME or an unwritable directory degrades to no +`InstallId`, never to disabled telemetry. + +Two guards are narrower than they look. The exporter is constructed in its own `try`, +because it parses the connection string — a credential — and a parse error can echo it +back, so the failure is logged WITHOUT interpolating the exception. And the command +decorator catches `BaseException` as well as `Exception`, because `KeyboardInterrupt` and +`SystemExit` derive from the former: without that branch a Ctrl-C would skip both handlers +and the `finally` would record the aborted command as "Succeeded". + +The events logger is a process-wide singleton that outlives a shutdown, so the handler is +tracked and detached on shutdown and any stale one is dropped before attaching — otherwise +a re-init double-emits. `SchemaVersion` is stamped on every event so the dashboard's +queries, a cross-system contract, can detect a schema change instead of silently breaking. + +## Cost joining + +For the open-weight backend the Claude binary's transport drops the provider's real cost +and per-call cache before Python can see it, so a proxy-side callback writes one JSONL +record per call and this module joins them back on at the TURN level: the turn's total cost +is overridden with the SUM of its calls' real cost, and the per-call breakdown is attached +as a deterministic audit record. + +Token buckets are LEFT UNTOUCHED, so the `EventCollector` remains the single writer of the +token-bucket invariant ([agents.md](agents.md) § Token accounting and the reconciliation +message). There is deliberately NO per-generation distribution: +matching a proxy call to a transcript generation has no deterministic key — only positional +or output-token heuristics — so that view lives in the per-call table rather than being +guessed onto the message stream. + +### Coverage, and why a gap keeps the estimate + +A turn's cost is overridden only when every call that reported usage is priced. A +degenerate call that reports NO usage at all — no cost and no tokens, seen occasionally on +some providers — is ignored, so one of them cannot revert a whole turn to the static +estimate. A call that reports usage but no cost is a genuine gap: the turn keeps its static +estimate, because overriding would bill it at $0, no breakdown is attached, and a warning +names the unpriced ids. A turn with no matching record keeps its estimate too. + +### Retry safety and the two phases + +Several turns can share an `iteration` — a crashed attempt and its retry — and both +attempts' proxy calls carry that tag. An iteration's calls are credited to a single +survivor (the last turn with that iteration THAT HAS GENERATIONS) and earlier siblings are +zeroed. The credit and the zero are decided TOGETHER: a sibling is zeroed only when the +survivor is actually credited, so if the survivor falls back to static the sibling keeps its +estimate and the iteration's spend is never dropped. + +The join is transactional. The whole plan is computed before any turn is mutated, so a +malformed record — which raises while building the per-call breakdown — aborts the join +with the run untouched, matching the caller's "keeping static pricing" contract. Spend +tagged with an iteration no turn has is surfaced rather than silently dropped. + +## Report rollups and the HTML twin + +`reports_html.py` is the evalboard's STATIC TWIN: the two render the same run and must +agree, so a rule implemented on one side belongs on the other. The arithmetic itself lives +in `reports_stats.py` and the renderers only format it. + +### An unmeasured value is never zero + +This is the rule the reporting surfaces exist to hold, and every violation of it has been +the same bug: a value that was never measured rendered as a confident zero. + +`analysis.py` returns `None` when nothing was timed, so a `0` there published "measured, +and instant" for a turn whose only tool call was force-closed and never timed. The renderer +must therefore test `is not None` rather than truthiness — a genuine measured `0.0` average +is a real measurement and must survive to the surface. The same reasoning governs the four +wall-clock buckets (an unmeasured one renders as an em dash, never `0ms` — CE058), the +ungraded score placeholder (deliberately not `0.000`), and a suite that graded nothing. + +`duration_seconds` is the awkward case: it is a non-optional float defaulting to `0.0`, so +there is no `None` arm to write — but a `0.0` duration is a run that was never timed, and +subtracting real buckets from it renders a fabricated negative residual. The evalboard +keeps that null; so does the report. + +### Read the stored value, do not re-derive it + +The four wall-clock buckets are computed ONCE by `turn_time_buckets` and carried on the row. +The `iterations` projection is deliberately 6-key, with no messages, no commands and no +harness timings, so a renderer CANNOT re-derive them from it — a second implementation of +that summation is exactly what the shared function exists to prevent. + +`_turn_tool_union_ms` shows the shape. It PREFERS the stored value, because the collector +writes it from the single span set it measures all four buckets against, so the two surfaces +are guaranteed to agree rather than merely observed to. The derivation is the LEGACY path +for a record written before the field existed, which must stay renderable. The two paths +cannot be told apart by value — both return `None` for a turn with no bounded span — which +is why the stored one is checked with `is not None`: a stored `0.0` is a measurement and +must not fall through to a re-derivation. The span SELECTION is the shared rule, not a copy +of it, for the same reason. + +### The ungraded row in every surface + +An ungraded run gets the explanatory line; an ordinary EMPTY run keeps its "n/a (0/0)" +rendering, because the two are different facts. The test is `pass_rate is None`, not +`not tasks_graded` — an execute night with a crashed row has `tasks_graded > 0` while still +having measured nothing, and that rendered `0.0%` beside `Error Share: 100.0%`, exactly the +total-failure reading the guard exists to prevent. + +Only the SCORE is dropped when there is none, never the row: duration, tokens and assistant +turns are facts about the run that grading has nothing to do with, and `execute`'s contract +withholds only the verdict. Skipping the row whole made an all-ungraded experiment render +every statistic as N/A. An experiment can be MIXED — `run --resume` grades rows +independently and folds a failed one back ungraded — so the series are consumed +independently and need not be index-aligned; pairing across variants is by task id. + +A fourth `Not Graded` row is rendered conditionally in every table that breaks down the +count, because without it Tasks Run / Succeeded / Failed / Errors stop summing to +`tasks_run` with nothing on the page to say where the rest went. The same reasoning drives +the JUnit `` element: it is JUnit's only "no verdict" shape, and reporting an +ungraded row as a failure would turn a healthy run red in CI while reporting it as a pass +would invent a verdict. An ungraded row is also excluded from the failure-reason list, +which is documented as failed and errored rows. + +### Per-instance aggregation, and what it buys + +Per-row results are sliced per criterion INSTANCE by position, because the checker appends +one result per criterion in declared order. Aggregating per-instance rather than pooling by +type is what lets a task stack many criteria of the SAME type — an activation suite's +per-skill `skill_triggered` criteria — and get a distinct aggregate for each, instead of +one type-pooled number repeated once per instance. The aggregate carries the criterion's +description so the stacked instances stay distinguishable downstream. + +The agent's own spend is broken out from the total only when there is overhead to +distinguish it from: judge spend is a property of the suite's criteria and identical across +harnesses, so comparing harnesses means comparing the agent line. **Total Cost** always +means the whole bill. + +### The claims the reports do NOT make + +An early-stopped row does not advertise "N turns avoided". That derived from +`max_turns - sdk_turn_index`, and on harnesses where one `communicate()` is a single SDK +turn it advertised dozens of avoided turns when all that was cut was a tool-call tail. The +upper bound is still persisted, labelled as the bound it is. + +Missing spend is worded cause-agnostically, because an unpriced turn and a hard kill reach +the same conclusion and the report cannot always tell which applied. + +## Harbor export + +Harbor is an outer harness with its own runtime contract: a fixed reward-file convention, +a fixed log layout, a trajectory format. `coder_eval.harbor` is deliberately narrow — it +translates coder-eval's artifacts into that contract and does not know how a task is +DEFINED. It is a core layer like `orchestration/`, so it must not import the CLI (CE004); +it raises plain exceptions and lets the CLI wrap them. + +Both directions exist. The packager exports a coder-eval task to run under Harbor with +coder-eval as the grader; `CoderEvalAgent` is the mirror image, making coder-eval Harbor's +AGENT against a fixed-path agent-phase task.yaml the packager bakes in. + +### Write the reward file, or do not + +Harbor's verifier reads a reward file and does NOT inspect the verifier script's exit code +— only whether the file exists, is non-empty and parses. A missing or malformed one raises +inside Harbor's own verify step, which its trial runner catches, records, and leaves at its +default rather than coalescing to a zero-reward object. That is Harbor's own +infra-versus-policy split, already built. + +So the whole job is: write the file, or do not. The "do not" case is the load-bearing one. +An unmeasured row must not become `reward=0.0` — that would train "the agent's behaviour +was bad" from a measurement that never happened. Not writing lets Harbor's own +missing-reward path mask the trial instead. It is CE049's principle (never coalesce a +possibly-unmeasured score to a numeric literal) one level up, at the artifact-writing +boundary rather than the in-process one. + +A grading-time INFRASTRUCTURE failure is the same case in disguise. The weighted-score +calculation short-circuits an empty criteria list to a hard `0.0` rather than `None`, so a +checker raising an escalating exception finalizes the row as ERROR with a score of `0.0`, +not `None`. That is not a measurement either, so it gets the same treatment via the row's +category. + +### Not every criterion can grade inside someone else's container + +A portability audit classifies each criterion type so the packager can refuse an +unsupported task AT EXPORT TIME, where the operator sees why, rather than at verify time, +where it is an unexplained low reward with no obvious cause. + +Filesystem and exit-code checks are portable — nothing about the verifier container changes +what they need. `reference_comparison` needs the reference tree, which the export always +places verifier-side when the task declares one, so that class never actually blocks. +Trajectory-reading criteria cannot work in the export direction at all: the verifier is a +separate process from the agent phase, and the agent may not even BE coder-eval, so there is +no iterations list to read without ATIF ingestion. `cli_called` reads a log written by a +recorder shim coder-eval's own sandbox installs, which the exported Dockerfile does not +provision. Credential-needing criteria need a model reachable from inside the verifier +container and a judge that does not follow the agent's route; they are refused with an +explicit opt-in escape hatch for an operator who has provisioned that themselves. + +Coverage is registry-derived, so a new criterion type added to the union without a +classification fails CLOSED rather than silently exporting as if it were portable. + +### The non-obvious constraint in the emitted task.yaml + +The verifier-side `tests/task.yaml` must NOT set a `none` agent type, even though the +verifier phase is conceptually exactly that. The `check_none_agent` validator rejects any +criterion with `requires_agent=True` — which includes `reference_comparison`, a criterion +the export treats as portable — the moment the type is literally `none`, regardless of +whether an agent actually runs. The bare-task path never instantiates an agent whatever the +type says, so a placeholder real type plus a placeholder prompt satisfies the validators +without changing behaviour. Verified directly rather than inferred. + +`--workspace-dir "$(pwd)"` on the agent side is the real fix, not a workaround: without it +the tempdir sandbox writes the agent's workspace to a throwaway directory elsewhere in the +container, never where the verifier looks. Confirmed live — the agent's output was real and +every criterion scored 0 as "file does not exist". `$(pwd)` is resolved by the container's +shell at exec time and equals the WORKDIR because the exec is given no explicit cwd. + +### What the export carries, and what it refuses to carry + +No Dockerfile is written unless the task sets `sandbox.docker.dockerfile_path` — only +real build steps (`RUN`) need one. Harbor's own `should_use_prebuilt_docker_image` +(`harbor/environments/definition.py`) pulls `task.toml`'s `[environment].docker_image` +and skips the build, confirmed against a real `harbor` install; `WORKDIR` needs no +Dockerfile line either, because `[environment].workdir` is what Harbor passes as the +`cwd` / `-w` at `docker exec` time, whether the image was built or pulled. So the +Dockerfile-less shape is a real choice rather than a null-vs-set distinction: +`docker_cfg.image` always has a value (`default_factory=get_default_docker_image_tag`). + +Nothing is `COPY`'d into the image any more. `environment/task.yaml`, each `type: local` +plugin, each `TemplateDirSource` and each `extra_mounts` entry are bind-mounted at their +own host path, mirroring `docker_runner.py`'s auto-mount — which is why the export warns +that it is NOT portable to a machine without those paths. + +The agentless case was found live: `coder-eval`'s schema forbids a `type: none` agent +from setting `initial_prompt`, and a `harbor run` against a real install surfaced that +`TaskDefinition` validation error before the guard existed. + +Run limits and the judge-route override are carried through, because grading has no agent +loop to cap but the task timeout still bounds the verifier invocation, and dropping the +route silently replaces a pinned backend with the verifier environment's default. + +`HOME` is in the default passthrough ONLY because the docker driver also bind-mounts the +host's `~/.claude` into the container at that path, so the value still resolves to a real +directory there. Harbor builds its own container with no such mount, so forwarding the +host's literal `HOME` would point the container at a directory that does not exist in it — a +real regression, not a no-op. + +A missing template is a hard failure, not a warning: the agent-phase task.yaml still +references it, so a silently-skipped copy ships an export whose agent has no starter code, +and every criterion then reads "file does not exist" indistinguishably from a real agent +failure — the CE039 anti-pattern one layer up, at the export boundary. A raw dataset-backed +task is refused for a similar reason: fan-out happens later in the pipeline, so exporting +one would emit a single Harbor task whose prompt and criteria still contain literal +placeholders — never expressible, but scored anyway. + +Symlinks are DROPPED rather than dereferenced in every task-authored tree the export +copies, because dereferencing writes a symlink target's content into a distributable +artifact. The reference copy wraps its failure in the export's own error type rather than +letting a bare `OSError` escape, because the CLI catches only the export errors and an +unreadable tree would otherwise abort a whole experiment export the docstring promises it +will not abort. + +The generated shell script quotes the workdir before interpolating it: that value comes +from a task-YAML field whose only validator checks for a leading slash, so it does not +reject quotes, substitutions, backticks or newlines. + +## The ATIF trajectory bridge + +ATIF models are VENDORED so coder-eval can emit and parse trajectories with zero runtime +dependency on the harbor package; fidelity is guarded by a frozen fixture validated once +against the real models. Three deviations are deliberate: the schema version is +pattern-validated rather than a closed literal, so a trajectory written by a FUTURE minor +version still parses (major bumps are still rejected) where harbor itself would refuse it; +harbor's `Agent` is renamed to avoid clashing with coder-eval's own; and an image payload is +an untyped dict, because coder-eval emits text only and merely needs to TOLERATE one on +read. These live outside `coder_eval.models` on purpose — they are interchange models for +Harbor interop, not evaluation models. + +### Emitting + +The converter is a PURE function of the models: no I/O, no agent-type branching, no mutation +of its input. Sub-agent generations are NESTED into embedded sub-trajectories rather than +flattened into the main thread, because flattening corrupts SFT data derived from the +trajectory. Reconciliation entries never become steps — their residuals are recorded in an +extra field and are already inside the authoritative totals. A turn with no message stream +degrades to one synthetic user step plus one agent step carrying all the turn's commands; +note that a stream with user or reconciliation entries but no generations takes the NORMAL +path, so those entries survive. + +Every user message is a genuine user utterance today. If a tool-result variant ever gains a +producing code path, the converter must learn to SKIP those — tool results already live in +step observations. + +### Hydrating + +The reverse direction reconstructs only what the trajectory-shaped criteria actually read: +the commands and the message stream. It is deliberately NOT a lossless round trip. +Per-generation token buckets are not recovered, so cost and token reporting for a hydrated +result is incomplete. Sub-agent nesting is flattened. Turn boundaries are recovered by +splitting on user steps, mirroring the emit side's convention, because ATIF carries no +explicit iteration marker. + +The one asymmetry that bites: a tool call's status rides on the call's own extra field and +its duration on the matching observation's, so both must be read back — otherwise +`command_executed` with `require_success` silently scores a successful command 0.0 on every +hydrated trajectory, because the status defaults to None rather than "success". diff --git a/src/coder_eval/agent.py b/src/coder_eval/agent.py index 355ea2c16..d4dddcaf8 100644 --- a/src/coder_eval/agent.py +++ b/src/coder_eval/agent.py @@ -62,38 +62,29 @@ def __init__(self, config: ClaudeCodeAgentConfig, ...): slot is always None. """ - # Shared turn-lifecycle bookkeeping. Class-level defaults so subclasses get - # the behavior without re-declaring them in __init__ (they may still set - # `_state` in start()). `_iteration_was_incremented` is set True right after - # the counter bump at the top of `communicate()` and consumed by - # `discard_pending_turn()`, which rolls the counter back exactly once per - # failed turn — even when partial-record assembly leaves `pending_turn=None`. + # Class-level defaults so a subclass gets the behaviour without re-declaring it. + # `_iteration_was_incremented` is consumed by `discard_pending_turn()`, which + # rolls the counter back exactly once per failed turn. + # Rationale: .claude/notes/reporting.md § The Agent ABC contract _state: AgentState = AgentState.WORKING _iteration: int = 0 _iteration_was_incremented: bool = False - # Capability flag: whether this agent honors the cooperative ``should_stop`` - # interrupt threaded through ``communicate()`` (early-stop-on-criterion). - # Default False — arming early-stop on an agent that does not set this True - # is rejected at resolution time. Concrete agents that check ``should_stop`` - # between messages override it to True. + # Whether this agent honors the cooperative ``should_stop`` interrupt. Default + # False: arming early-stop on an agent that does not set it True is rejected at + # resolution rather than silently never firing. supports_cooperative_stop: ClassVar[bool] = False - # Capability flag: whether this agent's constructor accepts the ``cost_log_tags`` - # kwarg (proxy-side actual-cost correlation headers for the LiteLLM backend). - # Default False — the agent-agnostic ``create_agent`` factory must only forward - # ``cost_log_tags`` to agents that set this True, else a route-driven kwarg would - # crash every agent (NoOp/Codex/Antigravity/plugins) whose ``__init__`` lacks it. + # Whether this agent's constructor accepts ``cost_log_tags``. The agent-agnostic + # factory must only forward it to agents that set this True, or a route-driven + # kwarg crashes every agent whose ``__init__`` lacks it. supports_cost_log_tags: ClassVar[bool] = False - # How this agent combines a configured ``system_prompt`` with its own default - # prompt, recorded per run as ``environment_info.system_prompt_semantics``. - # Declared on the base (not only on the agents that implement a regime) so the - # marker is present on EVERY run: dashboards can then read "absent" as one thing - # only — a run from before the marker existed — instead of conflating it with a - # plugin agent that never declared. Agents whose regime is fixed set this - # ClassVar; an agent whose regime depends on its config (Claude Code) overrides - # ``get_environment_info`` and emits the resolved value instead. + # How this agent combines a configured ``system_prompt`` with its own default, + # recorded per run. Declared on the BASE so the marker is present on every run + # and "absent" reads as one thing only. An agent whose regime depends on its + # config overrides ``get_environment_info`` and emits the resolved value. + # Rationale: .claude/notes/agents.md § The system_prompt_semantics marker system_prompt_semantics: ClassVar[SystemPromptSemantics] = "unknown" def _begin_turn(self) -> None: @@ -123,12 +114,10 @@ def _mark_stopped(self) -> None: # --- Shared mid-turn failure kernels -------------------------------------- # - # Tiny, byte-identical fragments that recur across (and within) the agent - # turn-loops. Each agent keeps its OWN outer try/except/finally bracket — the - # brackets genuinely differ (flat vs nested, finally vs not) — and calls these - # from inside its existing branches. They take the agent's own per-turn - # ``finalize`` callable (the turn-state's method) so the helper never needs to - # know how each agent assembles its AgentEndEvent payload. + # Each agent keeps its OWN outer try/except/finally bracket -- the brackets + # genuinely differ -- and calls these from inside its existing branches. They + # take the agent's own ``finalize`` callable, so the helper never needs to know + # how each agent assembles its end-event payload. def _finalize_and_raise_timeout( self, finalize: _FinalizeFn, timeout: float, *, cause: BaseException | None = None @@ -228,22 +217,17 @@ async def communicate( stream_callback: Optional callback for real-time event streaming timeout: Hard wall-clock deadline in seconds. When exceeded the agent must force-terminate any in-flight subprocess and raise - TurnTimeoutError. Implementations should not rely solely on - asyncio cancellation (the Claude Agent SDK uses anyio task - groups that swallow cooperative cancellation). + TurnTimeoutError. Do not rely solely on asyncio cancellation -- + some SDKs swallow it. max_turns: Hard cap on inner-loop turns within this single ``communicate()`` call. When the agent would exceed it, the returned ``TurnRecord`` has ``max_turns_exhausted=True``. None defers to the underlying SDK default. - should_stop: Cooperative early-stop poll for early-stop-on-criterion. - When provided, an implementation that supports cooperative - stopping (``supports_cooperative_stop=True``) should call it at - each safe message boundary and, when it returns True, stop - pulling further work and finalize the turn cleanly - (``crashed=False``, no raise). ``None`` (default) preserves the - pre-existing behavior exactly. Agents that do not support it - accept the argument and ignore it (the orchestrator only passes - it to a capable agent). + should_stop: Cooperative early-stop poll. An implementation with + ``supports_cooperative_stop=True`` calls it at each safe message + boundary and, when it returns True, stops pulling further work and + finalizes the turn cleanly (``crashed=False``, no raise). Agents + that do not support it accept and ignore the argument. Returns: TurnRecord containing the complete interaction diff --git a/src/coder_eval/analysis.py b/src/coder_eval/analysis.py index 54cf80edb..2839aefca 100644 --- a/src/coder_eval/analysis.py +++ b/src/coder_eval/analysis.py @@ -35,19 +35,14 @@ def calculate_command_statistics(turns: list[TurnRecord]) -> CommandStatistics: # Use `is not None` to include valid 0.0ms durations total_time = sum(cmd.duration_ms for cmd in all_commands if cmd.duration_ms is not None) timed_count = sum(1 for cmd in all_commands if cmd.duration_ms is not None) - # None, not 0, when NOTHING was timed: `avg_command_time_ms` is - # `float | None` precisely so it can say "no measurement" instead of - # publishing "measured, and instant". A 0 here rendered "0ms average - # command time" for a turn whose only tool call was force-closed and - # never timed at all. The `is not None` tests above are deliberately - # unchanged — a real 0.0ms duration IS a measurement and still counts. + # None, not 0, when NOTHING was timed. The `is not None` tests above are + # deliberately unchanged: a real 0.0ms duration IS a measurement. + # Rationale: .claude/notes/reporting.md § An unmeasured value is never zero avg_time = total_time / timed_count if timed_count > 0 else None - # Find slowest commands (type-safe using SlowestCommandInfo model). Only - # timed commands are eligible, and the duration travels alongside its - # command so it stays a float the whole way — an untimed command has no - # place in a "slowest" ranking, and coalescing it to 0.0 would only have - # hidden that. + # Only TIMED commands are eligible, and the duration travels alongside its + # command so it stays a float: an untimed one has no place in a "slowest" + # ranking, and coalescing it to 0.0 would only hide that. timed = [(c.duration_ms, c) for c in all_commands if c.duration_ms is not None] slowest_info = [ SlowestCommandInfo( diff --git a/src/coder_eval/argv_match.py b/src/coder_eval/argv_match.py index 6c8ed9973..91de9b83e 100644 --- a/src/coder_eval/argv_match.py +++ b/src/coder_eval/argv_match.py @@ -1,25 +1,18 @@ """Structured argv matching: the one engine both CLI surfaces share. -Two places ask the same question about one invocation. The ``cli_called`` -criterion reads a recorded ``argv`` back afterwards and asks *did this happen*; -a ``record_cli`` response rule asks it live, inside the sandbox, to choose which -canned response to serve. An author who writes ``verb: "ixp projects get"`` in a -rule and again in the criterion that grades it must get one semantic, not two -that drift. - -Everything here takes PLAIN DICTS rather than pydantic models, and imports -nothing beyond the standard library: ``Sandbox._generate_cli_recorders`` copies -this file into the recorder directory as a SIDECAR beside every shim that -declares response rules, and that shim imports it as a sibling while running -inside the sandbox, where ``coder_eval`` is not installed. Lint rule CE057 keeps -the imports stdlib-only. - -:class:`MatchSpec` is what ``CliMatch.match_spec`` emits. It is a ``TypedDict`` -rather than a bare dict on purpose: it is the seam where every guarantee the -pydantic models establish would otherwise be erased, and the fallback for a key -the reader failed to find is always "unconstrained" -- the direction that makes a -rule match everything, or a criterion score 1.0 against any log. TypedDict is -closed, so a key renamed on either side is a pyright error on both. +The ``cli_called`` criterion reads a recorded ``argv`` back afterwards and asks *did +this happen*; a ``record_cli`` response rule asks it live, inside the sandbox, to +choose which canned response to serve. An author who writes the same ``verb:`` in a +rule and in the criterion that grades it must get one semantic, not two that drift. + +HAZARD: everything here takes PLAIN DICTS and imports NOTHING beyond the standard +library. ``Sandbox._generate_cli_recorders`` copies this file into the recorder +directory as a SIDECAR beside every shim that declares response rules, and that shim +imports it as a sibling while running inside the sandbox, where ``coder_eval`` is not +installed. Lint rule CE057 enforces the imports; this comment is the only thing that +explains WHY before someone adds one. + +Rationale: .claude/notes/contracts.md § Recording a CLI invocation """ import re @@ -113,17 +106,15 @@ def record(name: str, value: str) -> None: name = token.lstrip("-") known = name in value_flags or name in known_names - # A bare negative number is a value, not a flag. Reading `-1` as a flag - # named `1` drops it from the positionals -- the same silent-disappearance - # that let `--yes proj-1` slip a delete past a guard. + # HAZARD: a bare negative number is a value, not a flag. Reading `-1` as a + # flag named `1` drops it from the positionals. if not known and is_number(name): positional.append(token) continue # Clustered short flags: `-rf` is `-r -f`. Declared names win, so a real - # multi-char short flag still matches, and `-fvalue` binds when `f` takes - # a value; otherwise each character is its own switch, which is what stops - # `-yf` escaping an `aliases: [y]` predicate. + # multi-char short flag still matches; otherwise each character is its own + # switch, which is what stops `-yf` escaping an `aliases: [y]` predicate. if not known and not token.startswith("--") and len(name) > 1: head, rest = name[0], name[1:] if head in value_flags: @@ -205,10 +196,9 @@ def argv_matches(spec: MatchSpec, argv: list[str]) -> bool: def names_of(flag: str, predicate: FlagPredicate) -> tuple[str, ...]: return (flag, *predicate["aliases"]) - # Declarations only. Folding `ignore_flags` into value_flags made ignored - # SWITCHES value-bearing, which swallowed the next positional and reopened a - # guard false-PASS; an ignored flag that takes a value declares it in - # value_flags. + # HAZARD: declarations only. Folding `ignore_flags` into value_flags made + # ignored SWITCHES value-bearing, swallowing the next positional and reopening + # a guard false-PASS. An ignored flag that takes a value declares it there too. ignore = frozenset(spec["ignore_flags"]) value_flags = frozenset( name @@ -225,10 +215,9 @@ def names_of(flag: str, predicate: FlagPredicate) -> tuple[str, ...]: offset = 0 spellings = spec["verb_spellings"] if spellings: - # Token-wise, not a subset and not a string startswith: `labellings confirm` - # must never be satisfied by `labellings unconfirm`. Taking the first match is - # safe because validation rejects one spelling prefixing another, so no argv - # can match two. + # HAZARD: token-wise, not a subset and not a startswith -- `labellings + # confirm` must never be satisfied by `labellings unconfirm`. First match is + # safe because validation rejects one spelling prefixing another. matched = next((tokens for tokens in spellings if positional[: len(tokens)] == list(tokens)), None) if matched is None: return False diff --git a/src/coder_eval/config.py b/src/coder_eval/config.py index c65bb1d69..d99f27edf 100644 --- a/src/coder_eval/config.py +++ b/src/coder_eval/config.py @@ -16,20 +16,11 @@ from coder_eval.models import AgentKind, ApiBackend -# Application Insights connection string baked into the application so a fresh -# install reports usage telemetry to the shared coder-eval resource with no -# configuration. An explicitly-set connection string (env or .env, via any of the -# field's aliases below) takes precedence — pydantic-settings always prefers an -# env value over a field default. -# -# This is an INGESTION-ONLY connection string (InstrumentationKey + IngestionEndpoint): -# it can only WRITE telemetry to the resource, never read/query/manage it — the same -# class of value embedded in every distributed telemetry client (VS Code, Azure CLI, -# gh, the UiPath CLI). Approved by security for embedding. It is base64-wrapped ONLY -# to avoid tripping naive secret scanners / push-protection and to mark it as an -# intentional, reviewed default — NOT for secrecy (base64 is trivially reversible). -# Residual risk is telemetry spoofing / ingestion-cost abuse, bounded by the resource -# being dedicated to coder-eval usage telemetry. +# HAZARD: an INGESTION-ONLY connection string, baked in so a fresh install reports +# usage telemetry with no configuration. It can only WRITE to the resource, never +# read, query or manage it, and it is base64-wrapped to avoid tripping naive secret +# scanners -- NOT for secrecy. An explicitly-set one takes precedence. +# Rationale: .claude/notes/reporting.md § On by default, and what that obliges _DEFAULT_TELEMETRY_CONNECTION_STRING = base64.b64decode( "SW5zdHJ1bWVudGF0aW9uS2V5PTgxZDBkOGI1LTg1ZjktNDMxNS1iYjJlLTg4ODg0Y2ZkYTVhNztJbmdlc3Rpb25FbmRwb2ludD1odHRwczovL3dlc3R1czItMi5pbi5hcHBsaWNhdGlvbmluc2lnaHRzLmF6dXJlLmNvbS87TGl2ZUVuZHBvaW50PWh0dHBzOi8vd2VzdHVzMi5saXZlZGlhZ25vc3RpY3MubW9uaXRvci5henVyZS5jb20vO0FwcGxpY2F0aW9uSWQ9MDRjN2U3ZjItYjg0OC00ZjhlLTkxNzMtZjI3NmE1YTAwMzk0" ).decode("utf-8") @@ -49,9 +40,8 @@ os.environ[key] = value -# Removed layer-5 `.env` knobs → their `-D` override paths. pydantic-settings -# silently ignores unknown env vars, so without an explicit guard a stale knob -# would silently stop having any effect — fail loud with a migration hint instead. +# pydantic-settings silently ignores unknown env vars, so without this guard a +# stale knob would quietly stop having any effect. Fail loud with a migration hint. _REMOVED_DEFAULT_KNOBS = { "DEFAULT_AGENT_MODEL": "agent.model", "DEFAULT_PERMISSION_MODE": "agent.permission_mode", @@ -104,36 +94,25 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: bedrock_model: str | None = None # Cross-region model ID bedrock_small_model: str | None = None # Cross-region small model ID - # LiteLLM (Anthropic-compatible) endpoint settings (used when api_backend == "litellm"). - # These map to ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN / ANTHROPIC_MODEL / - # ANTHROPIC_SMALL_FAST_MODEL, but ONLY inside the SDK subprocess env (see - # ClaudeCodeAgent._build_sdk_env). They are deliberately NOT named anthropic_* - # so the os.environ export loop below can't leak ANTHROPIC_BASE_URL process-wide - # (which would silently redirect the judge's in-process Anthropic() client). + # HAZARD: these map to the ANTHROPIC_* vars, but ONLY inside the SDK subprocess + # env. Deliberately NOT named anthropic_*, so the export loop below cannot leak + # ANTHROPIC_BASE_URL process-wide and redirect the judge's own client. litellm_base_url: str | None = None litellm_auth_token: str | None = None litellm_model: str | None = None litellm_small_model: str | None = None - # Path to the per-call cost/cache JSONL the LiteLLM proxy's cost_logger writes - # (LITELLM_COST_LOG). Must point at the SAME file the proxy uses (see - # litellm/start-litellm.sh). When set and the file exists, the harness joins each - # call's ACTUAL OpenRouter cost + cache onto the turn (litellm_cost.apply_actual_cost), - # overriding the static rate-card estimate; unset/missing => static pricing (fallback). + # Must point at the SAME file the proxy writes. When set and present, the harness + # joins each call's ACTUAL cost onto the turn; unset or missing => static pricing. + # Rationale: .claude/notes/reporting.md § Cost joining litellm_cost_log: str | None = None - # Codex settings (CodexAgent). CODEX_MODEL is the fallback model/deployment - # used when a task doesn't pin agent.model; CODEX_BASE_URL routes to a custom - # OpenAI-/responses-compatible endpoint (incl. Azure OpenAI). For Azure also - # set CODEX_API_VERSION (the required ``api-version`` query param) and use the - # deployment name as the model. CODEX_BASE_URL / CODEX_API_VERSION / - # CODEX_API_KEY are read directly via os.getenv in the agent, not mirrored here. + # CODEX_MODEL is the fallback when a task doesn't pin agent.model. For Azure set + # CODEX_API_VERSION too and use the deployment name as the model. CODEX_BASE_URL + # / CODEX_API_VERSION / CODEX_API_KEY are read via os.getenv in the agent. codex_model: str | None = None - # Antigravity settings (AntigravityAgent — Google's Gemini coding harness). - # GEMINI_API_KEY authenticates the local harness (read from .env here so the - # export loop below re-publishes it to os.environ, where the google-antigravity - # SDK looks for it). ANTIGRAVITY_MODEL is the fallback Gemini model used when a - # task doesn't pin agent.model. + # GEMINI_API_KEY is read from .env here so the export loop re-publishes it to + # os.environ, where the SDK looks for it. ANTIGRAVITY_MODEL is the fallback. gemini_api_key: str | None = None antigravity_model: str | None = None @@ -141,9 +120,8 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: log_level: str = "INFO" # Default log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) log_to_file: bool = False # Whether to enable file logging - # Usage telemetry (OpenTelemetry → Azure Application Insights customEvents). - # On by default via the baked-in connection string; see coder_eval/telemetry.py. - # telemetry_enabled (TELEMETRY_ENABLED) is the single canonical disable gate. + # On by default via the baked-in connection string. TELEMETRY_ENABLED is the + # single canonical disable gate. telemetry_enabled: bool = True # Defaults to the embedded coder-eval resource; any set value (env or .env, via # the aliases below) overrides it — pydantic-settings prefers env over default. @@ -155,11 +133,8 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: "uipath_ai_connection_string", ), ) - # Caller-settable origin stamp (TELEMETRY_SOURCE), emitted as the `Source` - # dimension on every event. Lets downstream pipelines tag themselves (e.g. - # `nightly-vm` / `skill-eval`) so internal runs are distinguishable from - # anonymous local ones — `IsCI` alone can't, and the framework's own CI is - # muted. Defaults to "coder-eval" for a plain local install. + # Emitted as the `Source` dimension so a downstream pipeline can tag itself and + # be told apart from an anonymous local run -- `IsCI` alone cannot. telemetry_source: str = "coder-eval" model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", case_sensitive=False, extra="ignore") @@ -175,10 +150,8 @@ def _validate_bedrock_settings(self) -> None: missing.append("AWS_BEARER_TOKEN_BEDROCK") if not self.aws_region: missing.append("AWS_REGION") - # BEDROCK_MODEL is the route-level model source. Without it, an - # invocation that doesn't override via --model / task.agent.model - # would send model=None to the SDK and Bedrock would return an - # opaque 400. Fail fast at startup with a clear error. + # Without it an invocation that overrides nothing sends model=None and + # Bedrock returns an opaque 400. Fail fast with a clear error. if not self.bedrock_model: missing.append("BEDROCK_MODEL") if missing: @@ -207,9 +180,8 @@ def _validate_litellm_settings(self) -> None: f"LiteLLM-endpoint routing is enabled but missing required settings: {', '.join(missing)}." + " Please set them in your .env file." ) - # base_url is present (not in `missing`); reject a malformed one so the - # downstream preflight (urlopen) and environment_info (urlparse hostname) - # get a well-formed absolute URL instead of a raw ValueError / empty host. + # Reject a malformed base_url here so the downstream preflight and + # environment_info get a well-formed absolute URL. parts = urlsplit(self.litellm_base_url or "") if parts.scheme not in ("http", "https") or not parts.hostname: raise ValueError( @@ -237,10 +209,8 @@ def validate_api_keys(self, agent_type: str) -> None: if self.api_backend == ApiBackend.LITELLM: self._validate_litellm_settings() - # Claude Code agent can use either: - # 1. ANTHROPIC_API_KEY environment variable - # 2. Cached CLI authentication from 'claude-code login' (subscription account) - # We don't validate the API key here because the SDK handles auth and fails clearly if missing. + # Either ANTHROPIC_API_KEY or cached CLI auth works, and the SDK fails + # clearly when neither does -- so no key validation here. if agent_type == AgentKind.CLAUDE_CODE.value: return @@ -248,9 +218,8 @@ def validate_api_keys(self, agent_type: str) -> None: # Global settings instance settings = Settings() -# Export settings to environment variables for external libraries (the Anthropic SDK, -# boto3/Bedrock) that use os.getenv() instead of reading from the Settings object. -# Only export non-None values and convert non-string types to strings. +# For external libraries that read os.getenv() rather than the Settings object. +# Non-None values only, stringified. for key, value in settings.model_dump().items(): if value is not None: env_key = key.upper() diff --git a/src/coder_eval/errors/categorization.py b/src/coder_eval/errors/categorization.py index 15e46d24a..dd6949053 100644 --- a/src/coder_eval/errors/categorization.py +++ b/src/coder_eval/errors/categorization.py @@ -36,9 +36,8 @@ def _categorize_by_exception_type(error: Exception, component: str) -> ErrorCate if isinstance(error, BudgetExceededError): return ErrorCategory.BUDGET_EXCEEDED - # AgentConfigError subclasses RuntimeError, so this typed check must precede - # any string-pattern fallback that could re-categorise a missing-prerequisite - # message as the retryable AGENT_API_ERROR. + # HAZARD: AgentConfigError subclasses RuntimeError, so this typed check must + # precede any pattern fallback that would re-categorise it as retryable. if isinstance(error, AgentConfigError): return ErrorCategory.AGENT_CONFIG_ERROR @@ -82,10 +81,9 @@ def _categorize_by_message(error_str: str, component: str) -> ErrorCategory | No if any(pat in error_str for pat in ["authentication", "unauthorized", "invalid api key", "401"]): return ErrorCategory.AGENT_AUTH_ERROR - # Billing/credit errors (NOT retryable - retrying wastes time) - # NOTE: Broad patterns like "insufficient" and "credit" are intentional. We prefer - # false positives (skipping retry on a non-billing error) over false negatives - # (wasting retries on a billing error that will never succeed). + # Billing/credit errors (NOT retryable). Broad patterns are intentional: a false + # positive skips one retry, a false negative wastes every retry on an error that + # will never succeed. if any( pat in error_str for pat in ["credit", "billing", "payment", "insufficient", "402", "quota exceeded", "spending limit"] @@ -141,11 +139,8 @@ def _categorize_by_component(error: Exception, error_str: str, component: str) - return ErrorCategory.SANDBOX_SETUP_ERROR if component == "agent": - # Substring heuristics for plain RuntimeErrors that mention a crash - # but aren't typed as AgentCrashError. The typed-AgentCrashError - # check below catches the agent's wrapped exceptions; this branch - # only fires for bare exceptions whose message happens to describe - # a crash. + # Heuristics for a plain RuntimeError that describes a crash without being + # typed as one; the typed check below catches the agent's wrapped ones. if ( "crash" in error_str or "killed" in error_str @@ -156,11 +151,9 @@ def _categorize_by_component(error: Exception, error_str: str, component: str) - if "invalid" in error_str or "malformed" in error_str: return ErrorCategory.AGENT_INVALID_OUTPUT - # Typed agent-crash exception is the LAST agent-side resort: by this - # point pattern matching has had a chance to recognise auth / rate- - # limit / billing / content-filter / api-network signatures stamped - # into the message, so anything still typed as AgentCrashError here - # is a genuinely unexpected failure that the user wants retried. + # LAST agent-side resort: by here the patterns have had their chance at the + # auth / rate-limit / billing signatures, so what remains is a genuinely + # unexpected failure the user wants retried. if isinstance(error, AgentCrashError): return ErrorCategory.AGENT_CRASH @@ -216,10 +209,9 @@ def categorize_error( component = context.get("component", "") - # Precedence: typed-exception group → message-string group → component group. - # A helper returning None means "no match in my group, fall through" — including - # the component-gated timeout/network arms for non-agent components, so a - # sandbox failure reaches the sandbox categories (and their retries) below. + # Precedence: typed-exception group -> message-string group -> component group. + # None means "no match in my group, fall through", so a sandbox failure reaches + # the sandbox categories and their retries below. if (category := _categorize_by_exception_type(error, component)) is not None: return category diff --git a/src/coder_eval/errors/executor.py b/src/coder_eval/errors/executor.py index 8949715b3..7cf7e5d83 100644 --- a/src/coder_eval/errors/executor.py +++ b/src/coder_eval/errors/executor.py @@ -22,41 +22,12 @@ async def execute_with_retry( ) -> Any: """Execute an operation with automatic retry on transient errors. - This is the core retry mechanism. It wraps any async operation and: - 1. Executes the operation - 2. Catches exceptions - 3. Categorizes the error - 4. Retries if eligible (with exponential backoff + jitter) - 5. Re-raises after exhausting retries - - Args: - operation: Async callable to execute (no arguments, use closures/lambdas) - operation_name: Human-readable operation name for logging - context: Context dict with: - - task_id: Task identifier (required) - - component: Component name (optional but recommended) - - agent_name: Agent name (optional) - max_attempts: Override max attempts (defaults to 10 as safety limit) - on_attempt_error: Async ``(exception, zero_indexed_attempt) -> None`` callback - invoked after every failed attempt (including the final non-retryable one), - for draining ``agent.pending_turn`` and calling ``agent.discard_pending_turn()``. - Callback exceptions are logged and swallowed so they cannot mask the original. - - Returns: - Result from operation - - Raises: - Last exception if all retries exhausted - - Example: - >>> async def flaky_api_call(): - ... return await agent.communicate(prompt) - >>> - >>> result = await execute_with_retry( - ... operation=flaky_api_call, - ... operation_name="Agent communication", - ... context={"task_id": "task-001", "component": "agent"}, - ... ) + Retries only what ``errors/categorization.py`` classifies as retryable; + everything else raises on the first attempt. ``on_attempt_error`` runs after each + failed attempt, before the backoff, so a caller can reset per-attempt state (the + orchestrator uses it to preserve a crashed partial ``TurnRecord``). + + Rationale: .claude/notes/agents.md § Shared turn lifecycle """ task_id = context.get("task_id", "unknown") last_error = None diff --git a/src/coder_eval/formatting.py b/src/coder_eval/formatting.py index a5cd052ee..91f8dd39d 100644 --- a/src/coder_eval/formatting.py +++ b/src/coder_eval/formatting.py @@ -96,11 +96,8 @@ def format_messages( continue type_name = type(msg).__name__ - # Known, non-transcript SDK types: StreamEvent carries token deltas - # (captured elsewhere) and RateLimitEvent is an out-of-band throttling - # notice the SDK interleaves into the stream. Neither is transcript - # content, so skip both rather than surfacing an "unhandled" warning. - # Matched by name (not import) to stay robust across SDK versions. + # Known non-transcript SDK types, skipped rather than warned about as + # "unhandled". Matched by NAME, not import, to survive an SDK version bump. if type_name in ("StreamEvent", "RateLimitEvent"): continue if type_name not in warned_unknown_types: diff --git a/src/coder_eval/harbor/__init__.py b/src/coder_eval/harbor/__init__.py index c726b0f64..f707096fd 100644 --- a/src/coder_eval/harbor/__init__.py +++ b/src/coder_eval/harbor/__init__.py @@ -1,22 +1,14 @@ """``coder_eval.harbor`` — the Harbor framework adherence layer. -Harbor (Laude Institute / Terminal-Bench 2.0, harborframework.com) is an outer -harness with its own runtime contract: a fixed reward-file convention the -verifier phase must satisfy, a fixed log layout, a trajectory format. This -package is deliberately narrow — it translates coder-eval's own artifacts -(``task.json``, ``weighted_score``) into that contract. It does not know how a -task is *defined*; that is the export/packager concern (``coder-eval export ---format harbor``), tracked separately. +Harbor (Laude Institute / Terminal-Bench 2.0) is an outer harness with its own +runtime contract: a fixed reward-file convention, a fixed log layout, a trajectory +format. This package translates coder-eval's own artifacts into that contract. It +does not know how a task is *defined* — that is the export/packager concern. -Direction-agnostic by design: the same shim serves a coder-eval task exported -to run under Harbor (coder-eval as the grader) and a coder-eval agent embedded -inside a Harbor-authored task (Harbor's own ``tests/test.sh`` as the grader, -Part A step 8 — coder-eval is purely the agent there and this package is -unused on that path). +A core layer like ``orchestration/``: it must not import ``coder_eval.cli`` (CE004). +Raise plain exceptions and let the CLI wrap them. -This package is a core layer like ``orchestration/`` — it must not import -``coder_eval.cli`` (CE004). Raise plain exceptions and let the CLI wrap them, -exactly as ``orchestration/regrade.py`` does. +Rationale: .claude/notes/reporting.md § Harbor export """ from __future__ import annotations diff --git a/src/coder_eval/harbor/agent.py b/src/coder_eval/harbor/agent.py index bccfcab07..40eb59b21 100644 --- a/src/coder_eval/harbor/agent.py +++ b/src/coder_eval/harbor/agent.py @@ -1,43 +1,20 @@ """``CoderEvalAgent`` — coder-eval as a Harbor agent (C1.2). -The mirror image of Part C's packager: instead of coder-eval grading a -Harbor-authored task (coder-eval as the verifier), this makes coder-eval -Harbor's AGENT — ``harbor run -a coder_eval.harbor.agent:CoderEvalAgent`` -invokes ``coder-eval execute --format harbor`` inside Harbor's own container -against the fixed-path agent-phase task.yaml the packager bakes in (see -``agent_paths.py``), then Harbor picks up the resulting ``trajectory.json`` -from ``self.logs_dir`` exactly as it does for its own ``ClaudeCode`` agent. - -Design, per ``tmp/harborframework.md``'s "Scoping note — Part A revisited": - -- The agent-phase task.yaml is at :data:`AGENT_TASK_YAML_PATH`, criteria-free - (see ``packager.py``'s ``_write_agent_phase_task_yaml``) — this agent never - sees ``success_criteria``, only the real ``agent``/prompt/sandbox config. -- ``coder-eval execute --format harbor --run-dir `` - writes ``task.json`` AND a ``trajectory.json`` (ATIF) sibling directly into - the container's ``/logs/agent/`` (``environment_logs_dir``), which Harbor - bind-mounts from ``self.logs_dir`` on the host — the same path Harbor's own - ``ClaudeCode`` agent writes its trajectory to (verified against the - installed ``harbor`` package's ``populate_context_post_run``: it writes to - ``self.logs_dir / "trajectory.json"`` on the HOST side, after the container - syncs back). This class instead has coder-eval write it directly inside the - container at the mirrored path. -- ``--workspace-dir "$(pwd)"`` (Gap 2's real fix, not a workaround): without - it, ``coder-eval execute``'s own ``tempdir`` sandbox writes the agent's - workspace to a throwaway ``mkdtemp()`` elsewhere in the container, never - where Harbor's verifier phase (``tests/test.sh``) looks (the container's - ``WORKDIR``) — confirmed live, agent output was real but every criterion - scored 0 as "file does not exist". See ``run()``'s docstring. - -Verified against a real ``harbor==0.22.0`` install (``tmp/harbor-venv``): -``BaseAgent.name()`` is a ``@staticmethod``; ``run()`` returns ``None`` and -must not return the trajectory itself (Harbor discovers it by reading -``self.logs_dir / "trajectory.json"`` after the container syncs back, the same -way ``ClaudeCode.populate_context_post_run`` does) — so this class overrides -``populate_context_post_run`` to parse that file and fill in -``AgentContext``'s token/cost fields, matching ``ClaudeCode``'s own pattern -exactly. ``environment.exec()`` takes a single shell command STRING (not an -argv list). +The mirror image of the packager: instead of coder-eval grading a Harbor-authored +task, this makes coder-eval Harbor's AGENT. ``harbor run -a +coder_eval.harbor.agent:CoderEvalAgent`` invokes ``coder-eval execute --format +harbor`` inside Harbor's own container against the fixed-path agent-phase task.yaml +the packager bakes in (see ``agent_paths.py``), writing ``task.json`` and a +``trajectory.json`` sibling directly into the container's ``/logs/agent/``, which +Harbor bind-mounts from ``self.logs_dir``. + +``BaseAgent.run()`` returns ``None`` and must not return the trajectory itself — +Harbor discovers it by reading ``self.logs_dir / "trajectory.json"`` after the +container syncs back — so this class overrides ``populate_context_post_run`` to parse +that file and fill in the context's token and cost fields, matching the installed +agents' own pattern. ``environment.exec()`` takes a single shell command STRING. + +Rationale: .claude/notes/reporting.md § Harbor export """ from __future__ import annotations @@ -99,25 +76,14 @@ async def install(self, environment: BaseEnvironment) -> None: async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None: """Run ``coder-eval execute --format harbor`` inside the environment. - ``instruction`` (Harbor's resolved ``instruction.md`` text) is NOT - forwarded — the agent-phase task.yaml at :data:`AGENT_TASK_YAML_PATH` - already carries the identical resolved prompt (``packager.py`` writes - both from the same source), so there is nothing to forward. Token/cost - totals are filled in afterward by ``populate_context_post_run``, not - here, matching every other installed agent's convention. - - ``--workspace-dir "$(pwd)"``: without it, ``coder-eval execute``'s own - ``tempdir`` sandbox (the agent-phase task.yaml forces ``driver: - tempdir`` — see ``packager._write_agent_phase_task_yaml``) writes the - agent's workspace to a fresh ``mkdtemp()`` elsewhere in the container, - NOT at the image's ``WORKDIR`` -- which is exactly where Harbor's own - verifier phase (``tests/test.sh``) looks for the agent's output. - ``$(pwd)`` is resolved by the container's shell at exec time, not by - this Python process, and equals the WORKDIR because ``environment.exec`` - is not given an explicit ``cwd`` (Docker execs default to the image's - configured WORKDIR). Confirmed live: without this flag the agent wrote - real output but the verifier scored every criterion 0 with "file does - not exist", because it never left the tempdir. + ``instruction`` is NOT forwarded: the agent-phase task.yaml already carries + the identical resolved prompt. Token and cost totals are filled in afterward + by ``populate_context_post_run``. + + ``--workspace-dir "$(pwd)"`` is load-bearing — without it the tempdir sandbox + writes the agent's workspace somewhere Harbor's verifier never looks. + + Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml """ del instruction, context # nothing to forward; context is populated post-run run_dir = self.environment_logs_dir.as_posix() diff --git a/src/coder_eval/harbor/atif_emit.py b/src/coder_eval/harbor/atif_emit.py index 99303ce36..51183524d 100644 --- a/src/coder_eval/harbor/atif_emit.py +++ b/src/coder_eval/harbor/atif_emit.py @@ -1,27 +1,17 @@ """EvaluationResult → ATIF Trajectory converter (the emit direction). -Maps coder_eval's persisted trajectory (``EvaluationResult.iterations`` — the -``TurnRecord`` envelope over the per-generation ``messages`` stream) onto the -vendored ATIF models, so every run can be consumed by ``harbor view``, Harbor -Hub, and ATIF-based SFT/RL pipelines. - -Mapping highlights: - -- ``UserMessage`` → ``Step(source="user")``; ``AssistantMessage`` (one per LLM - generation) → ``Step(source="agent")`` with per-generation ``Metrics``. -- ``CommandTelemetry`` joins its generation via ``assistant_turn_index`` and - becomes that step's ``tool_calls`` + ``observation``. -- Sub-agent generations (``parent_tool_use_id`` set) are NESTED into embedded - ``subagent_trajectories`` — flattening them into the main thread would - corrupt SFT data derived from the trajectory. -- ``ReconciliationMessage`` entries never become steps: their residuals are - recorded in ``Trajectory.extra["reconciliation"]`` and are already included - in the authoritative ``FinalMetrics`` totals (``total_token_usage``). -- Turns with no message stream (legacy task.json, minimal agents) degrade to - one synthetic user step + one agent step carrying all the turn's commands. - -The converter is a PURE function of the models: no I/O, no agent-type -branching, no mutation of the input result. +Maps coder_eval's persisted trajectory onto the vendored ATIF models, so every run +can be consumed by ``harbor view``, Harbor Hub, and ATIF-based SFT/RL pipelines. + +``UserMessage`` → ``Step(source="user")``; each ``AssistantMessage`` → +``Step(source="agent")`` with per-generation ``Metrics``; ``CommandTelemetry`` joins +its generation via ``assistant_turn_index``. Sub-agent generations are NESTED into +embedded ``subagent_trajectories``, and ``ReconciliationMessage`` entries never become +steps. + +A PURE function of the models: no I/O, no agent-type branching, no mutation. + +Rationale: .claude/notes/reporting.md § Emitting """ from __future__ import annotations diff --git a/src/coder_eval/harbor/atif_hydrate.py b/src/coder_eval/harbor/atif_hydrate.py index 729fed173..a1ea8885b 100644 --- a/src/coder_eval/harbor/atif_hydrate.py +++ b/src/coder_eval/harbor/atif_hydrate.py @@ -1,31 +1,16 @@ """ATIF Trajectory -> coder-eval ``TurnRecord`` list (the hydrate direction). -The reverse of ``atif_emit``: given a trajectory that was produced OUTSIDE this -process (a Harbor agent ran ``coder-eval execute --format harbor``, which wrote -``trajectory.json`` next to ``task.json``), reconstruct enough of coder-eval's -own trajectory shape to let the criteria checkers that read it -(``command_executed``, ``cli_called``, ``commands_efficiency``, -``skill_triggered``, ``llm_judge``'s transcript) work against it during a -separate ``coder-eval evaluate --format harbor`` invocation. - -Every criterion checker receives ``turn_records: list[TurnRecord] | None`` — -in a normal run this is ``EvaluationResult.iterations`` — and reads only -``TurnRecord.commands`` (tool calls) and ``TurnRecord.messages`` (for judge -transcripts); see ``criteria/command_executed.py``, ``criteria/skill_triggered.py``, -``criteria/commands_efficiency.py``. Those two fields are what this module -reconstructs. It does NOT attempt a lossless round-trip of ``atif_emit``'s -mapping: - -- Per-generation token buckets (``AssistantMessage.input_tokens`` etc.) are - NOT recovered from ``Step.metrics`` — ``TurnRecord.token_usage`` is left - unset. Cost/token reporting for a hydrated result is therefore incomplete; - only trajectory-shaped criteria are the target here. -- Sub-agent nesting is flattened: ``subagent_trajectories`` steps are appended - to the parent turn's commands (via their tool_calls) rather than - reconstructing a nested ``parent_tool_use_id`` relationship. -- Turn boundaries are recovered by splitting on ``source="user"`` steps - (mirroring ``atif_emit``'s "one synthetic user step per turn" convention), - not by any explicit iteration marker ATIF carries. +The reverse of ``atif_emit``: reconstructs enough of coder-eval's trajectory shape to +let the criteria that read it work during a separate ``coder-eval evaluate --format +harbor`` invocation. Those checkers read only ``TurnRecord.commands`` and +``TurnRecord.messages``, which is exactly what this rebuilds. + +Deliberately NOT a lossless round trip: per-generation token buckets are not +recovered (``token_usage`` is left unset, so cost reporting for a hydrated result is +incomplete), sub-agent nesting is flattened, and turn boundaries are recovered by +splitting on ``source="user"`` steps. + +Rationale: .claude/notes/reporting.md § Hydrating """ from __future__ import annotations diff --git a/src/coder_eval/harbor/atif_models.py b/src/coder_eval/harbor/atif_models.py index 22ab153b6..fa41558b7 100644 --- a/src/coder_eval/harbor/atif_models.py +++ b/src/coder_eval/harbor/atif_models.py @@ -1,26 +1,19 @@ """Vendored ATIF (Agent Trajectory Interchange Format) models, v1.7. -Mirrors the schema in ``harbor.models.trajectories`` (harbor 0.22.0) so -coder_eval can emit and parse ATIF trajectories with ZERO runtime dependency -on the ``harbor`` pip package. Fidelity is guarded by a frozen fixture in -``tests/fixtures/atif/`` that was validated once against the real harbor -models (see ``tests/test_atif_models.py`` for the reproducible procedure). - -Deliberate deviations from harbor's models: - -- ``schema_version`` is a pattern-validated ``str`` (``^ATIF-v1\\.\\d+$``) - instead of a closed Literal, so trajectories written by a FUTURE harbor - minor version (e.g. ``ATIF-v1.9``) still parse — harbor 0.22.0 itself - would reject them. Major-version bumps (``ATIF-v2.0``) are rejected. -- harbor's ``Agent`` model is named :class:`AtifAgent` here to avoid clashing - with ``coder_eval.agent.Agent``. -- ``ContentPart.source`` (image payloads) is an untyped dict — coder_eval - emits text-only content and only needs to *tolerate* image parts on read. - -Deliberate deviation from the repo convention "all models importable from -``coder_eval.models``": these are interchange-format models for Harbor -interop, not evaluation models — they are exported from ``coder_eval.harbor`` -to keep the core model namespace clean. +Mirrors the schema in ``harbor.models.trajectories`` so coder_eval can emit and parse +ATIF trajectories with ZERO runtime dependency on the ``harbor`` package. Fidelity is +guarded by a frozen fixture validated once against the real models (see +``tests/test_atif_models.py`` for the reproducible procedure). + +``schema_version`` is a pattern-validated ``str`` rather than a closed Literal, so a +FUTURE harbor minor version still parses; major-version bumps are rejected. harbor's +``Agent`` is renamed :class:`AtifAgent` here to avoid clashing with +``coder_eval.agent.Agent``. + +These live outside ``coder_eval.models`` on purpose: interchange models for Harbor +interop, not evaluation models. + +Rationale: .claude/notes/reporting.md § The ATIF trajectory bridge """ from __future__ import annotations diff --git a/src/coder_eval/harbor/experiment_packager.py b/src/coder_eval/harbor/experiment_packager.py index a8e3e6141..264d75aee 100644 --- a/src/coder_eval/harbor/experiment_packager.py +++ b/src/coder_eval/harbor/experiment_packager.py @@ -32,9 +32,7 @@ from coder_eval.orchestration.experiment import DEFAULT_EXPERIMENT_PATH, load_experiment, resolve_all_tasks -# Sources a resolved field's ConfigLineageEntry can carry (see -# models/results.py::ConfigLineageEntry). Only these two mean "the experiment -# introduced this, the task's own YAML did not." +# Only these two mean "the experiment introduced this, the task's own YAML did not". _EXPERIMENT_INTRODUCED_SOURCES = {"variant", "experiment-defaults"} @@ -106,9 +104,8 @@ def _out_subdir(out_dir: Path, resolved: ResolvedTask, *, needs_replicate_segmen dest = dest / resolved.task.row_id if needs_replicate_segment: dest = dest / f"rep{resolved.replicate_index:02d}" - # `resolve()` on a path that doesn't exist yet still normalizes `..` - # segments against its (existing) parents, so this catches traversal - # without requiring `dest` to already exist. + # `resolve()` normalizes `..` against existing parents even for a path that + # does not exist yet, so this catches traversal without requiring `dest`. if not dest.resolve().is_relative_to(out_dir_resolved): raise UnsafeExportPathError( f"resolved export path for variant {resolved.variant_id!r}, task {resolved.task.task_id!r} " diff --git a/src/coder_eval/harbor/packager.py b/src/coder_eval/harbor/packager.py index 239c4df8f..76a7cbb78 100644 --- a/src/coder_eval/harbor/packager.py +++ b/src/coder_eval/harbor/packager.py @@ -1,55 +1,32 @@ """C2 — the packager. ``coder-eval export --format harbor -o ``. -Emits a Harbor task directory from a coder-eval task, with coder-eval's own -criteria as the grader (C1.1's verifier shim). Task *definition* only; the -runtime contract (C1.1's reward writer) is what makes the emitted -``tests/test.sh`` actually work once Harbor runs it. +Emits a Harbor task directory from a coder-eval task, with coder-eval's own criteria +as the grader. Task *definition* only; ``harbor/reward.py`` is the runtime contract +that makes the emitted ``tests/test.sh`` work once Harbor runs it. -Emitted layout (see ``tmp/harborframework.md`` § C2 for the full mapping -table): +Emitted layout:: / ├── task.toml - ├── instruction.md # fixed placeholder -- the real prompt is in environment/task.yaml + ├── instruction.md # placeholder -- the real prompt is in environment/task.yaml ├── environment/ - │ ├── Dockerfile # ONLY when sandbox.docker.dockerfile_path is set (real - │ │ # RUN build steps needed) -- copied in, WORKDIR pinned. - │ │ # Otherwise absent entirely: task.toml's - │ │ # [environment].docker_image names sandbox.docker.image - │ │ # directly and Harbor skips building (see _write_environment) - │ ├── task.yaml # criteria-free copy for the CoderEvalAgent embed -- - │ │ # bind-mounted in, not COPY'd (see docker-compose.yaml) - │ └── docker-compose.yaml # ALWAYS written: read-only mount of task.yaml itself, - │ # plus one read-only bind mount per `type: local` - │ # agent.plugins[] entry and per TemplateDirSource in - │ # sandbox.template_sources, plus one bind mount (its own - │ # ro/rw mode kept) per sandbox.docker.extra_mounts entry -- - │ # same host path in and out (mirrors docker_runner.py's own - │ # auto-mount; see _write_docker_compose_mounts). Nothing is - │ # ever COPY'd into the image anymore. + │ ├── Dockerfile # ONLY when sandbox.docker.dockerfile_path is set; + │ │ # otherwise absent, and task.toml's + │ │ # [environment].docker_image names the image directly + │ ├── task.yaml # criteria-free copy for the CoderEvalAgent embed + │ └── docker-compose.yaml # ALWAYS written. Read-only mounts: task.yaml itself, + │ # each `type: local` agent.plugins[] entry, each + │ # TemplateDirSource; sandbox.docker.extra_mounts keep + │ # their own ro/rw mode. Nothing is COPY'd into the image. └── tests/ - ├── test.sh # C1.1's two-line shim - ├── task.yaml # the criteria, as authored - └── reference/ # task.reference, verifier-side only - -One design point worth stating explicitly because it is not obvious from -either side's docs: ``tests/task.yaml`` (uploaded whole into the container at -``/tests/`` by Harbor's own verifier — see ``_TEST_SH_TEMPLATE``, which -therefore references it as ``/tests/task.yaml``, not a cwd-relative path; -confirmed live against real docker, not assumed) must NOT set -``agent: {type: none}`` -even though C1.1's own docstring describes the verifier phase as exactly -that. coder-eval's own ``check_none_agent`` validator rejects any criterion -with ``requires_agent=True`` — which includes ``reference_comparison``, a -criterion this module treats as portable (C1.4) — the moment ``agent.type`` -is literally ``none``, regardless of whether an agent actually runs. -``coder-eval evaluate `` (the bare-task/work-dir path -C1.1's shim calls) never instantiates an agent no matter what ``agent.type`` -says (see ``evaluate_command.py``'s own comment: "no agent is created" is -true unconditionally on that path) — so a placeholder real agent type plus a -placeholder prompt satisfies coder-eval's validators without changing -behaviour, and unblocks ``reference_comparison``. Verified directly (not -inferred) before writing this module. + ├── test.sh # the two-line shim + ├── task.yaml # the criteria, as authored + └── reference/ # task.reference, verifier-side only + +``tests/task.yaml`` is uploaded whole into the container at ``/tests/``, which is why +``_TEST_SH_TEMPLATE`` references it absolutely rather than cwd-relative. + +Rationale: .claude/notes/reporting.md § Harbor export """ from __future__ import annotations @@ -75,9 +52,9 @@ DEFAULT_WORKDIR = "/app" _HARBOR_SCHEMA_VERSION = "1.4" # pinned to the harbor 0.22.0 findings in tmp/harborframework.md § C0 -# The task.yaml this module emits is graded via `coder-eval evaluate`, which -# never instantiates an agent on the bare-task/work-dir path regardless of -# `agent.type` (see module docstring) — this prompt is never read. +# Graded via `coder-eval evaluate`, which never instantiates an agent on the +# bare-task path regardless of `agent.type` -- this prompt is never read. +# Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml _VERIFIER_PLACEHOLDER_PROMPT = ( "(unused placeholder — this file is graded via `coder-eval evaluate`, which does not invoke an agent on this path)" ) @@ -185,12 +162,9 @@ def export_resolved_task( ) if task.dataset is not None: - # `export_task`/`export_resolved_task` calls `load_task`, which does - # NOT run `expand_dataset` -- fan-out happens later, in the experiment - # pipeline (`export_experiment` resolves it per row before reaching - # here). Exporting a raw dataset-backed task would silently emit ONE - # Harbor task whose prompt and criteria still contain literal - # `${row.}` placeholders: never expressible, but scored anyway. + # `load_task` does NOT run `expand_dataset` -- fan-out happens later, so + # exporting a raw dataset-backed task would emit ONE Harbor task still + # carrying literal `${row.}` placeholders. raise TaskNotExportableError( f"Task {task.task_id!r} has a `dataset:` block, which this function does not expand -- its " + "`${row.*}` placeholders would export unsubstituted. Export via `coder-eval export ... " @@ -277,36 +251,18 @@ def _write_environment( ) -> tuple[str, str | None]: """Derive ``environment/`` and return ``(workdir, docker_image)``: - - ``workdir``: the WORKDIR both it and test.sh must agree on. Per C0 § 5, - Harbor has no fixed workspace path — the verifier (default SHARED mode) - runs at whatever the container's own WORKDIR is. This function is the one - place that decides it, so nothing downstream can silently disagree. - - ``docker_image``: set only when no ``environment/Dockerfile`` was written at - all, so ``_write_task_toml`` can point ``task.toml``'s ``[environment].docker_image`` - straight at the pre-built image; ``None`` when a Dockerfile was written and - Harbor must build from it instead. - - Two shapes: - - - ``dockerfile_path`` set: the task needs real build steps (``RUN`` etc.) that - only a Dockerfile can express, so it's copied in as the base (appending a - `WORKDIR` if it declared none). Returns ``(workdir, None)``. - - unset: no Dockerfile is written at all. Harbor's own - ``should_use_prebuilt_docker_image`` (``harbor/environments/definition.py``) - already supports pulling ``task.toml``'s ``[environment].docker_image`` - directly and skipping the build step entirely (confirmed against a real - ``harbor`` install) — `WORKDIR` doesn't need a Dockerfile line either: - ``[environment].workdir`` is what Harbor's docker environment passes as the - ``cwd``/``-w`` at ``docker exec`` time (``docker.py``), independent of - whether the image was built or pulled prebuilt. Returns - ``(workdir, docker_cfg.image)`` — ``docker_cfg.image`` always has a value - (default_factory=``get_default_docker_image_tag``), so this is a real - choice, not a null-vs-set distinction. - - ``environment/task.yaml`` itself is bind-mounted, not ``COPY``'d, in at - :data:`AGENT_TASK_YAML_PATH` for the ``CoderEvalAgent`` Harbor-agent embed - to read (see ``_write_docker_compose_mounts``) — neither shape's Dockerfile - (when one exists at all) plays any part in getting it there. + - ``workdir``: the WORKDIR both it and test.sh must agree on. This function is + the one place that decides it, so nothing downstream can silently disagree. + - ``docker_image``: set only when no ``environment/Dockerfile`` was written, so + ``task.toml``'s ``[environment].docker_image`` points straight at the pre-built + image; ``None`` when a Dockerfile was written and Harbor must build from it. + + ``dockerfile_path`` set is the only shape that writes a Dockerfile: it is copied in + as the base, with a ``WORKDIR`` appended when it declared none. Unset writes none. + ``environment/task.yaml`` is bind-mounted at :data:`AGENT_TASK_YAML_PATH`, never + ``COPY``'d, so no Dockerfile is involved in getting it there. + + Rationale: .claude/notes/reporting.md § What the export carries, and what it refuses to carry """ env_dir = out_dir / "environment" env_dir.mkdir(parents=True, exist_ok=True) @@ -331,9 +287,8 @@ def _write_environment( ) if not _from_line_mentions_coder_eval_agent(dest_dockerfile): warnings.append(_MISSING_CODER_EVAL_WARNING) - # Non-Dockerfile build context (COPY sources etc.) is not carried over in - # v1 — a dockerfile_path build context beyond the Dockerfile itself needs - # its own decision (open question) before this is safe to widen. + # A build context beyond the Dockerfile itself is not carried over in v1; + # widening it needs its own decision. if source_dockerfile.parent != task_file.parent: other_files = [p for p in source_dockerfile.parent.iterdir() if p != source_dockerfile] if other_files: @@ -442,19 +397,14 @@ def _write_verifier_task_yaml(task: TaskDefinition, out_dir: Path) -> None: if task.reference is not None: payload["reference"] = {"directory": "reference"} if task.run_limits is not None: - # Carried through so `coder-eval evaluate` (run by tests/test.sh inside - # Harbor's verifier container) sees the same turn/token/USD caps the - # task author declared, rather than silently falling back to the - # packaged default experiment's -- grading itself has no agent loop to - # cap, but `run_limits.task_timeout` still bounds the verifier - # invocation, and a future criterion or judge call reading `run_limits` - # off the resolved task should see the real value, not the default. + # Carried through so the verifier sees the caps the task author declared + # rather than the packaged default experiment's: grading has no agent loop + # to cap, but `task_timeout` still bounds the verifier invocation. + # Rationale: .claude/notes/reporting.md § What the export carries, and what it refuses to carry payload["run_limits"] = task.run_limits.model_dump(mode="json", exclude_none=True) if task.checker_context is not None: - # The judge-route override (`checker_context.api_route`) determines - # which backend an `llm_judge`/`agent_judge` criterion dispatches - # through at verify time -- dropping it silently replaces a pinned - # route with the verifier environment's own default. + # Dropping the judge-route override silently replaces a pinned route with + # the verifier environment's own default. payload["checker_context"] = task.checker_context.model_dump(mode="json", exclude_none=True) (out_dir / "tests" / "task.yaml").write_text( yaml.safe_dump(payload, sort_keys=False, allow_unicode=True), encoding="utf-8" @@ -610,36 +560,23 @@ def _write_agent_phase_task_yaml( ) -> None: """``environment/task.yaml`` — the REAL agent config, but criteria-free. - Bind-mounted read-only in at :data:`AGENT_TASK_YAML_PATH` (see - ``_write_docker_compose_mounts``, called after this from ``_write_environment``) - for a ``CoderEvalAgent`` Harbor agent embed (``harbor/agent.py``) to run via - ``coder-eval execute --format harbor``. Unlike ``tests/task.yaml`` (the - verifier's placeholder-agent, real-criteria file), this is the mirror image: - ``task.agent`` and the resolved prompt are carried over VERBATIM (the whole - point is running the task's actual configured agent), but ``success_criteria`` - is forced to ``[]`` — never leaked into the agent-visible container, and never - read either (`coder-eval execute` never grades). ``TaskDefinition`` no longer - requires at least one criterion, so this no longer needs a placeholder. - - ``sandbox`` is the original task's ``sandbox`` block, field-merged with - ``driver: tempdir`` — everything else (``python.env_packages``, ``limits``, - ``template_sources``, ...) is preserved verbatim, not blanked. - ``template_sources``/``agent.plugins[]`` paths need no rewriting: they're - bind-mounted at their own unchanged host path (see ``_write_docker_compose_mounts``), - so whatever absolute path ``model_dump()`` already carries is correct as-is. - ``driver`` must be forced regardless of the original task's driver: this file - runs INSIDE the container Harbor already built, so re-declaring ``driver: - docker`` here would have ``coder-eval execute`` try to launch a second, nested - container rather than just using its own in-process sandbox at the container's - current workdir. ``docker`` config is dropped along with it — moot once - ``driver`` is forced to ``tempdir``. - - ``initial_prompt`` is omitted entirely for a ``type: none`` (agentless) - task: coder-eval's own schema forbids a no-op agent from setting a prompt - (no agent ever runs to read it) and raises at load time otherwise -- - verified live via ``harbor run`` against a real ``harbor`` install, which - surfaced exactly this ``TaskDefinition`` validation error before this - guard was added. + Bind-mounted read-only at :data:`AGENT_TASK_YAML_PATH` (see + ``_write_docker_compose_mounts``) for a ``CoderEvalAgent`` embed to run via + ``coder-eval execute --format harbor``. ``task.agent`` and the resolved prompt + carry over VERBATIM; ``success_criteria`` is forced to ``[]``, so it is never + leaked into the agent-visible container. + + ``sandbox`` is field-merged with ``driver: tempdir``, which MUST be forced + regardless of the original: this file runs inside the container Harbor already + built, so ``driver: docker`` here would launch a second, nested one. ``docker`` + config is dropped with it. Everything else is preserved verbatim, and + ``template_sources`` / ``agent.plugins[]`` paths need no rewriting -- they are + bind-mounted at their own unchanged host path. + + ``initial_prompt`` is omitted entirely for an agentless task -- the schema + forbids a no-op agent from setting one and raises at load time. + + Rationale: .claude/notes/reporting.md § What the export carries, and what it refuses to carry """ is_agentless = task.agent is not None and task.agent.type == "none" sandbox_dict = task.sandbox.model_dump(mode="json", exclude_none=True) @@ -682,12 +619,9 @@ def _write_agent_phase_task_yaml( def _write_test_sh(out_dir: Path, *, workdir: str) -> None: path = out_dir / "tests" / "test.sh" - # `workdir` comes from task-YAML-controlled `sandbox.docker.working_dir` - # (or a derived default), and the only validator on that field checks for a - # leading "/" -- it does not reject quotes, `$(...)`, backticks or - # newlines. shlex.quote it before interpolating into the generated /bin/sh - # script so a crafted working_dir can't break out of the argument it's - # meant to be. + # HAZARD: `workdir` is task-authored and its only validator checks for a + # leading "/" -- it does not reject quotes, `$(...)`, backticks or newlines. + # shlex.quote before interpolating into the generated /bin/sh script. path.write_text(_TEST_SH_TEMPLATE.format(workdir=shlex.quote(workdir)), encoding="utf-8") path.chmod(0o755) @@ -699,13 +633,9 @@ def _write_reference(task: TaskDefinition, task_file: Path, out_dir: Path) -> No dest = out_dir / "tests" / "reference" if dest.exists(): shutil.rmtree(dest) - # Same rule as the template copy above: drop symlinks rather than - # dereferencing them into the distributable export. Wrapped in - # TaskNotExportableError rather than left to raise a bare OSError: the - # `export` CLI only catches `(TaskNotExportableError, - # CriteriaNotExportableError)`, so an unreadable/missing reference tree - # would otherwise surface as an uncaught traceback (single-task path) or - # abort a whole experiment export the docstring promises it won't abort. + # Drop symlinks, as above. Wrapped in TaskNotExportableError rather than left + # to raise a bare OSError: the `export` CLI catches only the export errors, so + # an unreadable tree would abort an experiment export that promises not to. try: shutil.copytree(source, dest, ignore=ignore_patterns_and_symlinks(REFERENCE_COPY_IGNORE)) except OSError as e: @@ -740,13 +670,10 @@ def _write_task_toml(task: TaskDefinition, out_dir: Path, *, workdir: str, docke _HARBOR_ENV_PASSTHROUGH_EXCLUDE = { - # `HOME` is intentional in `env_passthrough`'s default ONLY because - # `docker_runner.py` also bind-mounts the host's `~/.claude` into the - # container at that same path, so the host `HOME` value still resolves to - # a real, populated directory there (see docs/DOCKER_ISOLATION.md). Harbor - # builds its own container with no such mount, so forwarding the host's - # literal `HOME` (e.g. `/Users/alice`) would point the container at a - # directory that doesn't exist in it -- a real regression, not a no-op. + # HAZARD: `HOME` is in the default passthrough ONLY because docker_runner + # bind-mounts ~/.claude at that same path. Harbor builds its own container + # with no such mount, so forwarding the host's literal HOME is a regression. + # Rationale: .claude/notes/reporting.md § What the export carries, and what it refuses to carry "HOME", } diff --git a/src/coder_eval/harbor/portability.py b/src/coder_eval/harbor/portability.py index 41896629d..db5d489de 100644 --- a/src/coder_eval/harbor/portability.py +++ b/src/coder_eval/harbor/portability.py @@ -1,37 +1,22 @@ """C1.4 — criteria portability audit for the Harbor export direction. -Not every criterion type can grade truthfully inside a verifier container -that another harness built. This module classifies each of the 15 criterion -types and lets the packager (C2) refuse an unsupported task **at export -time**, where the operator sees why, rather than at verify time, where the -failure is an unexplained low reward with no obvious cause. +Not every criterion type can grade truthfully inside a verifier container another +harness built. This module classifies each of the 15 types so the packager can refuse +an unsupported task **at export time**, where the operator sees why. Classification, v1: -- ``PORTABLE`` — filesystem/exit-code checks. Nothing about the verifier - container changes what these need: ``file_exists``, ``file_contains``, - ``file_matches_regex``, ``json_check``, ``file_check``, ``run_command``, - ``classification_match``. -- ``NEEDS_REFERENCE`` — ``reference_comparison``. Needs the reference tree, - which C2 places under ``tests/reference/`` (verifier-side only, never in - the agent's view — see C2's mapping table). +- ``PORTABLE`` — filesystem/exit-code checks. +- ``NEEDS_REFERENCE`` — ``reference_comparison``; never actually blocking, since the + export always emits ``tests/reference/`` when the task declares one. - ``NEEDS_TRAJECTORY`` — ``command_executed``, ``commands_efficiency``, - ``skill_triggered``. These read coder-eval's own ``TurnRecord`` iterations. - In the export direction the verifier is a separate process from the agent - phase, and the agent may not even be coder-eval (Harbor natively supports - many agents) — so there is no ``iterations`` list to read from without - C1.3 (ATIF ingestion + ``evaluate --trajectory``), which is not built yet. - Hard-error until it is. -- ``NEEDS_CLI_RECORDER`` — ``cli_called``. Reads coder-eval's own JSON Lines - invocation log (``invocation_log.py``), written by a recorder shim - coder-eval's OWN sandbox setup installs into `PATH` - (``_generate_cli_recorders``). The exported Dockerfile does not provision - that shim. Hard-error until C2 learns to bake it in. + ``skill_triggered``. Hard-error until ATIF ingestion lands. +- ``NEEDS_CLI_RECORDER`` — ``cli_called``. Hard-error until the export bakes the + recorder shim in. - ``NEEDS_CREDENTIALS`` — ``llm_judge``, ``agent_judge``, ``uipath_eval``. - Need model credentials and network reachable from inside the verifier - container, and the judge must not follow the agent's own route (the old - note's blocking issues). Hard-error in v1, with an explicit opt-in escape - hatch for an operator who has already provisioned that themselves. + Hard-error, with an opt-in escape hatch for an operator who has provisioned it. + +Rationale: .claude/notes/reporting.md § Not every criterion can grade inside someone else's container """ from __future__ import annotations diff --git a/src/coder_eval/harbor/reward.py b/src/coder_eval/harbor/reward.py index 265028ca5..c23744349 100644 --- a/src/coder_eval/harbor/reward.py +++ b/src/coder_eval/harbor/reward.py @@ -1,39 +1,17 @@ """Translate a graded coder-eval run into Harbor's reward-file contract. -Harbor's verifier (``harbor 0.22.0``, verified against source — see -``tmp/harborframework.md`` § C0) reads ``/logs/verifier/reward.json`` (a flat -``dict[str, float]``) or falls back to ``/logs/verifier/reward.txt`` (a bare -float, synthesized into the single key ``"reward"``). It does NOT inspect the -verifier script's exit code — only whether the reward file exists, is -non-empty, and parses. A missing/empty/malformed file raises inside Harbor's -own ``Verifier.verify()`` (``RewardFileNotFoundError`` / -``RewardFileEmptyError`` / ``VerifierOutputParseError``), which Harbor's -``Trial.run()`` catches: it records the exception on ``TrialResult`` and -leaves ``TrialResult.verifier_result`` at its default ``None`` — never -coalesced to a zero-reward object. That is Harbor's own infra-vs-policy split, -already built. - -This module's whole job, therefore, is: **write the file, or don't.** - -The "don't" case is not a degenerate corner — it is the load-bearing one. An -unmeasured row (``weighted_score is None`` — an ungraded row, or a task.json -that failed to load at all) must not become ``reward=0.0``: that would train -"the agent's behaviour was bad" from a measurement that never happened. Not -writing the file lets Harbor's own missing-reward path mask the trial -instead — the same principle as CE049 (never coalesce a possibly-unmeasured -score to a numeric literal), one level up, at the artifact-writing boundary -rather than the in-process one. - -A grading-time INFRASTRUCTURE failure is the same case in disguise: -``EvaluationResult.calculate_weighted_score`` (``models/results.py``) -short-circuits an empty ``success_criteria_results`` list to a hard ``0.0``, -not ``None`` -- so a checker that raises ``JudgeInfrastructureError`` / -``CheckerMisuseError`` / ``ReferenceTamperedError`` (escalating exceptions -that deliberately propagate out of grading rather than being captured into a -scored-0.0 result) finalizes the row ``FinalStatus.ERROR`` with -``weighted_score == 0.0``, not ``None``. That is not a measurement either, so -it gets the same "write nothing" treatment via ``final_status.category == -"error"``. +Harbor's verifier reads ``/logs/verifier/reward.json`` (a flat ``dict[str, float]``) +or falls back to ``reward.txt`` (a bare float). It does NOT inspect the verifier +script's exit code — only whether the file exists, is non-empty, and parses. A +missing or malformed one raises inside Harbor's own verify step, which its trial +runner records rather than coalescing to a zero-reward object. + +This module's whole job is therefore: **write the file, or don't.** An unmeasured row +must not become ``reward=0.0``, and a grading-time infrastructure failure (which +finalizes as ERROR with a score of ``0.0``, not ``None``) is the same case in +disguise. + +Rationale: .claude/notes/reporting.md § Write the reward file, or do not """ from __future__ import annotations diff --git a/src/coder_eval/invocation_log.py b/src/coder_eval/invocation_log.py index 420b55252..86ce543b8 100644 --- a/src/coder_eval/invocation_log.py +++ b/src/coder_eval/invocation_log.py @@ -25,9 +25,8 @@ from coder_eval.models import RECORD_CLI_LOG_NAME, SIDECAR_MODULES, RecordedCli -# The shim imports exactly one of them -- the argv matcher. A second sidecar -# would need its own import line, so this unpacks rather than indexing: adding -# one is then a loud failure here instead of a silently un-imported file. +# Unpacks rather than indexing, so adding a second sidecar is a loud failure here +# instead of a silently un-imported file. (_SIDECAR_MODULE,) = SIDECAR_MODULES _SIDECAR_MODULE_STEM = _SIDECAR_MODULE.removesuffix(".py") @@ -175,13 +174,10 @@ def main(argv): ''' -# Rendered into the shim only when the entry declares rules. Every line of the -# comment is addressed at whoever opens a generated shim inside a sandbox, which -# is why the reasoning lives in the emitted text rather than only here. -# -# The module name is derived from SIDECAR_MODULES rather than written out, so -# renaming the sidecar cannot leave this import pointing at a file that no -# longer exists -- the one failure the write side would not catch. +# Rendered into the shim only when the entry declares rules; the reasoning lives in +# the EMITTED text because its reader opens the shim inside a sandbox. The module +# name is derived from SIDECAR_MODULES, so renaming the sidecar cannot leave this +# import pointing at a file that no longer exists. _SIDECAR_IMPORT = f"""\ # {_SIDECAR_MODULE} is written beside this shim by coder_eval SandboxConfig.record_cli. # Loaded by ABSOLUTE PATH, not by name: a plain `import {_SIDECAR_MODULE_STEM}` resolves diff --git a/src/coder_eval/litellm_cost.py b/src/coder_eval/litellm_cost.py index 28c4bca11..e00410d58 100644 --- a/src/coder_eval/litellm_cost.py +++ b/src/coder_eval/litellm_cost.py @@ -1,44 +1,19 @@ """Join proxy-captured ACTUAL per-call cost/cache onto a run's turns. -For the open-weight (LiteLLM) backend the Claude binary's Anthropic transport -drops OpenRouter's real ``usage.cost`` + per-call cache before Python can see it, -so a proxy-side callback (``litellm/cost_logger.py``) writes one JSONL record per -call to ``LITELLM_COST_LOG``. This module reads those records back and joins them -onto the matching turns at the TURN level: - -* the turn's ``token_usage.total_cost_usd`` is overridden with the SUM of its - calls' real cost (the bill), replacing the static rate-card estimate; -* the per-call breakdown is attached as ``TurnRecord.provider_call_costs`` — a - deterministic audit record (one row per real proxy call, with its real cost + - cache buckets) that the evalboard renders as a per-call table. - -Token buckets are LEFT UNTOUCHED (SDK-authoritative): the join only writes cost, -so the ``EventCollector`` remains the single writer of the message token-bucket -invariant. There is deliberately NO per-generation distribution — matching a proxy -call to a transcript generation has no deterministic key (only positional / -output-token heuristics), so that view lives in the per-call table off -``provider_call_costs`` instead of being guessed onto the message stream. - -Coverage. A turn's cost is overridden only when every call that reported usage is -priced. Degenerate calls that report NO usage at all (no cost AND no tokens — seen -occasionally on some providers) are ignored, so one of them can't revert a whole -turn to the static estimate. A call that reports usage but no cost is a genuine -gap: the turn keeps its static estimate (overriding would bill it at $0), no -breakdown is attached, and a warning names the unpriced ids. A turn with no -matching record keeps its static estimate too. - -Retry safety: multiple ``TurnRecord``s can share an ``iteration`` (a crashed -attempt + its retry), and both attempts' proxy calls carry that iteration tag. An -iteration's calls are credited to a single survivor turn (the last with that -iteration THAT HAS GENERATIONS); earlier siblings are zeroed. Crucially the -credit/zero decision is made TOGETHER: a sibling is only zeroed when the survivor -is actually credited with real cost — if the survivor falls back to static, the -sibling keeps its static estimate too, so the iteration's spend is never dropped. - -Transactional: the full plan is computed before any turn is mutated, so a -malformed record (which raises while building the per-call breakdown) aborts the -whole join with the run left untouched — matching the caller's "keeping static -pricing" contract. +For the open-weight (LiteLLM) backend the real ``usage.cost`` never reaches Python, +so a proxy-side callback writes one JSONL record per call to ``LITELLM_COST_LOG``. +This module joins those records onto the matching turns at the TURN level: the +turn's ``token_usage.total_cost_usd`` is overridden with the SUM of its calls' real +cost, and the per-call breakdown is attached as ``TurnRecord.provider_call_costs``. + +Token buckets are LEFT UNTOUCHED (SDK-authoritative), so ``EventCollector`` remains +the single writer of the message token-bucket invariant. There is deliberately NO +per-generation distribution. + +A turn keeps its static estimate whenever the join cannot be trusted: a call that +reported usage but no cost, or no matching record at all. + +Rationale: .claude/notes/reporting.md § Cost joining """ from __future__ import annotations diff --git a/src/coder_eval/logging_config.py b/src/coder_eval/logging_config.py index 73c788bd7..e344900a9 100644 --- a/src/coder_eval/logging_config.py +++ b/src/coder_eval/logging_config.py @@ -25,9 +25,8 @@ APP_LOGGER_NAME = "coder_eval" -# Bounded ring-buffer used by ``_LogTailBuffer`` to capture a sanitised tail of -# task logs for the HTML report. Sized so a 200 KB tail comfortably covers the -# last few hundred lines of a typical run without bloating ``task.json``. +# Sized so a 200 KB tail covers the last few hundred lines of a typical run without +# bloating ``task.json``. DEFAULT_LOG_TAIL_MAX_BYTES = 200_000 # ANSI CSI escape sequences (e.g. ``\x1b[31m``). Stripped from the buffered tail @@ -75,11 +74,9 @@ def emit(self, record: logging.LogRecord) -> None: self._size -= len(old.encode("utf-8")) def get_text(self) -> str: - # Acquire the handler's lock so concurrent emit() calls (which also - # hold self.lock via Handler.handle()) cannot mutate _records while - # we iterate it for the join. Without this, a watchdog/proxy thread - # logging during the finally-block tail capture would raise - # "RuntimeError: deque mutated during iteration". + # HAZARD: hold the handler's lock, which concurrent emit() calls also take, + # or a watchdog thread logging during the tail capture raises "deque mutated + # during iteration". self.acquire() try: return _sanitise_log_text("".join(self._records)) @@ -87,9 +84,8 @@ def get_text(self) -> str: self.release() -# ContextVar that tracks the current task_id for the running async context. -# Each asyncio task gets its own copy, so parallel tasks are isolated. -# Set by task_log_handler; read by _TaskIdFilter to inject into plain-logger records. +# Each asyncio task gets its own copy, so parallel tasks are isolated. Set by +# task_log_handler; read by _TaskIdFilter. _current_task_id: ContextVar[str | None] = ContextVar("_current_task_id", default=None) # ANSI color codes for terminal output @@ -265,33 +261,9 @@ def task_log_handler( ) -> Generator[_LogTailBuffer]: """Context manager for task-specific logging. - Attaches a ``FileHandler`` (raw bytes, uncapped) and a sibling - ``_LogTailBuffer`` (sanitised, bounded) to the app logger at the start and - removes both at the end, guaranteeing cleanup even if exceptions occur. The - buffer is yielded directly so callers can call ``get_text()`` to capture a - sanitised tail of the log alongside the on-disk file. - - When task_id is provided, a filter is applied so that in parallel batch runs - each task's log file only contains its own messages. The ContextVar - ``_current_task_id`` is set so that plain loggers (no LoggerAdapter) - automatically get the correct task_id injected by ``_TaskIdFilter``. - - Thread-safe: uses a lock and reference counting so that concurrent handlers - correctly restore the original log level when the last handler exits. The - buffer is a passive sibling and does NOT participate in the refcount. - - Args: - task_log_file: Path to task log file - level: Logging level for file output (default: DEBUG) - task_id: Optional task ID for filtering in parallel runs - - Yields: - ``_LogTailBuffer`` exposing ``get_text()`` for the sanitised log tail. - - Example: - >>> with task_log_handler(Path("task.log"), task_id="my_task") as log_tail: - ... logger.info("This goes to both console and task.log") - ... tail_text = log_tail.get_text() + Attaches a file handler for the task's own log, sets the task-id ContextVar so + parallel tasks stay isolated, and yields the bounded tail buffer the HTML report + reads. """ # Create handler handler = logging.FileHandler(task_log_file, mode="w", encoding="utf-8") @@ -384,10 +356,9 @@ def aggregate_task_logs(run_dir: Path) -> None: outfile.write("\n" + "=" * 80 + "\n\n") for task_log_file in task_log_paths: - # Use the path relative to run_dir so nested task ids from dataset - # fan-out (variant/suite/row) render with full context, not just - # the leaf directory name. as_posix() keeps the header consistent - # across platforms (experiment.log is commonly shared / pasted). + # Relative to run_dir, so a dataset-fanned task id renders with full + # context rather than just its leaf. as_posix() keeps the header + # consistent across platforms. task_id = task_log_file.parent.relative_to(run_dir).as_posix() outfile.write(f"\n{'=' * 80}\n") outfile.write(f"TASK: {task_id}\n") diff --git a/src/coder_eval/plugins.py b/src/coder_eval/plugins.py index 79f9afe3c..e8dc7a1d6 100644 --- a/src/coder_eval/plugins.py +++ b/src/coder_eval/plugins.py @@ -28,9 +28,8 @@ PLUGIN_ENTRY_POINT_GROUP = "coder_eval.plugins" -# coder-eval's own built-in agents register through this same entry point. A -# failure registering it is a real breakage (empty registry), NOT a skippable -# third-party plugin error — so it is fatal rather than logged-and-skipped. +# The built-in agents register through this same entry point, so a failure here is +# a real breakage (empty registry), not a skippable third-party plugin error. BUILTIN_PLUGIN_NAME = "coder_eval" _loaded = False @@ -57,9 +56,8 @@ def load_plugins(*, force: bool = False) -> None: register = ep.load() register(AgentRegistry) except Exception: - # Built-in registration failing leaves the registry empty and would - # surface later as a misleading "No agent registered for 'claude-code'". - # Keep it fatal so the real (import/registration) cause fails loudly. + # Fatal: otherwise it surfaces later as a misleading "No agent + # registered for 'claude-code'" instead of the real cause. if ep.name == BUILTIN_PLUGIN_NAME: # Clear the flag so a caller that catches and retries re-runs the # scan instead of getting a no-op against an empty registry. diff --git a/src/coder_eval/pricing.py b/src/coder_eval/pricing.py index 4289ff404..a0838e741 100644 --- a/src/coder_eval/pricing.py +++ b/src/coder_eval/pricing.py @@ -7,6 +7,10 @@ https://ai.google.dev/gemini-api/docs/pricing, and OpenRouter's live ``/api/v1/models`` (every row re-verified 2026-09-03, except the Bedrock open-weight block: AWS publishes no eu-north-1 figures for those three). + +HAZARD: this table is a HAND-COPIED MIRROR of ``evalboard/lib/pricing.ts``. Editing +one means editing the other; ``evalboard/lib/__tests__/pricing-parity.test.ts`` fails +the build on drift in either direction. """ from collections.abc import Iterable @@ -83,20 +87,16 @@ class ModelPricing: "gpt-5.4-pro": ModelPricing(30.0, 180.0, 30.0, 3.0), "gpt-5.4-mini": ModelPricing(0.75, 4.5, 0.75, 0.075), "gpt-5.4-nano": ModelPricing(0.20, 1.25, 0.20, 0.02), - # GPT-5.6: sol flagship / terra balanced (Codex default) / luna economy. - # This table is a single current-rate card with no notion of an effective - # date, so a repriced model makes historical runs re-price at today's rate. - # Sol's rate is promotional through at least 2026-11-21; re-check then. + # HAZARD: a single CURRENT-rate card with no effective date, so a repriced + # model makes historical runs re-price at today's rate. Sol's rate is + # promotional through at least 2026-11-21; re-check then. "gpt-5.6-sol": ModelPricing(4.0, 20.0, 4.0, 0.40), "gpt-5.6-terra": ModelPricing(2.0, 12.0, 2.0, 0.20), "gpt-5.6-luna": ModelPricing(0.20, 1.20, 0.20, 0.02), - # Google Gemini (AntigravityAgent, via the Gemini Developer API), keyed on the - # literal ids the ListModels endpoint returns. No cache-write fee, so - # cache_write == input (unused: the agent maps cache_creation_tokens to 0). - # CAVEAT: Pro's >200K-token tier costs more ($4/$18, $0.40 cached), so a - # very-large-context run reads low. - # 3.6 / 3.7 / 3.8 Flash share one rate card. These are list rates; Google is - # discounting all three by half through 2026-12-31. + # Keyed on the literal ids ListModels returns. No cache-write fee, so + # cache_write == input (unused). CAVEAT: Pro's >200K-token tier costs more, so + # a very-large-context run reads LOW. List rates; discounted by half through + # 2026-12-31. "gemini-3.8-flash": ModelPricing(1.5, 7.5, 1.5, 0.15), "gemini-3.7-flash": ModelPricing(1.5, 7.5, 1.5, 0.15), "gemini-3.6-flash": ModelPricing(1.5, 7.5, 1.5, 0.15), @@ -110,20 +110,17 @@ class ModelPricing: # Off the public card (superseded by 3.1 Pro); last published rate kept so # historical runs still price. "gemini-3-pro-preview": ModelPricing(2.0, 12.0, 2.0, 0.20), - # Open-weight models on Bedrock, driven via the LiteLLM backend. These are the - # eu-north-1 rates, a ~20% premium over us-east-1 — do NOT "correct" them - # against the US column. Bedrock publishes no prompt-cache rate for these, so - # cache-creation is priced at input and cache-read at 0. + # HAZARD: eu-north-1 rates, a ~20% premium over us-east-1 -- do NOT "correct" + # them against the US column. No published prompt-cache rate, so cache-creation + # is priced at input and cache-read at 0. "deepseek.v3.2": ModelPricing(0.74, 2.22, 0.74, 0.0), "zai.glm-5": ModelPricing(1.2, 3.84, 1.2, 0.0), "moonshotai.kimi-k2.5": ModelPricing(0.72, 3.6, 0.72, 0.0), - # OpenRouter models. These providers cache prefixes implicitly (no - # cache_control, no write fee), so cache-creation is priced at input (unused) - # and cache-read at OpenRouter's published input_cache_read rate, read from - # the live /api/v1/models catalogue. Headline rates only: OpenRouter routes - # per request, so the real bill depends on the provider a call lands on — - # which is why the litellm path captures actual per-call cost proxy-side and - # overrides these (litellm_cost.apply_actual_cost). Static fallback. + # These providers cache prefixes implicitly, so cache-creation is priced at + # input (unused). HEADLINE rates only: OpenRouter routes per request, so the + # real bill depends on the provider a call lands on -- which is why the litellm + # path captures actual per-call cost and overrides these. Static fallback. + # Rationale: .claude/notes/reporting.md § Cost joining "moonshotai/kimi-k3": ModelPricing(3.0, 15.0, 3.0, 0.30), "z-ai/glm-5.2": ModelPricing(0.966, 3.036, 0.966, 0.1932), "deepseek/deepseek-v4-pro": ModelPricing(1.030776, 2.061552, 1.030776, 0.085898), @@ -168,10 +165,8 @@ def register_pricing(rates: dict[str, ModelPricing]) -> None: _REGISTERED_PRICING.update(rates) -# Bedrock cross-region inference-profile prefixes (mirrors -# models.routing._BEDROCK_KNOWN_PREFIXES). A Bedrock route qualifies a bare -# alias into e.g. ``eu.anthropic.claude-opus-4-8``; the pricing table is keyed -# on the bare alias, so we strip these back off before the lookup. +# Mirrors models.routing._BEDROCK_KNOWN_PREFIXES. A Bedrock route qualifies a bare +# alias; the table is keyed on the bare alias, so strip these before the lookup. _BEDROCK_REGION_PREFIXES: tuple[str, ...] = ("eu.", "us.", "apac.", "global.") @@ -183,13 +178,10 @@ def _normalize_model(model: str) -> str: so it is safe to apply unconditionally for every route. """ model = model.strip() - # LiteLLM/Bedrock routing prefixes (e.g. "converse/zai.glm-5", - # "bedrock/converse/deepseek.v3.2") → bare model id. ``openrouter/`` is here - # because agents that address OpenRouter natively (OpenCode) report the model - # WITH its provider prefix ("openrouter/deepseek/deepseek-v4-pro"), - # while the OpenRouter rate-card keys are the bare vendor/model ids that the - # LiteLLM route already uses — without this strip the same model prices under - # LiteLLM and silently goes unpriced under OpenCode. + # Routing prefixes -> bare model id. ``openrouter/`` is here because an agent + # that addresses OpenRouter natively reports the model WITH its provider + # prefix, while the rate-card keys are bare -- without the strip the same model + # prices under LiteLLM and silently goes unpriced under OpenCode. for routing_prefix in ("bedrock/converse/", "bedrock/", "converse/", "openrouter/"): if model.startswith(routing_prefix): model = model[len(routing_prefix) :] diff --git a/src/coder_eval/reports.py b/src/coder_eval/reports.py index 4981b9793..680fce266 100644 --- a/src/coder_eval/reports.py +++ b/src/coder_eval/reports.py @@ -244,16 +244,10 @@ def _pass_rate_lines(summary: RunSummary) -> list[str]: An ungraded run (``coder-eval execute``) has no pass rate at all, so it says so rather than rendering ``0.0% (0/N)`` — which reads as a total failure. """ - # Only an ungraded run gets the explanatory line. An ordinary EMPTY run keeps - # its original "n/a (0/0)" rendering — the two are different facts. - # - # `pass_rate is None` rather than `not tasks_graded`: an execute night with a - # crashed row has tasks_graded > 0 (an ERROR row is category `error`, not - # `ungraded`, so it stays in the denominator) while still having measured - # nothing — and this line then rendered `0.0% (0/5)` plus `Error Share: - # 100.0%`, exactly the total-failure reading the guard exists to prevent. - # Deferring to the model keeps one rule for run.md, run.json and the - # evalboard instead of three. + # `pass_rate is None`, not `not tasks_graded`: an execute night with a crashed + # row has tasks_graded > 0 while still having measured nothing. An ordinary + # EMPTY run keeps its "n/a (0/0)" rendering -- a different fact. + # Rationale: .claude/notes/reporting.md § The ungraded row in every surface if summary.tasks_not_graded and summary.pass_rate is None: return [f"- **Pass Rate**: n/a — {summary.tasks_not_graded} task(s) executed without grading"] lines = [f"- **Pass Rate**: {_fmt_rate(summary.pass_rate)} ({summary.tasks_succeeded}/{summary.tasks_graded})"] @@ -296,11 +290,10 @@ def _generate_command_statistics_section(stats: CommandStatistics) -> list[str]: pct = count / total * 100 if total > 0 else 0 lines.append(f"| {tool} | {count} | {pct:.1f}% |") - # `is not None`, not truthiness. `analysis.py` returns `None` when - # nothing was timed and a float otherwise, so a genuine measured `0.0` - # average — every command resolving faster than the clock's resolution — - # used to suppress the whole section. The distinction the producer makes - # has to survive to the surface that renders it. + # `is not None`, not truthiness: a genuine measured `0.0` average used to + # suppress the whole section. The distinction the producer makes has to + # survive to the surface that renders it. + # Rationale: .claude/notes/reporting.md § An unmeasured value is never zero if stats.avg_command_time_ms is not None: lines.extend( [ @@ -366,12 +359,10 @@ def _generate_generation_metrics_section(task_results: list[dict[str, Any]]) -> else: avg_turn_str = "N/A" - # READ, never summed here. The four values are computed once by - # `reports_stats.turn_time_buckets` and carried on the row by - # `reports_experiment.eval_result_to_task_dict`; `iterations` above - # is a 6-key projection that cannot support the arithmetic anyway. - # `.get()` because a `run.json` written before this phase has none - # of the four — which then renders as a dash, not as `0ms`. + # READ, never summed here -- computed once by `turn_time_buckets`. + # `.get()` because an older `run.json` has none of the four, which then + # renders as a dash, not as `0ms`. + # Rationale: .claude/notes/reporting.md § Read the stored value, do not re-derive it buckets = " | ".join( format_ms(task.get(key)) for key in ("startup_ms", "generation_ms", "tool_ms", "teardown_ms") ) @@ -542,12 +533,10 @@ def _runtime_notes_lines(summary: RunSummary) -> list[str]: ) if t.get("stopped_early"): reason = t.get("early_stop_reason") or "unknown" - # No "N turn(s) avoided" claim here. It derived from - # ``max_turns - sdk_turn_index``, and on Codex and Antigravity one - # ``communicate()`` is a single SDK turn — so an early-stopped row - # advertised dozens of avoided turns when all that was cut was a - # tool-call tail. ``turns_remaining_at_stop`` is still persisted on - # EarlyStopInfo, labelled there as the upper bound it is. + # No "N turn(s) avoided" claim: on harnesses where one + # `communicate()` is a single SDK turn it advertised dozens when all + # that was cut was a tool-call tail. The bound is still persisted. + # Rationale: .claude/notes/reporting.md § The claims the reports do NOT make notes.append(f"> **NOTE:** [{task_id}] stopped early ({reason}); {early_stop_gate_note(reason)}") if not notes: return [] @@ -664,10 +653,9 @@ def _generate_token_usage_section(task_results: list[dict[str, Any]]) -> list[st total_tokens = sum(t["total_tokens"] for t in tasks_with_tokens) agent_cost = sum_costs(*(t.get("agent_cost_usd") for t in tasks_with_tokens)) - # Same helpers RunSummary uses, so the report and run.json cannot disagree - # about the bill. Worded cause-agnostically ("spend missing") because an - # unpriced turn and a hard kill reach the same conclusion and the report - # cannot always tell which applied. + # The same helpers RunSummary uses, so the report and run.json cannot + # disagree about the bill. Worded cause-agnostically: an unpriced turn and a + # hard kill reach the same conclusion. incomplete = [t for t in task_results if row_cost_incomplete(t)] overhead = eval_overhead_cost(task_results) total_cost = sum_costs(*(t.get("total_cost_usd") for t in task_results)) @@ -675,10 +663,9 @@ def _generate_token_usage_section(task_results: list[dict[str, Any]]) -> list[st lines.append(f"**Total Tokens**: {total_tokens:,} (input: {total_input:,}, output: {total_output:,})") if total_cache_write > 0 or total_cache_read > 0: lines.append(f"**Cache Tokens**: write: {total_cache_write:,}, read: {total_cache_read:,}") - # The agent bill is broken out separately only when there is overhead to - # distinguish it from: judge spend is a property of the suite's criteria and - # identical across harnesses, so comparing harnesses means comparing the - # agent line. **Total Cost** always means the whole bill. + # Broken out only when there is overhead to distinguish it from. **Total + # Cost** always means the whole bill. + # Rationale: .claude/notes/reporting.md § Per-instance aggregation, and what it buys if overhead is not None: if agent_cost is not None: lines.append(f"**Agent Cost**: ${agent_cost:.4f}") @@ -795,9 +782,7 @@ def load_from_run_dir(run_dir: Path) -> tuple[str, Path]: if run_dir.is_symlink(): run_dir = run_dir.resolve() - # Check for reports in order of preference: - # 1. experiment.md/json (written by ExperimentReportGenerator) - # 2. run.md/json (written by batch-level _generate_run_summary) + # In order of preference: experiment.md/json, then run.md/json. for md_name, json_name in [("experiment.md", "experiment.json"), ("run.md", "run.json")]: report_md_path = run_dir / md_name summary_json_path = run_dir / json_name @@ -901,9 +886,8 @@ def _compute_suite_rollup( rows_graded = rows_total - rows_not_graded scored = [r.result.weighted_score for r in rows if r.result.weighted_score is not None] - # Verdict evidence, distinct from `rows_graded` (a bucket complement): the - # gate `nothing_was_measured` needs, since a TIMEOUT counts as graded while - # no criterion ever ran on it. + # Verdict evidence, distinct from `rows_graded` (a bucket complement): a + # TIMEOUT counts as graded while no criterion ever ran on it. rows_measured = len(scored) average_weighted_score = sum(scored) / len(scored) if scored else None @@ -926,16 +910,9 @@ def _compute_suite_rollup( for ctype, scores in sorted(by_type.items()) ] - # Drive each criterion's aggregate() + evaluate suite_thresholds. Per-row - # results are sliced per criterion INSTANCE by position: SuccessChecker. - # check_all_async appends one CriterionResult per criterion in declared order - # (evaluation/checker.py), so row.success_criteria_results[i] belongs to - # task_criteria[i]. Aggregating per-instance (not pooled by type) is what - # lets a task stack many criteria of the SAME type — e.g. activation's - # per-skill skill_triggered criteria — and get a distinct aggregate - # (per-skill recall / F1) for each, instead of one type-pooled number - # repeated once per instance. The aggregate carries the criterion's - # description so the stacked instances stay distinguishable downstream. + # Sliced per criterion INSTANCE by position: the checker appends one result per + # criterion in declared order, so `results[i]` belongs to `criteria[i]`. + # Rationale: .claude/notes/reporting.md § Per-instance aggregation, and what it buys criterion_aggregates: list[CriterionAggregate] = [] if task_criteria is not None: init_criteria(validate=False) @@ -960,9 +937,8 @@ def _compute_suite_rollup( # Thresholds declared but nothing produced — fail loudly. stub = _build_missing_aggregator(ctype, suite_thresholds, description) stub = _attach_row_accounting(stub, rows_total, len(per_rows)) - # completion_rate is now a real value in metrics; refresh the - # threshold checks so the rendered actual matches it, but keep - # the aggregate failed because the real metrics are absent. + # Refresh the threshold checks so the rendered actual matches, + # but keep the aggregate failed: the real metrics are absent. stub = _evaluate_thresholds(stub, suite_thresholds).model_copy(update={"passed": False}) criterion_aggregates.append(stub) continue @@ -975,11 +951,9 @@ def _compute_suite_rollup( # Sample up to K failed/errored rows for error analysis failed_samples: list[FailedRowSummary] = [] for row in rows: - # "succeeded" is not the only non-failure. An UNGRADED row was never - # measured, so it has no failure reasons to report and listing it here - # (in a field documented as failed/errored rows) contradicts the same - # function's own rule two blocks up, where it leaves both sides of the - # pass rate. + # "succeeded" is not the only non-failure: an UNGRADED row was never + # measured, so it has no failure reason and this field documents failed and + # errored rows. if row.result.final_status.category in ("succeeded", "ungraded"): continue if len(failed_samples) >= _FAILED_SAMPLE_LIMIT: @@ -1026,11 +1000,10 @@ def _compute_suite_rollup( rows_failed=rows_failed, rows_error=rows_error, rows_not_graded=rows_not_graded, - # None, never 0.0: a suite where nothing was graded has no pass rate, - # and 0.0 renders as "0.0%" beside a full set of rows. Routed through - # the SAME helper as RunSummary and VariantAggregate — this site was - # the third copy of the formula and had no ungraded guard at all, so a - # suite whose only non-ungraded row was a TIMEOUT published 0.0%. + # None, never 0.0, and routed through the SAME helper as RunSummary and + # VariantAggregate -- this site was the third copy of the formula and had no + # ungraded guard at all. + # Rationale: .claude/notes/reporting.md § An unmeasured value is never zero pass_rate=( None if nothing_was_measured(not_graded=rows_not_graded, measured=rows_measured) @@ -1103,10 +1076,8 @@ def _render_suite_markdown(rollup: SuiteRollup) -> str: lines.append(f"- error: {s.error_message[:_FAILURE_REASON_MAX_LEN]}") for r in s.failure_reasons: lines.append(f"- {r}") - # Strip the leading variant segment so the link resolves from the - # suite dir where suite.md lives. PurePosixPath keeps the separator - # POSIX on Windows too. Fall back to the raw relpath if it isn't - # prefixed with the variant (e.g. serialized from a legacy shape). + # Strip the leading variant segment so the link resolves from the suite + # dir. PurePosixPath keeps the separator POSIX on Windows. rel_path = PurePosixPath(s.task_json_relpath) try: suite_rel: PurePosixPath = rel_path.relative_to(rollup.variant_id) diff --git a/src/coder_eval/reports_experiment.py b/src/coder_eval/reports_experiment.py index acbd2a72c..d8724ffec 100644 --- a/src/coder_eval/reports_experiment.py +++ b/src/coder_eval/reports_experiment.py @@ -41,9 +41,8 @@ # Default pass_threshold from BaseSuccessCriterion — used for Wilson pass-rate in replicate stats. _REPLICATE_PASS_THRESHOLD = 0.9 -# Cap on the ``error_message`` carried into each run.json row: enough to identify -# a failure without fetching the task artifact, short enough that a wholly-errored -# run doesn't bloat run.json. The untruncated message stays on task.json. +# Enough to identify a failure without fetching the task artifact, short enough that +# a wholly-errored run doesn't bloat run.json. The full message stays on task.json. _ROW_ERROR_MESSAGE_MAX_CHARS = 400 @@ -72,9 +71,7 @@ def _cost_complete(result: EvaluationResult) -> bool: ) -# --------------------------------------------------------------------------- -# Helper: build task_result dict from EvaluationResult (for variant reports) -# --------------------------------------------------------------------------- +# Build a task_result dict from an EvaluationResult, for the variant reports. def eval_result_to_task_dict( @@ -115,10 +112,8 @@ def eval_result_to_task_dict( total_turns = sum((t.num_turns or 0) for t in result.iterations) - # Whether the agent emitted a text reply (becomes the trailing entry - # in the Turn timeline). Carried as a row-level boolean so evalboard - # grid/trends can compute the visible turn count without re-reading - # per-task content. + # Carried as a row-level boolean so the evalboard can compute the visible turn + # count without re-reading per-task content. has_reply = _has_final_reply(result) agent_cost = result.total_token_usage.total_cost_usd if result.total_token_usage else None @@ -156,13 +151,10 @@ def eval_result_to_task_dict( } for t in result.iterations ], - # The four wall-clock buckets, computed ONCE here through the canonical - # `turn_time_buckets` and carried as TASK-level keys. `iterations` below - # is a deliberate 6-key projection with no `messages`, no `commands` and - # no `harness_*_ms`, so the markdown report cannot re-derive them from - # it — and a second implementation of the summation is exactly what that - # function exists to prevent. Each stays `float | None`: an unmeasured - # bucket renders as a dash, never as `0ms` (CE049). + # Computed ONCE through `turn_time_buckets` and carried as TASK-level keys. + # `iterations` below is a deliberate 6-key projection, so no renderer can + # re-derive them. Each stays `float | None` (CE049). + # Rationale: .claude/notes/reporting.md § Read the stored value, do not re-derive it "startup_ms": _buckets.startup_ms, "generation_ms": _buckets.generation_ms, "tool_ms": _buckets.tool_ms, @@ -178,14 +170,11 @@ def eval_result_to_task_dict( result.total_token_usage.cache_read_input_tokens if result.total_token_usage else None ), "total_tokens": (result.total_token_usage.total_tokens if result.total_token_usage else None), - # What the task cost: agent + judge + simulator. `total_cost_usd` means the - # whole bill on every surface, so a consumer that reads it gets the real - # number without adding anything up. None when nothing was priced at all. + # agent + judge + simulator. `total_cost_usd` means the whole bill on every + # surface. None when nothing was priced at all. "total_cost_usd": row_total_cost, - # Subject-agent spend alone, broken out for harness-vs-harness comparison: - # judge cost is a property of the suite's criteria and identical across - # harnesses, so leaving it in would make two harnesses look closer than they - # are. Rolled up as RunSummary.agent_cost_usd. + # Subject-agent spend alone: judge cost is identical across harnesses, so + # leaving it in would make two look closer than they are. "agent_cost_usd": agent_cost, # False when the agent spend above is missing money, so it is a floor. # Rolled up as RunSummary.tasks_cost_incomplete / cost_complete. @@ -211,23 +200,20 @@ def eval_result_to_task_dict( "max_turns_exhausted": result.max_turns_exhausted, "expected_turns_overage": list(overage) if overage is not None else None, "total_turns": total_turns, - # Documented "visible turns" (tool calls + final reply) — the canonical - # turn count the run-level "within expected turns" metric compares against - # expected_turns. Distinct from total_turns (SDK num_turns). + # "Visible turns" (tool calls + final reply) -- what the "within expected + # turns" metric compares against. Distinct from total_turns (SDK num_turns). "visible_turns": visible_turn_count(result), "expected_turns": expected_turns_value, "has_final_reply": has_reply, - # Early-stop surfaces (opt-in per-criterion stop_early: blocks). None/False on the - # default path so downstream analysis never confuses a truncated run - # with a full one. + # None/False on the default path, so downstream analysis never confuses a + # truncated run with a full one. "stopped_early": result.early_stop is not None, "early_stop_reason": (result.early_stop.reason.value if result.early_stop is not None else None), "turns_remaining_at_stop": ( result.early_stop.turns_remaining_at_stop if result.early_stop is not None else None ), - # The threshold in effect for this stop, so a downstream consumer - # comparing early-stopped runs across an experiment sweep that varies - # it can tell which weighted-gate value produced a given verdict. + # The threshold in effect for this stop, so a sweep that varies it can tell + # which weighted-gate value produced a given verdict. "gate_threshold": (result.early_stop.gate_threshold if result.early_stop is not None else None), } d["variant_id"] = variant_id @@ -359,9 +345,8 @@ def _aggregate_count_rows(result: ExperimentResult, show_p_values: bool) -> list row += " | —" lines.append(row + " |") - # Row: Not Graded — conditional, like the budget sub-rows above. Without - # it Tasks Run / Succeeded / Failed / Errors stop summing to tasks_run on - # an ungraded run, with nothing in the table to say where the rest went. + # Conditional, like the budget sub-rows above. + # Rationale: .claude/notes/reporting.md § The ungraded row in every surface if any(result.variant_aggregates[vid].tasks_not_graded > 0 for vid in result.variant_ids): row = "| Not Graded" for vid in result.variant_ids: @@ -799,10 +784,8 @@ def write_reports( # bug in one report cannot mask the run outcome. from .reports_html import write_experiment_html, write_variant_html - # Build per-variant task link tables from task_summaries. Every - # variant_id in task_summaries is guaranteed to appear in - # ``result.variant_ids`` (the aggregator constructs them from the same - # source), so we pre-seed the dict with all known variants and extend. + # Every variant_id in task_summaries is guaranteed to appear in + # ``result.variant_ids``, so pre-seed with all known variants and extend. task_links_by_variant: dict[str, list[tuple[str, str, float | None, str]]] = { vid: [] for vid in result.variant_ids } diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index e41ac64a7..a60b19cfd 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -1,10 +1,10 @@ -"""HTML report generation for coder_eval runs. +"""Single-file HTML report — the evalboard's STATIC TWIN. -Produces self-contained HTML files (inline CSS/JS, no external fonts or -images) that visualize a single task's conversation trace and success -criteria, plus cross-variant experiment summaries. +This renderer and ``evalboard/`` show the same run and must agree, so a rule +implemented on one side belongs on the other. The arithmetic itself lives in +``reports_stats.py``; this module only formats it. -Designed for offline viewing and for upload as CI artifacts. +Rationale: .claude/notes/reporting.md § Report rollups and the HTML twin """ from __future__ import annotations @@ -39,9 +39,7 @@ logger = logging.getLogger(__name__) -# --------------------------------------------------------------------------- -# Styling — fully inline; dark theme with light override via `.light` class. -# --------------------------------------------------------------------------- +# Styling -- fully inline; dark theme with a light override via `.light`. _CSS = """ :root { @@ -235,9 +233,7 @@ """ -# --------------------------------------------------------------------------- -# Formatting helpers -# --------------------------------------------------------------------------- +# Formatting helpers. _MAX_VALUE_LEN = 400 @@ -285,9 +281,8 @@ def _status_badge(status: Any) -> str: status_str = getattr(status, "value", None) or str(status) try: fs = status if isinstance(status, FinalStatus) else FinalStatus(str(status)) - # "ungraded" -> neutral: the row carries no verdict, so it must render as - # neither green nor red. Same class an unrecognised status falls back to, - # reached deliberately here rather than by accident. + # "ungraded" -> neutral: no verdict, so neither green nor red. The same + # class an unrecognised status falls back to, reached deliberately. cls = {"succeeded": "success", "failed": "failure", "error": "error", "ungraded": "neutral"}[fs.category] except (ValueError, KeyError): cls = "neutral" # unknown / non-FinalStatus input @@ -318,9 +313,7 @@ def _format_params(params: dict[str, Any]) -> str: return repr(params) -# --------------------------------------------------------------------------- -# Section renderers -# --------------------------------------------------------------------------- +# Section renderers. def _render_header(result: EvaluationResult) -> str: @@ -871,9 +864,8 @@ def _render_error_details(result: EvaluationResult) -> str: if retryable else 'non-retryable' ) - # Prefer the in-result tail captured at run time (sanitised, bounded). Fall - # back to the legacy stack_trace from error_details so reports regenerated - # against archived runs (pre-error_log_tail) still surface diagnostics. + # Prefer the in-result tail captured at run time; the legacy stack_trace keeps + # an archived run's diagnostics renderable. log_text = result.error_log_tail or "" if not log_text and isinstance(details, dict): stack = details.get("stack_trace") @@ -956,10 +948,9 @@ def _render_generation_metrics(result: EvaluationResult) -> str: f'
Crashed Partials
' f'
{_esc(breakdown)}
' ) - # The four wall-clock buckets. The arithmetic is in reports_stats; this - # only formats it. An unmeasured bucket renders as an em dash, never 0ms — - # a run predating the head/tail capture measured nothing, and a zero would - # claim it measured instantly (CE058). + # The arithmetic is in reports_stats; this only formats it. An unmeasured bucket + # renders as an em dash, never 0ms (CE058). + # Rationale: .claude/notes/reporting.md § An unmeasured value is never zero buckets = turn_time_buckets(result) startup = format_ms(buckets.startup_ms) generation = format_ms(buckets.generation_ms) @@ -1140,9 +1131,7 @@ def _wrap_document(title: str, body: str) -> str: """ -# --------------------------------------------------------------------------- -# Variant / Experiment helpers -# --------------------------------------------------------------------------- +# Variant / experiment helpers. def _variant_stddev_lines(variant_id: str, result: ExperimentResult | None) -> str: @@ -1389,9 +1378,8 @@ def _row(label: str, values: list[str], p: str | None) -> str: ) rows.append(_row("Failed", [str(result.variant_aggregates[vid].tasks_failed) for vid in result.variant_ids], None)) rows.append(_row("Errors", [str(result.variant_aggregates[vid].tasks_error) for vid in result.variant_ids], None)) - # The fourth bucket, conditionally like its siblings elsewhere. Without it - # Tasks Run / Succeeded / Failed / Errors no longer sum to tasks_run on an - # ungraded run, with nothing on the page to say where the rest went. + # The fourth bucket, conditional like its siblings elsewhere. + # Rationale: .claude/notes/reporting.md § The ungraded row in every surface if any(result.variant_aggregates[vid].tasks_not_graded > 0 for vid in result.variant_ids): rows.append( _row( @@ -1528,9 +1516,7 @@ def _experiment_most_divergent(result: ExperimentResult) -> str: """ -# --------------------------------------------------------------------------- -# Public API -# --------------------------------------------------------------------------- +# Public API. class HTMLReportGenerator: @@ -1623,9 +1609,8 @@ def generate_variant_html( ) stddev_lines = _variant_stddev_lines(variant_id, result) rich_sections = _variant_rich_sections(variant_id, result, run_dir) - # Only rendered when non-zero, so an ordinary graded run's tile is - # unchanged — but a `coder-eval execute` run says where its tasks went - # instead of showing Succeeded/Failed/Errors all at zero. + # Non-zero only, so a graded run's tile is unchanged but an `execute` run + # says where its tasks went instead of showing three zeros. ungraded_stat = ( '
Not Graded
' + f'
{agg.tasks_not_graded}
' diff --git a/src/coder_eval/reports_junit.py b/src/coder_eval/reports_junit.py index 069dd53d1..890b6414d 100644 --- a/src/coder_eval/reports_junit.py +++ b/src/coder_eval/reports_junit.py @@ -34,19 +34,16 @@ logger = logging.getLogger(__name__) -# Characters outside XML 1.0's legal set. Kept as a plain (non-raw) ASCII-only -# string with doubled backslashes so this source file carries no literal astral -# or control characters. ``ElementTree`` will happily serialize control chars -# into *invalid* XML, so every agent-derived string is scrubbed before it enters -# the tree. +# HAZARD: ``ElementTree`` will happily serialize control chars into INVALID XML, so +# every agent-derived string is scrubbed before it enters the tree. Written +# ASCII-only so this file carries no literal astral or control characters. _ILLEGAL_XML = re.compile("[^\\x09\\x0A\\x0D\\x20-\\uD7FF\\uE000-\\uFFFD\\U00010000-\\U0010FFFF]") # Per-testcase failure/error body cap (chars). Agent detail dumps can be huge. _BODY_LIMIT = 10_000 -# Serialized status values we recognize, for distinguishing a known status from a -# schema-skewed one when labelling a failure/error (classification itself goes -# through FinalStatus.category — see _category_of). +# For telling a known status from a schema-skewed one when labelling; the +# classification itself goes through FinalStatus.category. _KNOWN_STATUSES = frozenset(s.value for s in FinalStatus) @@ -122,9 +119,8 @@ def _is_safe_relpath(value: str) -> bool: ``resolve()``-containment check in :func:`_load_task_json` is the belt-and-braces backstop (symlinks included). """ - # A Windows drive-qualified value (``C:/x``) reads as a plain relative path - # on POSIX but is absolute on Windows, so reject it explicitly rather than - # leaning on the resolve-containment backstop alone. + # HAZARD: ``C:/x`` reads as relative on POSIX and absolute on Windows, so reject + # it explicitly rather than leaning on the containment backstop alone. if not value or value.startswith("/") or "\\" in value or PureWindowsPath(value).drive: return False parts = PurePosixPath(value).parts @@ -305,10 +301,8 @@ def _task_case(row: dict[str, Any], run_dir: Path) -> ET.Element: return case if category == "ungraded": - # `coder-eval execute`: the task ran but was deliberately not scored. - # is JUnit's only "no verdict" element — reporting it as a - # would turn a healthy ungraded run red in CI, and reporting - # it as a pass would invent a verdict. _set_counts already counts these. + # is JUnit's only "no verdict" element. + # Rationale: .claude/notes/reporting.md § The ungraded row in every surface ET.SubElement(case, "skipped", {"message": "not graded (coder-eval execute)"}) return case @@ -416,9 +410,8 @@ def generate_junit_xml(run_dir: Path) -> str: summary = RunSummary.model_validate_json(run_json.read_text(encoding="utf-8")) root = ET.Element("testsuites", {"name": _xml_safe(summary.run_id)}) - # Guard the root time identically to per-testcase time: a corrupt/blob-pulled - # run.json can carry a NaN/inf total_duration_seconds (RunSummary has no - # finite validator), which would emit an invalid time="nan". + # Guarded like the per-testcase time: a corrupt run.json can carry a NaN/inf + # duration, which would emit an invalid time="nan". root.set("time", _time_attr(summary.total_duration_seconds)) # Group task rows by variant, preserving first-seen order. diff --git a/src/coder_eval/reports_stats.py b/src/coder_eval/reports_stats.py index 4f5ea832a..bb3b2de6e 100644 --- a/src/coder_eval/reports_stats.py +++ b/src/coder_eval/reports_stats.py @@ -321,11 +321,9 @@ class VariantSeries(NamedTuple): asst_turns: list[float] -# environment_info keys the Environment table must NOT render as ordinary rows. -# `installed_tools` has its own dedicated section; the rest are harness -# bookkeeping the reader did not ask for — `command_base_path` is a full PATH -# string on every row, and the graded_by_* provenance keys only appear on a -# re-graded row where they would read as facts about the run itself. +# Keys the Environment table must NOT render as ordinary rows: `installed_tools` has +# its own section, `command_base_path` is a full PATH string, and the graded_by_* +# keys appear only on a re-graded row, where they would read as facts about the run. ENV_TABLE_EXCLUDE = frozenset({"installed_tools", "command_base_path", "reference_digest"}) @@ -334,9 +332,9 @@ def is_env_table_key(key: str) -> bool: return key not in ENV_TABLE_EXCLUDE and not key.startswith("graded_by_") -# What an ungraded row shows where a score would go. Deliberately not "0.000": -# an ungraded task was never measured, and a zero is indistinguishable from a -# task that was measured and scored nothing. +# Deliberately not "0.000": a zero is indistinguishable from a task that WAS +# measured and scored nothing. +# Rationale: .claude/notes/reporting.md § An unmeasured value is never zero UNGRADED_SCORE_TEXT = "n/a" @@ -387,28 +385,24 @@ def turn_time_buckets(result: EvaluationResult) -> TurnTimeBuckets: turns = result.iterations or [] startup = _sum_measured(t.harness_startup_ms for t in turns) teardown = _sum_measured(t.harness_teardown_ms for t in turns) - # MAIN THREAD ONLY, the same filter the collector and the evalboard apply: - # a sub-agent's generations bubble into the same stream, and the spawning - # Agent call's own interval already spans them. + # MAIN THREAD ONLY, the same filter the collector and the evalboard apply: a + # sub-agent's generations bubble into the same stream, and the spawning call's + # own interval already spans them. generation = _sum_measured( m.generation_duration_ms for t in turns for m in t.messages if isinstance(m, AssistantMessage) and m.parent_tool_use_id is None ) - # `None` only when NO turn recorded a bounded tool span. A turn that ran - # tools and timed none is indistinguishable from a turn that ran none, so - # the presence of a SPAN — not the presence of a turn — is what decides - # measured-versus-not. `_sum_measured` over a list of plain floats could - # never return None, which made this read `0ms` ("measured and instant") - # for a run nobody timed. + # `None` only when NO turn recorded a bounded span: the presence of a SPAN, not + # of a turn, is what decides measured-versus-not. + # Rationale: .claude/notes/reporting.md § An unmeasured value is never zero per_turn = [_turn_tool_union_ms(t) for t in turns] tool = _sum_measured(per_turn) if any(ms is not None for ms in per_turn) else None - # `duration_seconds` is a non-optional float defaulting to 0.0, so there is - # no None arm to write — but a 0.0 duration is a run that was never timed, - # and subtracting real buckets from it renders a fabricated negative - # residual. The evalboard keeps that null for the same reason; so do we. + # `duration_seconds` defaults to 0.0 with no None arm to write, but a 0.0 + # duration is a run that was never timed, and subtracting real buckets from it + # renders a fabricated negative residual. The evalboard keeps that null too. unaccounted = ( result.duration_seconds * 1000.0 - (startup or 0.0) - (generation or 0.0) - (tool or 0.0) - (teardown or 0.0) if result.duration_seconds > 0.0 @@ -434,25 +428,15 @@ def _sum_measured(values: Iterable[float | None]) -> float | None: def _turn_tool_union_ms(turn: TurnRecord) -> float | None: """One turn's tool execution — the UNION of its main-thread command spans. - PREFERS THE STORED VALUE. ``EventCollector.build_turn_record`` writes - ``TurnRecord.tool_union_ms`` from the single span set it measures all four - buckets against, so reading it is how this surface and the collector are - guaranteed to agree rather than merely observed to. The derivation below is - the LEGACY path: a ``task.json`` written before that field existed carries - neither it nor any way to recover it except by recomputing, and every such - run must stay renderable. - - Note the two paths cannot be distinguished by value — both return ``None`` - for a turn with no bounded span and a float otherwise — which is why the - stored one is checked with ``is not None`` rather than by truthiness: a - stored ``0.0`` is a measurement (spans were recorded and occupied no - measurable time) and must not fall through to a re-derivation. - - The span SELECTION is ``timing.main_thread_tool_spans``, not a - copy of it. That rule (which commands count, and the sub-agent exclusion) - is what the collector measures the generation subtraction and the head and - tail against, so a second typed implementation here is how two surfaces - come to publish two different tool totals for one run. + PREFERS THE STORED ``TurnRecord.tool_union_ms``, which the collector writes from + the single span set it measures all four buckets against, so this surface and the + collector are guaranteed to agree rather than merely observed to. The derivation + is the LEGACY path for a ``task.json`` written before that field existed. + + The stored value is checked with ``is not None``, never truthiness: a stored + ``0.0`` is a MEASUREMENT and must not fall through to a re-derivation. + + Rationale: .claude/notes/reporting.md § Read the stored value, do not re-derive it """ if turn.tool_union_ms is not None: return turn.tool_union_ms @@ -474,20 +458,12 @@ def collect_variant_series(result: ExperimentResult) -> dict[str, VariantSeries] s = series.get(vr.variant_id) if s is None: # a task result for a variant not in variant_ids continue - # Only the SCORE is dropped when there is none — never the row. - # Duration, tokens and assistant turns are facts about the run that - # grading has nothing to do with, and `execute`'s stated contract is - # that only the verdict is withheld. Skipping the row whole made an - # all-ungraded experiment render `Avg Duration | N/A | N/A` with the - # Tokens and Assistant Turns rows absent entirely. - # - # The series are consumed independently (each statistic reads one - # list), so they need not be index-aligned with each other; - # `paired_comparison` pairs across VARIANTS by task id, not by index - # into these lists. An earlier note here claimed an experiment is - # either entirely graded or entirely ungraded because `grade` is - # run-level — `run --resume` grades rows independently and folds a - # failed one back ungraded, so mixed experiments are real. + # Only the SCORE is dropped when there is none, never the row. The + # series are consumed independently, so they need not be index-aligned; + # `paired_comparison` pairs across VARIANTS by task id. Mixed + # graded/ungraded experiments are real -- `run --resume` grades rows + # independently and folds a failed one back ungraded. + # Rationale: .claude/notes/reporting.md § The ungraded row in every surface if vr.weighted_score is not None: s.scores.append(vr.weighted_score) s.durations.append(vr.duration_seconds / vr.replicate_count) diff --git a/src/coder_eval/simulation/user_simulator.py b/src/coder_eval/simulation/user_simulator.py index 928c9f0fd..27f79d3b6 100644 --- a/src/coder_eval/simulation/user_simulator.py +++ b/src/coder_eval/simulation/user_simulator.py @@ -114,9 +114,8 @@ def _extract_system_prompt(config: SimulationConfig, task_description: str, init _OPENER_NUDGE = "Begin the conversation now: send your opening message as the user to the coding agent." -# Belt-and-suspenders deny list for the simulator agent. ``allowed_tools=[]`` -# is the primary safeguard; this list pins the security property against any -# future SDK change that might reinterpret an empty allow-list. +# SECURITY: ``allowed_tools=[]`` is the primary safeguard; this list pins the +# property against a future SDK change that reinterprets an empty allow-list. _SIMULATOR_DISALLOWED_TOOLS: list[str] = [ "Bash", "Read", @@ -189,20 +188,12 @@ def __init__( self._agent: Agent[Any] | None = None self._scratch_dir: Path | None = None - # The simulator's model is PINNED from config, not inherited from the route. - # Leaving it None meant BEDROCK_MODEL decided who the simulated user was, so - # an A/B that varied the subject model silently varied the interlocutor too — - # and `_simulator_cost_usd` had to price from environment_info["bedrock_model"] - # to compensate. `_resolve_model` translates the vendor-prefixed id into - # whatever the run's backend accepts, the same way the LLM judge does. + # PINNED from config, not inherited from the route: leaving it None let + # BEDROCK_MODEL decide who the simulated user was, so an A/B varying the + # subject model silently varied the interlocutor too. self._model = self._resolve_model(config.model, route) - # - # allowed_tools=[] is the primary guarantee that the simulator cannot - # touch files or run commands. The disallowed_tools list below is - # belt-and-suspenders against a future SDK change where an empty - # allow-list silently means "allow everything" — every common tool is - # named explicitly so a regression surfaces as a deny rather than as - # a security failure. + # SECURITY: allowed_tools=[] is the primary guarantee that the simulator + # cannot touch files or run commands; the deny list is the backstop. from coder_eval.models import ClaudeCodeAgentConfig agent_config = parse_agent_config( @@ -214,11 +205,9 @@ def __init__( setting_sources=[], permission_mode="default", system_prompt=self._system_prompt, - # The roleplay persona IS the simulator's entire identity: 'replace' - # keeps the claude_code coding-agent preset from prefixing it (which - # would contradict the persona's own "stay in character" instruction - # and change every dialog-mode evaluation). Mirrors the judge seam - # in criteria/agent_judge.py. + # The persona IS the simulator's entire identity, so the coding-agent + # preset must not prefix it. + # Rationale: .claude/notes/contracts.md § The judge's identity is its system prompt system_prompt_mode="replace", ) # parse_agent_config returns a union, but type=CLAUDE_CODE guarantees ClaudeCodeAgentConfig @@ -302,14 +291,9 @@ async def start(self) -> None: self._agent = ClaudeCodeAgent(self._agent_config, route=self._route, instance_name="simulator") await self._agent.start(str(self._scratch_dir)) except BaseException: - # Agent construction or _agent.start() failed (SDK/transport - # startup, missing CLI, bad config, or cancellation). The dialog - # loop's finally (its only caller of stop()) is never entered on - # this path, so clean up our own scratch dir here to avoid a sim-* - # tempdir leak that compounds across a batch of simulation tasks. - # Wrapping construction too (not just start) closes the leak for - # every failure after mkdtemp. Re-raise to preserve the original - # failure (incl. cancellation/interrupt) semantics. + # The dialog loop's `finally` is never entered on this path, so clean up + # the scratch dir here or a sim-* tempdir leaks per failed task. Wrapping + # construction too closes the leak for every failure after mkdtemp. await self._remove_scratch_dir() self._agent = None raise @@ -356,9 +340,8 @@ async def next_user_message(self, dialog_pairs: list[tuple[str, str]]) -> Simula stop_requested = self.config.stop_token in raw cleaned = strip_stop_token(raw, self.config.stop_token) if stop_requested else raw.strip() - # Guard against empty cleaned text after stripping the stop token — - # the agent still needs *something* to react to, and the dialog - # terminates on this turn anyway. + # The agent still needs SOMETHING to react to after the stop token is + # stripped, and the dialog terminates on this turn anyway. if stop_requested and not cleaned: cleaned = "(the user indicated the task is complete)" diff --git a/src/coder_eval/telemetry.py b/src/coder_eval/telemetry.py index 4c6d43f6d..acb41e4a5 100644 --- a/src/coder_eval/telemetry.py +++ b/src/coder_eval/telemetry.py @@ -1,44 +1,19 @@ """User telemetry via OpenTelemetry → Azure Application Insights ``customEvents``. -A self-contained, opt-out usage-telemetry side-channel. It emits discrete -lifecycle events (run-start, task-end, per-command) to the App Insights -``customEvents`` table and is **never** part of the eval data path. - -How customEvents routing works ------------------------------- -The Azure Monitor exporter routes an OpenTelemetry *log record* to the -``customEvents`` table (instead of the default ``traces`` table) **iff** the -record carries the attribute ``microsoft.custom_event.name``. The event name is -that attribute's value; every other record attribute becomes a -``customDimensions`` entry. We reach that attribute through plain stdlib -logging: an OTel ``LoggingHandler`` is attached to a dedicated logger, and -``track_event`` calls ``logger.info(name, extra={...})``. So the only OTel/Azure -imports live inside ``init_telemetry`` — ``track_event`` is pure stdlib and a -cheap no-op when telemetry is off. - -Posture -------- -Telemetry is **on by default**: an ingestion-only Application Insights connection -string is baked into the app (``config._DEFAULT_TELEMETRY_CONNECTION_STRING``) so a -fresh install reports usage to the shared coder-eval resource. An explicitly-set -``APPLICATIONINSIGHTS_CONNECTION_STRING`` / ``UIPATH_AI_CONNECTION_STRING`` / -``TELEMETRY_CONNECTION_STRING`` (env or ``.env``) takes precedence, routing -telemetry elsewhere. Telemetry is **off** only when ``TELEMETRY_ENABLED`` is set -false (the single canonical disable gate) or the connection string is cleared. -No prompts, file contents, or repo paths are ever captured — only enums, counts, -durations, an anonymous per-install id (a random UUID persisted in the user -config file — identifies an install, not a person), and non-PII platform -identity (OS / arch / Python version). Because telemetry is default-on, the first -run that initializes it prints a one-time stderr notice disclosing what is -collected and how to disable it (``_maybe_show_first_run_notice``). - -Non-fatal contract -------------------- -Every public function wraps its body in ``try/except Exception`` and logs a -warning rather than raising — telemetry must never break a run. This invariant -is enforced by the CE019 custom lint rule. Persisting the anonymous install id -is best-effort too: if its config file can't be written, telemetry still emits -events, just without the ``InstallId`` dimension. +A self-contained, opt-out usage-telemetry side-channel emitting discrete lifecycle +events. It is **never** part of the eval data path. + +Telemetry is **on by default** via a baked-in ingestion-only connection string; an +explicitly-set one takes precedence, and ``TELEMETRY_ENABLED`` is the single +canonical disable gate. No prompts, file contents or repo paths are captured — only +enums, counts, durations, an anonymous per-install id and non-PII platform identity. +The first run that initializes it prints a one-time stderr notice. + +**Non-fatal contract:** every public function wraps its body in +``try/except Exception`` and logs a warning rather than raising — telemetry must +never break a run. Enforced by lint rule CE019. + +Rationale: .claude/notes/reporting.md § Telemetry emission """ import atexit @@ -65,9 +40,8 @@ # The OTel LoggerProvider (typed Any: OTel/Azure are largely untyped and we keep # their symbols confined to init_telemetry to avoid leaking Unknown elsewhere). _provider: Any = None -# The OTel handler attached to the dedicated events logger; tracked so -# shutdown_telemetry can detach it (the logger is a process-wide singleton that -# outlives a shutdown, so leaving it attached would double-emit on a re-init). +# Tracked so shutdown_telemetry can detach it: the logger is a process-wide +# singleton, so leaving it attached would double-emit on a re-init. _handler: logging.Handler | None = None # Enrichment merged into every event — all scalar, no user content. The # per-process session id lives here under "SessionId" (no separate global). @@ -78,20 +52,17 @@ # of these makes logging raise KeyError on emit, so the coercer drops them. _RESERVED_LOGRECORD_ATTRS: frozenset[str] = frozenset(logging.makeLogRecord({}).__dict__) | {"message", "asctime"} -# The attribute the Azure Monitor exporter looks for to route a log record to -# the customEvents table. Hard-coded (matches the exporter's internal -# _MICROSOFT_CUSTOM_EVENT_NAME constant) so track_event needs no OTel import. +# What the Azure Monitor exporter looks for to route a record to customEvents. +# Hard-coded, matching the exporter's own constant, so track_event needs no OTel +# import. Rationale: .claude/notes/reporting.md § Telemetry emission _CUSTOM_EVENT_NAME_ATTR = "microsoft.custom_event.name" -# Version of the event property contract. Stamped as the `SchemaVersion` -# dimension on every event so the dashboard's Kusto queries (a cross-system -# contract) can detect a schema change instead of silently breaking. Bump on any -# breaking property rename/removal. +# Stamped on every event so the dashboard's queries (a cross-system contract) can +# detect a schema change instead of silently breaking. Bump on any breaking rename. _TELEMETRY_SCHEMA_VERSION = "1" -# One-time stderr notice shown on the first run that telemetry is on (default-on -# tooling must disclose collection). Persisted-once via a flag in the user config -# file; see _maybe_show_first_run_notice. +# Default-on tooling must disclose collection. Persisted-once via a flag in the +# user config file. _FIRST_RUN_NOTICE = ( "coder-eval collects anonymous usage telemetry (command names, outcomes, counts, durations, " "an anonymous per-install id, and platform info — never prompts, file contents, or repo paths) " @@ -99,9 +70,8 @@ "It is on by default. Disable it any time with TELEMETRY_ENABLED=false." ) -# The scalar contract for event properties. Public so producers (e.g. -# orchestrator.build_task_event) can annotate their event dicts with it and have -# pyright reject a non-scalar at the producing call site, not just at runtime. +# Public so a producer can annotate its event dict and have pyright reject a +# non-scalar at the producing call site rather than at runtime. Scalar = str | int | float | bool F = TypeVar("F", bound=Callable[..., Any]) @@ -180,9 +150,8 @@ def _get_or_create_install_id() -> str | None: path.write_text(json.dumps(data, indent=2) + "\n", encoding="utf-8") return install_id except Exception as exc: - # Never propagate: a missing HOME (Path.home() → RuntimeError), an - # unwritable dir (OSError), etc. must degrade to no InstallId, NOT disable - # telemetry. Keeping telemetry live without InstallId is the agreed behavior. + # Never propagate: a missing HOME or unwritable dir degrades to no + # InstallId, NOT to disabled telemetry. logger.debug("could not persist install id (%s); telemetry will omit InstallId", exc) return None @@ -237,9 +206,8 @@ def init_telemetry(version: str) -> None: if _initialized: return - # Single-init contract: settings is read once per process here. A - # shutdown → re-init cycle re-reads the same module-global settings - # singleton (only tests, which monkeypatch settings, exercise re-init). + # Single-init contract: settings is read once per process. Only tests, + # which monkeypatch it, exercise a re-init. from coder_eval.config import settings if not settings.telemetry_enabled or not settings.telemetry_connection_string: @@ -256,10 +224,9 @@ def init_telemetry(version: str) -> None: logger.debug("telemetry SDK unavailable; disabled") return - # Construct the exporter in its own guard: it parses the connection - # string (a credential — InstrumentationKey + IngestionEndpoint) and a - # parse error can echo it back, so on failure log a generic message - # WITHOUT interpolating the exception, never leaking the credential. + # HAZARD: its own guard. The exporter parses the connection string -- a + # credential -- and a parse error can echo it back, so log a generic + # message WITHOUT interpolating the exception. try: exporter = AzureMonitorLogExporter(connection_string=settings.telemetry_connection_string) except Exception: @@ -269,10 +236,9 @@ def init_telemetry(version: str) -> None: provider = LoggerProvider(resource=Resource.create({"service.name": "coder-eval"})) provider.add_log_record_processor(BatchLogRecordProcessor(exporter)) - # The SDK's LoggingHandler is deprecated in favor of a separate - # instrumentation package we don't depend on; the documented attribute - # bridge still works. Suppress the one-time warning so enabling - # telemetry doesn't print noise to a user's stderr. + # Deprecated in favour of a package we don't depend on; the documented + # attribute bridge still works. Suppressed so enabling telemetry prints + # no noise to a user's stderr. with warnings.catch_warnings(): warnings.simplefilter("ignore", DeprecationWarning) handler = LoggingHandler(level=logging.INFO, logger_provider=provider) @@ -280,8 +246,7 @@ def init_telemetry(version: str) -> None: events_logger = logging.getLogger("coder_eval.telemetry.events") events_logger.setLevel(logging.INFO) events_logger.propagate = False # never reach console/file handlers - # The logger is a process-wide singleton; drop any stale handler from a - # prior init/shutdown cycle before attaching this one. + # Process-wide singleton: drop any stale handler before attaching. for stale in list(events_logger.handlers): events_logger.removeHandler(stale) events_logger.addHandler(handler) @@ -311,9 +276,7 @@ def init_telemetry(version: str) -> None: # Telemetry is now live — disclose it once (default-on tooling must say so). _maybe_show_first_run_notice() except Exception as exc: - # Fully non-fatal — telemetry must never break a run. (An unpersistable - # install id is handled earlier as best-effort: telemetry stays on and - # just omits InstallId, so it doesn't reach here.) + # Fully non-fatal -- telemetry must never break a run (CE019). logger.warning("telemetry init failed: %s", exc) @@ -355,9 +318,9 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: status, error_type = "Failed", "Exit" raise except (KeyboardInterrupt, SystemExit) as exc: - # These derive from BaseException, not Exception, so without this - # branch a Ctrl-C / sys.exit would skip both handlers and the - # finally would mis-record the aborted command as "Succeeded". + # HAZARD: these derive from BaseException, so without this branch + # a Ctrl-C would skip both handlers and the `finally` would record + # the aborted command as "Succeeded". status, error_type = "Failed", type(exc).__name__ raise except Exception as exc: @@ -369,9 +332,8 @@ def wrapper(*args: Any, **kwargs: Any) -> Any: {"Status": status, "DurationMs": int((time.monotonic() - start) * 1000), "ErrorType": error_type}, ) - # functools.wraps copies func's signature onto wrapper for Typer/click, - # but the inferred type is Callable[..., Any], not the TypeVar F — the - # cast restores F so callers see the original command's type. + # functools.wraps copies the signature for Typer, but the inferred type + # is Callable[..., Any]; the cast restores F for callers. return wrapper # type: ignore[return-value] return deco diff --git a/src/coder_eval/utils.py b/src/coder_eval/utils.py index 2ec6cac2a..131111106 100644 --- a/src/coder_eval/utils.py +++ b/src/coder_eval/utils.py @@ -82,14 +82,11 @@ def process_plugins( resolved = Path(expanded).resolve() processed_plugin["path"] = str(resolved) - # Loud: claude-code loads a local plugin as a PLUGIN ROOT, so its skills must - # sit at /skills//SKILL.md. Point one level deeper — at the bare - # directory of skill directories — and the SDK loads NOTHING, with no error: - # every positive row of an activation suite scores 0 and the suite reports - # recall 0.0, which reads exactly like a skill that never triggers. This - # function is claude-code-only (see the module docstring); Codex and - # Antigravity scan both depths and warn for themselves. Warn rather than - # raise — a plugin may legitimately ship only agents/, commands/ or hooks/. + # HAZARD: a local plugin loads as a PLUGIN ROOT, so its skills must sit + # at /skills//SKILL.md. One level deeper and the SDK loads + # NOTHING, with no error -- every positive row of an activation suite + # then scores 0, which reads exactly like a skill that never triggers. + # Warn rather than raise: a plugin may ship only agents/ or hooks/. if plugin.get("type") == "local" and resolved.is_dir() and not (resolved / "skills").is_dir(): log.warning( f"Plugin path has no skills/ subdirectory, so it loads no skills: {resolved}. " @@ -236,9 +233,8 @@ def _git_short_sha(repo_path: Path) -> str: return "unknown" -# A semver-ish version token: major.minor.patch with an optional leading `v` -# and any prerelease/build tail (e.g. `1.196.0-alpha.20260605.7426`). Anchored -# at the start of a line so it rejects non-version `uip --version` output. +# major.minor.patch with an optional leading `v` and any prerelease tail. Anchored +# at line start so it rejects non-version `uip --version` output. _VERSION_TOKEN = re.compile(r"^v?\d+\.\d+\.\d+\S*$") @@ -309,9 +305,8 @@ def resolve_uipath_plugin_dir(search_path: str | None = None) -> Path | None: except (OSError, RuntimeError) as exc: logger.debug("Failed to resolve `uip` symlink %s: %s", resolved, exc) return None - # Walk up looking for `.../node_modules/@uipath`. We accept the first - # @uipath dir whose parent is named `node_modules` — the cli is always - # inside one, e.g. `~/.bun/.../node_modules/@uipath/cli/dist/index.js`. + # The first @uipath dir whose parent is named `node_modules` -- the cli is + # always inside one. for ancestor in real.parents: if ancestor.name == "@uipath" and ancestor.parent.name == "node_modules": logger.debug("Resolved @uipath plugin-tools dir=%s from `uip` at %s", ancestor, real) @@ -355,9 +350,8 @@ def _tool_plugin_versions(tools_dir: Path | None = None) -> dict[str, str]: return {} plugins: dict[str, str] = {} - # npm installs ``@uipath/`` into ``@uipath//``, so a ``-tool`` plugin - # dir is always named ``-tool``. Mirror the manifest-name contract - # (``name.endswith("-tool")`` below) in the glob to avoid statting unrelated dirs. + # A ``-tool`` plugin dir is always named ``-tool``; mirroring the + # manifest-name contract in the glob avoids statting unrelated dirs. for pkg_json in sorted(tools_dir.glob("*-tool/package.json")): try: data = json.loads(pkg_json.read_text(encoding="utf-8")) @@ -454,23 +448,20 @@ def get_version_info(sandbox_path: Path | None = None) -> dict[str, Any]: project_root = Path(__file__).resolve().parent.parent version_info["git_commit"] = _git_short_sha(project_root) - # Sibling repos that contribute to the agent's runtime context. - # Path resolution: env var first (CODER_EVAL_SKILLS_DIR), then sibling-of-coder_eval default. - # A downstream runner can set this env var to its configured path so custom layouts get the right SHA. + # Env var first (CODER_EVAL_SKILLS_DIR), then the sibling default, so a + # downstream runner with a custom layout still gets the right SHA. sibling_root = project_root.parent.parent skills_override = os.environ.get("CODER_EVAL_SKILLS_DIR") skills_path = Path(skills_override) if skills_override else sibling_root / "skills" version_info["skills_git_commit"] = _git_short_sha(skills_path) - # uip CLI is installed via npm; read its version from @uipath/cli's - # package.json (same source tool_plugins uses), falling back to a validated - # `uip --version`. Consumed by downstream run-summary tooling. + # Read from @uipath/cli's package.json (the source tool_plugins uses), falling + # back to a validated `uip --version`. tools_dir = resolve_uipath_plugin_dir() version_info["cli_version"] = _resolve_cli_version(tools_dir, None) - # The CLI shell (cli_version) and its `@uipath/*-tool` plugins (e.g. - # maestro-tool) version independently, so the shell version alone can - # mislead regression timelines. Record the installed plugin versions too. + # The shell and its `@uipath/*-tool` plugins version independently, so the + # shell version alone can mislead a regression timeline. version_info["tool_plugins"] = _tool_plugin_versions(tools_dir) # Get coder_eval version diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index b6376a583..693498df9 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 25_468 +_ESSAY_BASELINE_WORDS = 18_163 _SRC = Path("src/coder_eval") From ae47817e983ce08778d3573d35790a26cb2eea39 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 21:03:44 -0700 Subject: [PATCH 08/19] test(lint): guard the Rationale pointer's placement, not just its target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check_pointers proves a pointer RESOLVES. It cannot see the defect that actually shipped: a block replacement anchored on the wrong line leaves the tail of the replaced prose stranded after the pointer, where the $-anchored regex stops looking. The file parses, the pointer resolves, and the comment carries a severed half-sentence. Three phases of the prose-reduction work shipped that shape to review before this check existed; a fourth ran clean because the check was running from the first edit. It also reports an orphaned docstring terminator, which is worse than it looks: the file stops parsing, and the measurement silently reports it as zero words rather than failing. A docstring may still follow its pointer with Args:/Returns:/Raises: — that is the house shape, and the one case this must not flag. Co-Authored-By: Claude Opus 5 (1M context) --- tests/lint/prose_budget.py | 70 ++++++++++++++++++++++++++++++++++++++ tests/test_prose_budget.py | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+) diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 693498df9..994944740 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -52,6 +52,9 @@ _POINTER = re.compile(r"Rationale:\s*(\S+\.md)\s*§\s*(.+?)\s*$") +# The docstring delimiter, as a value: this module's own prose cannot spell it inline. +QUOTES = chr(34) * 3 + # ``##`` or ``###``: a pointer may target a SUBSECTION, so appending to an existing # section (the single-home rule) does not force the pointer up to the parent heading. _HEADING = re.compile(r"^#{2,3}\s+(.+?)\s*$") @@ -241,6 +244,70 @@ def check_pointers(repo_root: Path) -> list[str]: return failures +_TRAILING_SECTIONS = ("Args:", "Returns:", "Raises:", "Yields:", "Example:", "Examples:") + + +def check_pointer_placement(repo_root: Path) -> list[str]: + """A ``Rationale:`` pointer must be the LAST prose line of its block. + + Not style. A block replacement that anchors on the wrong line leaves the tail of the + replaced prose stranded AFTER the pointer, where the ``$``-anchored ``_POINTER`` + regex cannot see it -- so the pointer still resolves and the file still parses while + carrying a severed half-sentence. + + A docstring may follow its pointer with an ``Args:``/``Returns:``/``Raises:`` block, + which is the house shape; anything else is the defect. An orphaned docstring + terminator is reported too: a rewrite that leaves the original one behind makes the + file unparseable, which the measurement silently reports as zero words. + """ + failures: list[str] = [] + for path in sorted((repo_root / _SRC).rglob("*.py")): + rel = path.relative_to(repo_root / _SRC).as_posix() + source = path.read_text(encoding="utf-8") + lines = source.split("\n") + + for index in range(1, len(lines)): + if lines[index].strip() == QUOTES and lines[index - 1].strip().endswith(QUOTES): + failures.append(f"{rel}:{index + 1}: orphaned docstring terminator") + + try: + tree = ast.parse(source) + except SyntaxError: + continue + + for node in ast.walk(tree): + if not isinstance(node, _DOCSTRING_OWNERS): + continue + docstring = ast.get_docstring(node, clean=False) + if docstring is None: + continue + body = [line for line in docstring.split("\n") if line.strip()] + pointers = [i for i, line in enumerate(body) if _POINTER.search(line.strip())] + if pointers: + tail = body[pointers[-1] + 1 :] + if tail and not tail[0].strip().startswith(_TRAILING_SECTIONS): + name = getattr(node, "name", "") + failures.append(f"{rel}::{name}: prose after the Rationale pointer: {tail[0].strip()!r}") + + index = 0 + while index < len(lines): + if not lines[index].strip().startswith("#"): + index += 1 + continue + end = index + while end < len(lines) and lines[end].strip().startswith("#"): + end += 1 + run = lines[index:end] + for offset, line in enumerate(run): + if _POINTER.search(line.strip()) and offset != len(run) - 1: + failures.append( + f"{rel}:{index + offset + 1}: comment continues after the Rationale pointer: " + f"{run[offset + 1].strip()!r}" + ) + index = end + return failures + + def check(repo_root: Path) -> str | None: """``None`` when the tree is at or under the baseline, else the failure message.""" total = total_words(measure(repo_root).files) @@ -340,6 +407,9 @@ def main(argv: list[str]) -> int: for failure in check_pointers(repo_root): print(f"unresolved pointer: {failure}", file=sys.stderr) failed = True + for failure in check_pointer_placement(repo_root): + print(f"misplaced pointer: {failure}", file=sys.stderr) + failed = True if (message := check(repo_root)) is not None: print(message, file=sys.stderr) failed = True diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py index 9ead3a6b9..15bc10cea 100644 --- a/tests/test_prose_budget.py +++ b/tests/test_prose_budget.py @@ -264,3 +264,64 @@ def test_the_report_names_files_subsystems_the_total_and_the_essays(self, tmp_pa assert "ESSAYS" in report assert "a.py::essay_fn" in report assert "200" in report + + +# The docstring delimiter as a value, so a fixture can embed one without ending this file's +# own strings. +Q = chr(34) * 3 + + +class TestPointerPlacement: + """The guard promoted after three phases shipped this defect shape to review.""" + + def _root(self, tmp_path: Path, source: str) -> Path: + root = _tree(tmp_path, {"a.py": source}) + target = root / ".claude" / "notes" / "timing.md" + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("# Timing\n\n## close_window\n\nWhy.\n", encoding="utf-8") + return root + + def test_a_pointer_at_the_end_of_a_comment_run_passes(self, tmp_path: Path) -> None: + source = "# One.\n# Two.\n# Rationale: .claude/notes/timing.md \u00a7 close_window\nx = 1\n" + assert prose_budget.check_pointer_placement(self._root(tmp_path, source)) == [] + + def test_a_comment_run_that_continues_after_its_pointer_fails(self, tmp_path: Path) -> None: + source = "# One.\n# Rationale: .claude/notes/timing.md \u00a7 close_window\n# severed tail.\nx = 1\n" + failures = prose_budget.check_pointer_placement(self._root(tmp_path, source)) + assert len(failures) == 1 + assert "severed tail" in failures[0] + + def test_a_docstring_pointer_followed_by_args_passes(self, tmp_path: Path) -> None: + source = ( + "def f(a):\n" + f" {Q}Do it.\n\n" + " Rationale: .claude/notes/timing.md \u00a7 close_window\n\n" + " Args:\n" + " a: thing\n" + f" {Q}\n" + ) + assert prose_budget.check_pointer_placement(self._root(tmp_path, source)) == [] + + def test_a_docstring_with_prose_after_its_pointer_fails(self, tmp_path: Path) -> None: + source = ( + "def f():\n" + f" {Q}Do it.\n\n" + " Rationale: .claude/notes/timing.md \u00a7 close_window\n" + " and the stranded half of a sentence.\n" + f" {Q}\n" + ) + failures = prose_budget.check_pointer_placement(self._root(tmp_path, source)) + assert len(failures) == 1 + assert "stranded half" in failures[0] + + def test_an_orphaned_docstring_terminator_is_reported(self, tmp_path: Path) -> None: + source = f"def f():\n {Q}Do it.{Q}\n {Q}\n return 1\n" + failures = prose_budget.check_pointer_placement(self._root(tmp_path, source)) + assert any("orphaned docstring terminator" in failure for failure in failures) + + def test_a_file_with_no_pointer_is_not_flagged(self, tmp_path: Path) -> None: + source = "# One.\n# Two.\n# Three.\nx = 1\n" + assert prose_budget.check_pointer_placement(self._root(tmp_path, source)) == [] + + def test_a_syntax_error_is_skipped_not_fatal(self, tmp_path: Path) -> None: + assert prose_budget.check_pointer_placement(self._root(tmp_path, "def f(:\n")) == [] From a2d8e1d9498c12c6c17318b144bf686de597f7ba Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 21:06:47 -0700 Subject: [PATCH 09/19] docs: restore the self-containment clause on reports_html MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Self-contained (inline CSS/JS, no external fonts or images)" is caller-facing — it is why the file can be opened offline and uploaded as a CI artifact — and the 7/7 rewrite dropped it while adding the static-twin claim. Both belong. Co-Authored-By: Claude Opus 5 (1M context) --- src/coder_eval/reports_html.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index a60b19cfd..1364a0a13 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -1,4 +1,8 @@ -"""Single-file HTML report — the evalboard's STATIC TWIN. +"""HTML report generation — the evalboard's STATIC TWIN. + +Produces SELF-CONTAINED files (inline CSS/JS, no external fonts or images) for +offline viewing and CI artifact upload, covering a single task's conversation trace +and criteria plus cross-variant experiment summaries. This renderer and ``evalboard/`` show the same run and must agree, so a rule implemented on one side belongs on the other. The arithmetic itself lives in From ab7bc4923a45f6179421333327c10ec7e1551216 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 21:10:28 -0700 Subject: [PATCH 10/19] docs: record the four harness gaps the prose run did not close MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fifth was promoted (b8b1dcfc). These four are deferred with the reason, so the next person does not rediscover them: the general severed-fragment shape needs an allowlist to be usable, the single-home rule needs a similarity measure, a pointer landing unhelpfully is probably not mechanizable at all, and the generated-surface guard needs a commit-scoped diff the lint harness cannot see. The single-home one is the highest-value unbuilt guard in the notes design — it bit every phase of this run, once against a file the phase never opened. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/harness-candidates.md | 43 +++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index fcfb02b43..4521ec8b3 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -873,3 +873,46 @@ re-derive from scratch. timing work (the function is zero lines of its diff) and not a guardrail candidate — a small real bug needing its own change. Caught in: the turn-timing consolidation final review (gpt-5.6-sol). + +- [ ] **A comment line that opens mid-sentence directly after one that ended.** + The residue of a block replacement whose anchor matched the wrong line: the + tail of the replaced prose survives as a severed fragment. The exact-form half + of this — a `Rationale:` pointer that is not the last line of its block — was + PROMOTED in the prose-mass-reduction run and now ships as + `prose_budget.check_pointer_placement`. What remains is the general case, + where no pointer is involved, and it is heuristic: a legitimately wrapped + sentence looks identical to a severed one, so it needs an allowlist (a + continuation opening with a backtick, a quote, or a list marker is usually + fine). ~30 min plus the false-positive triage. Caught in: prose mass + reduction, Phases 4-7 review. + +- [ ] **Two `.claude/notes/` sections covering ONE topic under different + headings.** The single-home rule is the load-bearing invariant of the notes + tree and nothing enforces it. `check_pointers` proves a pointer resolves; + nothing proves the topic is not also argued three files away. It bit every + phase of the prose-mass-reduction run, including once against a file the + phase never opened (`reporting.md` vs `orchestration.md` on + `nothing_was_measured`). Needs a similarity measure over section bodies — + shared rare tokens, or a shared symbol name appearing as the subject of two + headings — so it is real work rather than a regex. Deferred on cost, not on + value: this is the highest-value unbuilt guard in the notes design. Caught + in: prose mass reduction, all phases. + +- [ ] **A `Rationale:` pointer that resolves to a heading which does not hold + the rationale that left the site.** The weaker sibling of the above and the + same shape of miss: the gate goes green while the reader arrives somewhere + unhelpful. Five instances in Phase 5 alone, all fixed by hand. Probably not + mechanizable without a semantic check, but worth recording as a known blind + spot of `check_pointers` so nobody reads its green as "the pointers are + good". Caught in: prose mass reduction, Phases 5-6 review. + +- [ ] **A criterion-class first docstring line changing without + `make plugin-reference` in the same commit.** CE033 already diffs the + generated `plugins/coder-eval/reference/criteria.md`, so drift IS caught — + but only for classes that reach the generated file, and only as "the + generated file is stale" rather than "you edited a generated surface". A + commit-scoped guard would name the cause. Deferred because the lint harness + has no access to a commit-scoped diff today; the ad-hoc version + (AST-comparing every `ClassDef` first line against a base ref) was written + and used throughout Phase 6 and is the thing to promote if that access + appears. Caught in: prose mass reduction, Phase 6. From e03a699c9948fe5d4cbeed9397ebda09c8afdcf3 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 22:11:06 -0700 Subject: [PATCH 11/19] fix(docs): restore contracts the prose refactor lost or misstated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An audit of the finished refactor found nine places where compression either dropped a contract or, worse, replaced it with a plausible wrong reason. The bad one is `_resolve_backend_route`. Its docstring said the function raises "so it must survive -O" — true of a different function, transplanted here — while the actual contract was gone from the source AND from the notes: an explicit backend override that cannot be honored must fail loudly, not degrade to a backend the task author never asked for. A reader told the raise is only about -O can reasonably "improve" it into a fallback, which is exactly the regression the deleted sentence forbade. The rest: - write_text_atomic claimed mode 0o644 as if guaranteed. It is a ceiling; the umask narrows it, and 0o077 yields 0600 — which breaks the docker driver, the failure the directive exists to prevent. - A NOTE in sandbox.py said three methods do not validate path traversal. Two of them go through resolve_files (and so through _within_sandbox and _reject_escaped); only list_files still does not. Someone reasoning about criterion containment from that comment reached the wrong conclusion. - enforces_permission_windows kept the clause making the window SAFE and lost the one making it REAL: without the container's DAC cap drops a mode-000 directory is still readable by root. - capture_to's copytree flags lost the reason one of them is load-bearing. - adopt no longer said criteria can still write to the adopted tree, under a docstring headlined "materializing nothing". - The docker log constants lost why there are two of them, and had no pointer. - armed_criteria_passed said "weighted score", which names a different quantity in the same class; the binarisation that makes gate_threshold=1.0 an exact equivalence was gone. - judge_context said "these tokens" with no list after it. - _build_run_command_env counted eight env facets and sets nine. The baseline rises 18,163 -> 18,549. That is the ratchet working as intended: restoring real contract is a legitimate reason to raise it, and the reason belongs here rather than in a silent constant bump. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/contracts.md | 7 ++-- src/coder_eval/evaluation/judge_context.py | 5 +-- src/coder_eval/models/results.py | 8 +++-- src/coder_eval/models/routing.py | 22 ++++++++++--- src/coder_eval/path_utils.py | 16 +++++++--- src/coder_eval/sandbox.py | 37 +++++++++++++++------- tests/lint/prose_budget.py | 2 +- 7 files changed, 67 insertions(+), 30 deletions(-) diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index edbf2dca4..09bd950ea 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -180,9 +180,10 @@ measured" rather than an instant 0.0 (CE058). ## Route resolution The agent's route and the evaluation side's route are resolved separately. -`resolve_evaluation_route` decides the `llm_judge` / `agent_judge` transport, and a -`model` lands on `checker_context.api_route.model` ONLY when a real override was given — -never the agent's own model. `criterion.model` is `None` rather than a materialized +`resolve_evaluation_route` decides the `llm_judge` / `agent_judge` transport. The +override travels one way: `checker_context.api_route.model` is the task-authored INPUT, +and it lands on the RESOLVED route's `model` — set only when a real override was given, +never from the agent's own model. `criterion.model` is `None` rather than a materialized default when unset, so the precedence (explicit per-criterion model, then the route's, then the default judge model) survives a `model_dump(mode="json")` and reload, which a `model_fields_set` check would not. diff --git a/src/coder_eval/evaluation/judge_context.py b/src/coder_eval/evaluation/judge_context.py index 4787a1c56..9b0f7e998 100644 --- a/src/coder_eval/evaluation/judge_context.py +++ b/src/coder_eval/evaluation/judge_context.py @@ -28,8 +28,9 @@ ) -# Paths beginning with one of these tokens resolve against a HOST directory rather -# than the sandbox, mirroring the same-named env vars `run_command` exposes. +# ``$TASK_DIR`` (the task YAML's own directory) and ``$REFERENCE_DIR`` (the per-run +# staged reference copy) resolve against a HOST directory rather than the sandbox, +# mirroring the same-named env vars `run_command` exposes. # `$REFERENCE_DIR` is readable here only because judges run outside the agent's # turn -- the directory sits at mode 000 for all of `agent.communicate`. # Rationale: .claude/notes/contracts.md § Judge context and untrusted text diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index ec9d6dc81..11f1c3780 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -780,10 +780,12 @@ def all_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: def armed_criteria_passed( self, criteria: list[SuccessCriterion], gate_threshold: float = DEFAULT_STOP_EARLY_GATE_THRESHOLD ) -> bool: - """True iff the ARMED subset's weighted score meets ``gate_threshold``. + """True iff the ARMED subset meets ``gate_threshold``. - The gate a run the watcher actually CUT is judged by, as opposed to the - strict-AND ``all_criteria_passed`` a naturally-completed run gets. + NOT ``calculate_weighted_score``'s quantity: each armed criterion is + BINARISED against its own ``pass_threshold`` first, then weighted. That is + what makes ``gate_threshold=1.0`` an EXACT equivalence with the strict-AND + ``all_criteria_passed``, rather than an approximation of it. Raises: ValueError: ``criteria`` does not correspond 1:1 with diff --git a/src/coder_eval/models/routing.py b/src/coder_eval/models/routing.py index 059543ed6..c0e9bd7da 100644 --- a/src/coder_eval/models/routing.py +++ b/src/coder_eval/models/routing.py @@ -215,10 +215,21 @@ def _resolve_backend_route( ) -> ApiRoute: """Build the ``ApiRoute`` for an EXPLICITLY-requested backend. - Raises rather than asserts on a missing credential: this is reached on the - evaluate-only path with no preceding key validation, so it must survive ``-O``. + Used only by the ``checker_context.api_route`` override path. It RAISES, + naming the missing env var, when the requested backend is not configured, + rather than silently falling back to a different one: an explicit override + that cannot be honored must fail loudly, not degrade to a backend the task + author never asked for. Raise rather than assert, because this is reached on + the evaluate-only path with no preceding key validation and must survive + ``-O``. + + ``ApiBackend.LITELLM`` is the exception to "credentials come from the + environment": a checker-side litellm route is built ENTIRELY from + ``params_override`` / ``env_params_override``, never from + ``settings.litellm_*``, and raises when ``model_override`` is absent — there + is no default gateway model to fall back to. - Rationale: .claude/notes/contracts.md § Route resolution + Rationale: .claude/notes/contracts.md § LiteLLM params and env_params """ match backend: case ApiBackend.BEDROCK: @@ -268,8 +279,9 @@ def resolve_evaluation_route( NOT: evaluation is pinned to a constant Claude backend instead — Bedrock when the credentials are present, else Direct. - ``model_override`` lands on ``checker_context.api_route.model`` only when a - real override was given, never the agent's own model. + ``model_override`` comes FROM ``checker_context.api_route.model`` (the + task-authored input) and lands on the RESOLVED route's ``model``. It is set + only when a real override was given, never from the agent's own model. Rationale: .claude/notes/contracts.md § Route resolution """ diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index c8536cfdf..d36c85bb1 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -31,9 +31,13 @@ PRIOR_RESULT_FILENAME = "prior.json" # The container's own stdout+stderr transcript, and the name it is folded back -# under after a GRADING container. Constants, not literals (CE053): the -# fold-back is guarded by ``is_file()``, so a rename on the producing side -# would degrade the copy to a silent no-op. +# under after a GRADING container. Named for the PHASE because on the +# ``run --resume`` path ``docker.log`` is already taken by the executed +# container's log -- folding a grading log back under it repeats the +# task.log/grade.log truncation bug one layer down. Constants, not literals +# (CE053): the fold-back is guarded by ``is_file()``, so a rename on the +# producing side would degrade the copy to a silent no-op. +# Rationale: .claude/notes/persistence.md § Run-directory filename constants DOCKER_LOG_FILENAME = "docker.log" GRADE_DOCKER_LOG_FILENAME = "grade.docker.log" @@ -61,8 +65,10 @@ def write_text_atomic(path: Path, text: str) -> None: is UNIQUE per call. ``O_NOFOLLOW`` closes a symlink-plant overwrite primitive; the unique name keeps ``O_EXCL``'s guarantee while making a leftover from a SIGKILLed predecessor inert instead of a permanent refusal - to write the record. Mode is ``0o644`` — do not narrow it; the docker driver - reads this file back as a different uid. + to write the record. Mode is ``0o644``, the widest that is never group- or + world-WRITABLE; do not narrow it, because the docker driver reads this file + back as a different uid. It is a CEILING, not a guarantee — the umask still + narrows it (0o077 yields 0600). Rationale: .claude/notes/persistence.md § write_text_atomic """ diff --git a/src/coder_eval/sandbox.py b/src/coder_eval/sandbox.py index dcf2d00ea..ce62c526e 100644 --- a/src/coder_eval/sandbox.py +++ b/src/coder_eval/sandbox.py @@ -173,9 +173,15 @@ def __init__( def enforces_permission_windows(self) -> bool: """Whether a chmod window is a real, safe control in this sandbox. - True only inside a ``driver: docker`` container, where the filesystem is - private to this one task. On the host (``driver: tempdir``) it is a - deliberate no-op. + True only inside a ``driver: docker`` container. SAFE because the + filesystem is private to this one task; REAL because that container drops + ``DAC_OVERRIDE`` / ``DAC_READ_SEARCH``, without which a mode-000 directory + is still readable by container root and the window is a silent no-op. + COUNTERPART to ``docker_runner._build_argv``'s cap drops. + + On the host (``driver: tempdir``) it is a deliberate no-op: parallel tasks + share the checked-out ``tasks//`` tree, so enforcing would chmod the + user's own working copy across tasks. NOTE the predicate is the ``CODER_EVAL_IN_CONTAINER`` env var, NOT ``config.driver``: the in-container entry point rewrites ``driver: docker`` @@ -272,7 +278,9 @@ def adopt(self, workspace: Path) -> Path: criteria need to resolve the same shimmed binaries the agent did. The caller keeps ownership: ``_cleanup_on_exit`` stays False, so - ``cleanup()`` never deletes an adopted directory. + ``cleanup()`` never deletes an adopted directory. Criteria CAN still + mutate the tree (a ``run_command`` that writes), which is why the copy + path stays the default for a bare user-supplied work dir. Rationale: .claude/notes/isolation.md § Detached grading and `Sandbox.adopt` @@ -1072,11 +1080,12 @@ def _maybe_remediate_home_plugins_pollution(self) -> Path | None: def _build_run_command_env(self) -> dict[str, str]: """Build the environment for ``run_command``. - Eight layers, each independent -- none breaks if another is absent: the - parent env, the agent's captured SDK PATH (PREPENDED, so system binaries - stay reachable), the sandbox venv, ``/node_modules/.bin``, + Each layer is independent -- none breaks if another is absent: the parent + env, the agent's captured SDK PATH (PREPENDED, so system binaries stay + reachable), the sandbox venv, ``/node_modules/.bin``, ``NODE_PATH=""``, a sandbox-scoped ``NPM_CONFIG_PREFIX``, ``TASK_DIR``, - and ``REFERENCE_DIR``. + ``REFERENCE_DIR``, and ``PLUGIN_TOOLS_DIR`` (which defers to an inherited + value). Rationale: .claude/notes/isolation.md § The criterion environment, layer by layer """ @@ -1212,9 +1221,11 @@ def run_command(self, command: str, timeout: float | int | None = None) -> tuple logger.warning(error_msg) return -1, "", error_msg - # NOTE: get_file_content, file_exists and list_files intentionally do NOT - # validate path traversal -- the agent legitimately reads installed packages and - # system headers. That protection lives at the agent permission level. + # HAZARD: only ``list_files`` skips containment -- it joins and rglobs directly. + # ``get_file_content`` and ``file_exists`` go through ``resolve_files``, hence + # ``_within_sandbox`` and ``_reject_escaped``. Do not read this as "containment + # lives elsewhere": for a criterion path it lives HERE. + # Rationale: .claude/notes/isolation.md § Criterion paths are contained, quietly def _within_sandbox(self, candidate: Path) -> bool: """Whether a resolved criterion path stays inside the sandbox. @@ -1484,6 +1495,10 @@ def capture_to(self, artifact_dir: Path) -> Path: noise entries in :data:`_WORKSPACE_CAPTURE_IGNORE`, because the WORKDIR can BE ``$HOME``. + HAZARD: ``symlinks=True`` + ``ignore_dangling_symlinks=True`` are both + required -- without the second, one dangling link raises ``shutil.Error`` + and fails artifact capture for the whole task. + Returns the destination path; unlike preserve_to it does NOT repoint ``self.sandbox_dir``. diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 994944740..eee1c7092 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 18_163 +_ESSAY_BASELINE_WORDS = 18_549 _SRC = Path("src/coder_eval") From 279e824e765a1cdd26b0f4aa48d14443f8340181 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 22:27:50 -0700 Subject: [PATCH 12/19] fix(docs): restore the qualifier that made each absolute true MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review B found one shape repeated six times: compression kept a categorical claim and dropped the words that made it true. Each is the only fact its comment now carries, sitting above the code someone would edit. - claude_code_agent: the task-notification guard is checked SECOND, not first. What matters is that it precedes _is_sdk_result_message. - opencode_agent: "the condition is the TELEMETRY, not the event vocabulary" — but one of the two arms IS the vocabulary, and a later comment says so. The dropped word was "alone". Deleting the arm as redundant re-opens scoring SUCCESS 1.0 on zero turns. - batch: the predicate is NOT_GRADED **or** executed. "Executed is a required half" reads as `and`, which would leave zero-iteration execute rows ungraded forever. - reference_comparison: "every failure below raises" — three branches below return a gating 0.0 on purpose, one of them carrying a CE039 noqa. - docker_runner: the stdout limit does not mirror _POST_RUN_STREAM_LIMIT; it is 256x larger, deliberately. Unifying them downward reinstates the mid-stream teardown that lost whole paid tasks. - cli_called: "harness fault, not agent behaviour" contradicted the block 50 lines down explaining that all five paths are agent-reachable. Also restores the judge ignore floor's MECHANISM: it is a copy-time control, the same list passed as copytree's `ignore`, not only an SDK setting. Without that, dropping .claude / .mcp.json from it looks redundant and lets an agent-planted settings file into the judge's own working directory. _resolve_backend_route's contract now has a home in the notes, and the docstring points at it rather than at the LiteLLM section. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/contracts.md | 14 +++++++++++--- src/coder_eval/agents/claude_code_agent.py | 3 ++- src/coder_eval/agents/opencode_agent.py | 10 ++++++---- src/coder_eval/criteria/cli_called.py | 6 ++++-- src/coder_eval/criteria/reference_comparison.py | 5 +++-- src/coder_eval/isolation/docker_runner.py | 4 +++- src/coder_eval/models/criteria.py | 7 +++++-- src/coder_eval/models/routing.py | 2 +- src/coder_eval/orchestration/batch.py | 9 +++++---- src/coder_eval/orchestrator.py | 5 ++--- tests/lint/prose_budget.py | 2 +- 11 files changed, 43 insertions(+), 24 deletions(-) diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index 09bd950ea..fd268399c 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -188,9 +188,17 @@ default when unset, so the precedence (explicit per-criterion model, then the ro the default judge model) survives a `model_dump(mode="json")` and reload, which a `model_fields_set` check would not. -Route resolution raises rather than asserts, because it is reached on the evaluate-only path -with no preceding key validation and must survive `-O`. The exhaustive final arm of each -match is unreachable but present, so every path returns explicitly. +An EXPLICIT backend override that cannot be honored RAISES, naming the missing env var, +rather than falling back to a different backend. That is the load-bearing rule of +`_resolve_backend_route`: the override exists because a task author named a backend, and +quietly grading on another one publishes a verdict from a route nobody asked for. The +tempting edit is a fallback (`or settings.bedrock_model`); it is the regression this rule +forbids. + +It raises rather than asserts for a second, independent reason: the path is reached on the +evaluate-only flow with no preceding key validation, so the check must survive `-O`. The +exhaustive final arm of each match is unreachable but present, so every path returns +explicitly. Under `DirectRoute` the judge transport is resolved at startup: `anthropic` when a key is present, `None` otherwise, in which case an enabled `llm_judge` fails at dispatch. The diff --git a/src/coder_eval/agents/claude_code_agent.py b/src/coder_eval/agents/claude_code_agent.py index c773e981e..bd1fb002e 100644 --- a/src/coder_eval/agents/claude_code_agent.py +++ b/src/coder_eval/agents/claude_code_agent.py @@ -160,7 +160,8 @@ def _is_task_notification(message: Any) -> bool: """Check if message is a TaskNotificationMessage (sub-agent terminal event). It carries ``session_id`` + ``usage``, so it would otherwise be misread as - the final ResultMessage — hence this guard, checked FIRST. Identified by the + the final ResultMessage — hence this guard, checked BEFORE + ``_is_sdk_result_message``. Identified by the SDK type or ``subtype`` rather than attribute-presence sniffing, so it cannot misfire on a mock; the ``subtype`` fallback is what lets a duck-typed mock be recognized. diff --git a/src/coder_eval/agents/opencode_agent.py b/src/coder_eval/agents/opencode_agent.py index 1d3a17a24..399fda793 100644 --- a/src/coder_eval/agents/opencode_agent.py +++ b/src/coder_eval/agents/opencode_agent.py @@ -1164,10 +1164,12 @@ async def _settle_turn( detail = stderr_bytes.decode("utf-8", "replace").strip() or f"exit code {proc.returncode}" self._crash_turn(state, collector, f"OpenCode exited non-zero: {detail}") - # A clean exit that captured NO token telemetry must not score. The - # condition is the TELEMETRY, not the event vocabulary: both arms below - # reach the same silent-empty-success outcome. Intentional cuts are exempt - # — either can land before the first event, or mid-step. + # A clean exit that captured NO token telemetry must not score. Keying on + # the token counts ALONE is what misses the second arm: an exit that + # recognized no events at all reaches the same silent-empty-success + # outcome. Intentional cuts are exempt — either can land before the first + # event, or mid-step. (The two arms are NOT interchangeable downstream; + # see the require_token_telemetry escape hatch below.) # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash nothing_recognized = state.recognized_events == 0 finished_without_tokens = state.steps_finished > 0 and state.usage.is_empty() diff --git a/src/coder_eval/criteria/cli_called.py b/src/coder_eval/criteria/cli_called.py index f9a8054a0..278dd9304 100644 --- a/src/coder_eval/criteria/cli_called.py +++ b/src/coder_eval/criteria/cli_called.py @@ -56,8 +56,10 @@ def _check_impl( # No pre-flight re.compile: `FlagMatch` compiles at validation, which is # where it has to happen -- the response-rule surface cannot report. if not sandbox.file_exists(criterion.log): - # Harness fault, not agent behaviour. Failing stops a max_count: 0 - # guard passing vacuously against a log that never existed. + # Scored 0.0, never raised: an agent can `rm` the log, so escalating + # would hand it a way to turn a failing run into an ERROR. Failing + # stops a max_count: 0 guard passing vacuously against a log that + # never existed. See the five uniform refuse-to-score paths below. return CriterionResult( criterion_type=criterion.type, description=criterion.description, diff --git a/src/coder_eval/criteria/reference_comparison.py b/src/coder_eval/criteria/reference_comparison.py index 54aa1347f..987c5b688 100644 --- a/src/coder_eval/criteria/reference_comparison.py +++ b/src/coder_eval/criteria/reference_comparison.py @@ -58,8 +58,9 @@ def _check_impl( # HAZARD: confined to the reference dir, unlike a judge's author-written # `files:` entry -- this names one file OF the solution, so traversal out of - # the staged copy is always a mistake. Every failure below is a - # TASK-DEFINITION error and raises rather than scoring a gating 0.0. + # the staged copy is always a mistake. Every REFERENCE_FILE failure is a + # TASK-DEFINITION error and raises rather than scoring a gating 0.0 — the + # agent-side failures further down deliberately do the opposite. # Rationale: .claude/notes/contracts.md § What escalates instead of scoring 0.0 ref_path = (reference_dir / criterion.reference_file).resolve() if not ref_path.is_relative_to(reference_dir.resolve()): diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index e259dcb4d..423c0ab1f 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -131,7 +131,9 @@ def _rewrite_loopback_for_container(url: str) -> str | None: HEARTBEAT_STALE_SECONDS = 20 # asyncio's StreamReader caps a line at 64 KiB by default, which a single stream -# event can exceed. Mirrors Orchestrator._POST_RUN_STREAM_LIMIT. +# event can exceed. The same KIND of guard as Orchestrator._POST_RUN_STREAM_LIMIT, +# but deliberately far larger (64 MiB vs 256 KiB) — do not unify them downward: a +# whole-file tool input on this stream tore the container down before task.json. # Rationale: .claude/notes/isolation.md § The stdout line limit STDOUT_LINE_LIMIT_BYTES = 64 * 1024 * 1024 # 64 MiB diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index a087efaf8..4a613e6a7 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -30,8 +30,11 @@ # SECURITY: the judge's ignore_patterns FLOOR, enforced unconditionally in # ``criteria/agent_judge.py::_build_agent_config`` even when the user supplied their -# own list. Imported from both call sites so the model defaults and the checker -# floor cannot drift. +# own list. These are COPY-TIME controls, not just SDK settings: the same list is +# the ``ignore`` passed to ``shutil.copytree`` in ``evaluation/sub_agent.py``, so +# dropping an entry lets an agent-planted ``.claude/`` or ``.mcp.json`` land in the +# judge's own working directory. Imported from both call sites so the model +# defaults and the checker floor cannot drift. # Rationale: .claude/notes/contracts.md § The security floor JUDGE_SECURITY_IGNORE_FLOOR: tuple[str, ...] = (".claude", ".mcp.json", "_reference") diff --git a/src/coder_eval/models/routing.py b/src/coder_eval/models/routing.py index c0e9bd7da..cc969824e 100644 --- a/src/coder_eval/models/routing.py +++ b/src/coder_eval/models/routing.py @@ -229,7 +229,7 @@ def _resolve_backend_route( ``settings.litellm_*``, and raises when ``model_override`` is absent — there is no default gateway model to fall back to. - Rationale: .claude/notes/contracts.md § LiteLLM params and env_params + Rationale: .claude/notes/contracts.md § Route resolution """ match backend: case ApiBackend.BEDROCK: diff --git a/src/coder_eval/orchestration/batch.py b/src/coder_eval/orchestration/batch.py index a3fed140d..0674669a0 100644 --- a/src/coder_eval/orchestration/batch.py +++ b/src/coder_eval/orchestration/batch.py @@ -356,10 +356,11 @@ def _owes_a_grade(result: EvaluationResult) -> bool: verdict at all. A row with a criteria vector or a score HAS been graded, whatever its status. - "Executed" is a required half, not decoration — a synthetic ERROR / - BUILD_FAILED row for a container that died before producing task.json carries - no verdict either, and routing those into grading replaced the real diagnostic - with a wrong-cause grading error. + The test is NOT_GRADED **or** executed, and the second half is what excludes + the synthetic rows: an ERROR / BUILD_FAILED record for a container that died + before producing task.json carries no verdict either, and routing those into + grading replaced the real diagnostic with a wrong-cause grading error. A + NOT_GRADED row qualifies on the first half alone, however few iterations it ran. Rationale: .claude/notes/orchestration.md § `--resume` is command-relative """ diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 39af0480b..6d978e89b 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -2184,9 +2184,8 @@ async def _evaluation_loop(self) -> bool: # simulator produces the opening utterance itself. return await self._simulation_dialog_loop(self.task.initial_prompt, sandbox_dir) - # Guaranteed for real agents by check_prompt_fields; a no-op task runs - # with no prompt, which NoOpAgent - # ignores it and returns an empty turn. + # Guaranteed for real agents by check_prompt_fields; a no-op task runs with + # no prompt, which NoOpAgent ignores, returning an empty turn. current_prompt = self.task.initial_prompt or "" iteration = 1 diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index eee1c7092..7551f1e37 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 18_549 +_ESSAY_BASELINE_WORDS = 18_673 _SRC = Path("src/coder_eval") From 4adca82c32911d482bdbd9b083d73ec1aee0e838 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 22:30:04 -0700 Subject: [PATCH 13/19] fix(docs): stop asserting a tmpfs mask that no longer exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--tmpfs` is never emitted. `_build_argv` does not construct one, `test_reference_inside_task_dir_needs_no_tmpfs_mask` asserts `not _tmpfs(argv)`, and docker_runner.py's own `_reference_mount_args` docstring opens "No tmpfs mask any more". Three of its neighbours said the opposite, two of them lines this refactor wrote. The claim predates the branch, so this is not a regression — but the refactor read every one of those lines and compressed them instead of questioning them, and it authored two new canonical homes for the falsehood. Re-ratifying it is worse than never having looked. What is actually true: the task dir is a shielded read-write COPY, so a reference embedded in it is covered by the same mode-000 window rather than hidden under a layered filesystem. `orchestration/evaluation.py`'s hard-fail was justified by the mask and by an EROFS on a `:ro` bind — both false now, and the real reason (the fallback resolves to a tree the window was not opened over) is the one that survives. Also restructures `permissions.md`'s Phase-1 blob. Its single 4,450-char line duplicated two of its own sibling sections, restated the FOWNER/CHOWN rationale that isolation.md explicitly delegates to docs/ saying "they are not restated here", and carried the tmpfs claim. All four inbound source pointers still resolve. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/permissions.md | 45 +++++++++++++++++++++- src/coder_eval/isolation/docker_runner.py | 13 ++++--- src/coder_eval/orchestration/evaluation.py | 18 ++++----- tests/lint/prose_budget.py | 2 +- 4 files changed, 61 insertions(+), 17 deletions(-) diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md index 0e32a4d8d..0f7fdd73c 100644 --- a/.claude/notes/permissions.md +++ b/.claude/notes/permissions.md @@ -4,7 +4,50 @@ ## Reference solutions and the anti-cheat window -- **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory **is** shielded alongside it: it previously was not, because under docker it was bind-mounted `:ro` and the chmod returned EROFS, but it is now a read-write throwaway copy (`docker_runner._prepare_task_dir_mount`) so the window applies — which matters because the task dir holds grading material beyond the reference (`run_command` fixtures, expected outputs, and, for a task laid out flat, every SIBLING task's reference). What the window does NOT hide is the task DEFINITION: `task.yaml` is also staged at `/work/input` for the in-container orchestrator, and that mount is untouched. Hiding the criteria from the agent is a separate, unsolved problem. Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half. +`task.reference` is a single required `directory:`, relative to the task YAML. The inline +`code:` and single-file `file:` forms are gone because a directory is the only shape that +can be permission-gated as a UNIT; a `model_validator(mode="before")` gives the removed +forms a migration error. + +The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, +symlinks stripped) into a tempdir and never preserves it into `run_dir/artifacts`. Cleanup +goes through `rmtree_restrictive`, keyed on a root recorded BEFORE the copy so a failed copy +still cleans up — `rmtree(ignore_errors=True)` silently declines on a tree left at 000. + +That copy is held at mode `000` for the whole of every `agent.communicate` call, via +`Sandbox.set_permissions`. Whether the window is a real control at all, and why it keys on +`CODER_EVAL_IN_CONTAINER` rather than `sandbox.driver`, is +[isolation.md](isolation.md) § Capability drops and the anti-cheat window; +`resolve_reference_dir` gates its `/work/references` branch on the same var for the same +reason. How the window stacks is § The stacked chmod window below, and why an +unappliable window is a hard error is § strict=True and the hard-fail path. + +The task directory is shielded ALONGSIDE the reference, which matters because it holds +grading material beyond it: `run_command` fixtures, expected outputs, and — for a task laid +out flat — every SIBLING task's reference. That is possible only because the task dir is a +read-write throwaway copy; when it was bind-mounted `:ro` the chmod returned EROFS. There is +no tmpfs mask and has not been one since: see +[isolation.md](isolation.md) § Why the framework mounts are writable copies. + +What the window does NOT hide is the task DEFINITION. `task.yaml` is also staged at +`/work/input` for the in-container orchestrator and that mount is untouched, so hiding the +criteria from the agent remains a separate, unsolved problem. + +Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as +`$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names +one file via `reference_file`. + +**KNOWN GAP — defense-in-depth, not a boundary.** Two holes, both documented in +[docs/DOCKER_ISOLATION.md](../../docs/DOCKER_ISOLATION.md#architecture) rather than restated +here: a deliberate re-chmod by the root agent, and waiting the window out (it spans +`agent.communicate` only, and nothing reaps agent child processes at turn end). + +The **write** half of the second is closed: `path_utils.digest_tree` hashes the tree at +staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising +`ReferenceTamperedError` on a mismatch — so an agent cannot overwrite the reference to drive +`reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full +containment needs the agent running as a non-root uid AND the window held for its whole +lifetime. `tasks/anti_cheat_reference` probes the passive-read half. ## The stacked chmod window diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 423c0ab1f..44c63dfe4 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -1407,10 +1407,11 @@ def _build_argv( if self.grade_workspace is not None: argv += ["-v", f"{self.grade_workspace.resolve()}:{CONTAINER_GRADE_WORKSPACE}"] - # ANTI-CHEAT: the reference normally lives INSIDE the task dir, so an empty - # tmpfs masks its path there and a writable COPY is mounted at - # /work/references instead. Docker applies mounts by target-path depth, so - # the deeper tmpfs wins regardless of argv order. + # ANTI-CHEAT: the reference normally lives INSIDE the task dir, and a + # writable COPY is mounted at /work/references for the window to chmod. + # There is NO tmpfs mask -- the task dir is itself a shielded copy now + # (see _reference_mount_args), so the embedded original is covered by the + # same window rather than hidden by a layered filesystem. # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies argv += self._reference_mount_args() # A throwaway lean COPY of ~/.claude, read-WRITE at the host's own path @@ -1468,8 +1469,8 @@ def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: _auto_mount(agent_cfg.system_prompt_file, dir_only=False) # HAZARD: task.reference.directory is deliberately NOT auto-mounted at its - # host path -- that would re-expose it through $TASK_DIR, the exact hole - # the tmpfs mask closes. + # host path. That would bind the REAL tree in beside the shielded copy, so + # the mode-000 window would leave it readable through $TASK_DIR. for mount in cfg.extra_mounts: normalized = _validate_extra_mount(mount) argv += ["-v", normalized] diff --git a/src/coder_eval/orchestration/evaluation.py b/src/coder_eval/orchestration/evaluation.py index 91dfe6fc4..f663d987d 100644 --- a/src/coder_eval/orchestration/evaluation.py +++ b/src/coder_eval/orchestration/evaluation.py @@ -64,9 +64,9 @@ def resolve_reference_dir(task: TaskDefinition, task_file: Path | None) -> Path if not task.reference: return None - # Under docker the host bind-mounts the reference at a fixed container path - # and masks its original location with an empty tmpfs, so resolving relative - # to task_file would find the MASK. Gated on the env var as well as the path, + # Under docker the host bind-mounts a private COPY of the reference at a fixed + # container path, and the task dir is a separate shielded copy -- so resolving + # relative to task_file would find the wrong tree. Gated on the env var as well as the path, # for the reason Sandbox.enforces_permission_windows is: a bare # `/work/references` probe would silently hijack every task's reference on any # host that happens to have that directory, invisibly. @@ -76,12 +76,12 @@ def resolve_reference_dir(task: TaskDefinition, task_file: Path | None) -> Path if container_mount.is_dir(): logger.debug("Reference resolved from the container mount at %s", container_mount) return container_mount - # HARD FAIL rather than falling back to task_file.parent: in-container - # that fallback resolves to the UN-masked reference under the `:ro` - # task-dir bind, which the mode-000 window cannot chmod (EROFS) — so the - # run would complete with the solution readable for the whole turn, - # reporting a normal pass/fail. A missing mount means the host-side wiring - # is broken, and that must be LOUD rather than silently unprotected. + # HARD FAIL rather than falling back to task_file.parent: in-container that + # fallback resolves to the reference embedded in the task-dir copy, which is + # NOT the path the window was opened over — so the run would complete with + # the solution readable for the whole turn, reporting a normal pass/fail. A + # missing mount means the host-side wiring is broken, and that must be LOUD + # rather than silently unprotected. raise FileNotFoundError( f"Task declares reference.directory={task.reference.directory!r} but {CONTAINER_REFERENCE_DIR} " + "is not mounted in this container; refusing to run unprotected. Most likely that path does not " diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 7551f1e37..df55654a3 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -27,7 +27,7 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 18_673 +_ESSAY_BASELINE_WORDS = 18_703 _SRC = Path("src/coder_eval") From 06b9d1efbe2e64b69a272786950ded4d6b782370 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 22:43:55 -0700 Subject: [PATCH 14/19] feat(lint): replace the prose baseline with two self-adjusting rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _ESSAY_BASELINE_WORDS was the weakest part of the gate. It was a single tree-wide number, hand-edited eleven times in this work alone, that a reviewer had to take on trust; it let growth in one file hide behind shrinkage in another; and it said nothing at all about a file that did not exist yet. It is gone. Two rules replace it, and neither has a number anyone maintains: own-line comments per file <= MAX(20, 0.15 * file length) no docstring over 150 words of PROSE The comment budget is proportional, so deleting code takes its budget with it and a new file is governed from its first commit. Own-line only: a trailing `# noqa` is a directive and a per-member annotation on an enum is the contract a dispatcher reads — counting either would push against documenting them. The floor is what protects a constants module at one comment per constant, which is where the tree's natural maximum sits. The essay rule now counts prose, not structure: an Args/Returns/Raises block is interface documentation, and counting it pushed exactly the docstrings that document their contract best over the line. Exemptions are categorical rather than numeric — an @abstractmethod docstring IS the contract implementers read, so the plugin SPI is covered by kind, and a new abstract method is covered automatically. The old "at most two, by fiat" allowance is unnecessary: the tree now has ZERO essays. Getting there took 44 comment lines out of 7 files. Most came out by reflowing two lines into one or cutting a section divider; the one real compression was a 22-line block in early_stop.py restating the floor bound that orchestration.md already owns, and the distractor-exclusion rule it uniquely held moved there rather than being dropped. One reflow silently merged `# pyright: reportImportCycles=false` into the prose line above it, which would have stopped pyright honouring it. --assert-code-unchanged caught it. That is the second time the directive multiset has earned its place. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/orchestration.md | 7 +- CLAUDE.md | 9 +- src/coder_eval/config.py | 35 ++---- src/coder_eval/criteria/agent_judge.py | 3 +- src/coder_eval/evaluation/sub_agent.py | 3 +- src/coder_eval/isolation/docker_runner.py | 15 +-- src/coder_eval/orchestration/config.py | 20 +--- src/coder_eval/orchestration/early_stop.py | 32 +----- src/coder_eval/path_utils.py | 7 +- src/coder_eval/pricing.py | 19 +--- tests/lint/prose_budget.py | 122 ++++++++++++++++++--- tests/test_prose_budget.py | 76 +++++++++++-- 12 files changed, 218 insertions(+), 130 deletions(-) diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index fd203e57b..200b534ed 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -236,7 +236,12 @@ trajectory continues. A `decide_within` timeout participates as an ordinary weig so a low-weight criterion's timeout that cannot doom the gate does not stop the run. A **pass-stop** fires once the `on_pass: stop` subset's FLOOR — worst case, every -still-undecided member scores 0 — already meets the threshold. +still-undecided member scores 0 — already meets the threshold. Criteria armed only on the +FAIL side (distractors) are excluded from both the numerator and the denominator of that +bound: they can never live-pass and exist only to guard the fail side, so folding them in +would veto every pass-stop and penalise the bound for a criterion it was never scoped to +cover. With no `on_pass: stop` criteria at all the bound is vacuous and returns nothing — +there is no pass-stop to take, and the run continues to the cap. At the default threshold both bounds collapse exactly to "any single armed criterion's effective fail stops the run" and "every `on_pass: stop` criterion has live-passed". diff --git a/CLAUDE.md b/CLAUDE.md index 7f139d69e..1f014078f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -208,7 +208,7 @@ make evalboard-verify # the JS half: tsc --noEmit + vitest + next build make docs-indexes # README/docs index tables from the mkdocs nav (CE028) make plugin-reference # the plugin's criteria reference from the models (CE033) -make docs-budget # prose budget report; fails `make verify` if the total grows +make docs-budget # per-file comment budget + docstring essay check (fails `make verify`) ``` Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is @@ -361,8 +361,11 @@ bandit, pre-commit, mcp code cannot say - **A docstring states the contract, not the history** — what a caller must know to call it correctly. Why the design is this shape belongs in `.claude/notes/`; what it used to - be belongs in git. `make docs-budget` reports the standing total and fails - `make verify` if it grows + be belongs in git. `make docs-budget` enforces two rules, both self-adjusting: a file's + own-line comments may not exceed `MAX(20, 0.15 × its length)`, and no docstring may + exceed 150 words of PROSE (an `Args:`/`Returns:`/`Raises:` block is structure, not + prose; an `@abstractmethod` is exempt because its docstring IS the interface contract). + There is no tree-wide total to hand-maintain — delete code and the budget shrinks with it ## Notes for AI Assistants diff --git a/src/coder_eval/config.py b/src/coder_eval/config.py index d99f27edf..651788145 100644 --- a/src/coder_eval/config.py +++ b/src/coder_eval/config.py @@ -26,11 +26,9 @@ ).decode("utf-8") -# Load .env file with override so .env values always win over shell environment +# override=True so .env always wins over the shell's possibly-stale credentials. load_dotenv(override=True) -# For certain keys, we want .env values to take precedence over shell environment -# because the shell may have outdated/different credentials env_values = dotenv_values(".env") for key in [ "ANTHROPIC_API_KEY", @@ -79,36 +77,30 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: _reject_removed_default_knobs() super().__init__(*args, **kwargs) - # API Keys (for Claude Code agent only) anthropic_api_key: str | None = None - # Paths runs_dir: Path = Path("runs") # Base directory for timestamped runs - # API Backend routing api_backend: ApiBackend = ApiBackend.DIRECT - # AWS Bedrock settings (used when api_backend == "bedrock") aws_bearer_token_bedrock: str | None = None aws_region: str | None = None bedrock_model: str | None = None # Cross-region model ID bedrock_small_model: str | None = None # Cross-region small model ID - # HAZARD: these map to the ANTHROPIC_* vars, but ONLY inside the SDK subprocess - # env. Deliberately NOT named anthropic_*, so the export loop below cannot leak - # ANTHROPIC_BASE_URL process-wide and redirect the judge's own client. + # HAZARD: these map to the ANTHROPIC_* vars ONLY inside the SDK subprocess env. + # NOT named anthropic_*, so the export loop cannot leak ANTHROPIC_BASE_URL + # process-wide and redirect the judge's own client. litellm_base_url: str | None = None litellm_auth_token: str | None = None litellm_model: str | None = None litellm_small_model: str | None = None - # Must point at the SAME file the proxy writes. When set and present, the harness - # joins each call's ACTUAL cost onto the turn; unset or missing => static pricing. + # Must point at the SAME file the proxy writes; unset or missing => static pricing. # Rationale: .claude/notes/reporting.md § Cost joining litellm_cost_log: str | None = None # CODEX_MODEL is the fallback when a task doesn't pin agent.model. For Azure set - # CODEX_API_VERSION too and use the deployment name as the model. CODEX_BASE_URL - # / CODEX_API_VERSION / CODEX_API_KEY are read via os.getenv in the agent. + # CODEX_API_VERSION too and use the deployment name as the model. codex_model: str | None = None # GEMINI_API_KEY is read from .env here so the export loop re-publishes it to @@ -116,15 +108,12 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: gemini_api_key: str | None = None antigravity_model: str | None = None - # Logging log_level: str = "INFO" # Default log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) log_to_file: bool = False # Whether to enable file logging - # On by default via the baked-in connection string. TELEMETRY_ENABLED is the - # single canonical disable gate. + # On by default via the baked-in connection string, which any set value (env + # or .env) overrides. TELEMETRY_ENABLED is the single canonical disable gate. telemetry_enabled: bool = True - # Defaults to the embedded coder-eval resource; any set value (env or .env, via - # the aliases below) overrides it — pydantic-settings prefers env over default. telemetry_connection_string: str | None = Field( default=_DEFAULT_TELEMETRY_CONNECTION_STRING, validation_alias=AliasChoices( @@ -180,8 +169,7 @@ def _validate_litellm_settings(self) -> None: f"LiteLLM-endpoint routing is enabled but missing required settings: {', '.join(missing)}." + " Please set them in your .env file." ) - # Reject a malformed base_url here so the downstream preflight and - # environment_info get a well-formed absolute URL. + # Reject a malformed base_url so the preflight and environment_info get a well-formed URL. parts = urlsplit(self.litellm_base_url or "") if parts.scheme not in ("http", "https") or not parts.hostname: raise ValueError( @@ -215,15 +203,12 @@ def validate_api_keys(self, agent_type: str) -> None: return -# Global settings instance settings = Settings() -# For external libraries that read os.getenv() rather than the Settings object. -# Non-None values only, stringified. +# For external libraries that read os.getenv(); non-None values only, stringified. for key, value in settings.model_dump().items(): if value is not None: env_key = key.upper() - # Convert Path objects and other types to strings if isinstance(value, Path): os.environ[env_key] = str(value) elif isinstance(value, bool): diff --git a/src/coder_eval/criteria/agent_judge.py b/src/coder_eval/criteria/agent_judge.py index 032cd2a75..896b39992 100644 --- a/src/coder_eval/criteria/agent_judge.py +++ b/src/coder_eval/criteria/agent_judge.py @@ -44,8 +44,7 @@ path_uses_token, ) -# Not part of the public coder_eval.models surface, but the single source of truth -# for both files. +# Not part of the public coder_eval.models surface, but the single source of truth for both files. from coder_eval.models.criteria import ( # noqa: CE001 JUDGE_SECURITY_IGNORE_FLOOR, _default_judge_agent_config, diff --git a/src/coder_eval/evaluation/sub_agent.py b/src/coder_eval/evaluation/sub_agent.py index 536370c71..f9e17425c 100644 --- a/src/coder_eval/evaluation/sub_agent.py +++ b/src/coder_eval/evaluation/sub_agent.py @@ -88,8 +88,7 @@ def __init__( # drop a user's own nested subdir of the same name. # Rationale: .claude/notes/contracts.md § The security floor self._reference_ignore_patterns = reference_ignore_patterns or [] - # Runtime-only in-process MCP injection. NOT routed through ``sdk_options`` - # -- ``mcp_servers`` is framework-owned. + # Runtime-only MCP injection, NOT via ``sdk_options`` -- ``mcp_servers`` is framework-owned. self._extra_mcp_servers = extra_mcp_servers or {} # Public so the criterion can read it after ``run_async()`` returns; absent # when the caller passed ``capture=None``. diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 44c63dfe4..fff43caf5 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -653,8 +653,7 @@ async def run(self) -> EvaluationResult: widened_workspace = await asyncio.to_thread(grant_container_access, self.grade_workspace, writable=True) argv = self._build_argv(input_dir, output_dir, container_name=container_name, image=image) logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) - # Prime the heartbeat before the container starts so the - # watchdog never sees an initial stale state. + # Prime the heartbeat before the container starts so the watchdog never sees an initial stale state. heartbeat_path = output_dir / HEARTBEAT_FILENAME await asyncio.to_thread(heartbeat_path.touch) heartbeat_task = asyncio.create_task(_heartbeat_loop(heartbeat_path)) @@ -673,13 +672,11 @@ async def run(self) -> EvaluationResult: returncode = await self._stream_container_output(proc, log_fh) finally: heartbeat_task.cancel() - # Narrowed so a genuine KeyboardInterrupt / SystemExit from a - # parallel sibling still propagates. + # Narrowed so a genuine KeyboardInterrupt / SystemExit from a parallel sibling still propagates. with contextlib.suppress(asyncio.CancelledError): await heartbeat_task await asyncio.to_thread(log_fh.close) - # Cancelled mid-flight: kill the container AND the docker CLI - # subprocess, best-effort. + # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort. if proc.returncode is None: await self._kill_container(proc, container_name) @@ -906,8 +903,7 @@ def _assert_grade_honored(self, result: EvaluationResult, task_json: Path | None """ if self.grade: return - # Keyed on EVIDENCE, not on the label: the question is not "what status is - # this" but "did it grade". + # Keyed on EVIDENCE, not on the label: the question is not "what status is this" but "did it grade". graded_anyway = bool(result.success_criteria_results) or result.weighted_score is not None if not graded_anyway and ( result.final_status.is_execution_fact or result.final_status is FinalStatus.NOT_GRADED @@ -1108,8 +1104,7 @@ def _prepare_task_dir_mount(self, staging: Path) -> None: return task_dir_copy = staging / "task_dir" shutil.copytree(source, task_dir_copy, ignore=ignore_patterns_and_symlinks(REFERENCE_COPY_IGNORE)) - # Read-only like the reference copy: criteria read fixtures here, nothing - # legitimately writes them. + # Read-only like the reference copy: criteria read fixtures here, nothing legitimately writes them. grant_container_access(task_dir_copy, writable=False) self._task_dir_mount_src = task_dir_copy diff --git a/src/coder_eval/orchestration/config.py b/src/coder_eval/orchestration/config.py index 79252a188..a3cfb9d48 100644 --- a/src/coder_eval/orchestration/config.py +++ b/src/coder_eval/orchestration/config.py @@ -69,7 +69,6 @@ class BatchRunConfig(BaseModel): ), ) - # Dataset sampling (for cheap smoke runs on dataset-backed tasks) max_rows: int | None = Field( default=None, ge=1, @@ -85,7 +84,6 @@ class BatchRunConfig(BaseModel): ), ) - # Replicate count override repeats: int | None = Field( default=None, ge=1, @@ -104,7 +102,6 @@ class BatchRunConfig(BaseModel): ), ) - # Logging verbose: bool = Field(default=False, description="Enable verbose (DEBUG level) logging for Docker output") # Docker WORKDIR alignment for the NON-docker-driver dispatch path — a host @@ -121,15 +118,8 @@ class BatchRunConfig(BaseModel): ), ) - # TODO(container-death-diagnostics): consider a run-level default resource - # cap. Containers run uncapped today (sandbox.limits.{max_memory_mb, - # max_cpus,max_pids} default to None -> _build_argv emits no --memory/ - # --cpus/--pids-limit), so at --max-parallel=20 a single runaway task can - # pressure the whole host. An opt-in default cap is already expressible - # via the EXISTING layered sandbox config -- defaults.sandbox.limits. - # max_memory_mb in the experiment YAML, or `-D sandbox.limits. - # max_memory_mb=N` on `coder-eval run` -- both flow through - # resolve_all_tasks and are overridden by per-task limits. If a dedicated - # CLI knob is ever wanted, add it as the FIRST (lowest-priority) layer in - # _build_sandbox_layers so per-task limits win, and do NOT default it to - # a non-None value (would change behavior for existing configs). + # TODO(container-death-diagnostics): containers run uncapped today, so at a + # high --max-parallel one runaway task can pressure the host. An opt-in + # default is already expressible through the layered sandbox config; a + # dedicated CLI knob would go in as the LOWEST-priority layer, never + # defaulted to a value (that would change existing configs). diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index 6666a8af2..c3f23e46a 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -284,8 +284,6 @@ def for_task(cls, task: TaskDefinition) -> EarlyStopWatcher: ) return cls(task.task_id, armed, max_turns=max_turns, gate_threshold=gate_threshold) - # --- StreamCallback -------------------------------------------------- # - def on_event(self, event: StreamEvent) -> None: """Fail-open wrapper around ``_on_event_impl``: any unexpected exception anywhere in the round — the collector reduction included, not just the @@ -361,8 +359,6 @@ def disarmed(self) -> bool: """True once a ``live_verdict`` raised and the watcher degraded to a full run.""" return self._disarmed - # --- Stop rule -------------------------------------------------- # - def _ceiling(self, verdicts: list[LiveVerdict]) -> float: """Best-case weighted score over the WHOLE armed set, given current verdicts. @@ -502,28 +498,12 @@ def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: self._fire(reason, self._armed[candidate_index][0], tool_call_index=tool_call_index) return - # Pass-stop: the on_pass=stop subset's own floor bound (worst case: - # every still-undecided member scores 0, weighted against only that - # subset's total weight) already meets ``gate_threshold`` — guaranteed - # regardless of what the rest of that subset still decides. Criteria - # armed only on the fail side (distractors) are excluded from both the - # numerator and the denominator: they can never live-pass and only - # guard the fail side above, so folding them in would veto every - # pass-stop and penalize this bound for a criterion it was never - # scoped to cover. At the default ``gate_threshold=1.0`` this requires - # every on_pass=stop criterion to actually be "pass" (any non-pass - # drops the floor below 1.0). The vacuous case (no on_pass=stop - # criteria at all) returns None — nothing to pass-stop on, the run - # continues to the cap. - # - # Recall deferral, mirrored from the fail-stop: the pass-stop is HELD - # while any pass-capable armed criterion OUTSIDE the on_pass=stop - # subset is still undecided (members of the subset are already priced - # into the floor). Cutting here would freeze a sibling - # ``on_pass: continue`` criterion's expected signal out of the - # trajectory — an unearned fail on the armed gate that a full run - # would not have produced. Once every such criterion decides (pass or - # fail), the still-satisfied floor fires the pass-stop on that round. + # Pass-stop: the on_pass=stop subset's FLOOR already meets + # ``gate_threshold``. Distractors are excluded from both the numerator and + # the denominator; no on_pass=stop criteria at all returns None. HELD while + # any pass-capable armed criterion OUTSIDE the subset is undecided -- + # cutting there would freeze a sibling's expected signal out of the run. + # Rationale: .claude/notes/orchestration.md § The ceiling and floor bounds pass_stop_indices = [i for i, armed_pass in enumerate(self._pass_trigger) if armed_pass] outside_pass_capable_undecided = any( v == "undecided" and "pass" in pol and not armed_pass diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index d36c85bb1..ec5572566 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -56,10 +56,9 @@ def write_text_atomic(path: Path, text: str) -> None: """Write ``text`` to ``path`` via a temp file + ``os.replace``. A plain ``write_text`` truncates first, so a crash mid-write leaves a - half-file — and a truncated ``task.json`` parses as *malformed*, which the - recovery paths read as "not complete", so ``--resume`` pays for the agent - again. One writer, so every producer of that file has the same crash - semantics. + half-file — and a truncated ``task.json`` parses as *malformed*, which + ``--resume`` reads as "not complete" and pays for the agent again. One + writer, so every producer has the same crash semantics. The temp file is opened ``O_CREAT | O_EXCL | O_NOFOLLOW`` under a name that is UNIQUE per call. ``O_NOFOLLOW`` closes a symlink-plant overwrite diff --git a/src/coder_eval/pricing.py b/src/coder_eval/pricing.py index a0838e741..a04b70794 100644 --- a/src/coder_eval/pricing.py +++ b/src/coder_eval/pricing.py @@ -27,8 +27,6 @@ class ModelPricing: cache_read_per_mtok: float # prompt caching read -# Official vendor rate cards, verified 2026-09-03. -# Key: CLI model name (before gateway mapping) _PRICING: dict[str, ModelPricing] = { # Fable 5.1 (and Mythos 5.1) price cache hits at 0.025x input, not the 0.1x # every other Claude model uses. Fable 5 pays $1 on the identical $10 base. @@ -45,8 +43,7 @@ class ModelPricing: "claude-opus-4-1": ModelPricing(15.0, 75.0, 18.75, 1.50), "claude-opus-4": ModelPricing(15.0, 75.0, 18.75, 1.50), "claude-opus-4-20250514": ModelPricing(15.0, 75.0, 18.75, 1.50), - # $2/$10, NOT the $3/$15 that Sonnet 4.6 and earlier pay. Do not copy the - # 4.x row onto it. + # $2/$10, NOT the $3/$15 that Sonnet 4.6 and earlier pay. Do not copy the 4.x row onto it. "claude-sonnet-5": ModelPricing(2.0, 10.0, 2.50, 0.20), "claude-sonnet-4-6": ModelPricing(3.0, 15.0, 3.75, 0.30), "claude-sonnet-4-5": ModelPricing(3.0, 15.0, 3.75, 0.30), @@ -56,19 +53,13 @@ class ModelPricing: "claude-haiku-4-5": ModelPricing(1.0, 5.0, 1.25, 0.10), "claude-haiku-4-5-20251001": ModelPricing(1.0, 5.0, 1.25, 0.10), "claude-haiku-3-5": ModelPricing(0.80, 4.0, 1.0, 0.08), - # Claude 3.7 Sonnet "claude-3-7-sonnet-20250219": ModelPricing(3.0, 15.0, 3.75, 0.30), - # Claude 3.5 Sonnet "claude-3-5-sonnet-20241022": ModelPricing(3.0, 15.0, 3.75, 0.30), "claude-3-5-sonnet-20240620": ModelPricing(3.0, 15.0, 3.75, 0.30), - # Claude 3 Opus "claude-3-opus-20240229": ModelPricing(15.0, 75.0, 18.75, 1.50), - # Claude 3 Sonnet "claude-3-sonnet-20240229": ModelPricing(3.0, 15.0, 3.75, 0.30), - # Claude 3 Haiku "claude-3-haiku-20240307": ModelPricing(0.25, 1.25, 0.30, 0.03), - # OpenAI GPT-5 / Codex (direct or via Azure OpenAI). OpenAI bills no separate - # cache-write fee, so cache_write == input on every entry below. + # OpenAI GPT-5 / Codex (direct or Azure). No cache-write fee: cache_write == input below. "gpt-5-codex": ModelPricing(1.25, 10.0, 1.25, 0.125), "gpt-5": ModelPricing(1.25, 10.0, 1.25, 0.125), "gpt-5.1-codex-max": ModelPricing(1.25, 10.0, 1.25, 0.125), @@ -107,8 +98,7 @@ class ModelPricing: "gemini-3.1-flash-lite": ModelPricing(0.25, 1.5, 0.25, 0.025), "gemini-3.1-flash-lite-preview": ModelPricing(0.25, 1.5, 0.25, 0.025), "gemini-3-flash-preview": ModelPricing(0.50, 3.0, 0.50, 0.05), - # Off the public card (superseded by 3.1 Pro); last published rate kept so - # historical runs still price. + # Off the public card; the last published rate keeps historical runs priced. "gemini-3-pro-preview": ModelPricing(2.0, 12.0, 2.0, 0.20), # HAZARD: eu-north-1 rates, a ~20% premium over us-east-1 -- do NOT "correct" # them against the US column. No published prompt-cache rate, so cache-creation @@ -127,8 +117,7 @@ class ModelPricing: } -# Plugin-contributed rates (e.g. coder_eval_uipath registers UiPath models). -# Merged over the built-in table at lookup time. +# Plugin-contributed rates, merged over the built-in table at lookup time. _REGISTERED_PRICING: dict[str, ModelPricing] = {} diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index df55654a3..730737afd 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -2,7 +2,7 @@ One gated number: ``essay_words`` — words in docstrings over 150 words (Typer command docstrings exempt, they render as ``--help``) plus words in comment runs of three or -more consecutive lines. It may not exceed ``_ESSAY_BASELINE_WORDS``, so the house style +more consecutive lines. Each file's comments are capped as a SHARE OF ITS LENGTH, so the house style is *no new essays*, not *no new documentation*. Also resolves every ``Rationale: § `` pointer, and — under @@ -19,6 +19,7 @@ import re import subprocess import sys +import textwrap import tokenize from collections import Counter from pathlib import Path @@ -27,7 +28,17 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -_ESSAY_BASELINE_WORDS = 18_703 +# Own-line comments a file may carry: a FLOOR for small files, then a share of its +# length. Proportional on purpose — there is no tree-wide total to hand-maintain, a +# file that loses code loses budget with it, and a NEW file is governed from its +# first commit. The tree's natural maximum sits just under this (a constants module +# at one comment per constant); the floor is what protects those. +_COMMENT_LINE_FLOOR = 20 +_COMMENT_LINE_RATIO = 0.15 + +# Docstring sections that are STRUCTURE, not prose: a parameter list is interface +# documentation and must not count against an essay budget aimed at narrative. +_DOCSTRING_SECTIONS = ("Args:", "Arguments:", "Returns:", "Yields:", "Raises:", "Attributes:") _SRC = Path("src/coder_eval") @@ -107,6 +118,44 @@ def _comment_runs(source: str) -> list[list[tokenize.TokenInfo]]: return runs +def prose_words(text: str) -> int: + """``docstring_words`` minus the contents of any ``Args:``/``Returns:``/``Raises:`` + block. + + The 150-word bar is about NARRATIVE. A function with eight documented parameters + is not writing an essay, and counting its parameter list pushed exactly the + docstrings that document their contract best over the line. + """ + lines = textwrap.dedent(text).split("\n") + kept: list[str] = [] + index = 0 + while index < len(lines): + if lines[index].strip() in _DOCSTRING_SECTIONS: + base = len(lines[index]) - len(lines[index].lstrip()) + index += 1 + while index < len(lines) and ( + not lines[index].strip() or (len(lines[index]) - len(lines[index].lstrip())) > base + ): + index += 1 + continue + kept.append(lines[index]) + index += 1 + return docstring_words("\n".join(kept)) + + +def _is_interface_contract(node: ast.AST) -> bool: + """True for an ``@abstractmethod``: its docstring IS the contract implementers read. + + The plugin SPI is the case. Exempting it by KIND rather than by name keeps the + rule principled — a new abstract method is covered, and a long docstring that is + not an interface contract still fails. + """ + return isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef) and any( + getattr(decorator, "id", getattr(decorator, "attr", "")) == "abstractmethod" + for decorator in node.decorator_list + ) + + def measure_source(source: str, rel: str) -> FileProse | None: """Measure one module's essay prose. ``None`` when it does not parse.""" try: @@ -126,7 +175,9 @@ def measure_source(source: str, rel: str) -> FileProse | None: name = "" if isinstance(node, ast.Module) else node.name if (rel, name) in _TYPER_COMMANDS and id(node) in top_level: continue - words = docstring_words(text) + if _is_interface_contract(node): + continue + words = prose_words(text) if words > _DOCSTRING_ESSAY_WORDS: essays.append((name, words)) essays.sort(key=lambda essay: (-essay[1], essay[0])) @@ -308,16 +359,54 @@ def check_pointer_placement(repo_root: Path) -> list[str]: return failures -def check(repo_root: Path) -> str | None: - """``None`` when the tree is at or under the baseline, else the failure message.""" - total = total_words(measure(repo_root).files) - if total <= _ESSAY_BASELINE_WORDS: - return None - return ( - f"prose budget exceeded: {total} essay words against a baseline of " - f"{_ESSAY_BASELINE_WORDS} (+{total - _ESSAY_BASELINE_WORDS}). " - "Move rationale to .claude/notes/, or lower the baseline if you removed prose." - ) +def comment_line_budget(total_lines: int) -> int: + """A file's own-line comment allowance.""" + return max(_COMMENT_LINE_FLOOR, round(_COMMENT_LINE_RATIO * total_lines)) + + +def check_comment_density(repo_root: Path) -> list[str]: + """Every file's own-line comments must fit :func:`comment_line_budget`. + + Own-line only. A trailing ``# noqa`` is a directive, not commentary, and a + per-member annotation on an enum is the contract a dispatcher reads — counting + either would push against documenting them. + """ + failures: list[str] = [] + for path in sorted((repo_root / _SRC).rglob("*.py")): + source = path.read_text(encoding="utf-8") + try: + tokens = list(tokenize.generate_tokens(io.StringIO(source).readline)) + except (SyntaxError, tokenize.TokenError, ValueError): + continue + lines = source.split("\n") + own = { + token.start[0] + for token in tokens + if token.type == tokenize.COMMENT and lines[token.start[0] - 1].strip().startswith("#") + } + budget = comment_line_budget(len(lines)) + if len(own) > budget: + rel = path.relative_to(repo_root / _SRC).as_posix() + failures.append( + f"{rel}: {len(own)} own-line comments against a budget of {budget} " + f"({len(lines)} lines). Move rationale to .claude/notes/." + ) + return failures + + +def check_essays(repo_root: Path) -> list[str]: + """No docstring may exceed the prose bar unless it is an interface contract. + + There is no numeric allowance. ``@abstractmethod`` and the Typer commands are + exempt by KIND; everything else that trips the bar is narrative with a home in + ``.claude/notes/``. + """ + return [ + f"{rel.as_posix()}::{name}: {words} prose words in a docstring " + f"(bar is {_DOCSTRING_ESSAY_WORDS}). Move the narrative to .claude/notes/." + for rel, prose in sorted(measure(repo_root).files.items()) + for name, words in prose.essays + ] def code_shape(source: str) -> str: @@ -410,8 +499,11 @@ def main(argv: list[str]) -> int: for failure in check_pointer_placement(repo_root): print(f"misplaced pointer: {failure}", file=sys.stderr) failed = True - if (message := check(repo_root)) is not None: - print(message, file=sys.stderr) + for failure in check_comment_density(repo_root): + print(f"comment budget: {failure}", file=sys.stderr) + failed = True + for failure in check_essays(repo_root): + print(f"docstring essay: {failure}", file=sys.stderr) failed = True return 1 if failed else 0 diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py index 15bc10cea..d46c5a46b 100644 --- a/tests/test_prose_budget.py +++ b/tests/test_prose_budget.py @@ -130,18 +130,70 @@ def test_total_words_sums_docstrings_and_comments(self, tmp_path: Path) -> None: assert prose_budget.total_words(prose_budget.measure(root).files) == 206 -class TestCheck: - def test_at_the_baseline_it_passes(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - root = _tree(tmp_path, {"a.py": f'def f():\n """{_words(200)}"""\n'}) - monkeypatch.setattr(prose_budget, "_ESSAY_BASELINE_WORDS", 200) - assert prose_budget.check(root) is None - - def test_one_word_above_the_baseline_fails(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - root = _tree(tmp_path, {"a.py": f'def f():\n """{_words(200)}"""\n'}) - monkeypatch.setattr(prose_budget, "_ESSAY_BASELINE_WORDS", 199) - message = prose_budget.check(root) - assert message is not None - assert "200" in message and "199" in message +class TestCommentBudget: + """The per-file comment budget that replaced the hand-maintained baseline.""" + + def test_the_floor_applies_to_a_small_file(self) -> None: + assert prose_budget.comment_line_budget(10) == 20 + assert prose_budget.comment_line_budget(100) == 20 + + def test_the_ratio_applies_once_it_beats_the_floor(self) -> None: + assert prose_budget.comment_line_budget(1000) == 150 + + def test_a_file_within_its_budget_passes(self, tmp_path: Path) -> None: + body = "\n".join(["x = 1"] * 200) + root = _tree(tmp_path, {"a.py": "# one\n# two\n" + body}) + assert prose_budget.check_comment_density(root) == [] + + def test_a_file_over_its_budget_fails(self, tmp_path: Path) -> None: + root = _tree(tmp_path, {"a.py": "\n".join(["# pad"] * 40) + "\nx = 1\n"}) + failures = prose_budget.check_comment_density(root) + assert len(failures) == 1 + assert "own-line comments" in failures[0] + + def test_a_trailing_comment_is_a_directive_not_commentary(self, tmp_path: Path) -> None: + """40 trailing `# noqa` must not consume the budget; 40 own-line ones would.""" + root = _tree(tmp_path, {"a.py": "\n".join(["x = 1 # noqa: E501"] * 40) + "\n"}) + assert prose_budget.check_comment_density(root) == [] + + def test_the_budget_shrinks_with_the_file(self) -> None: + """The point of a ratio: deleting code takes its comment budget with it.""" + assert prose_budget.comment_line_budget(2000) > prose_budget.comment_line_budget(1000) + + +class TestProseWords: + def test_an_args_block_does_not_count(self) -> None: + doc = f"Summary.\n\n{_words(140)}\n\nArgs:\n a: {_words(100)}\n" + assert prose_budget.prose_words(doc) < 150 + assert prose_budget.docstring_words(doc) > 150 + + def test_prose_after_a_section_still_counts(self) -> None: + """A trailing contract paragraph is prose, not structure.""" + doc = f"Summary.\n\nArgs:\n a: thing\n\n{_words(200)}\n" + assert prose_budget.prose_words(doc) > 150 + + def test_returns_and_raises_are_structure_too(self) -> None: + doc = f"Summary.\n\nReturns:\n {_words(90)}\n\nRaises:\n ValueError: {_words(90)}\n" + assert prose_budget.prose_words(doc) < 150 + + +class TestInterfaceContractExemption: + def test_an_abstractmethod_docstring_is_exempt(self) -> None: + source = ( + "from abc import ABC, abstractmethod\n\n" + "class A(ABC):\n" + " @abstractmethod\n" + f" def f(self):\n {Q}{_words(400)}{Q}\n" + ) + prose = prose_budget.measure_source(source, "m.py") + assert prose is not None + assert prose.essays == [] + + def test_a_plain_method_is_not_exempt(self) -> None: + source = f"class A:\n def f(self):\n {Q}{_words(400)}{Q}\n" + prose = prose_budget.measure_source(source, "m.py") + assert prose is not None + assert prose.essays == [("f", 400)] class TestPointers: From b72e663840044b193dff427a62016cb8bf45653e Mon Sep 17 00:00:00 2001 From: uipreliga Date: Mon, 14 Sep 2026 22:54:23 -0700 Subject: [PATCH 15/19] docs: update CLAUDE.md with communication style and development command clarifications Co-Authored-By: Claude --- CLAUDE.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1f014078f..c37d7d907 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,6 +2,9 @@ Working reference for AI assistants on the `coder_eval` codebase. +**Communication style:** use ASD-STE-100 when you speak to the user, and when you edit +this file. + Design *rationale* — why a subsystem is shaped the way it is, and which shipped defect shaped it — lives under **`.claude/notes/`**, which is not auto-loaded; start at [`.claude/notes/README.md`](.claude/notes/README.md). Read it before changing grading, @@ -117,7 +120,8 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes Defense-in-depth, not a boundary — the known gaps are documented in the notes. Authoring reference: [Reference Solutions](docs/TASK_DEFINITION_GUIDE.md#reference-solutions). - **Harness run-limit parity**: a shared config field must mean the same thing on every - backend, or the divergence is documented. Table: + backend, or the adapter rejects it at load time. A silently ignored field is a defect, + not a table row. Table: [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). Caps are authored under [Run Limits](docs/TASK_DEFINITION_GUIDE.md#run-limits). - **Execute vs. run**: `execute` is `run` with grading off — rows finalize as @@ -193,6 +197,8 @@ is `.claude/shared/run-layout.md`. ## Development Commands +If you run one command, run `make verify`. `make test` does not run the lint rules. + ```bash # MANDATORY: run after every implementation phase make format # ruff format @@ -231,7 +237,8 @@ When fixing a bug, ask: *could a custom lint rule have prevented this?* If the r cause is a mechanically detectable pattern, add a rule following the CE000+ pattern and wire it up. See `tests/test_custom_lint.py` for how rules are tested. Prefer removing the sharp edge over guarding it: a rule is right when the pattern is genuinely -unavoidable, not when a shared helper would do. Candidates not yet promoted to rules +unavoidable, not when a shared helper would do. A cap rule sets its limit below the +current value, never at it. Candidates not yet promoted to rules are collected in `.claude/harness-candidates.md`. A few rules constrain routine edits, so they are worth knowing before you start: @@ -355,7 +362,9 @@ bandit, pre-commit, mcp - **YAGNI** — don't add complexity until actually needed - **KISS** — keep it simple - **Clean code** — no dead code, all imports used, all tests passing -- **Greenfield project** — no backward-compatibility burden +- **Delete before you guard** — before you add a lint rule, doc paragraph, criterion + type or config field, try to delete the pattern that needs it. A new type that + subsumes an old one removes the old one in the same change (no back-compat burden) - **Comments are a last resort** — default to ZERO comments. Names, types and small functions carry the meaning. A comment is allowed ONLY when it records something the code cannot say @@ -365,11 +374,9 @@ bandit, pre-commit, mcp own-line comments may not exceed `MAX(20, 0.15 × its length)`, and no docstring may exceed 150 words of PROSE (an `Args:`/`Returns:`/`Raises:` block is structure, not prose; an `@abstractmethod` is exempt because its docstring IS the interface contract). - There is no tree-wide total to hand-maintain — delete code and the budget shrinks with it ## Notes for AI Assistants -- Communication style: use ASD-STE-100 when you speak to the user. - Temporary files go in `tmp/`, not `/tmp`. - Read `.claude/notes/` before changing grading, resume, early stop, timing, the reference anti-cheat, or any significant parts of this code's architecture. From 0713f6fa45872d21221e3742d0974dea6ddf2be1 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 15 Sep 2026 19:32:41 -0700 Subject: [PATCH 16/19] fix(lint): stop the prose budget penalising usage examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_TRAILING_SECTIONS` already accepted an `Example:` block after a pointer, but `_DOCSTRING_SECTIONS` counted one against the 150-word bar — the two lists disagreed about the same shape, so a docstring in the house style could fail the gate. Two call examples had been deleted to get under it, one of them carrying the `on_attempt_error` keyword contract for the retry helper. A call example is code, not narrative, so it joins the structure list. Both examples are restored, with `execute_with_retry`'s Args/Returns/Raises block. Also from review: - `_TYPER_COMMANDS` gains a staleness test. An allowlist entry whose function has moved exempts nothing while still reading as a deliberate exemption — the vacuous-guarantee shape CE057's membership test already guards against. - `Agent.communicate` trimmed from 315 to 273 words. Every obligation kept; the turn-lifecycle rationale becomes a pointer. Co-Authored-By: Claude Opus 5 (1M context) --- src/coder_eval/agent.py | 30 +++++++++++++----------------- src/coder_eval/errors/executor.py | 31 ++++++++++++++++++++++++++++--- src/coder_eval/logging_config.py | 13 +++++++++++++ tests/lint/prose_budget.py | 15 +++++++++++++-- tests/test_prose_budget.py | 27 +++++++++++++++++++++++++++ 5 files changed, 94 insertions(+), 22 deletions(-) diff --git a/src/coder_eval/agent.py b/src/coder_eval/agent.py index d4dddcaf8..8fb5f3cec 100644 --- a/src/coder_eval/agent.py +++ b/src/coder_eval/agent.py @@ -239,23 +239,19 @@ async def communicate( before raising if telemetry was captured. AgentCrashError: Agent failed mid-turn; same ``pending_turn`` contract. - On success, ``pending_turn`` must be None and the completed TurnRecord - is returned directly. On failure, ``pending_turn`` is set (if telemetry - was available) before raising — rollback of per-turn bookkeeping happens - exclusively in ``discard_pending_turn``, which the caller invokes after - every failed ``communicate()``. - - Streaming contract: the agent is the SOLE emitter of the standardized - event protocol (the orchestrator is a pure consumer). An implementation - MUST emit exactly one ``AgentStartEvent`` at the top of ``communicate()`` - and exactly one matching ``AgentEndEvent`` on every exit path (success, - crash, or timeout — emit it from ``finally``), with one ``TurnStartEvent`` - / ``TurnEndEvent`` pair per inner turn and ``ToolStartEvent`` / - ``ToolEndEvent`` for each tool call (every ``ToolStart`` closed by a - ``ToolEnd``, including ``status=unresolved`` for tools orphaned by a crash). - Events fan out through an internal ``EventCollector`` (which builds the - returned ``TurnRecord``) and the caller's ``stream_callback``; renderers - and the task-log handler consume the same stream. + On success ``pending_turn`` must be None. On failure it holds the partial + record, and only ``discard_pending_turn`` — which the caller invokes after + every failed call — rolls back per-turn bookkeeping. + + The agent is the SOLE emitter of the event protocol. Emit exactly one + ``AgentStartEvent`` at entry and one matching ``AgentEndEvent`` from + ``finally`` on every exit path, one ``TurnStartEvent`` / ``TurnEndEvent`` + pair per inner turn, and a ``ToolStartEvent`` closed by a ``ToolEndEvent`` + for every tool call (``status=unresolved`` when a crash orphans one). Fan + every event through an internal ``EventCollector``, which builds the + returned ``TurnRecord``, and through the caller's ``stream_callback``. + + Rationale: .claude/notes/agents.md § Shared turn lifecycle """ pass diff --git a/src/coder_eval/errors/executor.py b/src/coder_eval/errors/executor.py index 7cf7e5d83..80f3cce9e 100644 --- a/src/coder_eval/errors/executor.py +++ b/src/coder_eval/errors/executor.py @@ -23,11 +23,36 @@ async def execute_with_retry( """Execute an operation with automatic retry on transient errors. Retries only what ``errors/categorization.py`` classifies as retryable; - everything else raises on the first attempt. ``on_attempt_error`` runs after each - failed attempt, before the backoff, so a caller can reset per-attempt state (the - orchestrator uses it to preserve a crashed partial ``TurnRecord``). + everything else raises on the first attempt. Rationale: .claude/notes/agents.md § Shared turn lifecycle + + Args: + operation: Async callable taking no arguments -- pass state by closure. + operation_name: Human-readable name, for logging only. + context: Requires ``task_id``; ``component`` and ``agent_name`` are optional. + max_attempts: Overrides the safety limit of 10. + on_attempt_error: Async ``(exception, zero_indexed_attempt) -> None`` invoked + after every failed attempt, including the final non-retryable one, and + before the backoff. Its own exceptions are logged and swallowed so they + cannot mask the original. The orchestrator uses it to drain + ``agent.pending_turn`` and call ``agent.discard_pending_turn()``. + + Returns: + Whatever ``operation`` returned. + + Raises: + The last exception, once the retries are exhausted. + + Example: + >>> async def flaky_api_call(): + ... return await agent.communicate(prompt) + >>> + >>> result = await execute_with_retry( + ... operation=flaky_api_call, + ... operation_name="Agent communication", + ... context={"task_id": "task-001", "component": "agent"}, + ... ) """ task_id = context.get("task_id", "unknown") last_error = None diff --git a/src/coder_eval/logging_config.py b/src/coder_eval/logging_config.py index e344900a9..41d5fad00 100644 --- a/src/coder_eval/logging_config.py +++ b/src/coder_eval/logging_config.py @@ -264,6 +264,19 @@ def task_log_handler( Attaches a file handler for the task's own log, sets the task-id ContextVar so parallel tasks stay isolated, and yields the bounded tail buffer the HTML report reads. + + Args: + task_log_file: Path to the task log file. + level: Logging level for the file output. + task_id: Filters the file to this task's own records in a parallel batch. + + Yields: + ``_LogTailBuffer`` exposing ``get_text()`` for the sanitised log tail. + + Example: + >>> with task_log_handler(Path("task.log"), task_id="my_task") as log_tail: + ... logger.info("This goes to both console and task.log") + ... tail_text = log_tail.get_text() """ # Create handler handler = logging.FileHandler(task_log_file, mode="w", encoding="utf-8") diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 730737afd..518f5efe7 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -37,8 +37,19 @@ _COMMENT_LINE_RATIO = 0.15 # Docstring sections that are STRUCTURE, not prose: a parameter list is interface -# documentation and must not count against an essay budget aimed at narrative. -_DOCSTRING_SECTIONS = ("Args:", "Arguments:", "Returns:", "Yields:", "Raises:", "Attributes:") +# documentation and a call example is code, so neither counts against an essay budget +# aimed at narrative. Kept in step with _TRAILING_SECTIONS, which already accepts an +# Example: block after a pointer -- a shape the budget must not then penalise. +_DOCSTRING_SECTIONS = ( + "Args:", + "Arguments:", + "Returns:", + "Yields:", + "Raises:", + "Attributes:", + "Example:", + "Examples:", +) _SRC = Path("src/coder_eval") diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py index d46c5a46b..eee26ff6f 100644 --- a/tests/test_prose_budget.py +++ b/tests/test_prose_budget.py @@ -71,6 +71,20 @@ def test_same_name_nested_inside_the_exempt_module_is_counted(self) -> None: assert prose is not None assert prose.docstring_words == 400 + def test_every_exempt_pair_still_exists(self) -> None: + """An allowlist entry whose function has moved or been renamed exempts nothing + while still reading as a deliberate exemption -- the same vacuous-guarantee + failure CE057's membership test exists to catch.""" + import ast + + package = Path(__file__).resolve().parents[1] / "src" / "coder_eval" + for rel, name in sorted(prose_budget._TYPER_COMMANDS): + path = package / rel + assert path.is_file(), f"_TYPER_COMMANDS names {rel}, which does not exist" + tree = ast.parse(path.read_text(encoding="utf-8")) + top_level = {node.name for node in tree.body if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef)} + assert name in top_level, f"_TYPER_COMMANDS names {rel}::{name}, which is not defined there" + class TestCommentMeasurement: def test_three_consecutive_lines_are_a_block(self) -> None: @@ -176,6 +190,19 @@ def test_returns_and_raises_are_structure_too(self) -> None: doc = f"Summary.\n\nReturns:\n {_words(90)}\n\nRaises:\n ValueError: {_words(90)}\n" assert prose_budget.prose_words(doc) < 150 + def test_an_example_block_is_structure_not_prose(self) -> None: + """A call example is code. Counting it penalised exactly the docstrings that + show a caller how to use the thing -- and ``_TRAILING_SECTIONS`` already + accepts an ``Example:`` block after a pointer, so the two must agree.""" + doc = f"Summary.\n\nExample:\n >>> f({_words(200)})\n" + assert prose_budget.prose_words(doc) < 150 + assert prose_budget.docstring_words(doc) > 150 + + def test_the_two_section_lists_agree_on_example(self) -> None: + for section in ("Example:", "Examples:"): + assert section in prose_budget._DOCSTRING_SECTIONS + assert section in prose_budget._TRAILING_SECTIONS + class TestInterfaceContractExemption: def test_an_abstractmethod_docstring_is_exempt(self) -> None: From 97ced3b9c57917355235136a27df6d178d487829 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 15 Sep 2026 19:32:50 -0700 Subject: [PATCH 17/19] docs(notes): cut the duplicated catalogue and the unbuilt design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review cuts, and a readability fix for a fourth. - The CE lint-rule catalogue in README.md: 2,165 words restating each rule's own module docstring, disclaimed as non-authoritative by the two sentences directly above it, and targeted by no pointer. Deleted; the sentences naming where the authority lives stay. README.md drops 2,745 -> 589 words. - "Not wired up yet" in permissions.md: 769 words designing code that does not exist. Deleted — git holds it. - The wall bullets are reflowed to the notes' 88-column prose width. The largest was a single 19,453-character line, which no diff could show usefully. - reporting.md's pseudo-directory-tree, two entries carrying 2,174- and 506-character trailing comments, becomes `###` subsections of ordinary prose. The review also asked for the per-file abstracts to be deleted as restatements of the sections beneath them. Not done: 61 of the isolation abstract's 191 code identifiers appear nowhere else in notes/, docs/ or CLAUDE.md — among them `--allow-recorded-commands`, `task_config.resolved`, `_seed_from_prior_result` with its `early_stop` carry, and `FinalStatus.is_execution_fact`. Deleting them loses rationale rather than a duplicate. Re-homing those claims first is a follow-up. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/README.md | 14 +- .claude/notes/agents.md | 66 ++++++++- .claude/notes/contracts.md | 25 +++- .claude/notes/isolation.md | 238 ++++++++++++++++++++++++++++++++- .claude/notes/orchestration.md | 158 +++++++++++++++++++++- .claude/notes/permissions.md | 19 --- .claude/notes/reporting.md | 66 ++++++++- 7 files changed, 537 insertions(+), 49 deletions(-) diff --git a/.claude/notes/README.md b/.claude/notes/README.md index 55bffc8df..37b8a5aa0 100644 --- a/.claude/notes/README.md +++ b/.claude/notes/README.md @@ -57,12 +57,10 @@ exists to shrink. Do not "fix" this by promoting it. Nothing here states how many `CE` rules exist. `tests/lint/rules/` owns that count, and a number written down anywhere else is a second declaration that will be wrong. -## The CE lint-rule catalogue +## Where a CE rule's rationale lives -Each rule's authoritative rationale lives in its own module docstring under -`tests/lint/rules/` (or, for the doc-surface and whole-tree rules, in the -corresponding `@pytest.mark.lint` class in `tests/test_custom_lint.py`). The prose -summary below is kept for orientation only — when it disagrees with a rule file, -the rule file is correct. - -Recent additions, each traceable to a shipped defect: **CE064** (in `src/coder_eval/agents/`, a module that imports `TurnClock` must pass an explicit `timestamp=` to `AgentStartEvent` and `AgentEndEvent` — the turn's OUTER bounds, which no other rule looks at, since CE058-CE061 all scope to `AssistantMessage` and the bracket is not one. `timing.decompose_turn` produces `harness_startup_ms` / `harness_teardown_ms` by subtracting a generation-window bound from a bracket timestamp, so the two must share a basis; all three clocked harnesses derived their bounds from the `TurnClock` and let the bracket fall back to `StreamEvent.timestamp`'s `default_factory=datetime.now`, putting a monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact split `TurnClock` exists to remove, reintroduced at the one seam the clock did not own. Measured on a live antigravity turn: an `AgentEndEvent` stamped **17 us BEFORE its own last message finished**, which cannot happen (the event is constructed strictly after the final flush), and `decompose_turn` clamped that negative and published `0.0` — "measured, and instant", the CE058 confusion arrived at from the other direction — for a harness whose real tail is ~0.1 ms; it now records 0.035 ms. It surfaced on one harness only because the drift is tens of microseconds and antigravity is the only one that holds its process across turns, so nothing happens between its last flush and its end event; every other harness books a tail of 7-543 ms, where the drift is invisible rather than absent — which is why the fix is at every clocked site rather than at that one. SCOPE IS DERIVED, never a harness list: codex and opencode take their spans from the CLI's own epoch stamps, deliberately have no `TurnClock`, and are correctly invisible to the rule — a raw bracket is CONSISTENT with their bounds — and the day either adopts a clock the rule starts applying with no edit. BLIND SPOT, in the rule's docstring: presence, not correctness. It cannot tell `self.clock.now()` from a `datetime.now()` spelled out at the call site, because the three harnesses legitimately reach their clock three ways; the guard for the SOURCE is behavioural (`tests/_bracket_clock.py` injects a stand-in anchored a year from real time, so a reverted argument fails by a year rather than by the microseconds that separate the two clocks), which is the division of labour CE060 states — a rule removes the SILENT case, a default nobody chose), **CE063** (no module in `src/coder_eval/agents/` may import `busy_ms` — tool execution comes out of a generation window in exactly ONE place, `timing.py::subtract_tool_time`. Five reducers used to do it themselves while the head and tail were already computed centrally at the same seam, and that asymmetry is where every timing defect on this branch lived — none of them in the arithmetic, all of them in the bookkeeping AROUND it: when to reset a per-step span list (clearing it at `step_start` wiped a span before the flush could subtract it, a 100% overstatement of that window), when to clear a spent start stamp (a second flush with no intervening start republished the previous span — 3000 ms of generation for a 2000 ms turn), when to advance the mark. A sixth harness reaching for `busy_ms` rebuilds that, and its tool time is then subtracted TWICE — by the reducer and again by the collector — under-reporting generation on one harness only, which takes a corpus comparison to notice. A separate id from CE061 rather than a rebody: CE061 asks where a window's ARITHMETIC came from and four reducers still call `close_window`, so its property is live and unsuperseded; this asks whether a reducer subtracts at all. It deliberately does NOT reuse CE061's `_imports_the_helper`, whose bare-module-import branch exists so `timing.close_window(...)` counts as reaching the helper — inverted into a ban that branch flags four of the five reducers. CE061 is now **exemption-free**: claude-code was its one permanent `# noqa` and, with the subtraction moved, calls the shrunken `close_window` like the other four), **CE060** (in `src/coder_eval/agents/`, every `AssistantMessage(...)` must pass `message_id` explicitly — an identity invariant, which is why it is its own id rather than a second arm of CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field defaulted to `None` on every message it ever recorded, and the evalboard — which groups assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock gap when either side lacks one — collapsed a whole turn's generations into ONE timeline row as soon as the harness's generation windows became contiguous (the gap is then exactly 0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on the day they were written — a snapshot is regenerated from whatever the code currently does, so it catches a later change and never an initial omission. The damage was not confined to the timeline, which is why "only granularity is lost" was the wrong way to describe it: a grouped emission is one API call to the evalboard's thinking-cost simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the 10 s slow-generation bar were per-turn too. Unlike its two siblings it **derives its constructor set from each module's own `coder_eval.models` imports** instead of hardcoding the spelling, which closes exactly the blind spot CE058's clause below concedes: `claude_code_agent.py` binds only `AssistantMessage as AssistantMessageTelemetry`, so a name list guards that file's two construction sites purely by coincidence, and an arbitrary `as Msg` is missed outright. Widening CE058/CE059 the same way is recorded in `.claude/harness-candidates.md`. BLIND SPOT, in the rule's docstring: the runtime `None` — the kwarg must be PRESENT, not statically non-`None`, because OpenCode's `messageID` and Pi's `responseId` legitimately evaluate to `None` when the CLI omits them, and passing a fallback expression *is* deciding), **CE058** (in `src/coder_eval/`, an unknown timing value may not become a numeric literal — `duration_ms is None` means *never timed* and `0.0` means *timed and instant*, so writing the literal publishes the second while meaning the first. One invariant, one id, five syntactic forms — a zero constructor keyword, `x or 0`, `x if x is not None else 0.0`, `if x.duration_ms is None: x.duration_ms = 0.0`, and a `model_copy(update={...})` dict (the shape the Antigravity DONE path writes through, which a keyword-only rule cannot see). Antigravity constructed EVERY message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` and its breakdown rendered `0%` for months with nothing failing; Codex published the SDK's `0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds by a command count of which 70 of 211 in one nightly had never been timed. The fourth form is the one no existing rule shape covered and is where a live instance was hiding — `claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a tool result, in the one harness a timing audit had called healthy. BLIND SPOT, stated in the rule's docstring: form 1 keys on the callee's spelling, so renaming the `AssistantMessageTelemetry` import alias silently disarms it there), **CE059** (in `src/coder_eval/agents/`, an `AssistantMessage` may not receive the same `ast.Name` for both `started_at` and `completed_at` — the Antigravity reducer read `datetime.now()` once and passed it as both bounds, so `started_at == completed_at` on 368 of 368 sampled messages. A separate id from CE058 because it is a separate invariant, a zero-length window whatever the duration field says, and one invariant per id is what makes a `# noqa` mean one thing. It does NOT fire when the same call passes `generation_duration_ms=None`: a call that says, in the field built to say it, that no window was measurable is not claiming one — that exemption is what keeps the rule pointed at the misleading case instead of accumulating four permanent suppressions on the rollout-rebuild and sub-agent-synthesis sites), **CE056** (no bare `CODER_EVAL_IN_CONTAINER` literal outside `models/container_paths.py` — the CE053 shape again: a rename-safety constant that shipped beside the literal it replaced, and the straggler was the single WRITER, so a rename would have disarmed four security/correctness gates at once with nothing failing; CE052 cannot catch it because that rule inspects `if` guards and the writer is not one), **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record or run-LOG filename literal outside `path_utils` — widened to `docker.log` / `grade.docker.log` / `task.log` / `grade.log` after the same shape recurred: `docker.log` was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, and because the consumer guards its copy with `is_file()`, a rename would have silently discarded the only record of why a grading container failed — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed), **CE057** (a module copied into the recorder directory beside a generated sandbox shim — `models.sandbox.SIDECAR_MODULES`, currently `argv_match.py` — may import stdlib only. The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not installed, so one package import makes every shadowed CLI die with an ImportError the agent reads as "the tool is broken", costing a whole run to diagnose. The rule derives its target set from that exported tuple and a test asserts it matches a file that exists — a lint rule guarding zero files must fail, not pass). +Each rule's authoritative rationale is its own module docstring under +`tests/lint/rules/` — or, for the doc-surface and whole-tree rules, the corresponding +`@pytest.mark.lint` class in `tests/test_custom_lint.py`. Read that before editing, +suppressing or widening a rule. No prose summary is kept here: a second copy is a second +declaration, and it is the one that goes stale. diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index 382b1185c..7265cf9a4 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -4,9 +4,44 @@ ## Token accounting and the reconciliation message -- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout; both harnesses' turn totals already include sub-agent cost. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - -- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. The residual is almost always positive; a NEGATIVE one means the captured generations over-report some bucket, which is why the note's wording is branched — a `-512` entry must not read as "billed but not surfaced". +- **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every + sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` + in the turn transcript, so per-sub-agent usage is derived by grouping those messages + on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude + bubbles its sub-agent's intermediate generations into the parent stream natively, and + the **terminal** generation (delivered as the Agent tool result, never streamed) is + synthesized into one via `_synthesize_subagent_terminal_message` from + `tool_use_result.usage`. Codex reconstructs all child generations from the child + rollout; both harnesses' turn totals already include sub-agent cost. + `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so + sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every + raw SDK event to the task log for inspection. + +- **Reconciliation message (stream self-reconciles to the turn total)**: The per-message + stream consistently under-reports the authoritative turn total — a fixed prompt slice + (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent + input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends + one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the + `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` + − Σ(assistant message buckets). The invariant: **summing the four token buckets across + `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for + both Claude and Codex (Codex's stream is already complete after + `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). + This is what lets the evalboard SUM the message stream as the source of truth instead + of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the + stream sum whenever a reconciliation entry is present, and the timeline renders it as + its own row. It is agent-agnostic (booked at the single `EventCollector` seam), + carries no cost (cost stays on `token_usage`), and is excluded from generation/turn + counts and the cost simulator. The LiteLLM open-weight actual-cost join + (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only + (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call + `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token + buckets, so `EventCollector` stays the single writer and this invariant holds on every + backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still + authoritative for budget/judges/reports. The residual is almost always positive; a + NEGATIVE one means the captured generations over-report some bucket, which is why the + note's wording is branched — a `-512` entry must not read as "billed but not + surfaced". ### The result_tokens measure and CE043 @@ -24,9 +59,28 @@ intentionally brief and out of scope; trimming for DISPLAY belongs in the render ## Harness run-limit parity -- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode and Pi each keep a native unit too, because their CLIs stream a real multi-step loop per `communicate()` (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. - - The **known unfixed divergences** — which config fields each harness does and does not enforce, and the per-harness `agent.plugins[].path` depth (claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, which is the costly direction: no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, reading exactly like a skill that never triggers; held to the plugin-root shape for `SKILL_SOURCE_PATH` by CE045) — are the table's to state, not this file's. Full table + rationale: docs/agents/HARNESS_PARITY.md. +- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same + thing on every backend, so a divergence is either fixed or documented — never silent. + **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool + calls, read live off the shared `EventCollector.visible_turn_count`, the same list + `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, + so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit + (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT + the same budget across harnesses. OpenCode and Pi each keep a native unit too, because + their CLIs stream a real multi-step loop per `communicate()` + (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same + loop boundary as the cooperative early stop and finalizes cleanly as + `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in + `_drain()`, so the background-work poll loop honors it too. + + The **known unfixed divergences** — which config fields each harness does and does not + enforce, and the per-harness `agent.plugins[].path` depth (claude-code REQUIRES a + plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, + which is the costly direction: no error, every positive row of an activation suite + scores 0, and the suite reports recall 0.0, reading exactly like a skill that never + triggers; held to the plugin-root shape for `SKILL_SOURCE_PATH` by CE045) — are the + table's to state, not this file's. Full table + rationale: + docs/agents/HARNESS_PARITY.md. ## Shared turn lifecycle diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index fd268399c..980f3edba 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -4,9 +4,28 @@ ## Datasets and aggregation -- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.) - -- **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure. +- **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a + single task into N row-tasks with `${row.}` substitution in `initial_prompt` + and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` + **before** variant resolution, so variants cannot override the dataset. Row sampling: + CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides + `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random + N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for + classification suites like activation). Stratified sampling (whether the N-per-stratum + count comes from the **CLI** `--sample-per-stratum` flag or **YAML** + `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each + run (so the nightly activation suite broadens coverage over time). Set + `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. + (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each + run.) + +- **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes + `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits + `count / mean / median / std / min / max` so every criterion is suite-thresholdable + for free. Classification-style criteria return `ClassificationCriterionResult` + (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared + `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` + gates the suite on those metrics; CLI exits non-zero on any gate failure. ### Criterion aggregation diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index 8b7d9dfa6..c9e2957d5 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -4,7 +4,243 @@ ## Detached grading and `Sandbox.adopt` -- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because that is the only place its criteria mean what they meant during the run: `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the host — the host answering a question nobody asked. The grading container gets TWO mounts and their separation is the design: the grading pass's own fresh `run_dir` at `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, preserving `task.execute.json` exactly as on the host path) and the executed workspace at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written over. The container half reuses the same `regrade_in_place` (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` flag plus a staged `prior.json`) rather than restating it. A container-graded row carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which is the parity that makes the split honest. `--allow-host-grading` survives as the ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still stamps. **The dispatch is itself inside the trust gate**: the record names the image, and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the gate already refuses, and it shipped reachable with no flags because `embedded_commands` walked only `success_criteria` and `post_run`. That is the same blind spot the function's own docstring already described for `--copy` provisioning ("a shared run directory whose criteria were all `file_exists` sailed through"), one layer up, so `include_container_dispatch` scans it on the in-place path exactly as `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry reported one `docker build` as four shell commands), and naming every HOST PATH it exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to consent to a strict subset of what happens. Which families the gate discloses follows from ONE parameter (see § Detached grading from the CLI). Three further properties are load-bearing and were not free: the grading container gets a **scratch** run dir, never the caller's — `run --resume` passes the executed row's OWN directory, where `_parse_result_or_raise` (which keys on `task.json` existing and discards `returncode`) read a dead container's stale pre-grade record back as a successful grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion silently resolving against the wrong tree; and the dispatch is guarded against an image that ignores the `regrade` key (see § The two honored-request guards). The grading container is a SECOND, fresh container: only the workspace crosses and `pre_run` is not re-run, so a criterion depending on out-of-workspace state (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` scores 1.000 — warned at dispatch AND stamped onto the row as `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why ("a console warning does not travel with `task.json` into `run.json`, the reports or the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs `docker build` under the deterministic tag `coder-eval-task-:built`, so the grading image REPLACES the run's, and nothing pins image identity on either side — a `reference_digest`-style pin is the real fix and needs the RUN path to record it first, so for now the row says it happened rather than the guide claiming a control that does not exist. The grading container's own logs are folded out of the scratch dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on the resume path that name is the executed run's) and `grade.log`, which is a documented run-layout artifact holding the per-criterion detail. Folding out only on success deleted exactly the evidence, while DockerRunError's own text said `See {log_path}` — a path already gone by the time it printed. Both copies refuse a symlinked destination, because `shutil.copy2` follows one and the sibling verdict write goes through `write_text_atomic` for precisely that reason; and the verdict write raises `RegradeError`, never a bare `OSError`, since it sits outside the dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into Typer AFTER a successful grade while `run --resume` caught it and reported a correct verdict as a grading failure. A container grade also emits its own `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: the grading path had inherited only the silent half of the container-silent invariant (§ Environment forwarding). The dispatch is gated on `IN_CONTAINER_ENV`, never on the driver — the in-container entry point rewrites `docker` -> `tempdir` before building its Orchestrator, so a driver-based test would read an already-changed value and a grading container would dispatch a grading container. That env var now has ONE definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it that way — the migration converted all four READERS and left the single WRITER (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one site that produces the value the gates consume: a rename would have updated every consumer and left the container exporting the old name, disarming the reference anti-cheat window, the reference mount, the grading-container recursion guard and the watchdog together, all silently. CE052 accepts both spellings — a rule that saw only the literal would read a constant-based gate as no gate and tell the author to paste the literal back, arguing against the SSOT it exists to reinforce. The earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate ` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `Sandbox.resolve_files` is containment-checked for the same reason (see § Criterion paths are contained, quietly). A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate `) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (see .claude/notes/orchestration.md § The in-container driver rewrite), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate ` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run — and `recorded_task_file` is its path twin, which must travel with it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its `source_file`, reintroducing the defect one caller down; both seams are now pinned by a test that drives the in-container regrade branch end to end, because deleting either left the whole suite green. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy. +- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval + evaluate` takes two shapes, told apart by a **pure** resolver + (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run + directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, + **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` + / dataset expansion are already baked in and re-loading the source would silently + grade a *different* task (fallback to `source_file` only when `resolved` no longer + validates, and loudly). It seeds the fresh result from the prior one via + `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the + trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — + recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` + — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a + truncated trajectory under the full-run strict-AND gate and can flip the verdict. + Carrying it is only half the fix — **both** grading paths select the gate through the + single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade + actually takes originally called `all_criteria_passed` inline, so the seeded field was + written and never read. `tests/test_seed_from_prior_result.py` partitions every + `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and + asserts the two `_select_gate()` call sites. A prior status that + `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is + **preserved**, never overwritten: grading may only move `NOT_GRADED` to + SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. + **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. + `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads + it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued + the opposite, and the disagreement pinned a re-graded max-turns row at + MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination + `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` + puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to + the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — + so it is not knowable without grading. Consuming it first made it terminal AND + permanent (the `is_execution_fact` arm then pinned it), so identical agent output + scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. + The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` + carries, so the detached grade walks the identical chain. The CLI must also branch on + WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All + criteria passed" (a CI wrapper reading the exit code went green on a row run.json + counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as + though grading had crashed, claimed the row was "left ungraded" (false — the restored + record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host + `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting + the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map + by the HTML report and typed as one by the evalboard, so a nested capture prints as a + Python dict repr). The route recorder follows the same rule: on a detached grade it + writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's + `api_routing` alone — writing in place contradicted the "prior wins" contract and left + a self-contradictory record (a direct route named beside the run's stale + `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now + persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and + restored in the evaluate-only branch (closing the PATH gap that method's docstring + already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set + (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would + clobber already-correct costs). The verdict is written back into the run's + `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place + write is what makes plain `coder-eval aggregate ` rebuild a graded `run.json` + with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place + primitive: it reuses `setup`'s adoption half but skips every *materializing* step + (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive + `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv + *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is + never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it + to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, + and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which + would overwrite the agent's deliverables before the criteria read them) and to KEEP + `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives + cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the + prior run instead. **`post_run` is the opposite case and moved phases**: it is defined + as running after the verdict and may mutate the workspace the criteria read (`rm -rf + node_modules` is the archetype), so running it under `execute` inverted its own + contract and broke round-trip equivalence — the criteria had not read the tree yet, so + `execute` + `evaluate` graded a workspace `post_run` had already modified and could + return a different verdict than a single `run` for the identical trajectory (the + in-tree tasks all escaped it only because their `post_run` touches nothing a criterion + reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — + `_skip_post_run` skips on `grade=False`, and skips again when the prior row already + recorded results, since nothing declares these commands idempotent. That makes it a + capability of the in-place path, so `embedded_commands` scans it OUTSIDE + `include_setup_phase` (which is False in place) — minus + `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` + contribution, which every task carries and the record therefore did not choose; + without that exemption the refusal fired on 100% of run directories, and a refusal + that always fires is waved through. In-place is **more correct**, not merely faster: + `_setup_template` filters the copy through `_should_ignore_template_file`, which drops + `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion + like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict + (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a + run dir, copy for a bare work dir (criteria can mutate it and it is the user's own + tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a + container workspace is unreachable from the host), and grading a `driver: docker` task + is DISPATCHED INTO A CONTAINER of the task's own image (`_should_grade_in_container` + -> `_grade_in_container` -> `DockerRunner(prior_result=, grade_workspace=)`), because + that is the only place its criteria mean what they meant during the run: + `tasks/byod_smoke_test.yaml` asserts `test -f /opt/byod_marker`, baked into its image, + and the IDENTICAL row scores SUCCESS 1.000 in a container and FAILURE 0.000 on the + host — the host answering a question nobody asked. The grading container gets TWO + mounts and their separation is the design: the grading pass's own fresh `run_dir` at + `CONTAINER_OUTPUT_DIR` (whose `task.json` the host then folds back into the row, + preserving `task.execute.json` exactly as on the host path) and the executed workspace + at `CONTAINER_GRADE_WORKSPACE`, read-WRITE and NOT a copy, adopted rather than written + over. The container half reuses the same `regrade_in_place` + (`run_task_internal_command._grade_recorded_run`, driven by `context.json`'s `regrade` + flag plus a staged `prior.json`) rather than restating it. A container-graded row + carries NO `graded_on_host` stamp, so it is indistinguishable from a `run` row, which + is the parity that makes the split honest. `--allow-host-grading` survives as the + ESCAPE HATCH (no docker on this machine; criteria known to be host-portable) and still + stamps. **The dispatch is itself inside the trust gate**: the record names the image, + and a container of it runs with the default credential allowlist (`ANTHROPIC_API_KEY`, + `UIPATH_ACCESS_TOKEN`, `AWS_BEARER_TOKEN_BEDROCK` ...) forwarded in and a copy of + `~/.claude` mounted — a strictly WIDER capability than the `run_command` strings the + gate already refuses, and it shipped reachable with no flags because + `embedded_commands` walked only `success_criteria` and `post_run`. That is the same + blind spot the function's own docstring already described for `--copy` provisioning + ("a shared run directory whose criteria were all `file_exists` sailed through"), one + layer up, so `include_container_dispatch` scans it on the in-place path exactly as + `post_run` is — rendering the whole dispatch as ONE command string (the prompt joins + with `"; "` and counts `len(commands)`, so an argv fragment appended as its own entry + reported one `docker build` as four shell commands), and naming every HOST PATH it + exposes: the task DIRECTORY copied from the recorded `source_file`'s parent (a record + naming `~/.ssh/config` copies all of `~/.ssh` in), every auto-mounted + `agent.plugins[].path` / `TemplateDirSource.path` / `system_prompt_file`, and the + writable `~/.claude` copy. Disclosing only `sandbox.docker.*` asked the operator to + consent to a strict subset of what happens. Which families the gate discloses follows + from ONE parameter (see § Detached grading from the CLI). Three further properties are + load-bearing and were not free: the grading container gets a **scratch** run dir, + never the caller's — `run --resume` passes the executed row's OWN directory, where + `_parse_result_or_raise` (which keys on `task.json` existing and discards + `returncode`) read a dead container's stale pre-grade record back as a successful + grade, and where `docker.log` was truncated; the recorded `source_file` is the HOST's + path (`Orchestrator.recorded_task_file`, the path twin of `recorded_task`), because a + container run recorded `/work/task_dir/task.yaml`, which exists on no host, so the + dispatch guard's `task_file is None` test passed and `_prepare_task_dir_mount`'s `if + not source.is_dir(): return` then mounted NOTHING — every `$TASK_DIR` criterion + silently resolving against the wrong tree; and the dispatch is guarded against an + image that ignores the `regrade` key (see § The two honored-request guards). The + grading container is a SECOND, fresh container: only the workspace crosses and + `pre_run` is not re-run, so a criterion depending on out-of-workspace state + (`tasks/samples/skillsbench/3d-scan-calc` symlinks `/root/mass_report.json` in + `pre_run` and its verifier asserts that path) scores 0.000 for a trajectory `run` + scores 1.000 — warned at dispatch AND stamped onto the row as + `environment_info.graded_without_pre_run`, since re-running `pre_run` would trade it + for the deliverable-clobbering bug `_skip_pre_run_for_adopted` exists to prevent. The + stamp is the load-bearing half: `stamp_host_grading`'s own docstring already says why + ("a console warning does not travel with `task.json` into `run.json`, the reports or + the evalboard"), and 3 of the 10 in-tree docker tasks match the pattern, reachable + with NO flags via `execute` -> `run --resume`. `dockerfile_path` is the second, weaker + gap and is stamped the same way (`graded_with_rebuilt_image`): `_build_image` re-runs + `docker build` under the deterministic tag `coder-eval-task-:built`, so the + grading image REPLACES the run's, and nothing pins image identity on either side — a + `reference_digest`-style pin is the real fix and needs the RUN path to record it + first, so for now the row says it happened rather than the guide claiming a control + that does not exist. The grading container's own logs are folded out of the scratch + dir in a `finally`, not only on success: `docker.log` (as `grade.docker.log`, since on + the resume path that name is the executed run's) and `grade.log`, which is a + documented run-layout artifact holding the per-criterion detail. Folding out only on + success deleted exactly the evidence, while DockerRunError's own text said `See + {log_path}` — a path already gone by the time it printed. Both copies refuse a + symlinked destination, because `shutil.copy2` follows one and the sibling verdict + write goes through `write_text_atomic` for precisely that reason; and the verdict + write raises `RegradeError`, never a bare `OSError`, since it sits outside the + dispatch `try` where `evaluate` (which guards only `RegradeError`) let it escape into + Typer AFTER a successful grade while `run --resume` caught it and reported a correct + verdict as a grading failure. A container grade also emits its own + `CoderEval.Task.End` host-side (`_emit_task_telemetry`), mirroring `batch.py`: the + grading path had inherited only the silent half of the container-silent invariant (§ + Environment forwarding). The dispatch is gated on `IN_CONTAINER_ENV`, never on the + driver — the in-container entry point rewrites `docker` -> `tempdir` before building + its Orchestrator, so a driver-based test would read an already-changed value and a + grading container would dispatch a grading container. That env var now has ONE + definition (`models/container_paths.py::IN_CONTAINER_ENV`), and **CE056** keeps it + that way — the migration converted all four READERS and left the single WRITER + (`docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`) on the literal, which is the one + site that produces the value the gates consume: a rename would have updated every + consumer and left the container exporting the old name, disarming the reference + anti-cheat window, the reference mount, the grading-container recursion guard and the + watchdog together, all silently. CE052 accepts both spellings — a rule that saw only + the literal would read a constant-based gate as no gate and tell the author to paste + the literal back, arguing against the SSOT it exists to reinforce. The earlier + behavior silently rewrote the driver to `tempdir`, which ran a container task's + criteria against a host filesystem lacking `/verifier` and the image's toolchain + (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the + grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is + stamped `graded_on_host` so it is never silently comparable with a container-graded + one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the + digest is persisted into `environment_info` at staging time by `_stage_reference` (it + shipped once as a read with no writer anywhere, so the guard was dead code; then it + shipped with a writer whose value was **discarded before it reached disk**, because + `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred + lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. + `_setup` now `update()`s that dict rather than rebinding it, and + `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end + run, not just that `_staged_digest` works in isolation), and + `verify_reference_unchanged` now takes the task file it resolves against and RAISES on + a vanished or unresolvable reference instead of returning silently. That comparison + digests a STAGED copy of the source, not the raw tree: the recorded digest is taken + over the staged copy, which `stage_reference_dir` filters through + `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two + differently-filtered trees and reported a permanent false mismatch for any reference + that is a git checkout — exactly the case the ignore list exists for. **The recorded + config is untrusted input**: `evaluate ` rebuilds the task from a shareable + artifact, so a rebuilt config that carries shell (`run_command` criteria, + `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the + in-place path skips them — `pre_run`/`post_run` **and the sandbox's own + provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The + provisioning half was the one the gate originally missed, and the worst: + `grading_sandbox_config` carries the recorded `sandbox` block through untouched and + the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install ` / `npm install` / `git clone ` — arbitrary code at install + time — so a shared run dir whose criteria were all `file_exists` sailed through a scan + that walked only `success_criteria`. `Sandbox.resolve_files` is containment-checked + for the same reason (see § Criterion paths are contained, quietly). A warning is not a + control: it prints as the command is already being prepared. Passing the task file + explicitly (`evaluate `) also bypasses it, since that config came + from the operator. The workspace fallback `artifacts / prior.task_id` is + containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated + string, and `"../../.."` joins to a real directory `is_dir()` confirms), + `_sanitize_restored_path` drops relative entries (they resolve against the grader's + cwd) and anything inside the run dir rather than only the workspace, and + `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted + `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard + entirely. The record must also describe the task as AUTHORED, not as executed: + `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the + in-container Orchestrator (see .claude/notes/orchestration.md § The in-container + driver rewrite), and recording that rewrite made a docker run's own `task.json` claim + `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the + record, `evaluate ` on a container row skipped BOTH the + `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container + task against the host filesystem silently — the exact outcome that gate exists to + prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct + from what is run — and `recorded_task_file` is its path twin, which must travel with + it through EVERY caller. `regrade_in_place` and `_grade_recorded_run` shipped without + it, so every container-graded row re-recorded `/work/task_dir/task.yaml` as its + `source_file`, reintroducing the defect one caller down; both seams are now pinned by + a test that drives the in-container regrade branch end to end, because deleting either + left the whole suite green. NOTE the Typer command is a thin wrapper over + `run_evaluation(...)`, which has real Python defaults — calling a Typer command + function in-process hands unspecified options an `OptionInfo` sentinel, which silently + made `in_place=None` truthy. ### What a graded row inherits, and what it does not diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index 200b534ed..c05196984 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -4,13 +4,58 @@ ## Config merging and CLI overrides -- **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly. - -- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only. +- **Single declarative merge resolver**: All five config layers merge through ONE engine + (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots + (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the + model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: + nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). + `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` + lists and call the same `resolve_root`, so a field merges identically regardless of + which layer supplied it (the unification invariant, enforced by + `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare + its strategy explicitly. + +- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper + (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D + agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved + `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with + did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) + survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` + targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, + lighter alias that does NOT route through that collision check — `--type` and `-D + agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, + and SDK options are `-D`-only. ## Execute vs. run: the grading switch -- **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` with grading removed — the agent runs and the full trajectory is captured, but no criterion is checked, `weighted_score` is `None` (never `0.0`, which would be indistinguishable from "graded and scored zero"), and the row finalizes as **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are facts about the *run*, not about grading, and still apply (so `execute` still exits non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog check, and post-failure diagnostics); it crosses the docker boundary in `context.json` (defaulting to `True` in-container, so a host predating `execute` keeps grading). It is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because a task YAML must never declare itself ungraded; only the invoking command decides. `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code path. Three things are refused rather than degraded: `--junit-xml` (a report of verdicts, and there are none — though `reports_junit` still emits `` for an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades nothing), and simulation tasks (their turn-continuation logic reads criteria results, so an ungraded dialog would silently change its own stopping behavior). `stop_early:` blocks are inert under `execute` for the same reason the kill switch exists: the full trajectory is the deliverable. Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and grades with its own tests. +- **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` + with grading removed — the agent runs and the full trajectory is captured, but no + criterion is checked, `weighted_score` is `None` (never `0.0`, which would be + indistinguishable from "graded and scored zero"), and the row finalizes as + **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. + Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` + and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - + tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` + invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** + — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are + facts about the *run*, not about grading, and still apply (so `execute` still exits + non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` + → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog + check, and post-failure diagnostics); it crosses the docker boundary in `context.json` + (defaulting to `True` in-container, so a host predating `execute` keeps grading). It + is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because + a task YAML must never declare itself ungraded; only the invoking command decides. + `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in + that flag, so there is no third code path. Three things are refused rather than + degraded: `--junit-xml` (a report of verdicts, and there are none — though + `reports_junit` still emits `` for an ungraded row it encounters), + `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades + nothing), and simulation tasks (their turn-continuation logic reads criteria results, + so an ungraded dialog would silently change its own stopping behavior). `stop_early:` + blocks are inert under `execute` for the same reason the kill switch exists: the full + trajectory is the deliverable. Motivating consumer: an external harness (Harbor / + Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and + grades with its own tests. ### The terminal-status chain @@ -137,7 +182,54 @@ differently from what `run` would have produced. ## `--resume` is command-relative -- **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate ` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`. +- **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a + four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / + `prior_resolved`), because **"finished" is not absolute — it depends on what the + resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so + the original "has any final status" test called it complete: right for `execute + --resume` (it finished executing), and wrong for `run --resume`, which was asked to + grade and would instead report "already complete", grade nothing, and **exit 0**. The + routing test is the row's **evidence** (`weighted_score is None and not + success_criteria_results`), not its category: keying on `category == "ungraded"` + missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget + stop aborts before grading, so it lands unscored with category `error`/`failed`, and + resume filed it as complete while `evaluate ` graded the identical bytes + happily. Under `grade=True` those rows route to `to_grade`, where + `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already + on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, + which is the entire reason `execute` and `run` are separate. The carve-out is **only** + for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has + never retried failures — delete the task.json), and `clear_rerun_artifacts` + deliberately skips `to_grade`, whose artifacts are the very thing being graded. A + per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` + (the console line alone is not durable), and folded back in with its ORIGINAL ungraded + result, so one bad row neither aborts the resume nor vanishes from run.json — and the + exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded + nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an + ungraded row is the expected outcome and never fails the command. A row is owed a + grade only when it was **executed** AND is unscored: evidence of "no verdict" alone + routed every dead container and failed image build (`_write_synthetic_task_json` + writes those with no verdict either) into grading, where the fold-back replaced the + real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` + disagreeing about the same row — so the test is `final_status is NOT_GRADED or + iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of + replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: + `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated + the agent trajectory log the run had already paid for — contradicting + `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in + `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not + config drift — and the warning's "already-finalized tasks keep their original-config + results" text is actively wrong for it. **`orchestration/regrade.py` is the single + implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to + `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every + replicate but the first was relabelled — which is exactly how two copies become two + verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since + `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: + a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the + grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into + `average_duration`, the report tables and the evalboard; the grading cost is preserved + separately as `environment_info["grading_duration_seconds"]`. ### When a resumed grade crashes @@ -165,7 +257,61 @@ silent. A missing stamp (a run predating the feature) is tolerated. ## Early stop on criterion -- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged. +- **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block + (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's + **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the + smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — + there is **no run-level master switch**: `run_limits.stop_early: false` is the + run-level KILL SWITCH that force-disarms every block (the one-line + experiment-variant/`-D` override for an authoritative full run), and + `run_limits.stop_early: true` (the removed master arm) is a hard + `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only + (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion + is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit + trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: + stop` (pass-stop the moment the criterion live-passes; default `continue` just + latches) and `decide_within: N` (still undecided after N tool-call steps latches an + **effective fail**, fed through the same fail-stop rule, reported as + `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing + force-fail; cumulative across retry attempts of the same turn). A trigger whose + polarity the instance can't decide (per the abstract, checker-independent + `live_decidable_polarities()`, a pure function of the criterion's own fields, paired + with the checker's `live_verdict` override by lint rule CE025, a registry-based + whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both + positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: + once a criterion decides, its `live_verdict` is never polled again. Stop rule is + weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, + reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ + weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires + once the armed set's **ceiling** (best case for everything still undecided) can no + longer reach the threshold — so a low-weight fail or timeout that can't doom the gate + is absorbed and the run continues — and is **deferred while any pass-capable armed + criterion is undecided** (a distractor misfire never truncates a positive row's recall + signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) + already meets the threshold, and is symmetrically **deferred while any pass-capable + armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass + never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). + A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor + misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. + Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when + `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the + agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live + verdicts only *trigger* the stop — the standard `check_all_async` on the frozen + trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut + gates on the **armed subset** via the weighted + `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or + not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the + verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED + (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores + `should_stop`, or a stop firing on the final message, still gates armed-only. Every + resolution-time guardrail violation is a hard error at resolution (plan *and* run); + the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic + schema error at task load, which the run surface reports as a skipped task like any + other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: + `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, + `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked + rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior + byte-for-byte unchanged. ### Gate selection is fired-only diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md index 0f7fdd73c..680509b60 100644 --- a/.claude/notes/permissions.md +++ b/.claude/notes/permissions.md @@ -89,25 +89,6 @@ static reference mid-turn cannot break the `LiveVerdict` monotonicity contract (contracts.md § The live_verdict contract); reading the half-written sandbox can, and is the "end-state peeking" `live_verdict` rules out. -### Not wired up yet - -The remaining work, for whoever picks it up: - -- `EarlyStopWatcher._evaluate_impl` wraps its verdict loop in a `READ_ONLY_MODE` window - over the reference. One window per round, around the loop rather than per criterion — - that is the tightest placement, which matters because a chmod is global filesystem state - and the agent is running CONCURRENTLY: the re-grant is visible to it too, for as long as - it is open. -- That loop is a `StreamCallback` (plain `def`), so it needs a synchronous twin of - `set_permissions` pushing onto the same registry — the stack is already thread-safe, so - the twin is small. -- `live_verdict` gains NO parameter. It reads the reference from a per-task accessor - instead. That accessor must be a `ContextVar`, NOT `os.environ`: `run_batch -j 8` runs - many orchestrators in one process, so a process-global would leak one task's reference - into a sibling's verdict, silently and only under parallelism. (`REFERENCE_DIR` today is - set only in the `env=` dict handed to `run_command` subprocesses, so it is not readable - in-process.) - ## Locking and crash safety The registry is keyed by the *resolved* path so a directory reached by two different diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md index 791395a13..00b9c304e 100644 --- a/.claude/notes/reporting.md +++ b/.claude/notes/reporting.md @@ -4,15 +4,69 @@ ## Published rates and run-time caps -- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)` — see [orchestration.md](orchestration.md) § Rates need verdict evidence, not bucket counts for why, and why `measured` is counted evidence rather than a bucket count. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender. - -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see [orchestration.md](orchestration.md) § Early stop on criterion) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. +- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE + models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through + the single `models/results.py::nothing_was_measured(not_graded=, measured=)` — see + [orchestration.md](orchestration.md) § Rates need verdict evidence, not bucket counts + for why, and why `measured` is counted evidence rather than a bucket count. The + evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured + task renders "—" and sorts LAST in the worst-first Trends view rather than to the very + top as the worst offender. + +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` + (`RunLimits` model) is the single namespace for all *task-level* run-time caps — + `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / + `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD + breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` + (both `category == "failed"`). Structural caps are set from the CLI via `-D + run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D + run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D + run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block + overrides individual keys without replacing the task's block. The one *per-criterion* + cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead + (see [orchestration.md](orchestration.md) § Early stop on criterion) — the watcher + must attribute a decision-step timeout to a specific criterion, which `RunLimits` + (task-scoped, criterion-agnostic) cannot express. ## Plugin and GitHub Action layout -plugins/coder-eval/ # The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a derived pin of pyproject's, bumped by release.yml, guarded by tests/test_action_version_pin.py), `skills//SKILL.md` × 6 (`/coder-eval:init`, `/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, `/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads (`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct from `run-layout.md`, which describes what is inside a run directory. Every skill must appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the skill listing's budget is shared with every skill the user has installed. **Skill naming is verb-first imperative** — a skill is a command you issue (`/coder-eval:`) and every one of them takes an action, so name it for the action: a bare verb where that is unambiguous (`init`, `analyze` — the object comes from the argument), otherwise `-` (`lint-tasks`, `check-skill`). Never `-`: `skill-check` was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. `task` and `ci` predate the rule and stay — renaming a published skill breaks every user's muscle memory for no functional gain, since activation keys on the `description`, never the name. Distinct from `.claude/commands/`, which stays repo-local contributor tooling. - -action.yml # Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the `v` tag + cuts the Release, so nothing consumer-visible moves before the wheel is on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing. +### plugins/coder-eval/ + +The published Claude Code plugin: `.claude-plugin/plugin.json` (its `version` is a +derived pin of pyproject's, bumped by release.yml, guarded by +tests/test_action_version_pin.py), `skills//SKILL.md` × 6 (`/coder-eval:init`, +`/coder-eval:check-skill`, `/coder-eval:task`, `/coder-eval:lint-tasks`, +`/coder-eval:analyze`, `/coder-eval:ci`), and `reference/` — everything a skill reads +must live here, since an installed plugin is copied to ~/.claude/plugins/cache/ WITHOUT +its parent dirs (address it via `${CLAUDE_PLUGIN_ROOT}`). `reference/criteria.md` is +generated (`make plugin-reference`, CE033); `reference/run-layout.md` is a verbatim +mirror of `.claude/shared/run-layout.md`; `reference/task-rubric.md` is the shared +task-quality rubric that `task` and `lint-tasks` both read (plugin-only — no repo-side +twin); `reference/repo-layout.md` is the eval-tree DISCOVERY policy every skill reads +(`SKILL_NEEDS_EVAL_ROOT_DISCOVERY`, which a new skill must declare a stance in) — glob +for `task_id:` files and `run.json`, never assume `tasks/`/`runs/latest` — as distinct +from `run-layout.md`, which describes what is inside a run directory. Every skill must +appear in all four surfaces in `SKILL_DOC_SURFACES` (derived test), and their combined +frontmatter `description` length is capped (`SKILL_LISTING_BUDGET_CHARS`) because the +skill listing's budget is shared with every skill the user has installed. **Skill naming +is verb-first imperative** — a skill is a command you issue (`/coder-eval:`) and +every one of them takes an action, so name it for the action: a bare verb where that is +unambiguous (`init`, `analyze` — the object comes from the argument), otherwise +`-` (`lint-tasks`, `check-skill`). Never `-`: `skill-check` +was renamed to `check-skill` precisely because it read backwards next to `lint-tasks`. +`task` and `ci` predate the rule and stay — renaming a published skill breaks every +user's muscle memory for no functional gain, since activation keys on the `description`, +never the name. Distinct from `.claude/commands/`, which stays repo-local contributor +tooling. + +### action.yml + +Published composite GitHub Action (coder-eval as a CI gate). release.yml's `release` job +maintains its `version:` default; its `promote` job (gated on publish-pypi) moves the +`v` tag + cuts the Release, so nothing consumer-visible moves before the wheel is +on PyPI. verify-published-action.yml then verifies the published composite +(tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and +nightly. Runbook: CONTRIBUTING.md § Releasing. ## The Agent ABC contract From aeb1de403679e5c6ae37f0ff37e08ad310294f80 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 15 Sep 2026 19:33:02 -0700 Subject: [PATCH 18/19] docs: slim CLAUDE.md, and drop a directory that never existed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CLAUDE.md is loaded into every session, so a copy kept there costs context on every turn and goes stale with nothing to sense it. Three sections were copies. - **Directory Structure** was wrong. It listed `optimize/`, which exists on neither this branch nor main, and omitted `errors/` and `plugins.py`, both present since the initial public release. The tree was never the value — the annotations were. It becomes a list of what a filename cannot tell you (the CE048 twins, the CE063 seam, the CE057 sidecar, CE053's filename ownership, CE056's env var, the pricing.ts mirror), opening with "run `ls`". 454 -> 176w. - **Success Criteria** was a third copy of one registry, behind the CE033-generated plugin reference and the Task Definition Guide, with no parity test — the shape CE030's own CLAUDE.md test exists for. Accurate today; stale eventually. Now the shared-field paragraph plus three pointers. 259 -> 76w. - **Extension Points** restated docs/EXTENDING.md's checklists step for step while the section 190 lines above it says "Each entry is a pointer". Keeps the three orientation facts — pkgutil discovery, the plugin SPI with no closed enum, pricing on the same register hook — and the obligations (CE036, CE047, parity). 48 -> 21 lines. Also reverts the run-limit parity sentence claiming an adapter rejects an unsupported field at load time. It does not: opencode_agent.py:785 and pi_agent.py:738 both warn and continue. Neither lint-pinned surface is touched — the CE030 model sentence and the six skill names are unchanged — and nothing links into the cut sections. CLAUDE.md: 2,802 -> 2,028 words. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 181 +++++++++++++++--------------------------------------- 1 file changed, 49 insertions(+), 132 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c37d7d907..5cf9f61e5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,69 +26,28 @@ data-driven analysis. ## Directory Structure -``` -coder_eval/ -├── agent.py # Agent ABC (start, communicate, stop, get_state) -├── config.py # Settings via pydantic-settings (.env loading) -├── sandbox.py # Sandbox manager (tempdir, venv, templates, adopt) -├── orchestrator.py # Main evaluation loop -├── reports.py # Markdown/JSON run reports + per-suite rollups -├── reports_experiment.py # Cross-variant experiment reports -├── reports_junit.py # JUnit XML from a finalized run dir (CI ingestion) -├── reports_html.py # Single-file HTML report (the evalboard's static twin) -├── reports_stats.py # Shared report statistics + ungraded rendering helpers -├── formatting.py # Number/duration formatting shared by the renderers -├── analysis.py # Command statistics aggregation -├── logging_config.py # Structured logging setup -├── path_utils.py # Run IDs, path utilities, atomic writes, tree digests -├── fs_permissions.py # set_permissions: stacked chmod window -├── pricing.py # Model pricing (mirrored by evalboard/lib/pricing.ts) -├── litellm_cost.py # Join proxy-captured actual per-call cost onto turns -├── timing.py # TurnClock + turn decomposition (single subtraction seam) -├── invocation_log.py # record_cli recording shim + JSON Lines reader -├── argv_match.py # Structured argv matcher (STDLIB-ONLY sidecar — CE057) -├── telemetry.py # App Insights / OpenTelemetry emission -├── isolation/ # driver: docker — one container per task -├── harbor/ # Harbor export + coder-eval as a Harbor agent -├── optimize/ # Prompt/config optimization helpers -├── utils.py # Version info helpers -│ -├── agents/ # Agent implementations (claude_code, codex, antigravity, -│ # opencode, pi, noop) + registry, watchdog -│ -├── models/ # Pure Pydantic data models (see __init__ for exports) -│ ├── enums.py # AgentKind, AgentState, FinalStatus, ApiBackend -│ ├── criteria.py # 15 success criterion types + base + union -│ ├── experiment.py # ExperimentDefinition, ExperimentVariant, ResolvedTask -│ ├── cli_match.py # FlagMatch + CliMatch (cycle-free leaf) -│ ├── container_paths.py # IN_CONTAINER_ENV + container path constants (CE056) -│ ├── mutations.py # PromptMutation variants -│ ├── results.py # CriterionResult, TurnRecord, EvaluationResult, rollups -│ ├── routing.py # ApiRoute (DirectRoute/BedrockRoute) -│ ├── sandbox.py # SandboxConfig, ResourceLimits, RecordedCli, CliResponse -│ ├── tasks.py # TaskDefinition, AgentConfig, Dataset, RunLimits -│ ├── telemetry.py # CommandTelemetry, TokenUsage, TranscriptMessage -│ └── templates.py # RepoSource, TemplateDirSource, StarterFilesSource -│ -├── criteria/ # Criterion checker plugins (one file per type) -│ ├── __init__.py # CriterionRegistry with auto-discovery -│ └── base.py # BaseCriterion + @handle_criterion_errors -│ -├── evaluation/ # checker.py (SuccessChecker), judge_context, judge_verdict, -│ # sub_agent, summaries -├── orchestration/ # batch, config, config_merge, early_stop, evaluation, -│ # regrade, experiment, overrides, task_loader -├── cli/ # Typer commands; each has a plain-Python twin (CE048) -├── scoring/ # AST / token / signature / complexity / quality similarity -├── streaming/ # Event protocol, EventCollector, renderers -├── simulation/ # Multi-turn user simulation (dialog mode) -└── resources/ # Package resources - -experiments/ tasks/ tests/ docs/ templates/ evalboard/ -plugins/coder-eval/ # Published Claude Code plugin (six skills) -action.yml # Published composite GitHub Action -.claude-plugin/marketplace.json -``` +`src/coder_eval/` — run `ls` for the current layout. What a filename does not tell you: + +- **`models/`** is the pure-Pydantic layer: the dependency arrow runs `agents` → + `models`, so it may reach `agents` / `plugins` only lazily (CE017). All core models + import from `coder_eval.models`, never from its submodules. +- **`criteria/`** auto-discovers one checker per type via `pkgutil`. +- **`cli/`** holds Typer commands; each has a plain-Python twin (CE048). +- **`timing.py`** owns the single subtraction seam (CE063). +- **`argv_match.py`** is a STDLIB-ONLY sidecar copied beside the recorder (CE057). +- **`fs_permissions.py`** is `set_permissions`, the stacked chmod window. +- **`path_utils.py`** owns run ids, atomic writes and tree digests — and every run-record + filename literal (CE053). +- **`models/container_paths.py`** owns `IN_CONTAINER_ENV` (CE056). +- **`pricing.py`** is hand-mirrored by `evalboard/lib/pricing.ts`; a parity test fails on + drift either way. +- **`reports_html.py`** is the evalboard's static twin. +- **`isolation/`** is `driver: docker`, one container per task. +- **`streaming/`** is the event protocol and `EventCollector`. + +Outside the package: `tasks/`, `experiments/`, `templates/`, `tests/`, `docs/`, +`evalboard/`, `plugins/coder-eval/` (the published plugin), `action.yml` (the published +composite Action). ## Key Architectural Patterns @@ -120,8 +79,7 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes Defense-in-depth, not a boundary — the known gaps are documented in the notes. Authoring reference: [Reference Solutions](docs/TASK_DEFINITION_GUIDE.md#reference-solutions). - **Harness run-limit parity**: a shared config field must mean the same thing on every - backend, or the adapter rejects it at load time. A silently ignored field is a defect, - not a table row. Table: + backend, or the divergence is documented. Table: [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). Caps are authored under [Run Limits](docs/TASK_DEFINITION_GUIDE.md#run-limits). - **Execute vs. run**: `execute` is `run` with grading off — rows finalize as @@ -145,37 +103,20 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes - **Dialog mode**: `simulation/` drives a multi-turn LLM user — see [Dialog Mode](docs/DIALOG_MODE.md). -## Success Criteria (15 types) - -| Type | Scoring | Description | -|------|---------|-------------| -| [`file_exists`](docs/TASK_DEFINITION_GUIDE.md#file_exists) | Binary | File must exist | -| [`file_contains`](docs/TASK_DEFINITION_GUIDE.md#file_contains) | Fractional | String presence/absence | -| [`file_check`](docs/TASK_DEFINITION_GUIDE.md#file_check) | Fractional | Unified file existence + content + regex check | -| [`json_check`](docs/TASK_DEFINITION_GUIDE.md#json_check) | Fractional | JSON validation + JSON Schema + JMESPath assertions | -| [`run_command`](docs/TASK_DEFINITION_GUIDE.md#run_command) | Binary / Continuous | Exit code + optional stdout matching or float scoring | -| [`file_matches_regex`](docs/TASK_DEFINITION_GUIDE.md#file_matches_regex) | Binary | Regex match on file | -| [`reference_comparison`](docs/TASK_DEFINITION_GUIDE.md#reference_comparison) | Continuous | AST/token/complexity similarity | -| [`command_executed`](docs/TASK_DEFINITION_GUIDE.md#command_executed) | Fractional | Agent tool usage verification | -| [`cli_called`](docs/TASK_DEFINITION_GUIDE.md#cli_called) | Binary | Structured match over the `record_cli` invocation log | -| [`commands_efficiency`](docs/TASK_DEFINITION_GUIDE.md#commands_efficiency) | Continuous | Tool-call efficiency against an expected budget | -| [`uipath_eval`](docs/TASK_DEFINITION_GUIDE.md#uipath_eval) | Fractional | UiPath agent evaluation results | -| [`classification_match`](docs/TASK_DEFINITION_GUIDE.md#classification_match) | Binary | File-based label match; emits suite-level P/R/F1 | -| [`skill_triggered`](docs/TASK_DEFINITION_GUIDE.md#skill_triggered) | Binary | Did the agent engage the target skill? Agent-agnostic | -| [`llm_judge`](docs/TASK_DEFINITION_GUIDE.md#llm_judge) | Continuous | LLM grades artifacts + optional trajectory/reference | -| [`agent_judge`](docs/TASK_DEFINITION_GUIDE.md#agent_judge) | Continuous | Sandboxed SDK agent investigates with tools. Expensive | +## Success Criteria -All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9). Live -criteria also accept `stop_early:`. Dataset-backed tasks may set `suite_thresholds:` -(see [Suite-level scoring](docs/DATASETS.md#suite-level-scoring)). - -Each type above links to its own section in the -[Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md#success-criteria), which is the -authoritative per-field reference. Path and env-var resolution inside a checker is +Every criterion type registers in `criteria/` and is listed by +`CriterionRegistry.list_types()`. The authoritative per-field reference is +[Task Definition Guide § Success criteria](docs/TASK_DEFINITION_GUIDE.md#success-criteria); +path and env-var resolution inside a checker is [Checker Context](docs/TASK_DEFINITION_GUIDE.md#checker-context). The plugin ships a generated copy at `plugins/coder-eval/reference/criteria.md` — regenerate it with `make plugin-reference`; never hand-edit it (CE033). +All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9). Live +criteria also accept `stop_early:`. Dataset-backed tasks may set `suite_thresholds:` +(see [Suite-level scoring](docs/DATASETS.md#suite-level-scoring)). + ## Evaluation Flow ``` @@ -284,50 +225,25 @@ documentation: [Claude Code plugin](docs/PLUGIN.md). ## Extension Points -### Adding a New Criterion - -1. Define the model in `models/criteria.py` inheriting `BaseSuccessCriterion`. -2. Add it to the `SuccessCriterion` union. -3. Create the checker in `criteria/` inheriting `BaseCriterion`, decorated with - `@register_criterion` — auto-discovered at runtime. -4. If it is a live criterion, add `ContractCase`s (CE036) and run `make plugin-reference`. +**A new criterion**: model in `models/criteria.py` → the `SuccessCriterion` union → +checker in `criteria/` decorated `@register_criterion`, auto-discovered via `pkgutil`. +A live criterion also needs `ContractCase`s (CE036) and `make plugin-reference`. -Worked example: [Custom success criteria](docs/EXTENDING.md). Document the new type in -the [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md#success-criteria). - -### Adding a New Agent - -Agents register through the plugin SPI (entry-point group `coder_eval.plugins`) — there -is no closed enum or dispatch to edit. In-tree and third-party agents use the same path. -Full walkthrough: [Extending Coder Eval](docs/EXTENDING.md). Per-agent setup and -credentials: [Claude Code](docs/agents/CLAUDE_CODE.md), [Codex](docs/agents/CODEX.md), -[Antigravity](docs/agents/ANTIGRAVITY.md), [OpenCode](docs/agents/OPENCODE.md), -[Pi](docs/agents/PI.md). A new agent must also be added to every onboarding surface -CE047 tracks, and its run-limit behaviour recorded in +**A new agent**: agents register through the plugin SPI (entry-point group +`coder_eval.plugins`) — there is no closed enum or dispatch to edit, and in-tree and +third-party agents take the same path. A new agent must be named on every onboarding +surface CE047 tracks, and its run-limit behaviour recorded in [Run-Limit Parity](docs/agents/HARNESS_PARITY.md). -1. Define a `BaseAgentConfig` subclass (its own `type: Literal["your-kind"]`) and - implement the `Agent` ABC. -2. Bind them with `registry.register("your-kind", YourConfig)(YourAgent)` inside a - `register(registry)` hook exposed via a `coder_eval.plugins` entry point. -3. Use the shared turn lifecycle on the base class — `self._begin_turn()`, - `self._end_turn_ok()`, `self._mark_stopped()`. Do not reimplement it. -4. Before raising on a mid-turn failure, set `self.pending_turn` to a `crashed=True` - `TurnRecord`, then raise `AgentCrashError` or `TurnTimeoutError` bare. -5. Emit the standardized event protocol and fan it through an internal `EventCollector` - plus the caller's `stream_callback`. One `AgentStartEvent` and one matching - `AgentEndEvent` on *every* exit path, from `finally`. -6. If the agent shells out or holds OS resources, implement real `stop()` / `kill()` / - `kill_sync()`. `kill_sync()` runs on a non-asyncio thread and must not await. - -### Registering Model Pricing (plugins) - -Call `register_pricing(YOUR_RATES)` from the same `register(registry)` hook — there is -no separate entry-point group. Keys are bare model ids; vendor/Bedrock prefixes are -normalized off at lookup. Registration is idempotent for identical rates and raises on a -conflicting rate for an existing key, so plugin load order can never silently reprice a -model. `coder_eval_uipath/pricing.py` is the worked example; see also -[Model pricing](docs/EXTENDING.md). +**Model pricing**: `register_pricing(YOUR_RATES)` from the same `register(registry)` +hook — no separate entry-point group. + +Steps, checklists and worked examples: [Extending Coder Eval](docs/EXTENDING.md). +Document a new criterion type in the +[Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md#success-criteria). Per-agent setup +and credentials: [Claude Code](docs/agents/CLAUDE_CODE.md) · [Codex](docs/agents/CODEX.md) +· [Antigravity](docs/agents/ANTIGRAVITY.md) · [OpenCode](docs/agents/OPENCODE.md) · +[Pi](docs/agents/PI.md). ## Task Definition @@ -362,6 +278,7 @@ bandit, pre-commit, mcp - **YAGNI** — don't add complexity until actually needed - **KISS** — keep it simple - **Clean code** — no dead code, all imports used, all tests passing +- **Greenfield project** — no backward-compatibility burden - **Delete before you guard** — before you add a lint rule, doc paragraph, criterion type or config field, try to delete the pattern that needs it. A new type that subsumes an old one removes the old one in the same change (no back-compat burden) From 494168164f2b187e28d36d2c5ac8e83affa796c9 Mon Sep 17 00:00:00 2001 From: uipreliga Date: Tue, 15 Sep 2026 19:38:26 -0700 Subject: [PATCH 19/19] docs(harbor): apply the branch's prose rules to the bind-mount rewrite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bind-mount rewrite (#175) landed on main while this branch was open, so its new prose in `harbor/packager.py` never met the gate this branch introduces. Two docstrings were over the 150-word bar. - The module's emitted-layout diagram carried an eight-line annotation restating what `_write_environment` and `_write_docker_compose_mounts` already say. It is three lines now: Dockerfile only when `dockerfile_path` is set, and everything bind-mounted rather than COPY'd. - `_plugin_volume_specs` keeps its contract and its one real HAZARD -- a plugin path is carried UNEXPANDED and `docker_runner.py` expands it at launch, so the two sides must move together. The rest is a pointer. The rationale moves to `reporting.md § What the export carries, and what it refuses to carry`: why a prebuilt image needs no Dockerfile (Harbor's own `should_use_prebuilt_docker_image`, and `[environment].workdir` reaching `docker exec` independently of the build), why nothing is COPY'd, why a bind mount cannot self-nest into the export's `-o` directory, and the live `harbor run` that surfaced the agentless `initial_prompt` validation error. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/notes/reporting.md | 3 +++ src/coder_eval/harbor/packager.py | 39 ++++++++++++------------------- 2 files changed, 18 insertions(+), 24 deletions(-) diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md index 00b9c304e..4d25c97b8 100644 --- a/.claude/notes/reporting.md +++ b/.claude/notes/reporting.md @@ -350,6 +350,9 @@ Dockerfile line either, because `[environment].workdir` is what Harbor passes as Dockerfile-less shape is a real choice rather than a null-vs-set distinction: `docker_cfg.image` always has a value (`default_factory=get_default_docker_image_tag`). +A bind mount also removes a hazard the `COPY` approach had: it never walks the export's +own `-o` directory, so a plugin source that contains it cannot self-nest. + Nothing is `COPY`'d into the image any more. `environment/task.yaml`, each `type: local` plugin, each `TemplateDirSource` and each `extra_mounts` entry are bind-mounted at their own host path, mirroring `docker_runner.py`'s auto-mount — which is why the export warns diff --git a/src/coder_eval/harbor/packager.py b/src/coder_eval/harbor/packager.py index 76a7cbb78..5f83c46cb 100644 --- a/src/coder_eval/harbor/packager.py +++ b/src/coder_eval/harbor/packager.py @@ -10,14 +10,9 @@ ├── task.toml ├── instruction.md # placeholder -- the real prompt is in environment/task.yaml ├── environment/ - │ ├── Dockerfile # ONLY when sandbox.docker.dockerfile_path is set; - │ │ # otherwise absent, and task.toml's - │ │ # [environment].docker_image names the image directly + │ ├── Dockerfile # only when sandbox.docker.dockerfile_path is set │ ├── task.yaml # criteria-free copy for the CoderEvalAgent embed - │ └── docker-compose.yaml # ALWAYS written. Read-only mounts: task.yaml itself, - │ # each `type: local` agent.plugins[] entry, each - │ # TemplateDirSource; sandbox.docker.extra_mounts keep - │ # their own ro/rw mode. Nothing is COPY'd into the image. + │ └── docker-compose.yaml # always written; every input is bind-mounted, never COPY'd └── tests/ ├── test.sh # the two-line shim ├── task.yaml # the criteria, as authored @@ -460,23 +455,19 @@ def _template_volume_specs(task: TaskDefinition, warnings: list[str]) -> list[st def _plugin_volume_specs(task: TaskDefinition) -> list[str]: """Return one ``src:src:ro`` compose volume spec per ``type: local`` ``agent.plugins[]``. - Mounted at its own host path, unmodified — exactly mirroring ``docker_runner.py``'s - own auto-mount for non-Harbor runs (``-v {target}:{target}:ro``) — so - ``environment/task.yaml``'s ``agent.plugins[].path`` needs no rewriting: the path is - identical inside and outside the container. This also means a bind mount never - touches this export's own output directory (unlike an earlier ``COPY``-based - approach, which had to guard against the plugin source containing the export's - ``-o`` directory) — there's nothing to walk or copy, so no self-nesting hazard here. - - Forced read-only (``:ro``), unconditionally: this mounts the skill/plugin content - an agent reads, never writable state, and the same host path may be mounted into - unrelated concurrent containers. - - Unlike ``TemplateDirSource.path`` (already resolved to an absolute host path by - ``load_task``), a plugin's ``path`` is carried unexpanded (e.g. literal - ``"$SKILLS_REPO_PATH"``) — ``docker_runner.py``'s own auto-mount expands it the same - way (``os.path.expandvars`` + ``os.path.expanduser``) at container-launch time, so this - mirrors that rather than requiring the export-time environment to already have it resolved. + Mounted at its own host path, unmodified, mirroring ``docker_runner.py``'s auto-mount + (``-v {target}:{target}:ro``), so ``environment/task.yaml``'s ``agent.plugins[].path`` + needs no rewriting -- the path is identical inside and outside the container. + + Always ``:ro``: this carries skill/plugin content an agent reads, never writable + state, and the same host path may be mounted into unrelated concurrent containers. + + HAZARD: a plugin ``path`` is carried UNEXPANDED (a literal ``"$SKILLS_REPO_PATH"`` + reaches the spec), unlike ``TemplateDirSource.path``, which ``load_task`` has already + resolved. ``docker_runner.py`` expands it at container-launch time; changing either + side alone breaks the mirror. + + Rationale: .claude/notes/reporting.md § What the export carries, and what it refuses to carry """ plugins = (task.agent.plugins if task.agent is not None else None) or [] local_plugins = [p for p in plugins if isinstance(p, dict) and p.get("type") == "local"]