diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index e550452ad..03bd25a39 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -962,3 +962,33 @@ re-derive from scratch. leaves them stale with nothing failing. Needs a backtick-path extractor scoped to one section, which is the narrow case of the prose-path candidate above. — caught during the reports consolidation rebase. + +- [ ] **A prose "see X's docstring" citation whose target no longer holds the + claim.** Moving rationale out of a docstring leaves every citation of that + docstring pointing at text that is gone, and `check_pointers` cannot see it + because the citation is prose, not a `Rationale:` pointer. Not mechanised: the + match is heuristic ("see the module docstring", "see CE063's docstring", + "see that class's docstring" all read differently), and most citations are + in-file and still valid, so a rule would need per-site triage rather than a + regex. Instances found (line numbers at 946ca968): + `src/coder_eval/harbor/packager.py:382` and `:386`, + `tests/test_harbor_packager.py:163` and `:187`, + `tests/lint/rules/ce064_turn_bracket_on_the_clock.py:40`. Caught in: tests + prose slimming, Phases 3 and 7. + +- [ ] **Narrative after a docstring's `Args:`/`Returns:` block that follows a + `Rationale:` pointer.** `check_pointer_placement` accepts any tail that STARTS + with a trailing section, so pointer → `Args:` → an indented entry → a new + base-indented paragraph passes although the pointer is no longer the last prose + line. Guarding it means walking section blocks by indentation (the shape + `prose_words` already uses), plus false-positive triage over every src/ and + tests/ docstring that ends in a section — more than a quick add. Caught in: + tests prose slimming, final review (gpt-5.6-sol). + +- [ ] **A `Rationale:` pointer whose target is not under `.claude/notes/`.** + `check_pointers` joins the captured path onto the repo root, so an absolute + path, a `..` segment or any other Markdown file resolves, while CLAUDE.md and + `.claude/notes/README.md` define the pointer as a repo-relative notes path. No + such pointer exists today. Deferred because restricting the target is a design + decision (a `docs/` guide heading is a plausible SSOT target) rather than a + mechanical guard. Caught in: tests prose slimming, final review (gpt-5.6-sol). diff --git a/.claude/notes/README.md b/.claude/notes/README.md index 37b8a5aa0..641e4f1db 100644 --- a/.claude/notes/README.md +++ b/.claude/notes/README.md @@ -15,6 +15,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`. - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading +- [lint-rules.md](lint-rules.md) — why each CE lint rule exists - [orchestration.md](orchestration.md) — config merge, resume, early stop, execute vs. run - [permissions.md](permissions.md) — the chmod window and the reference anti-cheat - [persistence.md](persistence.md) — atomic writes and judge persistence @@ -46,12 +47,31 @@ Path relative to the repo root, then `§`, then the target `##` heading text ver There is no other accepted form: `tests/lint/prose_budget.py` parses this one and fails `make docs-budget` when the file or the heading does not exist. -## The prose budget is one number, not a lint rule - -`make docs-budget` reports the standing total and fails when it grows. It is deliberately -**not** a `CE` rule: `tests/lint/rules/` polices per-pattern invariants one AST at a time, -while this is a single whole-tree total. Making it a rule would mean a rule class, a rule -test and a catalogue entry to enforce one integer — enlarging the harness the budget +## The prose budget is not a lint rule + +`make docs-budget` runs `tests/lint/prose_budget.py` over `src/coder_eval` and `tests`. +It enforces three rules with no baseline to maintain: no docstring over 150 prose words +(an `@abstractmethod` and the Typer commands are exempt), no own-line comment run over 8 +lines, and a file's own-line comments within `MAX(20, 0.15 × lines)`. It also fails when a +`Rationale:` pointer does not resolve or is not the last prose line of its block. + +The run cap is the primary rule and the file total is the backstop under it. The order +matters, because the total alone was inverted: it blocked `isolation/docker_runner.py` at +229/229 for carrying 62 short annotations pinned to the lines they explain, while a 16-line +essay in `tests/test_regrade.py` sat at 29% of its budget. Four files had settled at exactly +100% of the cap — a budget with an allowance becomes a target. The run cap has no allowance +and catches the shape the total could not see; the total still catches a file that is mostly +commentary however it is broken up. + +Measured when the cap was chosen: 78.5% of comment runs in the tree are 1-2 lines, 2.5% are +6 or more, and the longest was 16. After the essays moved out, 21 runs sat at exactly 8 +lines. They were cut to 7 or fewer so that no run starts at the cap, and an ordinary edit +to one of them does not fail the gate. The four files above still sit at 100% of their total +budget, so a comment added to one of them has to be paid for by deleting another. + +It is deliberately **not** a `CE` rule: `tests/lint/rules/` polices per-pattern invariants +one AST at a time, while this measures prose across whole trees. Making it a rule would +mean a rule class, a rule test and a catalogue entry — enlarging the harness the budget exists to shrink. Do not "fix" this by promoting it. Nothing here states how many `CE` rules exist. `tests/lint/rules/` owns that count, and a @@ -59,8 +79,8 @@ number written down anywhere else is a second declaration that will be wrong. ## Where a CE rule's rationale lives -Each rule's authoritative rationale is its own module docstring under -`tests/lint/rules/` — or, for the doc-surface and whole-tree rules, the corresponding -`@pytest.mark.lint` class in `tests/test_custom_lint.py`. Read that before editing, -suppressing or widening a rule. No prose summary is kept here: a second copy is a second -declaration, and it is the one that goes stale. +A rule's invariant, scope and blind spots live in its rule file under `tests/lint/rules/` +(or its `@pytest.mark.lint` class in `tests/test_custom_lint.py`). Read that before you +edit, suppress or widen a rule. The defect that motivated it lives in +[lint-rules.md](lint-rules.md), under the rule's id. That file does not repeat the +invariant, scope or blind spots. diff --git a/.claude/notes/agents.md b/.claude/notes/agents.md index 7265cf9a4..8344a33c4 100644 --- a/.claude/notes/agents.md +++ b/.claude/notes/agents.md @@ -231,6 +231,27 @@ overlaps, so a call closing in the gap before a step start needs nobody to remem The rule that used to live there was wrong once on OpenCode — clearing at `step_start` wiped the span before `step_finish` could subtract it, a 100% overstatement of that window. +The OpenCode gap case is reachable only through the NON-TERMINAL tool path. The CLI +normally emits one already-`completed` event per call, and that call closes inside the +step that opened it. That is why the measured corpus shows 0.00% for this case, and why +`TestToolSpansSurviveTheStepBoundary` drives `_OpenCodeTurnState` directly instead of a +recorded stream. + +The Pi inter-turn gap is small in practice: measured across 25 real window pairs, the +median was 0.25 ms and the maximum 0.75 ms. The fix is worth +keeping mainly because the tool spans must keep working once the gap closes, which +`tests/test_pi_agent.py::TestToolSpansSurviveTheTurnBoundary` pins. + +An Antigravity call that is still running when a flush cuts the window must have its +already-elapsed part taken out of that window. Subtracting only CLOSED intervals published +that part as model time while the call's own `duration_ms` counted it again, and because +the windows cover the turn end to end, there is no slack to absorb it. Measured on `tasks/hello_date` +with a live gemini-3.1-pro-preview: a `Bash` opening 1.7 ms before the flush drove +Σ generation + Σ command 0.26 ms PAST the turn's own `duration_seconds`, on a turn whose +entire headroom was 1.4 ms. Four sibling runs passed by 1.2-8.7 ms out of ~12 s, so the +defect was a coin flip per run. `tests/test_antigravity_agent.py::test_a_tool_still_open_at_the_flush_is_not_generation_time` +pins it. + ## Why a clean exit can still be a crash An exit code of 0 with no telemetry is indistinguishable from a real pass in every @@ -267,6 +288,14 @@ reports no usage at all — where crashing every turn makes the harness unusable merely imprecise. It deliberately does NOT cover vocabulary drift: that arm has silently zeroed a whole run before, and no provider quirk explains it. +The OpenCode golden fixtures (`tests/_fixtures/golden_streams/opencode_fixtures.py`) mirror +events captured from a live `opencode run --format json`, in the CLI's own compact +vocabulary (`step_start` / `step_finish` / `text` / `tool_use`, payload under `part`). The +`session.next.*` names in the server's OpenAPI schema describe `opencode serve`'s SSE +surface, not the CLI stream, so "correcting" a fixture toward them re-creates the +zero-telemetry vocabulary drift above. The fixtures need no import guard because +`pyproject.toml` declares `opencode = []`: there is no Python package to skip on. + ## Token accounting, per harness Keep the buckets straight: `uncached_input_tokens` is the FRESH prompt slice only, because diff --git a/.claude/notes/contracts.md b/.claude/notes/contracts.md index 980f3edba..0c8ee894c 100644 --- a/.claude/notes/contracts.md +++ b/.claude/notes/contracts.md @@ -158,6 +158,23 @@ interpreter and its directory goes FIRST on a PATH the orchestrator also reuses `run_command`, so `tool: python3` made the shim re-resolve its own interpreter to itself — an exec loop that spins to the task timeout. +### Why the record_cli probe reads a real shim log + +The `tasks/record_cli_responses.yaml` probe needs a detector that proves per-invocation +response dispatch, and most of its signals cannot. `cli_called` matches argv only, so it +passes whether a response rule answered or the entry fallback did. The agent can +transcribe `captured.txt`, because the task YAML is serialised to `/work/input` and mounted +at `/work/task_dir`, and both are readable. A codegen regression that renders `RULES = []` +raises nothing, so neither `rule_error` nor `sidecar_error` is booked. Only the +`"rule": N` key in the log catches it. + +`tests/test_tags.py::TestRecordCliProbeIntegrity` therefore does not compare the YAML +against itself. It runs each stubbed command through a real shim and matches the task's +regexes against the lines the shim wrote. The needle's spelling (`"rule": 0`, with the +space) comes from the default `json.dumps` separators in `invocation_log.record`. A test +that copied the needle would stay green after a switch to compact separators, while the +blocking CI probe failed. + ## The checker base class ### Exactly one of _check_impl or _check_impl_async diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index c9e2957d5..70f4b3774 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -781,6 +781,16 @@ shape this host got is logged rather than left to be inferred. `setup` is: discovering a venv a task never asked for grades it under a PATH it never ran under, and would let an agent shadow binaries by writing `.venv/bin/` into its own workspace. +The unit test in `tests/test_sandbox.py` reads `pyvenv.cfg`. That proves the flag is set, +not that the result is correct. Every task image installs packages globally: the +framework image uses `uv pip install --system`, and skillsbench task images use +`RUN pip install ...`. `tests/test_sandbox_venv_live.py` checks the result in the +`coder-eval-agent` base image. It mounts this checkout's `src/` over the image's copy, so +the test runs the code under test and not the version the image was built with. Measured +on that test's own scenario, `python -c "import pydantic"` exits 1 with an isolated venv +and exits 0 with `--system-site-packages`. `pydantic` is a coder_eval runtime dependency, +so the base image already has it globally, and the check needs no build and no network. + ### The criterion environment, layer by layer Each layer is independent — none breaks if another is absent. diff --git a/.claude/notes/lint-rules.md b/.claude/notes/lint-rules.md new file mode 100644 index 000000000..7d73e8bc5 --- /dev/null +++ b/.claude/notes/lint-rules.md @@ -0,0 +1,964 @@ +# Lint rules + +> Conventions and authority order: see [README.md](README.md). + +A rule's module docstring (or, for a doc-surface or whole-tree rule, its +`@pytest.mark.lint` class in `tests/test_custom_lint.py`) states the invariant, the scope, +the exemptions and the blind spots. This file does not repeat them. It holds only why each +rule exists: the defect that caused it, the measurements behind it, and why each trade-off +was chosen. + +## CE004 + +CE004 once borrowed CE066's core predicate whole and inherited its `reports/` exemption. +Nothing had imported `cli` from `reports/` yet, so the hole was latent rather than live. +Re-enumerating the packages in the rule is what let that list rot in the first place, +which is why the scope now comes from `_layers`. + +`harbor/` belongs in scope because its reward writer raises a plain exception +(`RewardWriteSkippedError`, or the re-exported `RegradeError`) and lets the CLI wrap it +into an exit code — the `orchestration/regrade.py` -> `evaluate` shape. + +## CE009 + +Without `extra='forbid'` a misspelled key, for example `directry: foo/` for `directory: +foo/` on `ReferenceSource` or a typo on a criterion field, is dropped with no signal that +it was ignored. Forbidding extras surfaces the typo at load time with the concrete field +name. + +Result and persistence models are exempt as deliberate round-trip leniency. + +Base chains stop at the file boundary because that keeps the rule cheap: it trusts the +convention that every model class in the scoped files eventually derives from `BaseModel`. + +## CE013 + +The 2026-05-20 judge refactor replaced text-with-JSON verdict parsing with a typed tool +channel (`submit_verdict`). The legacy parser ran regex against structural transcript tags +(`[ASSISTANT]`, `[RESULT - …]`) and JSON-shape literals (`\{`, `"score"`, `"rationale"`), +all coupled to `ClaudeCodeAgent._format_messages` rendering choices that have no business +being a correctness contract. The rule keeps that pattern from returning. + +## CE014 + +The declarative merge engine reads a per-field strategy off the Pydantic `FieldInfo` and +falls back to a type-aware default: nested `BaseModel` / free-form `dict` merge `deep`, +`list` and scalars merge `replace`. A `list` is the one type whose default (`replace`) is +easy to mean otherwise (`append`), so a list field that silently keeps `replace` when it +meant `append` is a latent resolution bug. For nested-model and dict fields the +nested-replace regression is structurally impossible, so they may keep a plain `Field`. + +Scope is the set of model classes the engine feeds through `merge_layers` / +`resolve_root`: the three `-D`-reachable roots, the sandbox sub-models reached by deep +merge, and the two models merged outside the `-D` roots (`TaskDefinition` for +`pre_run`/`post_run`, `SimulationConfig` for `constraints`). Scoping by class name rather +than by file keeps the rule pinned to the engine's real roots and avoids flagging unrelated +list fields that share a file, such as `PreRunCommand` in `tasks.py`. + +## CE020 + +`BaseAgentConfig` is the vendor-neutral Pydantic base shared by every agent kind, +including third-party BYOA configs. A field on it typed against `claude_agent_sdk` leaks a +Claude-Code-specific type onto agents that have nothing to do with the Claude SDK. The +refactor that removed that leak turned `plugins: list[SdkPluginConfig]` into a local +`LocalPluginConfig` and moved `setting_sources` down to `ClaudeCodeAgentConfig`. The +boundary is mechanically detectable, so the rule guards it. + +## CE021 + +`task.json` is the harness's always-produce/always-consume artifact: the only thing that +crosses the container boundary, and the per-task record every dashboard and timeline reads. +A bare `EvaluationResult.model_validate_json(text)` turns a present-but-malformed file +into an uncaught exception that crashes the run. Two causes produce such a file: schema +skew between a stale `:latest` image and the host (the docker version checks only warn), +and a truncated or torn write. The incident was at `docker_runner.py`: the parse re-bucketed +the task to a non-persisted in-memory ERROR with no per-task report. The fix degrades: +catch `ValueError` and persist a synthetic ERROR record (`batch.py::_load_completed_result`, +`recover_task_results`, `docker_runner.py::_handle_malformed_task_json`). +`recover_task_results` catches `(OSError, ValueError)`, which is why a tuple member counts +as a guard. + +Scope stays narrow to `EvaluationResult` because it is the always-produce/consume +contract. + +## CE022 + +`Orchestrator._simulation_dialog_loop` is a sequential dialog driver whose residual length +is irreducible without a state-object rewrite (the 2026-06-23 decompose-god-functions +plan, Phase 5), so it keeps its `# noqa: PLR0915`. + +`_CAP` is the measured post-decomposition count (122) plus 6 headroom: ordinary edits do +not trip it, a real regrowth does. Ruff's ceiling for every other function is 80/25. + +## CE024 + +A bare `A | B | C` union of tagged Pydantic models validates via smart-union: input with a +missing or typo'd `type` tag silently coerces to whichever variant happens to fit, instead +of raising a crisp discriminator error. The `SuccessCriterion` union shipped this way — +tag-less criterion dicts coerced to the structurally-nearest variant — and the fix wraps it +in `Annotated[..., Field(discriminator="type")]`. + +The callable `Discriminator(...)` form is accepted because `CriterionResultUnion` in +`models/results.py` uses it. + +## CE026 + +Several surfaces introduce the same composite Action — `README.md`, `docs/CI_GATE.md`, +`docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow +users copy verbatim — and each was hand-maintained, so they drifted. The motivating bug: +`docs/CI_GATE.md` claimed "there is nothing to install" and offered a copy-pasteable +`uses:` step with no agent runtime. The action is agent-agnostic, so an integrator who +copied it got a run that died on a missing `claude` binary. The correcting paragraph was +11 lines away; the tutorial's snippet showed the prerequisite steps, the reference +page's did not. + +Only the first Action block on a page is held to prerequisite parity because it is the +page's quickstart; later blocks are single-input illustrations, and skipping them is what +keeps the rule quiet. + +The zero-install clause does not judge whether a paragraph 11 lines later states a real +prerequisite: that is semantic reasoning no static rule should attempt. It forces the +absolute to be scoped where it is written instead ("no Marketplace install step"). + +Slug parity exists because `action.yml`'s `name:` is the Marketplace listing title, and a +rename silently 404s every Marketplace link and leaves every badge naming the old listing. + +Input parity exists because GitHub does not fail a workflow on an unknown input: a +renamed input leaves every snippet promising something the step does not do — silently, +and worst in the `ci` skill, whose output lands in other people's repositories where +this repo's CI never sees it. + +## CE027 + +A documented env var whose name matches no `Settings` field or `AliasChoices` is dropped +with zero signal. That is the failure behind the `CODER_EVAL_API_BACKEND` doc bug: the real +field is `API_BACKEND`, so the prefixed spelling selected no backend and the run fell back +to Direct Anthropic. The framework also reads a handful of vars directly through +`os.getenv` (for example `CODER_EVAL_SKILLS_DIR`, `CODEX_BASE_URL`, +`CODER_EVAL_IN_CONTAINER`), and those are legitimately documentable, so a `src/` consumer +also backs a name. + +Prose mentions are not scanned because they are too prone to false positives (markdown +links like `CODEX_AGENT_GUIDE.md`, secret references like `secrets.BEDROCK_TOKEN`, +regex-pattern examples like `API_KEY = "…"`), and `NAME=value` is the form users copy into +a workflow, so it carries the real risk. Broad third-party namespaces (`AWS_`, +`ANTHROPIC_`, `GEMINI_`, `GITHUB_`, `EVALBOARD_`, `PLUGIN_`) are skipped because SDKs and +CI consume them, not necessarily through `Settings`, so scanning them gives false +positives on names that are legitimately external. + +The `src/` scan recognises a consumer through a named constant because +`CODER_EVAL_IN_CONTAINER` has one definition (`models/container_paths.py::IN_CONTAINER_ENV`) +and its consumers spell it `os.environ.get(IN_CONTAINER_ENV)`. A literal-only scanner +reports the repo's own gate as unbacked and pushes the author to paste the literal back: +the scanner argues against the SSOT it should reinforce. Resolution is two-step so the scan +stays strict: a constant that nothing reads is still unbacked. + +## CE028 + +The docs overhaul's root cause was doc/code drift, and the flat index surfaces drift the same +way: a page is added to the nav and forgotten in the three flat lists, or a page is deleted +and left dangling in them. Generating them from the nav removes the second source. + +A separate `--check` entry point would be untested duplication, so CE028 is the only +checker. The "every published `docs/*.md` is in the nav" check is the one that would have +caught the overhaul's whole bug class. The `docs/tutorials/README.md` table carries a +"you'll learn" column the nav does not, so generating it would need a second per-page +field and bring back a dual source of truth. + +## CE029 + +A published example that does not parse is worse than no example: readers copy it, hit a +`ValidationError`, and conclude the feature is broken. The rule caught exactly that. The +`prompt_mutations` recipe in `docs/AB_EXPERIMENTS.md` used `text:` where the field is +`content:`, and every mutation model declares `extra="forbid"`, so the published snippet +raised `variants.1.prompt_mutations.0.suffix.content Field required`. + +Scope is deliberately narrow: a false positive on an illustrative fragment would make +`make lint` a nuisance and get the rule deleted. A bare `success_criteria:` list is the +single most common doc shape, and the task guide's overview block uses the schematic +`agent: { ... }` form on purpose. The `` marker is for an +example that is intentionally partial in a way the heuristic cannot see. + +## CE030 + +The defects that a docs overhaul fixed were all one failure: a doc claim that does not +match the code. The two worst (P0 and P1) were "a Pydantic field the user must set, +documented nowhere." CE030 is the sensor that keeps that class from coming back, for a +small, explicit registry of user-facing models. + +Nested models are not walked because that would silently expand the documentation +commitment to dozens of models nobody signed up for (`AgentConfig`, `SandboxConfig`, +criteria). `CliMatch` is absent for that reason: its fields are documented in the +`cli_called` reference, and registering a third nested model under `SandboxConfig` would +start exactly that tree-walk. + +The inline-code match stays simple because the rule exists to catch *entirely +undocumented* fields; a fuzzier "documented in the right section" rule invites false +passes that erode trust in the gate. + +`RecordedCli` and `CliResponse` both declare `exit_code` / `stdout` / `stderr`, so +registering the second newly guards only `when`. Registering both is still net-positive. + +## CE031 + +A config field that users set in a task YAML but that no code reads is dead config: it +silently does nothing, and the author has no way to know. `SimulationConfig.parallel_trials` +was that — documented, set in a shipped task YAML, defaulting to `True`, and read nowhere +(trial concurrency is entirely `--max-parallel`'s job). + +"Consumed" means an attribute access by name because that is how a behavior-driving config +takes effect: the orchestrator and validators must read the field. `TaskDefinition` stays +unregistered because the dataset expander round-trips its fields through `model_dump`. + +The name-collision blind spot is accepted because a sensor with false negatives only never +wrongly breaks the build. + +## CE032 + +`Sandbox.resolve_files` is the single place criterion `path` semantics live: literal-first +resolution (a real file named `report[2024].json` is not reinterpreted as a character +class), glob expansion for artifacts whose location the prompt does not pin, ignore-pattern +filtering (`.venv` / `node_modules` / `dist` cannot be graded as agent output), and +exactly-one enforcement on content reads. + +A checker that builds its own path with `sandbox.sandbox_dir / ` and reads it +directly silently opts out of all of that, so path semantics differ per criterion. That is +how `reference_comparison.agent_file` drifted from every other path field. + +## CE033 + +An installed Claude Code plugin is copied to `~/.claude/plugins/cache/` without its parent +directories, so a skill cannot read `docs/TASK_DEFINITION_GUIDE.md` at runtime — every +reference a skill needs ships inside `plugins/coder-eval/`. A bundled copy of the +criterion vocabulary is exactly the kind of file that drifts: a criterion gains a field, +or a new criterion lands, and the copy keeps teaching the old schema to every plugin user. +A separate `--check` entry point would be untested duplication. + +A hardcoded inherited-field list would be a second declaration of the base schema. When +`stop_early:` replaced `stop_when` + `max_steps_to_decide`, such a list would have +rendered the new field into all 14 per-criterion sections and leaked two dead names; the +computed set absorbed the change with no edit. + +Full descriptions matter because what a field means is the half of the schema an authoring +agent gets wrong (that `min_count: 0` lets a criterion pass when nothing matched, that +`weight: 0` makes a criterion informational). A truncated or curated subset would need a +hardcoded name list, and so a second declaration of the schema. Defaults and types are +left out because rendering defaults means handling `default_factory` (whose +`FieldInfo.default` is `PydanticUndefined`), and rendering types means normalizing +`X | None` annotations — two helpers serving the half of the reference an authoring agent +needs least. `coder-eval plan` and the model docstrings cover the rest. + +## CE034 + +`require_success` defaults to False, so a `command_executed` criterion counts an +invocation that CRASHED. On an unarmed criterion that is merely generous. On an armed one +it corrupts the run's verdict, because three behaviours compose: + +1. `live_verdict` and `_check_impl` share `_matching_commands`, so a failed invocation + live-PASSES a positive criterion (`min_count > 0`, no `max_count`) the moment it is + observed; +2. `stop_early.on_pass: stop` ends the run on that pass — and `decide_within` latches + it, so the timeout never fires either; +3. gating is FIRED-ONLY: a run the watcher cut gates on the ARMED SUBSET + (`armed_criteria_passed`), so unarmed criteria are never consulted. + +The defect this caught was in `tasks/early_stop_weighted_low_weight_absorbed.yaml`: an +agent that ran `python app.py` BEFORE it created app.py scored a weighted 1.0 over the +armed subset and reported SUCCESS — with no app.py and a crashed script — because the +unarmed `file_exists` was bypassed. Running the plugin's own `lint-tasks` skill against +this repository's tasks found it. + +Pass-capability is read off the model's `live_decidable_polarities()` so the rule does not +re-derive the shape a second time. + +## CE035 + +The motivating bug shipped in `verify-published-action.yml`: two steps read +`steps.parity.outputs.version`, but the `parity` step writes only `pin` / `newest` / +`lagging` (the shell variable was `VERSION`, the output key was `newest`). GitHub expands +an unwritten output to the empty string, so `TAG_REF: v${{ steps.parity.outputs.version }}` +became the bare string `v`, `git show "v:action.yml"` exited 128 under +`set -euo pipefail`, and the preflight job was red on 100% of triggers. Through +`needs: preflight`, the paid end-to-end tier could never run at all. + +Nothing caught it. The workflow is invisible to ruff, pyright, pytest and the AST lint +runner, and `actionlint` models `steps.*.outputs` as an open string map, so an unwritten +shell key is untyped and unflagged there too. + +A missing step id or an undeclared `needs` output is always a finding because both are +fully enumerable from the file. + +## CE036 + +The contract is documented on `LiveVerdict` / `BaseCriterion.live_verdict` +(`criteria/base.py`), but nothing enforced it before CE036: a criterion (in-tree or +plugin) that implements `live_verdict` non-monotonically type-checks, passes CE025, and +silently corrupts `EarlyStopWatcher`'s deferred fail-stop, verdict latching and +`_prev_verdicts` flip-attribution — it latches a verdict the run then contradicts. See +GitHub issue #61 item 2. Why replay is the only sound check is in +`contracts.md § The live_verdict contract` (subsection "Why replay is the only sound +check"); it is not repeated here. + +### Design choices + +Fixtures are mandatory. A property test over random trajectories returns `"undecided"` +almost always and passes vacuously. So each live type supplies cases in `CASES`, and +`missing_case_types`, driven by the `SuccessCriterion` union like CE025, fails when a +new `LiveSuccessCriterion` has none. The author demonstrates the contract in the same +change that adds the criterion. + +Each case declares what it reaches. `ContractCase.reaches` pins the verdict on the FULL +trajectory, so a fixture that stops exercising its decision path (a renamed tool, a +changed regex) fails loudly instead of degrading into a vacuous all-`undecided` replay. + +Polarity honesty. `live_decidable_polarities` is documented as a subset of what the +checker's `live_verdict` can emit for that instance. A case that terminally decides a +polarity the instance does not claim is a real bug: the watcher treats that trigger as +inert while the checker decides it. + +It is a pytest class because it reasons over the criteria registry and executes checkers, +which an AST-only runner cannot do. + +### Honest limits, expanded + +The rule raises the cost of the bug and puts the contract in front of the next +implementer; it does not close the hole, and nothing short of a proof would. A plugin +copies the pattern — a `ContractCase`-style fixture plus the prefix walk — because this +module lives under `tests/` and is not shipped in the wheel; this module is the reference +implementation. The determinism probe does catch RNG and per-call mutable state. For a +`datetime.now()` dependency the monotonicity replay is a likelier tripwire, and only if +the fixture straddles the flip. + +### The prefix walk + +A raise is reported as a labeled violation (case and prefix length) instead of crashing +the walk, and the remaining prefixes still replay, so one bad prefix does not mask +breaches elsewhere. The watcher runs mid-turn, where a raise takes down the stop logic; +the shape `command_executed` pins for a malformed regex — degrade to `"undecided"`, +never raise — is the contract for every implementation. The terminal verdict is `None` +when the last prefix raised because there is then no verdict to compare against, and the +previous prefix's stale value would stack a bogus `reaches` breach on the real one. + +### Seeded permutations + +`contract_violations` walks one ordering, the one the fixture author wrote, but the +contract quantifies over any trajectory. The orderings an author does not think of are +where an order-sensitive bug hides — a verdict computed from the latest command instead +of the accumulated set can look monotone on the authored ordering and flip on a +reordering. Seeded shuffles probe those orderings for free. + +Each shuffle is renumbered because `EarlyStopWatcher._collect_verdicts` keeps its partial +trajectory sorted by `sequence_number`, so `live_verdict` never sees a list whose order +contradicts those numbers. Without the renumber the layer reports breaches on inputs the +watcher cannot construct, and it degrades to a silent no-op for any checker that sorts by +`sequence_number` itself — the shuffle sorts straight back to the authored ordering. +`case.reaches` and polarity honesty are not checked on shuffles because pinning either +makes the layer unsound for exactly the order-sensitive criteria it exists to probe. + +## CE037 + +A helper whose docstring documents a bug the live code still has is worse than none. + +The motivating `_rmtree_restrictive` explained, correctly and in detail, why +`rmtree(..., ignore_errors=True)` orphans a mode-000 reference tree, while both live +cleanup sites called exactly that. A reader auditing the cleanup path found a function +asserting the shipped code was broken, and a test that called the helper directly made the +real path read as covered. + +The rule turns "helper written, never wired" into a `make lint` failure at the commit that +introduces it, which is the only moment anyone knows where it was meant to be called from. + +Scope is narrow so the rule stays a bug detector rather than a style nag: methods are +reached via `self`, which a name search cannot see; public names have out-of-tree callers; +dunders are protocol; a decorator (`@register_criterion`, `@field_validator`, +`@app.command`) is a registration. A whole-tree text search rather than an import graph +means a helper referenced anywhere — called, passed as a callback, aliased — counts as +wired, and a name that merely appears in a docstring is an accepted false negative for a +rule that must never block a legitimate refactor. + +## CE038 + +In the leaking shape the awaiting coroutine receives `CancelledError`, propagates it out +of `__aenter__`, and never reaches the `finally`. + +The motivating bug held a reference directory at mode 000 with no matching restore: the +directory stayed unreadable for the rest of the run, and a stale registry entry poisoned the +next window on the same path. The comment above the acquire claimed shielding prevented +exactly that. The rule requires all four conditions so it stays specific. + +## CE039 + +A gating `score=0.0` is a strict AND term in `all_criteria_passed`, and it flows into every +downstream count that consumes criterion scores: `CriterionAggregate` mean/median, +`suite_thresholds` gates on dataset-fanned suites, run and experiment pass rates, the JUnit +report, the evalboard. + +The motivating case: a typo in `reference_comparison.reference_file` raised +`FileNotFoundError` (an `OSError`), which became a gating 0.0 and counted against the +agent's pass rate. It silently zeroed every row of a dataset-fanned suite while looking +like a genuine similarity failure. + +## CE043 + +`result_tokens` is the cost simulator's cache-independent measure of tool-output size, so an +agent that clips captured output before storing it silently under-reports every tool +result. The Codex agent shipped with `f"Output: {output[:100]}"`, which pinned ~77% of its +Bash results at ~31 tokens and skewed the cost model. + +Storing the output whole is safe because the harness's own exec-output truncation already +bounds it. + +## CE044 + +Eight fields are byte-identical duplicates across `marketplace.json` and `plugin.json` +(`name`, `displayName`, `description`, `keywords`, `author`, `homepage`, `repository`, +`license`), and nothing else compares them: the only other test that reads `plugin.json` +is `tests/test_action_version_pin.py`, and it reads only `version`. A one-sided edit, such +as retitling the plugin in the marketplace but not in the manifest, ships silently and +shows two different one-liners in the wild. + +The allowlist half is the one that has already bitten. The marketplace schema allows both +`keywords` ("Tags for plugin discovery and categorization") and `tags` ("Tags for +searchability and discovery"); the plugin-manifest schema has no `tags` property. Split +discovery strings drop half of them from the installed copy, and leave a future editor +with no rule for which list a new term belongs in. The `MARKETPLACE_ONLY` allowlist keeps +that decision in code, not in tribal knowledge. + +## CE045 + +Probed against the real CLI, from a cwd that is not the skill's own repo (project +discovery would otherwise find the skill regardless of `--plugin-dir`, and the namespace +prefix is the real signal): + + claude --plugin-dir /skills -> nothing + claude --plugin-dir -> `root:probe-beta` + +The cost is invisible and total: every activation suite the plugin generated reported +recall 0.0, which the bundled template's own comment calls "reads exactly like a broken +skill", and `ci` wrote the same path into users' SCHEDULED workflows, where it renders as a +permanent red indistinguishable from the drift the schedule exists to detect. + +Six wrong-value lines across five files shipped at once: `docs/PLUGIN.md`, tutorial 07, +`activation.yaml` (comment and example), `check-skill`, and `ci`. Nothing held them in +agreement, which is why they drifted together. + +## CE046 + +`Agent.get_environment_info` emits the `system_prompt_semantics` marker from the ClassVar of +the same name, so every run, including out-of-tree SPI agents, records which system-prompt +regime built its prompts. Dashboards read an ABSENT marker as "a run from before the marker +existed" and pool it into a legacy bucket, so a bare-dict override does not merely omit a +key: it mis-buckets every run of that agent. + +The motivating bug: `OpenCodeAgent.get_environment_info` returned `{"opencode_model": ..., +"opencode_pure": ...}` with no `super()` spread, so no OpenCode run carried the marker and +no test caught it. The base's docstring states the contract ("Overrides should spread +`super().get_environment_info()` rather than returning a bare dict"); the rule makes it +mechanical. + +## CE047 + +OpenCode shipped while four of the seven surfaces still listed three harnesses, and +nothing failed. + +The roster is restated in prose on surfaces that nothing ties to the code, so adding a +harness means remembering all seven. A reader, a crawler or an LLM answering "which agents +does Coder Eval support?" is told the missing agent does not exist, and the surface quietly +under-sells the framework. The rule only makes "we forgot this harness exists" impossible to +ship. + +Plugin agents are out of scope because a built-in is what these surfaces promise. + +## CE048 + +The failure is silent, which is what makes it worth a rule. `evaluate`'s +`in_place: bool | None = typer.Option(None, "--in-place/--copy")` reads as "no preference" +and selects copy-vs-in-place from the target shape. Called in process, the tests graded in +place and the default they meant to cover was never exercised. + +The fix is the `run_pipeline` / `run_evaluation` / `run_plan` split. In each half the +Typer signature is a thin wrapper and the body is a plain function with real Python +defaults. Tests are in scope because that is the only place the defect occurs. + +## CE049 + +The motivating bug: `build_task_event` published +`Score = float(result.weighted_score or 0.0)` on every `CoderEval.Task.End`. Four App +Insights tiles compute `avg(todouble(customDimensions.Score))` with no status filter, so one +`coder-eval execute` night dragged every score tile toward zero, indistinguishable from a +genuinely bad night. `orchestrator.py` documents the same hazard in prose ("every +downstream `score or 0.0` would launder it into a real-looking failure"); the rule makes it +mechanical. + +## CE050 + +The motivating bug: `orchestration/regrade.warn_on_embedded_commands` — the only +disclosure of what shell a rebuilt, untrusted run config would execute on the grader's +host — probed with `getattr(c, "command", None)`. Besides being rename-fragile it +structurally could not name `agent_judge`, the criterion that spawns a tool-using agent +and so has the widest blast radius of all. `models/tasks.py` states the same convention +in prose ("isinstance narrowing, NOT getattr(c, 'files'/'command')"); this rule promotes it +to a gate. + +A field-name-only rule would fire a dozen times on agent code that legitimately probes raw +SDK event objects for `command`/`tool`/`prompt`, and would be turned off within a week. +Scoping to the receiver's name catches the real shape (`for c in task.success_criteria: +... getattr(c, "command", None)`). + +## CE051 + +The driver IS the isolation boundary. Rewriting `docker` to `tempdir` behind the caller's +back does not degrade gracefully: it moves execution from a container onto the operator's +own machine, where the task's criteria address paths and toolchains that do not exist. +They score 0.0, the row is written back FAILURE for a trajectory that passed, and the same +commands (`rm -rf /verifier`, `mkdir -p /logs/verifier`) run unsandboxed on the grading +host. + +The motivating bug: `regrade.grading_sandbox_config` rewrote the driver unconditionally +on BOTH new grading entry points, which also neutralized the `driver: docker` refusal in +`Sandbox.adopt` — a guard added in the same change specifically to catch this. The +legitimate suppressions are the in-container rewrite in `run_task_internal_command` and +the opt-in host-grading branch, which refuses by default and stamps `graded_on_host` on +the row. + +## CE052 + +`run_task_internal_command` armed its heartbeat watchdog — a daemon thread whose whole +authority is `os._exit(137)` — unconditionally. A test that invoked the command in-process +left the pytest worker holding that thread, which exited the worker 40 s later inside an +unrelated test file, naming a different test on each run and on each platform with no +traceback. The dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, +naming neither the test nor the cause. + +`os._exit` also drops buffered IO. Outside the container's own disposable main process, +its outcome is not degraded, it is unattributable. + +The test that armed the watchdog invoked `run_task_internal_command` in-process on +purpose: the command must refuse a malformed `context.json`, and asserting that means +calling it. The 40 s is the 20 s grace plus the 20 s stale window. The failure was +invisible at low load: with 14 local workers the file finished and the run ended before +the timer fired, so it reproduced only on CI's 2 workers. + +`CODER_EVAL_IN_CONTAINER` is the established in-container predicate +(`Sandbox.enforces_permission_windows`, `orchestration/evaluation.resolve_reference_dir`). +A lexical check is enough because the missing property was a guard written down at the +site. + +## CE053 + +The rule was widened to the four run-log names after the same shape recurred: `docker.log` +was produced in `isolation/` and consumed in `orchestration/` as three unrelated literals, +and because the consumer guards its copy with `is_file()`, a rename would have silently +discarded the only record of why a grading container failed. The original defect: +`TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals +stayed unmigrated, including all three `rglob("task.json")` sites the constant's own +comment cites as its reason to exist, so it created the second source of truth it argues +against. + +Two half-copies of the same string in different packages is how a rename becomes a silent +no-op on the sites it missed. The unmigrated literals sat in `orchestrator.py`, `batch.py`, `docker_runner.py`, +`reports.py`, `reports_junit.py`, `reports_stats.py` and `report_command.py`, while only +the new modules used the constant. A rationale that only a human remembers is not a rule. + +## CE054 + +The `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: +`.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user +guide both described it as protection it never provided. + +The reader was `verify_reference_unchanged`, which read +`environment_info.get("reference_digest")` to refuse a re-grade whose answer key had +changed. A whole-tree grep found exactly one occurrence of the key — the read itself — and +every automated gate in the repo was green. + +## CE055 + +Two in-tree tasks shipped an absolute criterion `path:`, and the pair is the argument for +a static rule on top of the runtime `CheckerMisuseError`. `byod_smoke_test` IS in a CI +bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not +exist" for a file that existed. `dockerfile_build_example` is in NO bucket, so nothing ran +it and no runtime guard was ever reached. + +The two broken tasks: `tasks/byod_smoke_test.yaml` checked `/opt/byod_marker`, and the +real cause sat in a warning inside a task log; +`tasks/dockerfile_build_example/dockerfile_build_example.yaml` checked `/opt/greeting.txt` +and `/opt/secret_check.txt`. + +## CE056 + +The CE053 shape again: a rename-safety constant shipped beside the literal it replaced, +and the straggler was the single WRITER, so a rename would have disarmed four +security/correctness gates at once with nothing failing. + +`models/container_paths.py` states why the constant exists: "two half-copies of the same +string in different packages is how a rename becomes a silent no-op". Every reader moved +to `IN_CONTAINER_ENV`; the writer, `docker_runner`'s `--env CODER_EVAL_IN_CONTAINER=1`, +did not. A rename would have left the container exporting the old name, and each gate +would read "not in a container": + +- `Sandbox.enforces_permission_windows`: the reference-solution anti-cheat window stops + being applied, and an unprotected run scores like a protected one; +- `resolve_reference_dir`: the `/work/references` branch is skipped; +- `_should_grade_in_container`: a grading container dispatches another grading container; +- the orphan-container heartbeat watchdog's `os._exit(137)` gate. + +Prose that names the variable is left alone because a docstring or error message names it +on purpose, and the rule must not push authors to obfuscate their own explanations. + +## CE057 + +The failure is silent: the sidecar runs where `coder_eval` and its dependencies are not +installed, so one package import makes every shadowed CLI die with an ImportError the +agent reads as "the tool is broken", costing a whole run to diagnose. + +Import-time enforcement, a test that renders and executes a shim, catches the defect only +when some test happens to declare a response rule; the lint rule catches it the moment the +import is written. `STDLIB_ALLOWED` stays small because every entry must exist in whatever +interpreter the sandbox's shebang resolves to. + +`from __future__ import ...` is not an import hazard: every interpreter that can run the +shim supports it. Widening the allowlist for it would retire the rule's own guard on +future-import syntax. + +## CE058 + +The `model_copy(update={...})` dict form exists because the Antigravity DONE path writes +through that shape, which a keyword-only rule cannot see. Antigravity constructed EVERY +message with `generation_duration_ms=0.0`, so the task page's Generation cell read `0ms` +and its breakdown rendered `0%` for months with nothing failing. Codex published the SDK's +`0.0` as a measured command duration, so `avg_command_time_ms` divided real milliseconds +by a command count of which 70 of 211 in one nightly had never been timed. The fourth form +is the one no existing rule shape covered, and it is where a live instance was hiding: +`claude_code_agent._finalize_commands` set `0.0` on every command force-closed without a +tool result, in the one harness a timing audit had called healthy. + +Every consumer downstream of a timing field (an average, a breakdown percentage, a +timeline cell) treats an invented literal as a measurement. This is the same reasoning as +CE049 on the score side. + +`TurnRecord.harness_startup_ms` and `harness_teardown_ms` are the turn's head and tail +buckets, and they are the same invariant one level up. A turn whose stream carried no +assistant message was never timed at either end. A `0.0` there claims the harness started +instantly, and that reading sends a real gap into the evalboard's `Unaccounted` cell while +a named bucket says it was measured at zero. `tool_union_ms` is the third bucket, under +the same None-vs-0.0 contract. A measured `0.0` remains a legitimate answer: a window +subtracted down to nothing by the tool execution inside it, or a clamped inversion where +both ends really were observed. + +A sixth form, `cmd.duration_ms = 0.0` as a plain assignment, exists because form 4 passed +the live `_finalize_commands` defect only by coincidence. Form 4 keys on the `if` test +naming a timing attribute, and the shipped bug spelled it `if cmd.duration_ms is None:`. +But the assignment sat inside an outer `if cmd.result_status is None:` block. Setting the +literal under THAT guard instead reads just as naturally, books the identical lie, and is +invisible to forms 1-5. A guard is evidence about the value only when it names the value; +with no such guard there is no evidence at all, which is strictly worse. + +Form 6 uses `_zero_literal`, not form 4's broader `_numeric_literal`. Under `if x is None` +the guard proves the value was never measured, so any invented number is a defect. A bare +assignment proves nothing: `cmd.duration_ms = elapsed_ms` is how a measured value is +written, and a literal `1234.0` is a legitimate test factory or replay. Only the +placeholder zero is the tell, the same narrowing form 1 makes for the same reason. + +## The CE id space + +CE062 IS DELIBERATELY UNUSED and must stay that way — the ids in `runner.py` jump 061 to +063. It was claimed during the turn-timing work and then folded into CE063 rather than +shipped. An id is a permanent documentation anchor: a suppression comment carrying 062 in +an older branch, review or commit message must never start meaning something new. CE023 is +retired the same way, after the rule was deleted with the package it guarded. + +Claim 068 next, and note 065 IS TAKEN without being in `ALL_RULES`: doc-surface and +whole-tree rules are `@pytest.mark.lint` classes in `tests/test_custom_lint.py` rather than +`BaseRule`s, so `runner.py`'s uniqueness assert cannot see them. Enumerating them in a +comment is how that note fell behind CE044, so grep instead: +`grep -E '^class Test(CE[0-9]{3})' tests/test_custom_lint.py`. Spell it `[0-9]`, not `\d` +— GNU and BSD `grep -E` read `\d` as a literal `d` and report zero hits, which reads as +"no ids taken". The whole id space is unioned in one place, by +`TestRuffExternalCoversEveryRule._known()`. + +### CE058 field families + +THREE field families, not two. `tool_union_ms` is the turn's third wall-clock bucket, on +the same model and under the same None-vs-0.0 contract as the `harness_*` pair — and it +matched NO arm of `_TIMING_NAME`, so `TurnRecord(tool_union_ms=0.0)` was invisible although +`TurnRecord` was already in `_TIMING_CONSTRUCTORS`. Naming the field +`tool_union_duration_ms`, to inherit the generic `_duration_ms` arm for free, was +considered and rejected: the two fields beside it needed their own arm for exactly this +reason, and one spelling across the four buckets is worth two lines of regex. + +The `_startup_ms` / `_teardown_ms` arms need a leading segment for the same reason the +`_duration_ms` arm does: the shipped fields are `harness_*`, and a bare `startup_ms` is +more likely a budget than a measurement. + +## CE059 + +The Antigravity reducer read `datetime.now()` once and passed it as both bounds, so +`started_at == completed_at` on 368 of 368 sampled messages. It is a separate id from +CE058 because it is a separate invariant — a zero-length window whatever the duration +field says — and one invariant per id is what makes a `# noqa` mean one thing. The +`generation_duration_ms=None` exemption keeps the rule pointed at the misleading case +instead of accumulating four permanent suppressions on the rollout-rebuild and +sub-agent-synthesis sites. + +Every consumer that derives a window from the two stamps saw nothing at all on those +messages; a window needs two reads at two moments. The exempt sites are Codex's rollout +rebuild and the sub-agent syntheses on Codex and Claude Code, where the generation arrives +as a tool result and is never streamed, so collapsing both bounds to one `now()` is a +formatting choice, not a false measurement. + +The two-names blind spot has a live shape in Codex: `started = +_ms_to_dt(self.open_start_ms)` and `completed = _ms_to_dt(self.open_end_ms if ... is not +None else self.open_start_ms)` collapse to one instant whenever `open_end_ms` is None. + +## CE060 + +An identity invariant, which is why it is its own id rather than a second arm of +CE058/CE059, both of which are about timing. Antigravity omitted the kwarg, so the field +defaulted to `None` on every message it ever recorded, and the evalboard — which groups +assistant emissions by `message_id` and falls back to a `SAME_EMISSION_GAP_MS` wall-clock +gap when either side lacks one — collapsed a whole turn's generations into ONE timeline +row as soon as the harness's generation windows became contiguous (the gap is then exactly +0 ms, always). Nothing failed: the consumer SUMS the group, so the totals and the +reconciliation invariant stayed right, and the golden snapshots had ratified the `null` on +the day they were written — a snapshot is regenerated from whatever the code currently +does, so it catches a later change and never an initial omission. The damage was not +confined to the timeline, which is why "only granularity is lost" was the wrong way to +describe it: a grouped emission is one API call to the evalboard's thinking-cost +simulator, whose prompt-cache cascade is quadratic in that count, so a single-shot +Antigravity run had every cascade coefficient pinned at zero; the `Messages` count and the +10 s slow-generation bar were per-turn too. Deriving the constructor set from each +module's own imports closes exactly the blind spot CE058's clause concedes: +`claude_code_agent.py` binds only `AssistantMessage as AssistantMessageTelemetry`, so a +name list guards that file's two construction sites purely by coincidence, and an +arbitrary `as Msg` is missed outright. Widening CE058/CE059 the same way changes two +shipped rules, needs its own mutation checks, and is recorded in +`.claude/harness-candidates.md`. + +The mechanism and its blast radius (evalboard grouping, the thinking-cost simulator) are +specified in `docs/agents/HARNESS_PARITY.md` § Timing capture. + +The shared resolver, and why it lives outside this rule, is under `_model_ctor` below. + +On the runtime `None`: `opencode_agent.py` passes `str(part.get("messageID") or "") or None` +and `pi_agent.py` the same shape for `responseId`. The golden snapshot +`pi_a_single_text_turn.json` carries that shape, but its `null` comes from a fixture that +emits no `responseId`, not from a live CLI omission. + +A `**`-expanded call has no carve-out because no site in `src/coder_eval/agents/` uses +`**` expansion for these constructors. + +## CE061 + +Pi shipped measuring its window from its own `turn_start` while four sibling reducers +tiled from a mark, so the wall clock between one turn's end and the next turn's start (the +model time that PRODUCED that turn) fell into no bucket at all. Why nothing failed: +timing.md § Why the ms-exact identity contract exists. Pi's own tests passed because they +were written against Pi's own arithmetic. The hazard is therefore not a reducer that +computes the window wrongly but one that computes it AT ALL: a new harness whose author +reimplements the arithmetic inline arrives with a green test suite by construction. +`tests/test_timing_identity_contract.py` is the two-sided check. + +A separate id from CE058, CE059 and CE060: those are about the VALUES a message carries — +an unknown duration published as a literal, a window built from one clock read, a missing +identity. This one is about PROVENANCE, where the arithmetic came from, and one invariant +per id is what makes a `# noqa` mean one thing. + +Proving the import is the cheap structural half that the arithmetic tests cannot reach — a +sixth harness rolling its own. + +Tool subtraction is not part of a window's geometry: a call issued by an earlier emission +can still be running when the next window closes, and folding that into `close_window` +would put a mode flag on a helper whose value is having one shape. The collector already +knows every span, so `timing.subtract_tool_time` removes tool time once for every harness. +claude-code calls the same `close_window` as the other reducers, with no suppression. The +explicit-`None` exemption covers codex's rollout rebuild and claude-code's sub-agent +synthesis. + +## CE063 + +Five reducers each subtracted tool time themselves while the head and tail were computed +centrally at the same seam, and that asymmetry is where every timing defect on this branch +lived (measured cases: agents.md § Per-harness generation marks). A sixth harness reaching +for `busy_ms` rebuilds that, and the double subtraction under-reports generation on one +harness only, which takes a corpus comparison to notice. It is a separate id from CE061 +rather than a rebody: CE061 asks where a window's ARITHMETIC came from, and all five +reducers call `close_window`, so its property is live and unsuperseded; this asks whether +a reducer subtracts at all. All five reducers today import `close_window` by name, so a +ban built from `_imports_the_helper` would fire on the first working call site spelled +`timing.close_window(...)`. + +In `_imports_the_helper`'s own words, a rule that missed `timing.close_window(...)` "would +tell an author to change a working call site." CE063 therefore keys on the `busy_ms` name +binding plus an `ast.Attribute` match for `timing.busy_ms`. + +`tests/test_timing_identity_contract.py` drives every harness off a scripted clock and +asserts that the four buckets tile the turn to the millisecond. CE063 adds only the cheap +structural half that a static check can reach. + +## CE064 + +The turn's OUTER bounds are something no other rule looks at: CE058-CE061 all scope to +`AssistantMessage`, and the bracket is not one. All three clocked harnesses derived their +window bounds from the `TurnClock` and let the bracket fall back to `datetime.now`, +putting a monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact +split `TurnClock` exists to remove, reintroduced at the one seam the clock did not own. +Measured on a live antigravity turn: an `AgentEndEvent` stamped **17 us BEFORE its own +last message finished**, which cannot happen (the event is constructed strictly after the +final flush), and `decompose_turn` clamped that negative and published `0.0` — "measured, +and instant", the CE058 confusion arrived at from the other direction — for a harness +whose real tail is ~0.1 ms. It surfaced on one harness only because the drift is tens of +microseconds and antigravity is the only one that holds its process across turns, so +nothing happens between its last flush and its end event; every other harness books a tail +of 7-543 ms, where the drift is invisible rather than absent — which is why the fix is at +every clocked site rather than at that one. Codex and opencode take their spans from the +CLI's own epoch stamps and deliberately have no `TurnClock`, so a raw bracket is +CONSISTENT with their bounds. `tests/_bracket_clock.py` anchors its stand-in a year from +real time, so a reverted argument fails by a year rather than by the microseconds that +separate the two clocks — the division of labour CE060 states: a rule removes the SILENT +case, a default nobody chose. + +The measurement came from instrumenting `decompose_turn` on a live antigravity turn: + +``` +PROBE tail: elapsed=-0.017000ms busy=0.000000ms raw=-0.017000ms + last_completed = 09:05:22.033099 + agent_end = 09:05:22.033082 +``` + +Every other harness also books a head of 0.2-6 s. The fix belongs at every clocked site +because that makes the subtraction single-basis rather than usually-close, and +"usually-close" is not a property a millisecond field can rest on. The noop agent has no +windows at all. + +The three clock expressions are a local `clock` in `communicate`, `state.clock` from the +caller, and `self.clock` inside the state. Demanding one spelling would make the rule a +syntax check on three harnesses' internal structure. + +`tests/_bracket_clock.py::AnchoredClock` advances on the REAL monotonic clock instead of +stepping by hand. That is what lets the same fixture also assert the head and tail: with +the bracket and the window bounds on one basis, `decompose_turn` returns small positive +measurements instead of the clamped `0.0` the cross-basis subtraction produced (the PROBE +above). + +The two ends of the bracket fail differently, so `assert_overhead_is_measured` needs one +assertion for each. A defaulted `AgentStartEvent` lands ~365 days before the clock-derived +first window, so the head blows any sane UPPER bound by the whole offset. A defaulted +`AgentEndEvent` lands ~365 days BEFORE its own last message, so `decompose_turn` clamps the +negative to `0.0` — "measured, and instant", which passes an upper bound. Only a strict +`> 0.0` catches it. It holds on all three clocked harnesses because real work separates a +turn's last flush from its end event. The margin is smallest on antigravity, which holds +its process across turns: it measures 0.007-0.035 ms there, 7-35 ticks of the 1 us +resolution that both `datetime` and `time.monotonic()` have on Linux, macOS and Windows. +That is the magnitude the clamped defect hid, which is why `>= 0.0` is not an acceptable +relaxation. + +## CE065 + +`evalboard/lib/pricing.ts` used to carry a hand-copied mirror of the Python rate card. +Keeping a hand-copy honest needed five layers of bookkeeping: a regex parser that re-read +`pricing.py` at test time, a meta-guard against that regex silently narrowing, a +`DELIBERATELY_UNMIRRORED` exemption set, a staleness guard for the exemption set, and a +comment begging the next reader to keep the set honest. It still shipped a real bug — +`claude-sonnet-5`, `gpt-5.6-sol`, `gpt-5.6-terra` and `gpt-5.6-luna` sat in the exemption +set under "the evalboard never runs them" while appearing tens of thousands of times in the +run corpus, so every one of those runs rendered `—` for cost with nothing failing. + +If a *test* can read the table, a *generator* can emit it. + +The old exemption set encoded TWO different things, and they survive differently. That +three OpenRouter models must stay unpriced, so that `runs.ts`'s apportionment of the +provider's real bill still fires, is a property of the RATE, so it is now data on the rate +itself (`ModelPricing.per_request_billing`) and nothing has to remember it. That four +heavy frontier variants are not priced on the board is a property of the FRONTEND, so +`DELIBERATELY_UNMIRRORED` keeps the stale-membership guard the deleted test carried — an +exemption nobody re-reads is what shipped the bug above. + +### Keeping DELIBERATELY_UNMIRRORED honest + +Membership silences the mirror for one id indefinitely, so a stale entry hides a live bug +rather than a non-issue. Before adding an id, grep the run corpus for it — absence from run +data is the ONLY justification, and it expires the moment a harness adopts the model. +`_assert_exemptions_are_live` fails the build once an id leaves `pricing.py`. + +## CE066 + +A metric, a statistic and a serializer that core needed were pulled out of `reports*`, which +put `turn_time_buckets` and the run.json serializer in a rendering module. They now live in +`result_metrics.py`, `stats.py` and `run_record.py`. + +Core legitimately writes reports: `orchestrator.py` writes the per-task HTML and +`orchestration/batch.py` drives `ReportGenerator`. + +The allowlist is purely writers, and an allowlist rather than a denylist is the CE018 +rationale. `eval_result_to_task_dict` is deliberately absent, because carrying a +serializer on it would be the rule documenting a wart instead of the wart being removed. + +Checking the RELATIVE spelling is not a detail. The relative form is the local idiom — +both surviving edges are `from .reports import write_task_html` and `from ..reports import +ReportGenerator` — and an earlier draft matched only `node.module`, which for a relative +import holds `"reports"` with the dots in `node.level`. It fired on nothing the codebase +actually writes, and its own tests passed because they used the absolute form. An unrun +assertion is documentation, not enforcement. + +## TestRuffExternalCoversEveryRule + +`[tool.ruff.lint] external` is what stops ruff reporting RUF102 "Invalid rule code" for a +suppression it does not own. It was hand-maintained and had fallen ~14 ids behind — +including CE054 and CE048, whose own docstrings advertise `# noqa: CE054` / `# noqa: CE048` +as the supported escape hatch. The first person to use the documented exemption got a red +`make check` for doing exactly what the rule told them to. + +The list then drifted a SECOND time, and this class is why it drifted quietly: it read +`ALL_RULES` alone, so it could not see a rule that is a `@pytest.mark.lint` class rather +than a `BaseRule`. CE044 and CE065 are both such rules, both were missing, and only CE065 +was noticed — by a human reading a diff. Nothing was red for want of those two entries (no +`# noqa: CE044` or `# noqa: CE065` exists in the tree), so the fix was pre-emptive. + +The reverse direction matters because a declared id for a deleted rule is the +exemption-set rot that the generated pricing table exists to remove. + +## TestRunRecordFieldVocabulary + +Both run-analysis surfaces shipped six wrong field names at once (`turns`, `total_tokens`, +`assistant_turn_count`, `max_turns`, `criteria_count`, `all_criteria_perfect`), and the +failure is worst exactly where the instruction applies: the >20-task path, where the agent +is told NOT to fall back to reading whole files. + +Unioning the run-level and criterion-level models is a weakening that still catches every +one of the six shipped names, since none of them exists on any of those models. + +## _layers + +A second copy of "where does this file sit in the package" is how a package added to one +regex silently escapes the other. The two rules keep separate exemption sets because they +do not ask the same question — see [CE004](#ce004) for the cycle that inheriting one +opened. + +The denylist form left holes twice over. An earlier draft named only `orchestrator.py` as +the top-level core module, which exempted `result_metrics.py` — the very module CE066's +fix message tells a violator to move their metric into — along with `run_record.py`, +`stats.py` and `timing.py`. Its successor listed ten core directories and `isolation/` was +not one of them, so `isolation/docker_runner.py`, the `driver: docker` evaluation path, +could import anything with both rules silent. + +The package regex is anchored on `src/` because the unanchored form made a repo-root file +core: this project's own checkout directory is named `coder_eval`, so +`…/coder_eval/conftest.py` matched the package. A clone under a parent directory literally +named `src` still matches, and so does that clone's `tests/` tree. No path substring +separates the package from a checkout laid out like it; closing it properly means +relativising every rule's path against the repo root. It is unreachable today because +CE004 and CE066 are only ever handed paths under the runner's `SRC`, and +`_ALSO_SCAN_TESTS` is `{"CE048"}`, which uses neither predicate. + +## _model_ctor + +CE060 and CE061 ask the same first question: is this call building an `AssistantMessage`? +CE060 worked out the answer first. Duplicating the resolver into CE061 means a model +rename or a new import spelling needs two fixes in two rules, and the second fix is the +one that gets missed. CE064 asks the identical question about +`coder_eval.streaming.events` and `coder_eval.timing`, which is why `bindings_from` takes +the module as a parameter instead of a third copy. + +Taking the class name from the model follows CE056 importing `IN_CONTAINER_ENV` and CE057 +deriving its target set from `SIDECAR_MODULES`. Alias resolution removes the *local +binding* spelling, not every rename; taking the name from the model covers the rest. diff --git a/.claude/notes/orchestration.md b/.claude/notes/orchestration.md index 733aab932..a3042f95f 100644 --- a/.claude/notes/orchestration.md +++ b/.claude/notes/orchestration.md @@ -80,6 +80,12 @@ unscored. Nothing is lost by deferring: the fact lives on `result.max_turns_exha which the seeding carries. The statuses that ARE execution facts differ in kind — they abort the run before a verdict is reachable, so preserving them overturns nothing. +The `FinalStatus.is_execution_fact` table must agree with this order: +`_EXECUTION_FACT_STATUSES` maps MAX_TURNS_EXHAUSTED to `False` (the defect a disagreement +produces is under isolation.md § Detached grading and `Sandbox.adopt`). +`tests/test_seed_from_prior_result.py` pins the status in a test of its own, apart from +the loop over the real execution facts. + ### The four grading sites `grade=False` is checked in exactly four places, and they do not behave alike: @@ -821,5 +827,3 @@ bucket was a manual step no checker could verify — an untyped `dict[str, int]` neither that every category is present nor that no stray key is — while the `-1` default it leaned on was already unreachable, and, though documented as "fail-closed", sorted BELOW error, so a fifth category would have silently outranked ERROR as the worst status. - - diff --git a/.claude/notes/permissions.md b/.claude/notes/permissions.md index 680509b60..fc1d64c3e 100644 --- a/.claude/notes/permissions.md +++ b/.claude/notes/permissions.md @@ -89,6 +89,20 @@ static reference mid-turn cannot break the `LiveVerdict` monotonicity contract (contracts.md § The live_verdict contract); reading the half-written sandbox can, and is the "end-state peeking" `live_verdict` rules out. +## Why the chmod tests are Linux-only + +`tests/test_reference_permissions.py` drives `chmod` against the HOST filesystem, and +Windows `chmod` honours only the read-only bit, so mode 000 never takes and every +assertion reads back 0o555/0o777. + +This is NOT a coverage gap for Windows users. The window is enforced only when +`CODER_EVAL_IN_CONTAINER=1`, which only `DockerRunner` sets — and Docker Desktop on +Windows runs LINUX containers (WSL2), so the in-container orchestrator that performs the +chmod is on Linux and behaves exactly as these tests assert. A Windows host only ever sees +the window under `driver: tempdir`, where it is a deliberate no-op regardless of platform. +The real behaviour is covered by the Linux CI jobs and by `tasks/anti_cheat_reference`, +which runs in the container. + ## Locking and crash safety The registry is keyed by the *resolved* path so a directory reached by two different diff --git a/.claude/notes/reporting.md b/.claude/notes/reporting.md index a694d49a8..a8fdf1b01 100644 --- a/.claude/notes/reporting.md +++ b/.claude/notes/reporting.md @@ -68,10 +68,38 @@ on PyPI. verify-published-action.yml then verifies the published composite (tag/pin/PyPI/Marketplace parity, plus a real consumer run) after each Release and nightly. Runbook: CONTRIBUTING.md § Releasing. +### Why the action argv tests run the shipped script + +`tests/test_action_inputs.py` runs the real `run:` bodies from `action.yml` and does not reimplement them. The two bash steps build a `uv tool install` and a `coder-eval run` command line from string inputs, and every failure there is silent. If an extra drops out of the requirement string, the install gives a working CLI that has no agent. If word splitting or pathname expansion changes a value, the CLI gets a different value than the workflow wrote. In both cases the run measures something else and still exits 0. A test that keeps its own copy of the script tells you nothing about what consumers get. + +`args` is one argv entry per line, appended verbatim, so a `-D` value like `key=[A,B,C]` survives. That value is a bash character class. If the input were split on whitespace, bash would silently replace the value with a single name whenever a file in the working directory matched. + +The argv-recording stub is a bash script, not a Python one. On a Windows runner, `shell: bash` is Git Bash. Git Bash rewrites arguments that look like absolute POSIX paths when it passes them to a native Windows binary, so a Python-shebang stub receives `/action-checkout` as `C:/Program Files/Git/action-checkout`. Turning that conversion off does not fix it, because then the shebang launcher cannot pass Python its own script path. A bash stub never crosses into a native binary, so argv arrives byte for byte on every platform. The stub writes argv NUL-delimited rather than as JSON, so a value with a quote, a backslash or a space needs no escaping when it leaves bash. `CE_PROBE` is how the env-passthrough test sees what the child received. The passthrough is collected into an array and handed to `env -- … coder-eval`, never exported into the step's own shell, so only a process the script starts can report the value. + +Collecting rather than exporting matters because the loop is line-based, so an `env` value carrying a newline splits into a second `NAME=VALUE` entry. Exported, that entry could overwrite `CE_ARGS`, `CE_RUN_DIR` or `GITHUB_OUTPUT`, which the step reads AFTER the loop — an argv rewrite. Collected, it reaches the child as data only. No hostile author is needed: any interpolated value or a rotated multi-line secret does it. `test_a_newline_in_a_value_cannot_rewrite_the_step` pins it. + +### The claude-pr-review hardening invariants + +`claude-pr-review.yml` runs with privileges over attacker-controlled PR content. `tests/test_pr_review_workflow.py` pins its hardening, so an edit that weakens it fails the build instead of passing silently. + +`include_comments_by_actor` is a hand-maintained copy of the CODEOWNERS `*` owners. If the two drift, a maintainer's review guidance silently disappears from Claude's context. The tool allowlist bans tools that give secret, re-ingest or network reach. For example, a shell `cat` can read a token persisted in `.git/config`, and `gh pr view` reads every comment verbatim, which bypasses the actor allowlist. `persist-credentials: false` removes the on-disk token that the action's own `git fetch` needs, so an env-only credential helper restores auth. The test checks both halves. If the helper is dropped, fetch fails with "could not read Username". If a literal `secrets.*` goes into the helper instead of an env reference, the token is written to disk again. + +### Couplings of verify-published-action.yml + +You cannot run `verify-published-action.yml` before merge, so every link it has to another file is a place where a rename passes `make verify` and the gate silently stops working in production. `tests/test_verify_published_workflow.py` gives each link a test that runs before merge. + +GitHub does not report an error when `workflow_run: workflows: ["Release"]` names a workflow that does not exist. The trigger never fires, and the gate falls back to the nightly schedule with no signal. The workflow derives the Marketplace slug with a shell pipeline. That is a second slugger next to `marketplace_slug`, the one CE026 uses for doc links. The two agree only because `action.yml`'s `name:` is `coder_eval`, the one input that both leave unchanged. The `# <-- kept in sync` pin anchor has three readers with different whitespace tolerances, so after a reformat one reader can report "parity OK" for a pin that another reader did not bump. The inline consumer task YAML is a full `TaskDefinition` document. CE029 checks that shape in Markdown, but nothing checks it in the workflow, so a field rename or an `extra="forbid"` violation would show up only as an unclear failure in the paid nightly run. + +### Why the derived version pins are tested on every commit + +`pyproject.toml` is the only source of the version. Two files hold a pin derived from it, and one `release.yml` step bumps both inside the release commit. The first is the `version:` default in `action.yml`. The composite action installs `coder-eval==`, so a consumer who pins `UiPath/coder_eval@vX.Y.Z` (or the moving `@v0`) must get X.Y.Z. The second is `version` in `plugin.json`. `claude plugin validate --strict` rejects a manifest with no version, and Claude Code keys plugin updates off this value, so a stale pin leaves users on a cached copy. + +The seds run only on the release path. A hand edit, or a release that skipped the amend step, drifts with no signal. That is how `action.yml` once shipped pinned to 0.8.6 while main was at 0.8.9. `tests/test_action_version_pin.py` makes the check run on every commit. It also checks the line shape each sed matches: the `# <-- kept in sync` trailing comment in `action.yml`, and a `"version"` line with a trailing comma in `plugin.json`. If a reformat moves `version` to the last key or puts the JSON on one line, the bump becomes a no-op. A `grep -q` guard in `release.yml` catches this, but only after the tag exists. + ## The Agent ABC contract `agent.py` is the plugin SPI: everything a third-party agent author must satisfy. The -authoring walkthrough is [docs/EXTENDING.md](../../docs/EXTENDING.md) and the five +authoring walkthrough is [docs/EXTENDING.md](../../docs/EXTENDING.md) and the numbered lifecycle requirements are in CLAUDE.md § Adding a New Agent; what follows is why the seams are shaped the way they are. @@ -253,6 +281,19 @@ literal `50` beside its own `SLOW_PARAMS_PREVIEW_CHARS`, and a hand-rolled and moving them in would give the package an SDK dependency and force CE066 to exempt the orchestrator's `analysis` import. +## Ungraded rows in a rollup + +Only the SCORE is dropped from an ungraded row, never the row itself. Duration, tokens and +assistant turns are facts about the run that grading has nothing to do with, and `execute`'s +stated contract is that only the verdict is withheld. Skipping the row whole made an +all-ungraded experiment render `Avg Duration | N/A | N/A` with the Tokens and Assistant +Turns rows absent entirely. + +An earlier note in `reports/helpers.py` claimed an experiment is either entirely graded or +entirely ungraded, because `grade` is run-level. It is not: `run --resume` grades rows +independently and folds a failed one back ungraded, so mixed experiments are real. That is +why the series are consumed independently rather than index-aligned. + ## Report rollups and the HTML twin `reports/html.py` is the evalboard's STATIC TWIN: the two render the same run and must diff --git a/.claude/notes/timing.md b/.claude/notes/timing.md index 49a3ad769..879fb0a48 100644 --- a/.claude/notes/timing.md +++ b/.claude/notes/timing.md @@ -140,6 +140,91 @@ marker between those parts. Naming these for the interval they MEASURE rather th what they contain is the whole point; `docs/agents/HARNESS_PARITY.md` holds the per-harness composition. +### The golden-stream timing sensor + +`tests/_fixtures/golden_streams/_scrub.py::assert_timing_captured` is a replay-based sensor +for what an AST rule cannot see, such as an SDK that returned `0.0`. It runs on the +UNSCRUBBED dump because `scrub()` keeps `None` and masks every other value: a scrubbed +snapshot shows that a field was set, never that it was set to something meaningful. A +force-closed `"unknown"` orphan is exempt from the command check because it was never +timed, and saying so is the honest record. A scenario that resolves no command passes that +check vacuously, which is correct rather than weak. + +The bounds half of the generation check is not redundant. A reducer computes the duration +and the two bounds in separate expressions, so it can report a healthy duration beside two +stamps that collapsed to one instant. CE059 catches that +statically only when both bounds are the same `ast.Name`; two different names holding the +same value pass CE059, and this check catches them. + +The head/tail check is keyed on assistant messages with a MEASURABLE window, because that +is what the collector measures the head and tail against. Both halves of that key are +load-bearing. The `expect_generation_window` flag is the wrong key: `codex_e_orphan_tool` +streams a generation whose window subtracts to zero, so it clears the flag while still +having a head and a tail to report. "Any assistant message" is too weak: +`codex_g_items_rebuild` rebuilds its transcript from the rollout after the turn ended, with +`generation_duration_ms=None` and placeholder `now()` bounds, so there is nothing to measure +an end against and `None` for both is the honest answer. + +The head/tail check asserts PRESENCE only, which is all the fixtures support: the replays +run in ~0.3 ms of synthetic wall clock, so head and tail are microseconds and any bound or +ordering check is noise. A `>= 0` check is worse than noise — `decompose_turn` clamps with +`max(..., 0.0)`, so it would restate the implementation and could never fail. + +The four-bucket identity is the one assertion here that catches a DOUBLE-COUNT rather than +an absence (the orphaned-tool case under § decompose_turn). It is off for the +`FICTIONAL_DURATIONS` scenarios, which inject integer-millisecond SDK item durations of +17-900 ms while the replay takes ~0.3 ms of real wall clock, so no rebasing can make the +two commensurable. + +Before the identity, the stored `tool_union_ms` is compared with a union that +`_tool_union_ms` recomputes. That helper validates the dump into a `TurnRecord` and calls +the collector's own `main_thread_tool_spans` and `union_ms`, so it cannot drift on the +selection rule (the sub-agent-id derivation, the stamp parse, the `end >= start` filter). +It still builds its own span set and union — the bookkeeping where the per-reducer defects +lived (lint-rules.md § CE063) — so it verifies the producer's bookkeeping instead of +reading the producer's answer. + +It is a scenario-level floor, not a per-entry rule, because no per-entry form works against +the real snapshots. `claude_d_subagent_terminal` holds two content-bearing assistant +messages of which exactly one is legitimately `None` (the synthesized sub-agent generation, +delivered as a tool result and never streamed), so no scenario-level flag can say "this one +but not that one". "Never exactly 0.0" conflicts with the clamps that legitimately produce a +measured zero. The per-message contract lives in each agent's own unit tests; this is the +cross-harness floor. + +### Why the ms-exact identity contract exists + +`tests/test_timing_identity_contract.py` is the committed MAGNITUDE sensor for the +four-bucket identity, because nothing else in the suite is one. The golden corpus masks +every timing value and its identity check is one-sided (see The golden-stream timing +sensor above), so an UNDERCOUNT, which is the defect class this area keeps producing, +passes it silently. `scripts/timing/decompose_run.py --max-residual-pct` IS two-sided, but +needs live `task.json` files. + +Magnitudes are only real where a scripted clock makes them real, so each case drives the +harness's own reducer with a hand-moved clock and feeds its output through a real +`EventCollector`, the seam production uses to compute the head and the tail. The module +reuses the three clock-injection idioms the per-harness suites already use rather than +inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code), a +`datetime` subclass patched onto the module (opencode), and scripted SDK epoch-millisecond +stamps (codex). OpenCode needs a `datetime` SUBCLASS rather than a stub because the +reducer also calls `datetime.fromtimestamp` through the same module global to convert the +CLI's epoch stamps (see `tests/test_opencode_agent.py`'s `_SteppedClock`). claude-code +needs `time.monotonic` patched on top of the injected clock because `turn_start_time`, the +turn deadline and measured tool durations still read it; scripting only the clock leaves +the tool span a monotonic duration subtracted off a scripted reading. Codex takes its +stamps from SDK epoch milliseconds rather than any host clock, so its case scripts those +stamps directly. + +End-to-end tests that run on the REAL clock assert no RELATIVE lower bound on the four +buckets — only that each is measured and that their sum does not exceed the turn. +A relative lower bound (`>= share * turn_ms`) is a scheduler-noise detector: under parallel +load the turn's `duration_seconds` inflates while the measured buckets do not. It was one — +a `>= 0.5 *` bound in `tests/test_antigravity_agent.py::test_generation_and_tool_time_account_for_the_turn` +survived only while the sum excluded the head, and failed under `-n auto` once the head +joined it. The exact share belongs in `tests/test_timing_identity_contract.py`, on a +scripted clock. + ## main_thread_tool_spans The span set the generation subtraction, the head and the tail are all measured against, @@ -268,6 +353,19 @@ a sub-agent message therefore shrinks the head or the tail by time no other buck and Codex's recovered child messages carry the CHILD's clock, so the bracket can move either way. +### Why the query build sits outside all four buckets + +claude-code's `_build_claude_query` runs inside `communicate` before `AgentStartEvent` is +emitted, so it precedes the head's own start stamp: its cost is inside `duration_seconds` +but outside all four buckets, as unexplained residual. Measured at 0.03 ms bare and +0.10 ms with four plugin roots, which is noise, and +`tests/test_agent_telemetry.py::TestClaudeHeadIsMeasuredAtFirstOutput` keeps it that way +with a 50 ms budget, so the gap between the buckets and the turn stays the harness's +rather than ours. + +Why the head starts at the first observed model output: +[agents.md § First-generation window seeding](agents.md). + ## Why the subtraction and the head/tail may run in either order `EventCollector.build_turn_record` calls `subtract_tool_time` before `_overhead_ms`, and @@ -308,3 +406,18 @@ the message, and Codex, whose `_flush_message` window extends to the last item's `completed_at_ms`) while the other three tile the turn contiguously, so a call open at a boundary runs inside two windows. Central subtraction handles both without either reducer knowing which it is. + +### Why a coincident tool result cannot catch an un-tiled window + +The claude-code mark-reset defect (measured numbers in +[agents.md § Per-harness generation marks](agents.md)) survived its own identity case. +`_claude_turn` scripts the tool result at the instant the tool ends, so the interval the +reset discarded and the tool's own span were the SAME milliseconds: `subtract_tool_time` +removed them either way and the identity closed with or without the bug. Live probes had +the same blind spot from the other direction: three concurrent `sleep 3` calls make the +tool union so large that the round trip rounds away (measured: 0.05% residual). + +`_claude_slow_result_turn` separates the two: a 20 ms tool whose result arrives through +two user messages ~2 s apart, the shape traced off `tasks/dataset_example.yaml`. Reverting +the fix fails that case and leaves every other case in the file green. `_claude_turn` +deliberately keeps the coincident shape so the two read as a pair. diff --git a/CLAUDE.md b/CLAUDE.md index b92f39b64..977c1772e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -163,7 +163,7 @@ make docs-indexes # README/docs index tables from the mkdocs nav (CE028) make plugin-reference # the plugin's criteria reference from the models (CE033) make pricing-mirror # the evalboard's rate table from pricing.py (CE065) -make docs-budget # per-file comment budget + docstring essay check (fails `make verify`) +make docs-budget # comment-run cap + comment budget + essay check (fails `make verify`) ``` `src/coder_eval/pricing.py` is the single source of truth for rates on both halves of @@ -183,10 +183,10 @@ and whole-tree rules — those reasoning over Markdown/YAML or the entire `src/` rather than one AST at a time — are instead `@pytest.mark.lint` classes in `tests/test_custom_lint.py`. -**Every rule carries its own rationale in its module docstring**, including the defect -that motivated it and its known blind spots. That docstring is the authoritative -explanation; read it before editing, suppressing, or widening a rule. Run `make lint` — -`make test` deliberately excludes these. +**Each rule's module docstring states its invariant, its scope and its known blind +spots.** Read it before you edit, suppress or widen a rule. The defect that caused each +rule is in `.claude/notes/lint-rules.md`. Run `make lint` — `make test` does not run +these rules. When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern, add a rule following the CE000+ pattern and @@ -308,10 +308,15 @@ bandit, pre-commit, mcp code cannot say - **A docstring states the contract, not the history** — what a caller must know to call it correctly. Why the design is this shape belongs in `.claude/notes/`; what it used to - be belongs in git. `make docs-budget` enforces two rules, both self-adjusting: a file's - own-line comments may not exceed `MAX(20, 0.15 × its length)`, and no docstring may - exceed 150 words of PROSE (an `Args:`/`Returns:`/`Raises:` block is structure, not - prose; an `@abstractmethod` is exempt because its docstring IS the interface contract). + be belongs in git. `make docs-budget` applies three self-adjusting rules to `src/` and + `tests/`: no own-line comment RUN may exceed 8 lines (a run reads through one blank line, + so a paragraph split on one blank line does not duck it), a file's own-line comments may + not exceed `MAX(20, 0.15 × its length)` in total, and no docstring may exceed 150 words of PROSE (an + `Args:`/`Returns:`/`Raises:` block is structure, not prose; an `@abstractmethod` is exempt + because its docstring IS the interface contract). The run cap governs the SHAPE of any one + comment — put the paragraph in `.claude/notes/` behind a `Rationale:` pointer — and the + file total is the backstop under it, for a file that is mostly commentary however it is + broken up. ## Notes for AI Assistants diff --git a/Makefile b/Makefile index 44b2d3f44..0350c7824 100644 --- a/Makefile +++ b/Makefile @@ -39,7 +39,7 @@ plugin-reference: ## Regenerate the plugin's bundled criteria reference from th pricing-mirror: ## Regenerate the evalboard's rate table from pricing.py (SSOT) uv run python -m tests.lint.pricing_mirror -docs-budget: ## Report the docstring/comment prose budget and check it against the baseline +docs-budget: ## Check the docstring/comment prose rules in src/ and tests/ uv run python -m tests.lint.prose_budget typecheck: ## Run type checking with pyright diff --git a/pyproject.toml b/pyproject.toml index 60bd7e60b..4e331ce1e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -262,7 +262,6 @@ external = [ "CE020", "CE021", "CE022", - "CE023", "CE024", "CE025", "CE026", diff --git a/src/coder_eval/cli/evaluate_command.py b/src/coder_eval/cli/evaluate_command.py index d6db81719..d024f905a 100644 --- a/src/coder_eval/cli/evaluate_command.py +++ b/src/coder_eval/cli/evaluate_command.py @@ -511,14 +511,10 @@ def _report_and_exit( status handling landed. Always raises ``typer.Exit``. """ - # BEFORE the count guard below: a grading crash returns a populated ERROR - # result with an EMPTY criteria list, so the count check fired first and the - # real error was never printed. - # - # Whether the terminal status describes THIS pass or was carried over from the - # run being graded. `Orchestrator._terminal_status` preserves a prior execution - # fact, so reading `result.final_status` as this pass's own outcome misreports - # both arms below. + # BEFORE the count guard below: a grading crash returns a populated ERROR result + # with an EMPTY criteria list, so the count check fires first and hides the real + # error. `inherited` says whether the terminal status describes THIS pass or was + # carried over from the run being graded. # Rationale: .claude/notes/isolation.md § Detached grading from the CLI inherited = prior is not None and prior.final_status.is_execution_fact diff --git a/src/coder_eval/criteria/base.py b/src/coder_eval/criteria/base.py index 03f3d76dc..75b58b132 100644 --- a/src/coder_eval/criteria/base.py +++ b/src/coder_eval/criteria/base.py @@ -26,16 +26,10 @@ # A criterion's verdict from a PARTIAL, mid-run trajectory (early-stop observability). # "undecided" means the outcome is not yet knowable from the events seen so far. -# -# CONTRACT every live_verdict override must satisfy (see BaseCriterion.live_verdict): -# - Deterministic: a pure function of the ``turn_records`` prefix passed in — no -# wall-clock, randomness, or other hidden state. -# - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST -# return that SAME verdict for every longer prefix (i.e. every later call in the -# same run). "undecided" is the only verdict allowed to change on a later call. -# -# Enforced by lint rule CE036, which replays every live criterion against every prefix -# of recorded trajectories; CE025 checks only the subclassing/pairing shape. +# Every live_verdict override must be DETERMINISTIC (a pure function of the +# ``turn_records`` prefix, no hidden state) and MONOTONIC (a "pass"/"fail" holds for +# every longer prefix; only "undecided" may change). CE036 replays every live +# criterion against every prefix; CE025 checks the subclassing shape. # Rationale: .claude/notes/contracts.md § The live_verdict contract LiveVerdict = Literal["pass", "fail", "undecided"] diff --git a/src/coder_eval/criteria/cli_called.py b/src/coder_eval/criteria/cli_called.py index 278dd9304..539b4131a 100644 --- a/src/coder_eval/criteria/cli_called.py +++ b/src/coder_eval/criteria/cli_called.py @@ -110,7 +110,6 @@ def _check_impl( # Booked when the shim's own rule evaluation RAISED: the responses the agent # saw were not the ones the task described, so no verdict over this log # means anything. - # # HAZARD: all five of this checker's refuse-to-score paths are uniform at a # gating 0.0 and NONE may raise -- every one is agent-reachable, and an # escalation to ERROR is a better outcome for a failing agent than FAILED. diff --git a/src/coder_eval/harbor/packager.py b/src/coder_eval/harbor/packager.py index 5f83c46cb..afd6b8e5b 100644 --- a/src/coder_eval/harbor/packager.py +++ b/src/coder_eval/harbor/packager.py @@ -381,11 +381,14 @@ def _find_workdir(dockerfile: Path) -> str | None: def _write_verifier_task_yaml(task: TaskDefinition, out_dir: Path) -> None: - """``tests/task.yaml`` — the criteria, as authored. See the module docstring for the agent-type caveat.""" + """``tests/task.yaml`` — the criteria, as authored. It must not set a ``none`` agent type. + + Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml + """ payload: dict[str, object] = { "task_id": task.task_id, "description": task.description, - "agent": {"type": "claude-code"}, # placeholder; never instantiated (see module docstring) + "agent": {"type": "claude-code"}, # placeholder; never instantiated (see this function's docstring) "initial_prompt": _VERIFIER_PLACEHOLDER_PROMPT, "success_criteria": [c.model_dump(mode="json", exclude_none=True) for c in task.success_criteria], } @@ -586,16 +589,10 @@ def _write_agent_phase_task_yaml( if not is_agentless: payload["initial_prompt"] = initial_prompt if task.pre_run: - # `pre_run` runs "inside the sandbox after setup completes but before the - # agent starts" (PreRunCommand's own docstring) -- exactly the phase - # `coder-eval execute` still performs for the CoderEvalAgent embed (it shares - # `run`'s entire pipeline minus grading, see execute_command.py's module - # docstring), so this is a real translation, not a Dockerfile-build-time - # stand-in. Unlike `post_run` (belongs to the GRADING phase -- see - # orchestrator.py's own comment -- which `coder-eval execute` never runs at - # all), `pre_run` has a real place to run here. Commands are relative to the - # sandbox cwd, resolved the same way template_sources/`_setup_template` are, - # so no path rewriting is needed. + # A real translation, not a build-time stand-in: `coder-eval execute` still + # runs the pre-agent phase for the CoderEvalAgent embed, unlike `post_run`, + # which belongs to grading and never runs there. Commands are relative to the + # sandbox cwd, resolved as template_sources are, so no path rewriting. payload["pre_run"] = [c.model_dump(mode="json", exclude_none=True) for c in task.pre_run] if task.run_limits is not None: # `CoderEvalAgent.run()` invokes `coder-eval execute` against this diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index 4a613e6a7..9b6b8ba11 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -29,12 +29,10 @@ # SECURITY: the judge's ignore_patterns FLOOR, enforced unconditionally in -# ``criteria/agent_judge.py::_build_agent_config`` even when the user supplied their -# own list. These are COPY-TIME controls, not just SDK settings: the same list is -# the ``ignore`` passed to ``shutil.copytree`` in ``evaluation/sub_agent.py``, so -# dropping an entry lets an agent-planted ``.claude/`` or ``.mcp.json`` land in the -# judge's own working directory. Imported from both call sites so the model -# defaults and the checker floor cannot drift. +# ``criteria/agent_judge.py::_build_agent_config`` even over a user-supplied list. It is a +# COPY-TIME control too: the ``shutil.copytree`` ignore in ``evaluation/sub_agent.py``, so +# dropping an entry lets an agent-planted ``.claude/`` or ``.mcp.json`` land in the judge's +# working directory. Both call sites import it, so the defaults and the floor cannot drift. # Rationale: .claude/notes/contracts.md § The security floor JUDGE_SECURITY_IGNORE_FLOOR: tuple[str, ...] = (".claude", ".mcp.json", "_reference") diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index c3f23e46a..9a59e2ec7 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -165,15 +165,13 @@ def validate_early_stop(task: TaskDefinition) -> None: + "Disarm with run_limits.stop_early: false to run this agent anyway." ) - # (4) A threshold of exactly 0 trivially satisfies both the pass-stop - # floor check and the final weighted gate regardless of whether any armed - # criterion has actually decided — neutralizing the armed pass/fail gate - # with one YAML line (coder-eval is used as a CI gate). Checked here - # (not on RunLimits itself) because this is the whole-task, hard-stop - # surface: an EarlyStopConfigError here flips the plan exit code and - # aborts run, whereas a plain ValueError on the merged RunLimits model - # would land in the CLI's generic "resolution failed" branch, which - # prints red text but does not flip the exit code. + # (4) A threshold of exactly 0 trivially satisfies both the pass-stop floor and + # the final weighted gate however the armed criteria decided, neutralizing the + # gate with one YAML line. Checked here, not on RunLimits: an + # EarlyStopConfigError flips the plan exit code and aborts run, where a plain + # ValueError lands in the CLI's generic "resolution failed" branch, which prints + # red text but does not flip the exit code. + # Rationale: .claude/notes/orchestration.md § Early stop on criterion if limits is not None and limits.stop_early_gate_threshold <= 0.0: raise EarlyStopConfigError( f"run_limits.stop_early_gate_threshold ({limits.stop_early_gate_threshold}) must be " diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index cdcfba490..819d3c940 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -1982,13 +1982,11 @@ async def _communicate_attempt() -> TurnRecord: iteration=iteration, ) from None - # ANTI-CHEAT WINDOW. Both the reference and the task dir sit at mode 000 - # for the whole of every communicate attempt — retries included, since this - # wrapper is outside execute_with_retry — and are restored on every exit - # path. Routed through the SANDBOX, which owns whether a chmod window means - # anything for its driver. It does NOT hide the task DEFINITION: task.yaml - # is also staged at /work/input, and hiding the criteria from the agent is - # a separate, unsolved problem. + # ANTI-CHEAT WINDOW. The reference and the task dir sit at mode 000 for every + # communicate attempt, retries included (this wrapper is outside + # execute_with_retry), and are restored on every exit path. The SANDBOX owns + # whether a chmod window means anything for its driver. It does NOT hide the + # task DEFINITION: task.yaml is also staged at /work/input, an unsolved gap. # Rationale: .claude/notes/permissions.md § Reference solutions and the anti-cheat window assert self.sandbox is not None async with self.sandbox.set_permissions([self._reference_dir, self.sandbox.task_dir]): @@ -2997,15 +2995,10 @@ async def _cleanup(self) -> None: # Deliberately NOT preserved into run_dir/artifacts: run directories get # archived, uploaded and shared, and the reference must not ride along. - # # Keyed on the staging root recorded BEFORE the copy — NOT on - # `_reference_dir.parent`, which is only set once the copy succeeds. The - # field is None under docker, where the reference is the host-owned bind - # mount: that one is NOT OURS TO DELETE, and rmtree'ing its parent would - # take `/work` with it. - # - # rmtree_restrictive, because a run killed mid-turn leaves the tree at - # mode 000, where plain rmtree silently declines. + # `_reference_dir.parent`, which is set only once the copy succeeds, and is + # None under docker where the reference is a host-owned bind mount that is + # NOT OURS TO DELETE. # Rationale: .claude/notes/persistence.md § rmtree_restrictive staging_root = self._reference_staging_root self._reference_dir = None diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index ec5572566..1324d4ae7 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -30,13 +30,11 @@ # container's read-only input mount. Never written by a run. PRIOR_RESULT_FILENAME = "prior.json" -# The container's own stdout+stderr transcript, and the name it is folded back -# under after a GRADING container. Named for the PHASE because on the -# ``run --resume`` path ``docker.log`` is already taken by the executed -# container's log -- folding a grading log back under it repeats the -# task.log/grade.log truncation bug one layer down. Constants, not literals -# (CE053): the fold-back is guarded by ``is_file()``, so a rename on the -# producing side would degrade the copy to a silent no-op. +# The container's stdout+stderr transcript, and its fold-back name after a GRADING +# container. Named for the PHASE: on ``run --resume`` ``docker.log`` already holds the +# executed container's log, and reusing it repeats the task.log/grade.log truncation bug. +# Constants, not literals (CE053): the fold-back is guarded by ``is_file()``, so a +# rename on the producing side would silently skip the copy. # Rationale: .claude/notes/persistence.md § Run-directory filename constants DOCKER_LOG_FILENAME = "docker.log" GRADE_DOCKER_LOG_FILENAME = "grade.docker.log" diff --git a/src/coder_eval/reports/helpers.py b/src/coder_eval/reports/helpers.py index 14f944afa..5183a25c7 100644 --- a/src/coder_eval/reports/helpers.py +++ b/src/coder_eval/reports/helpers.py @@ -111,20 +111,11 @@ def collect_variant_series(result: ExperimentResult) -> dict[str, VariantSeries] s = series.get(vr.variant_id) if s is None: # a task result for a variant not in variant_ids continue - # Only the SCORE is dropped when there is none — never the row. - # Duration, tokens and assistant turns are facts about the run that - # grading has nothing to do with, and `execute`'s stated contract is - # that only the verdict is withheld. Skipping the row whole made an - # all-ungraded experiment render `Avg Duration | N/A | N/A` with the - # Tokens and Assistant Turns rows absent entirely. - # - # The series are consumed independently (each statistic reads one - # list), so they need not be index-aligned with each other; - # `paired_comparison` pairs across VARIANTS by task id, not by index - # into these lists. An earlier note here claimed an experiment is - # either entirely graded or entirely ungraded because `grade` is - # run-level — `run --resume` grades rows independently and folds a - # failed one back ungraded, so mixed experiments are real. + # Only the SCORE is dropped when there is none — never the row. The + # series are consumed independently (each statistic reads one list), so + # they need not be index-aligned; `paired_comparison` pairs across + # VARIANTS by task id, not by index into these lists. + # Rationale: .claude/notes/reporting.md § Ungraded rows in a rollup if vr.weighted_score is not None: s.scores.append(vr.weighted_score) s.durations.append(vr.duration_seconds / vr.replicate_count) diff --git a/src/coder_eval/timing.py b/src/coder_eval/timing.py index 25c7c77fb..71f6fdd48 100644 --- a/src/coder_eval/timing.py +++ b/src/coder_eval/timing.py @@ -282,7 +282,6 @@ def subtract_tool_time( raw_total = sum(raw for _, raw in members) # Nothing to apportion, and the loop below divides by it — a group already # at zero stays at zero. - # # ORDER IS LOAD-BEARING: this skip runs BEFORE the equality check below, # because `close_window` clamps an inverted window to 0.0 while its bounds # still say `completed_at < started_at` — a measured inversion the check diff --git a/tests/_bracket_clock.py b/tests/_bracket_clock.py index 46faa1ea2..e8ba8057b 100644 --- a/tests/_bracket_clock.py +++ b/tests/_bracket_clock.py @@ -1,23 +1,13 @@ """A ``TurnClock`` stand-in anchored far from real time, for the CE064 tests. CE064 checks only that ``timestamp=`` is PRESENT on an ``AgentStartEvent`` / -``AgentEndEvent`` emit — its own declared blind spot is that it cannot tell -``self.clock.now()`` from a ``datetime.now()`` written out at the call site. -This is the guard for the SOURCE of that stamp, on the three harnesses that own -a clock. - -ANCHORED FAR FROM NOW, and that is the whole trick. A bracket left on -``StreamEvent.timestamp``'s ``default_factory=datetime.now`` lands within -microseconds of a clock-derived one, so an assertion written against real time -would pass either way. Anchoring the stand-in a year out (the same device as -``tests/test_timing_identity_contract.py``'s ``EPOCH_MS``) makes a reverted -``timestamp=`` fail by a year rather than by a microsecond. - -It advances on the REAL monotonic clock instead of stepping by hand, which is -what lets the same fixture assert the second half: with the bracket and the -window bounds finally on one basis, ``decompose_turn``'s head and tail come out -as small positive measurements rather than as the clamped ``0.0`` a cross-basis -subtraction produced (see ``ce064_turn_bracket_on_the_clock``'s measured probe). +``AgentEndEvent``; this guards the SOURCE of that stamp on the three harnesses +that own a clock. The stand-in sits a year from ``datetime.now()``, so a +bracket that fell back to ``StreamEvent``'s default fails by a year, not by a +microsecond. It advances on the real monotonic clock, so the head and tail +come out as small positive measurements. + +Rationale: .claude/notes/lint-rules.md § CE064 """ import time @@ -68,22 +58,12 @@ def assert_bracket_on_the_clock(events: list[StreamEvent]) -> None: def assert_overhead_is_measured(record: TurnRecord) -> None: """The turn's head and tail are real measurements taken on one basis. - The two ends fail differently, and each needs its own assertion. - - A defaulted ``AgentStartEvent`` lands ~365 days before the clock-derived - first window, so the HEAD blows any sane bound by that whole offset — the - upper bound is what catches it. - - A defaulted ``AgentEndEvent`` fails the other way: it lands ~365 days - BEFORE its own last message, so ``decompose_turn`` clamps the negative and - publishes ``0.0`` — "measured, and instant", which sails through an upper - bound. Only a strict ``> 0.0`` catches it, and it holds on all three - harnesses because a turn's last flush and its end event are separated by - real work. The margin is small where it is smallest: antigravity holds its - process across turns and measures 0.007-0.03 ms here, which is 7-30 ticks - of the 1 us resolution both `datetime` and `time.monotonic()` have on - Linux, macOS and Windows. That is the magnitude the clamped defect hid, so - do not relax this to ``>= 0.0`` — a zero is the defect. + A defaulted ``AgentStartEvent`` blows the head's upper bound by the whole + anchor offset. A defaulted ``AgentEndEvent`` makes ``decompose_turn`` clamp + the tail to ``0.0``, which only the strict ``> 0.0`` catches. Do not relax + it to ``>= 0.0`` — a zero is the defect. + + Rationale: .claude/notes/lint-rules.md § CE064 """ assert record.harness_startup_ms is not None, "harness_startup_ms was never measured" assert record.harness_teardown_ms is not None, "harness_teardown_ms was never measured" diff --git a/tests/_fixtures/golden_streams/_scrub.py b/tests/_fixtures/golden_streams/_scrub.py index 0c19a806f..1593cc3c5 100644 --- a/tests/_fixtures/golden_streams/_scrub.py +++ b/tests/_fixtures/golden_streams/_scrub.py @@ -123,19 +123,14 @@ def assert_reconciliation(record: dict[str, Any]) -> None: def _tool_union_ms(record: dict[str, Any]) -> float: """Wall ms this turn's MAIN-THREAD tools occupied — the union, never the sum. - Validates the raw dump into a ``TurnRecord`` and calls the SAME typed - selector the collector uses, rather than reimplementing the selection rule - (the sub-agent-id derivation, the stamp parse, the ``end >= start`` filter) - over dicts. Three copies of that rule existed and agreed only because - someone kept checking; the collector's own version once passed every command - while filtering only its generations, and the two agreed by luck. - - What is shared with production is the SELECTION and ``union_ms``. What is - NOT shared is the bookkeeping around them — this still builds its own span - set and computes its own union, which is where every timing defect on this - branch actually lived (see CE063's docstring). Do not "simplify" it into - reading ``tool_union_ms``: that would make the sensor a restatement of the - producer's answer, and the cross-check below is what verifies that field. + Shares the collector's SELECTION (``main_thread_tool_spans``) and + ``union_ms``, but not the bookkeeping around them — where the per-reducer + timing defects lived (see CE063's docstring). Do not + "simplify" it into reading ``tool_union_ms``: that would make the sensor a + restatement of the producer's answer, and the cross-check below is what + verifies that field. + + Rationale: .claude/notes/timing.md § The golden-stream timing sensor """ turn = TurnRecord.model_validate(record) return union_ms(main_thread_tool_spans(turn.messages, turn.commands)) @@ -151,86 +146,21 @@ def assert_timing_captured( ) -> None: """Assert a TurnRecord dump actually recorded the timing it could measure. - Run on the UNSCRUBBED dump. ``scrub()`` masks values but preserves ``None`` - (see its docstring), and present-vs-absent IS the whole assertion here — a - scrubbed snapshot can tell you a field was set, never that it was set to - something meaningful. - - An AST rule cannot see that an SDK returned ``0.0``; this replay-based - sensor can. Two checks: - - **Unconditional.** Every command that RESOLVED (``result_status`` of - ``"success"`` or ``"error"``) carries ``execution_started_at``, - ``execution_completed_at`` and ``duration_ms``. A force-closed orphan - (``"unknown"``) is exempt: it was never timed, and saying so is the honest - record. Where a scenario resolves no command the check is vacuously true, - which is correct rather than weak — the scenario is asserting nothing - about commands because it has none. - - **Flagged.** When ``expect_generation_window``, at least one assistant - entry reports a ``generation_duration_ms`` that is non-``None`` AND - greater than zero AND whose recorded bounds actually span it - (``completed_at > started_at``). - - The bounds half is not redundant. Two harnesses derive the duration from a - MONOTONIC clock and the bounds from the wall clock, so the two can - disagree: a reducer could report a healthy duration beside two stamps that - collapsed to one instant. CE059 catches that statically only when both - bounds are the same ``ast.Name``; when they are two different names - holding the same value it cannot, and this is the check that does. - - **Unconditional, and keyed on the messages rather than on the flag.** A - turn's head and tail (``harness_startup_ms`` / ``harness_teardown_ms``) are - set exactly when the turn produced an assistant message with a MEASURABLE - window, because that is what the collector measures them against — so both - are non-``None`` when one exists and both are ``None`` when none does. - - Both halves of that key are load-bearing. The flag is the wrong one: - ``codex_e_orphan_tool`` streams a generation whose window subtracts to - zero, so it clears the flag while still having a head and a tail to report. - And "any assistant message" is too weak: ``codex_g_items_rebuild`` rebuilds - its transcript from the rollout after the turn ended, with - ``generation_duration_ms=None`` and placeholder ``now()`` bounds, so there - is nothing there to measure an end against and the honest answer is - ``None`` for both. - - PRESENCE is all the fixtures can support, and it is the thing worth - asserting: the replays run in ~0.3 ms of synthetic wall clock, so their - head and tail are microseconds and any bound or ordering check would be - noise. A ``>= 0`` check would be worse than noise — ``decompose_turn`` - clamps with ``max(..., 0.0)``, so it would restate the implementation and - could never fail. - - **The four-bucket identity**, when ``check_identity``. Generation plus the - UNION of the tool intervals plus the head plus the tail cannot exceed the - turn's ``duration_seconds``, because the four are disjoint: the windows are - tool-subtracted and so are the head and tail. This is the one assertion - that catches a DOUBLE-COUNT rather than an absence — it is how an orphaned - tool force-closed inside the tail, booked both as tool and as teardown, was - found reconciling at -86% of wall clock while all 72 golden tests passed. - - The check is ONE-SIDED on purpose and stays that way. A symmetric bound - would be a sensor in name only here: the replays run in ~0.3 ms of - synthetic wall clock, so ``abs(residual) <= max(0.1 ms, 20% x wall)`` - passes essentially any magnitude. The two-sided, millisecond-exact check - lives in ``tests/test_timing_identity_contract.py``, where a scripted clock - makes the magnitudes real, and the live two-sided gate is - ``scripts/timing/decompose_run.py --max-residual-pct``. - - ``check_identity`` is off for the scenarios that inject their own SDK - timestamps (see ``FICTIONAL_DURATIONS``): those declare integer-millisecond - item durations of 17-900 ms while the replay itself takes ~0.3 ms of real - wall clock, so no rebasing can make the two commensurable — the SDK's - stamps are milliseconds and the replay is faster than one. - - Why a scenario-level floor rather than a per-entry rule: no per-entry form - works against the real snapshots. ``claude_d_subagent_terminal`` holds two - content-bearing assistant messages of which exactly one is legitimately - ``None`` (the synthesized sub-agent generation, delivered as a tool result - and never streamed), so no scenario-level flag can express "this one but - not that one". And "never exactly 0.0" conflicts with the clamps that can - legitimately produce a measured zero. The detailed per-message contract - lives in each agent's own unit tests; this is the cross-harness floor. + Run on the UNSCRUBBED dump: ``scrub()`` masks values but preserves ``None`` + (see its docstring), and present-vs-absent is the assertion. + + - Every RESOLVED command (``"success"`` / ``"error"``) carries both + execution stamps and ``duration_ms``; an ``"unknown"`` orphan is exempt. + - When ``expect_generation_window``, some assistant entry has a positive + ``generation_duration_ms`` whose bounds span it. + - The head and tail are both set exactly when an assistant message has a + non-``None`` ``generation_duration_ms`` — keyed on the messages, not the flag. + - When ``check_identity``: a stored ``tool_union_ms``, when present, matches + the recomputed union; and the four buckets overshoot the wall clock by no + more than the tolerance. One-sided on purpose: the two-sided check is + ``tests/test_timing_identity_contract.py``. + + Rationale: .claude/notes/timing.md § The golden-stream timing sensor """ for command in record.get("commands") or []: if command.get("result_status") not in ("success", "error"): diff --git a/tests/_fixtures/golden_streams/codex_fixtures.py b/tests/_fixtures/golden_streams/codex_fixtures.py index e0a157fdf..70b1d89da 100644 --- a/tests/_fixtures/golden_streams/codex_fixtures.py +++ b/tests/_fixtures/golden_streams/codex_fixtures.py @@ -36,14 +36,12 @@ # --- Notification factories (mirror test_codex_agent) ----------------------- -# Fixed epoch milliseconds, so every derived duration is deterministic and the -# golden snapshots pin a real value rather than a scrubbed clock read. It is a -# BASE, not a wall-clock claim: ``_rebase_notifications`` shifts the whole -# timeline onto the replay's own clock before the scenario runs, so the SDK -# stamps and the agent's own event stamps are commensurable. Left absolute, -# a codex replay recorded a ``harness_startup_ms`` of ~126 DAYS — the agent -# events are stamped ``now()`` while these sat in 2027 — which is a number no -# presence-only assertion can catch. +# Fixed epoch milliseconds, so every derived duration is deterministic and the golden +# snapshots pin a real value. It is a BASE, not a wall-clock claim: ``_rebase_notifications`` +# shifts the timeline onto the replay's own clock, so SDK stamps and agent stamps are +# commensurable. Left absolute, a codex replay recorded a ``harness_startup_ms`` of +# ~126 DAYS (agent events stamped ``now()``, these in 2027), which no presence-only +# assertion can catch. _T0_MS = 1_800_000_000_000 diff --git a/tests/_fixtures/golden_streams/opencode_fixtures.py b/tests/_fixtures/golden_streams/opencode_fixtures.py index 5f1381479..eb2662980 100644 --- a/tests/_fixtures/golden_streams/opencode_fixtures.py +++ b/tests/_fixtures/golden_streams/opencode_fixtures.py @@ -1,22 +1,16 @@ """OpenCode golden-master scenarios: recorded CLI event lines + a runner. -The agent shells out to ``opencode run --format json`` and reduces its -newline-delimited JSON, so a scenario is an ordered list of event LINES and the -driver is a fake process that replays them. No Python package to guard on: -``pyproject.toml`` declares ``opencode = []``. - -The event helpers, the ``HAPPY_STREAM`` sample and the ``_FakeProcess`` / -``_RunningProcess`` fakes live HERE and are imported back into -``test_opencode_agent`` — one definition, two consumers. The ``patch_exec`` -pytest FIXTURE stays in that module (it needs ``monkeypatch``); the runner -below does the same patching with a plain context manager. - -The lines mirror events CAPTURED FROM A LIVE run — the CLI's own compact -vocabulary (``step_start`` / ``step_finish`` / ``text`` / ``tool_use``, payload -under ``part``). Do NOT "correct" them toward the ``session.next.*`` names in -the server's OpenAPI schema: those describe ``opencode serve``'s SSE surface, -and an earlier version of this harness parsed them and silently captured zero -telemetry on a real run. +A scenario is an ordered list of ``opencode run --format json`` event LINES, +replayed by a fake process. The event helpers, ``HAPPY_STREAM`` and the +``_FakeProcess`` / ``_RunningProcess`` fakes live HERE and are imported by +``test_opencode_agent``; its ``patch_exec`` fixture needs ``monkeypatch``, so +the runner below patches with a context manager. + +The lines mirror a LIVE capture (``step_start`` / ``step_finish`` / ``text`` / +``tool_use``, payload under ``part``). Do NOT "correct" them toward the +``session.next.*`` names of ``opencode serve``'s SSE surface. + +Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash """ from __future__ import annotations @@ -310,15 +304,13 @@ def _build_catalogue() -> list[OpenCodeScenario]: ) ) - # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` - # by the orphan sweep at finalization. It carries NO `state.time`, which is - # the honest shape for a call that never returned: with no + # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` by + # the orphan sweep at finalization. It carries NO `state.time`: with no # `execution_started_at` there is no `duration_ms` and no span. - # - # READ THE SNAPSHOT: the sweep still stamps `execution_completed_at`, which - # it does on every close path, so the record holds an end with no - # beginning. Compare `pi_d_orphaned_tool`, where the start IS stamped and a - # manufactured duration follows from it. + # READ THE SNAPSHOT: the sweep still stamps `execution_completed_at`, as it does + # on every close path, so the record holds an end with no beginning. Compare + # `pi_d_orphaned_tool`, where the start IS stamped and a manufactured duration + # follows from it. scenarios.append( OpenCodeScenario( name="d_orphaned_tool", diff --git a/tests/_fixtures/golden_streams/pi_fixtures.py b/tests/_fixtures/golden_streams/pi_fixtures.py index 2d26d7d6d..95d953975 100644 --- a/tests/_fixtures/golden_streams/pi_fixtures.py +++ b/tests/_fixtures/golden_streams/pi_fixtures.py @@ -270,18 +270,13 @@ def _build_catalogue() -> list[PiScenario]: ) ) - # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` - # by the orphan sweep at finalization. - # - # READ THE SNAPSHOT: the command carries `execution_started_at` (the CLI - # really did emit that start) and NEITHER `execution_completed_at` NOR - # `duration_ms`. Nothing observed this call finishing, so the instant the - # sweep runs is not a completion; stamping it used to manufacture both, and - # the pair then read as a measured span that the collector subtracted from - # a generation window the tool never occupied. One bound alone forms no - # span (`main_thread_tool_spans` requires both), so the window is left - # whole. Same rule as claude-code's `_finalize_commands`: unknown status - # and unknown duration are one fact (CE058). + # (d) a tool the CLI opens and never resolves — force-closed as `unresolved` by the + # orphan sweep at finalization. + # READ THE SNAPSHOT: the command carries `execution_started_at` and NEITHER + # `execution_completed_at` NOR `duration_ms`. Nothing observed this call finishing, + # so stamping the sweep instant would manufacture a span the tool never occupied. + # One bound alone forms no span, so the window is left whole. + # Rationale: .claude/notes/timing.md § main_thread_tool_spans scenarios.append( PiScenario( name="d_orphaned_tool", @@ -313,20 +308,13 @@ def _build_catalogue() -> list[PiScenario]: ) ) - # (f) a duplicate `turn_end` with no `turn_start` between — a transport - # hiccup this reducer explicitly promises to survive, since pi retries - # internally. A spent `turn_started_at` left in place reopens the next - # window at the PREVIOUS turn's start and republishes that whole span: - # reproduced as 3000 ms of generation for a 2000 ms turn. It had a unit test - # and no golden. - # - # READ THE SNAPSHOT: both halves of that reset are now in `on_turn_end`. - # The second assistant message carries NO content block and an empty - # `tool_use_ids` — it booked the duplicate's own usage and nothing else. - # `turn_text_parts` / `turn_tool_ids` used to be cleared in `on_turn_start` - # only, so the replayed line published the first turn's text a second time - # as its own message; the argument `on_turn_end`'s comment makes for - # `turn_started_at` applies to those two lists unchanged. + # (f) a duplicate `turn_end` with no `turn_start` between — a transport hiccup this + # reducer promises to survive, since pi retries internally. A spent + # `turn_started_at` left in place reopens the next window at the PREVIOUS turn's + # start: reproduced as 3000 ms of generation for a 2000 ms turn. + # READ THE SNAPSHOT: both halves of that reset are in `on_turn_end`, and the second + # assistant message carries no content block and an empty `tool_use_ids`. + # Rationale: .claude/notes/timing.md § Where a reducer's window opens scenarios.append( PiScenario( name="f_duplicate_turn_end", diff --git a/tests/lint/action_docs.py b/tests/lint/action_docs.py index f3274bdda..7457166e4 100644 --- a/tests/lint/action_docs.py +++ b/tests/lint/action_docs.py @@ -1,37 +1,22 @@ """CE026 — the GitHub Action's onboarding surfaces must stay truthful and self-sufficient. -Several surfaces introduce the same composite Action — ``README.md``, -``docs/CI_GATE.md``, ``docs/tutorials/02-ci-pipeline.md``, and now the Claude Code -plugin's ``ci`` skill, whose emitted workflow users copy verbatim — and each was -hand-maintained, so they drifted. The motivating bug: ``docs/CI_GATE.md`` claimed "there is nothing to -install" and offered a copy-pasteable ``uses:`` step with no agent runtime — the action -is agent-agnostic, so an integrator who copied it got a run that dies on a missing -``claude`` binary. The correcting paragraph was 11 lines away; the tutorial's snippet -showed the prerequisite steps; the reference page's did not. - -Four clauses, all mechanical: - -1. **Prerequisite parity.** The *first* fenced ``yaml`` block on a doc page that - references the action (``uses: /coder_eval@…``) is the page's quickstart, so - it must also show the agent-runtime steps. Later blocks on the same page are - single-input illustrations and are skipped, which is what keeps the rule quiet. -2. **No unqualified zero-install absolute** in prose near such a block. This catches - the *phrase*, not the *contradiction*: judging whether a paragraph 11 lines later - states a real prerequisite is semantic reasoning no static rule should attempt, so - the rule instead forces the absolute to be scoped where it is written - ("no *Marketplace* install step"). -3. **Marketplace slug parity.** Every ``github.com/marketplace/actions/`` link and - the shields badge label must match ``action.yml``'s ``name:`` — the listing title, - which a rename would silently 404 in four places at once. -4. **Input parity.** Every ``with:`` key on a snippet's ``uses: /coder_eval@…`` - step must be a real ``action.yml`` input. GitHub does not fail a workflow on an - unknown input, so a renamed input leaves every snippet promising something the step - no longer does — silently, and worst of all in the ``ci`` skill, whose output lands - in *other people's* repositories where our CI can never see it. - -Like CE027-CE031 this is deliberately NOT a ``BaseRule`` in ``tests/lint/runner.py``: -that runner is AST-only over ``.py`` files, whereas this rule reasons over Markdown and -YAML. It is wired as ``tests/test_custom_lint.py::TestCE026ActionDocSurfaces``. +Scope: ``README.md`` and every ``docs/`` and ``plugins/`` Markdown file (the ``ci`` +skill emits a workflow users copy verbatim). Four clauses: + +1. **Prerequisite parity.** The FIRST fenced ``yaml`` block on a page that references + ``uses: /coder_eval@…`` must show ``REQUIRED_PREREQ_TOKENS`` (kept in sync + with the ``action-dogfood`` CI job). Later blocks are skipped. +2. **No unqualified zero-install absolute** ("nothing to install") near such a snippet. + BLIND SPOT: it matches the phrase, not a contradiction elsewhere on the page. +3. **Marketplace slug parity.** Marketplace links and the shields badge label match + ``action.yml``'s ``name:``. +4. **Input parity.** Every ``with:`` key on the Action step is a declared + ``action.yml`` input. + +Opt out with ``PREREQ_SKIP_MARKER`` (above the fence) or ``CLAIM_SKIP_MARKER`` (on the +line). Wired as ``tests/test_custom_lint.py::TestCE026ActionDocSurfaces``. + +Rationale: .claude/notes/lint-rules.md § CE026 """ from __future__ import annotations diff --git a/tests/lint/agent_roster_parity.py b/tests/lint/agent_roster_parity.py index 4dba2ac6c..388a0009f 100644 --- a/tests/lint/agent_roster_parity.py +++ b/tests/lint/agent_roster_parity.py @@ -1,32 +1,19 @@ """CE047 — every marketing/onboarding surface must name every built-in agent. -The roster of supported harnesses is restated in prose on a handful of surfaces -that nothing mechanically ties to the code: the README, the docs home, the -comparison page, ``llms.txt``, the ``mkdocs.yml`` site description, the Pages -stub, and the packaging metadata. Adding a harness means remembering all seven — -which is exactly how **OpenCode shipped while being absent from most of them**: -the agent worked, but a reader (or a crawler, or an LLM answering "which agents -does Coder Eval support?") was told it did not exist. There is no error, no test -failure, and no user report for that; the surface just quietly under-sells the -framework. - -``AgentKind`` is the framework's own list of built-ins (it is deliberately NOT the -closed set of valid ``agent.type`` values — the ``AgentRegistry`` is authoritative -and plugins extend it — but every built-in *is* in the enum, and a built-in is -what these surfaces promise). This rule derives the expected roster from that -enum and asserts each surface mentions each agent by name. - -Scope note: this is a **presence** check over a file (or, where a file is mostly -unrelated content, over one extracted region — ``mkdocs.yml``'s -``site_description``, ``pyproject.toml``'s ``description`` + ``keywords``). It -cannot tell a good sentence from a bad one; it only makes "we forgot this harness -exists" impossible to ship. Third-party plugin agents are out of scope — they own -their own docs. - -It is intentionally NOT a ``BaseRule`` in ``tests/lint/runner.py``: that runner is -AST-only over ``.py`` files, whereas this reasons over Markdown/YAML/TOML/HTML -surfaces. It is wired as a dedicated test in -``tests/test_custom_lint.py::TestCE047AgentRosterParity`` (precedent: CE026-CE031). +The expected roster is ``AgentKind`` minus ``NON_ROSTER_KINDS``. Each surface in +``ROSTER_SURFACES`` must name each agent by one of its ``AGENT_DISPLAY_NAMES`` +spellings. ``AgentKind`` lists the built-ins only; the ``AgentRegistry`` stays +authoritative for valid ``agent.type`` values. Third-party plugin agents are out of +scope. + +Blind spot: this is a presence check over a file, or over one extracted region +(``mkdocs.yml``'s ``site_description``, ``pyproject.toml``'s ``description`` + +``keywords``). It cannot tell a good sentence from a bad one. + +Not a ``BaseRule``: it reasons over Markdown/YAML/TOML/HTML, and is wired as +``tests/test_custom_lint.py::TestCE047AgentRosterParity``. + +Rationale: .claude/notes/lint-rules.md § CE047 """ from __future__ import annotations diff --git a/tests/lint/dead_config_fields.py b/tests/lint/dead_config_fields.py index 5c5ddb6f0..f376fa329 100644 --- a/tests/lint/dead_config_fields.py +++ b/tests/lint/dead_config_fields.py @@ -1,33 +1,21 @@ """CE031 — behavior-driving config fields must be consumed somewhere in ``src/``. -A Pydantic field on a config model that users set in a task YAML but that no code -ever reads is *dead config*: it silently does nothing, and the author has no way -to know. This is what ``SimulationConfig.parallel_trials`` was — documented, set -in a shipped task YAML, defaulting to ``True``, and read nowhere (trial -concurrency is entirely ``--max-parallel``'s job). CE031 makes that class -impossible to reintroduce for a small, explicit registry of models. - -"Consumed" here means the field name appears as an **attribute access** -(``x.field``) anywhere under ``src/`` — the consumption contract for a -*behavior-driving* config: the orchestrator/validators must read the field by -name for it to have any effect. A field read only via ``model_dump()`` / -serialization is NOT caught by this definition, which is exactly why the registry -is restricted to behavior models (``SimulationConfig``, ``RunLimits``, -``Dataset``) and does **not** include serialization/telemetry models or the -sprawling ``TaskDefinition`` (whose fields are largely round-tripped through -``model_dump`` in the dataset expander). - -Known floor (documented, accepted): attribute names collide across models — if -``RunLimits`` and ``SimulationConfig`` both declare ``max_turns`` and only one is -read by name, both count as consumed. Collisions cause **false negatives** (a dead -field masked by a same-named live one elsewhere), never false positives, so the -rule can never wrongly break the build. An ``EXEMPT`` map covers any field that is -legitimately consumed only via serialization, with a reason. - -Like CE027 through CE030, this is not a ``BaseRule`` in the AST runner (that runner walks -one file at a time and reports line-level violations; this rule reasons over the -*whole* ``src/`` tree at once). It is wired as -``tests/test_custom_lint.py::TestCE031DeadConfigFields``. +For every model in ``CONSUMED_MODELS``, each field must appear as an **attribute +access** (``x.field``) somewhere under ``src/``, or be listed in ``EXEMPT`` with a +reason it is consumed only via serialization. + +Register only behavior-driving models. A field read only through ``model_dump()`` is +not an attribute access, so a serialization or telemetry model, or ``TaskDefinition``, +would report live fields as dead. + +BLIND SPOT: attribute names are not tied to a model. A dead field is masked by a +same-named attribute read anywhere under ``src/``. This causes false negatives, never +false positives. + +Wired as ``tests/test_custom_lint.py::TestCE031DeadConfigFields``, not the AST runner, +because it reasons over the whole ``src/`` tree at once. + +Rationale: .claude/notes/lint-rules.md § CE031 """ from __future__ import annotations diff --git a/tests/lint/doc_env_parity.py b/tests/lint/doc_env_parity.py index ba9e27527..c34bb76a7 100644 --- a/tests/lint/doc_env_parity.py +++ b/tests/lint/doc_env_parity.py @@ -1,30 +1,21 @@ """CE027 — documented framework env vars must be backed by a real consumer. -``coder_eval.config.Settings`` sets no ``env_prefix`` and uses ``extra="ignore"``, -so a documented env var whose name does not match a ``Settings`` field (or one of -its ``AliasChoices``) is **silently dropped** at runtime with zero signal — the -exact failure mode behind the ``CODER_EVAL_API_BACKEND`` doc bug (the real field -is ``API_BACKEND``, so the ``CODER_EVAL_``-prefixed spelling selected no backend -and the run fell back to Direct Anthropic). - -This rule scans the doc/config surfaces (``README.md``, ``action.yml``, -``docs/**``) for env-var **assignments** (``NAME=value`` — the copy-pasteable, -dangerous form) carrying a **framework-owned prefix** and flags any whose name is -neither a ``Settings`` env name/alias nor referenced anywhere in ``src/`` — the -framework also reads a handful of vars directly via ``os.getenv`` (e.g. -``CODER_EVAL_SKILLS_DIR``, ``CODEX_BASE_URL``, ``CODER_EVAL_IN_CONTAINER``), and -those are legitimately documentable. - -Scope note: only *assignments* are checked, not bare prose mentions. Prose -scanning is too false-positive-prone (markdown links like -``CODEX_AGENT_GUIDE.md``, secret RHS references like ``secrets.BEDROCK_TOKEN``, -regex-pattern examples like ``API_KEY = "…"``), and the assignment form is the -one users actually copy into a workflow, so it carries the real risk. - -It is intentionally NOT a ``BaseRule`` registered in ``tests/lint/runner.py``: -that runner is AST-only and walks ``.py`` files, whereas this rule reasons over -Markdown/YAML doc surfaces. It is wired as a dedicated test in +``Settings`` sets no ``env_prefix`` and uses ``extra="ignore"``, so a documented env +var that nothing consumes is silently dropped at runtime. + +The rule scans ``README.md``, ``action.yml`` and ``docs/**`` for assignments +(``NAME=value``) whose name carries a ``FRAMEWORK_ENV_PREFIXES`` prefix. It flags a +name that is neither a ``Settings`` field or alias nor consumed in ``src/``: an +``os.getenv`` / ``os.environ`` read, a ``"NAME=VALUE"`` child-process literal, or +either of those through a named constant. + +Blind spot: only assignments are checked, never bare prose mentions, and third-party +prefixes (``AWS_``, ``ANTHROPIC_``, ...) are not scanned. + +Not a ``BaseRule``: it reasons over Markdown/YAML, and is wired as ``tests/test_custom_lint.py::TestCE027DocEnvVarParity``. + +Rationale: .claude/notes/lint-rules.md § CE027 """ from __future__ import annotations @@ -35,11 +26,8 @@ from pydantic import AliasChoices -# Framework-owned env-var name prefixes. A documented token starting with one of -# these is owned by coder-eval and MUST be consumed by it. Deliberately EXCLUDES -# broad third-party namespaces (AWS_, ANTHROPIC_, GEMINI_, GITHUB_, EVALBOARD_, -# PLUGIN_) whose vars are consumed by SDKs / CI, not necessarily via Settings, so -# scanning them would produce false positives on legitimately-external names. +# A documented token with one of these prefixes is owned by coder-eval and MUST be consumed by it. +# Third-party namespaces (AWS_, ANTHROPIC_, GEMINI_, GITHUB_, EVALBOARD_, PLUGIN_) are excluded on purpose. FRAMEWORK_ENV_PREFIXES: tuple[str, ...] = ( "CODER_EVAL_", "API_", @@ -51,35 +39,20 @@ _PREFIX_ALT = "|".join(FRAMEWORK_ENV_PREFIXES) -# A framework-prefixed env-var ASSIGNMENT in doc/config text: ``NAME=`` where NAME -# carries a framework prefix. The negative lookbehind rejects a name embedded in -# a larger token — attribute access (``secrets.BEDROCK_TOKEN``), a hyphenated -# token (``X-API_KEY=``), a path/URL segment (``dir/API_X=``, ``http://API_Y=``), -# or a Windows path (``C:\\API_Z=``). The single ``=`` (not ``==``) with no space -# before it also rejects ``API_KEY = "…"`` regex-pattern examples. +# A framework-prefixed ``NAME=`` assignment. The lookbehind rejects a name inside a larger token: +# ``secrets.BEDROCK_TOKEN``, ``X-API_KEY=``, ``dir/API_X=``, ``http://API_Y=``, ``C:\\API_Z=``. +# A single ``=`` with no space before it rejects ``API_KEY = "…"`` regex-pattern examples. _ENV_ASSIGNMENT = re.compile(r"(? set[str]: const_values.update(dict(_SRC_ENV_CONST_DEF.findall(text))) const_reads.update(_SRC_ENV_CONST_READ.findall(text)) const_reads.update(_SRC_ENV_CONST_VALUE.findall(text)) - # Step two: a constant is backed only if it is BOTH defined as an env name and - # read somewhere. Defined-but-unread stays unbacked, exactly as before. + # Step two: a constant is backed only if it is BOTH defined as an env name and read somewhere. names.update(const_values[ident] for ident in const_reads & const_values.keys()) return names diff --git a/tests/lint/doc_examples.py b/tests/lint/doc_examples.py index 70b1fbf36..ca0ddd930 100644 --- a/tests/lint/doc_examples.py +++ b/tests/lint/doc_examples.py @@ -1,34 +1,22 @@ """CE029 — self-contained YAML examples in the docs must validate against their models. -A published example that does not parse is worse than no example: readers copy it, -hit a ``ValidationError``, and conclude the feature is broken. This rule caught -exactly that — the ``prompt_mutations`` recipe in ``docs/AB_EXPERIMENTS.md`` used -``text:`` where the field is ``content:``, and every mutation model declares -``extra="forbid"``, so the published snippet raised -``variants.1.prompt_mutations.0.suffix.content Field required``. - -Scope is deliberately narrow — the rule only validates blocks it can *prove* are -whole documents, because a false positive on an illustrative fragment would make -``make lint`` a nuisance and get the rule deleted: +The rule validates only blocks it can prove are whole documents: * a block is a **task** when it has ``task_id`` + ``initial_prompt`` + ``success_criteria``, and an **experiment** when it has ``experiment_id`` + - ``variants``. Anything else is a fragment and is skipped — including a bare - ``success_criteria:`` list, which is the single most common doc shape. -* **schematic** blocks are skipped: a doc that writes ``agent: { ... }`` to mean - "and so on" parses to the literal key ``"..."``. The task guide's overview block - uses that form deliberately. -* a block that is not valid YAML at all is skipped rather than reported — it - cannot be classified, so the rule has no basis for claiming it is a broken - *example* as opposed to deliberately-invalid illustrative text. -* escape hatch: a block preceded by ```` is skipped, - for a future example that is intentionally partial in a way the heuristic - cannot see. - -Like CE027, this is intentionally NOT a ``BaseRule`` registered in -``tests/lint/runner.py``: that runner is AST-only and walks ``.py`` files, whereas -this rule reasons over Markdown. It is wired as a dedicated test in -``tests/test_custom_lint.py::TestCE029DocYamlExamples``. + ``variants``. Anything else, including a bare ``success_criteria:`` list, is a + fragment and is skipped. +* a **schematic** block (``agent: { ... }``, parsed as key ``"..."``) is skipped. +* a block that is not valid YAML is skipped, not reported: it cannot be classified, + so it may be deliberately invalid illustrative text. +* a block preceded by ```` is skipped. + +BLIND SPOT: a broken example missing a classifying key is never validated. + +Wired as ``tests/test_custom_lint.py::TestCE029DocYamlExamples``, not the AST runner: +it reasons over Markdown. + +Rationale: .claude/notes/lint-rules.md § CE029 """ from __future__ import annotations diff --git a/tests/lint/doc_indexes.py b/tests/lint/doc_indexes.py index 612a5f7bd..98ecf888a 100644 --- a/tests/lint/doc_indexes.py +++ b/tests/lint/doc_indexes.py @@ -1,26 +1,22 @@ """CE028 — the flat doc-index surfaces are generated from the mkdocs nav. -The docs overhaul's root cause was doc/code drift; the index surfaces -(``README.md``'s Documentation table, ``docs/index.md``'s "Where to go next" -table, and the ``## Docs`` / ``## Tutorials`` sections of ``docs/llms.txt``) drift -the same way — a page is added to the nav and forgotten in the three flat lists, -or a page is deleted and left dangling in them. This module makes ``nav`` (plus -the ``extra.docs_index`` blurb map) in ``mkdocs.yml`` the single source of truth: -``write()`` renders all three surfaces from it, ``make docs-indexes`` calls -``write()``, and CE028 (``check()``) re-renders and diffs against disk. There is -deliberately **no ``--check`` mode and no arg parser** — CE028 *is* the checker; a -second entry point would be untested duplication. - -CE028 also enforces the invariants the render depends on: every nav page has a -blurb and every blurb has a nav page (bijection, tutorial leaves exempted), every -published ``docs/*.md`` is in the nav (the check that would have caught this whole -overhaul's bug class), and the hand-written ``docs/tutorials/README.md`` table -stays in parity with the nav's tutorial pages (that table is **checked, not -generated** — generating it would need a second per-page field). - -Like CE027/CE029/CE030 this is not a ``BaseRule`` in the AST runner; it reasons -over Markdown/YAML and is wired as +``nav`` plus the ``extra.docs_index`` blurb map in ``mkdocs.yml`` is the single source +of truth for ``README.md``'s Documentation table and ``docs/index.md``'s "Where to go +next" table (between ``docs-index:start`` / ``docs-index:end``), and for the +``## Docs`` / ``## Tutorials`` sections of ``docs/llms.txt`` (between the ``:docs`` / +``:tutorials`` marker pairs). ``write()`` fills the markers (``make docs-indexes``); +``check()`` re-renders and diffs against disk. There is no ``--check`` mode: CE028 is +the checker. + +CE028 also checks that every nav page has a blurb and every blurb a nav page (tutorial +leaves exempt), that every published ``docs/*.md`` is in the nav, and that the +hand-written ``docs/tutorials/README.md`` table matches the nav's tutorial leaves +(checked, not generated). + +Not a ``BaseRule``: it reasons over Markdown/YAML, and is wired as ``tests/test_custom_lint.py::TestCE028DocIndexParity``. + +Rationale: .claude/notes/lint-rules.md § CE028 """ from __future__ import annotations diff --git a/tests/lint/doc_schema_parity.py b/tests/lint/doc_schema_parity.py index 80fc44a98..2100c8faf 100644 --- a/tests/lint/doc_schema_parity.py +++ b/tests/lint/doc_schema_parity.py @@ -1,43 +1,21 @@ """CE030 — models the project commits to documenting must have no undocumented fields. -Every defect this docs overhaul fixed was the same failure: a doc claim that no -longer matched (or never matched) the code. P0 and P1 were literally "a Pydantic -field the user must set, documented nowhere." CE030 is the sensor that makes that -class impossible to reintroduce: for a small, explicit registry of user-facing -models, every field must appear in the model's doc page as inline code, or be -listed in ``EXEMPT`` with a reason it is not user-authored. - -Design choices, each load-bearing: - -* **Allowlist, not denylist.** A new field on a registered model that is neither - documented nor exempted *fails* — which is the point. Adding a user-facing field - now forces a doc update or a reasoned exemption in the same change. -* **Explicit registry, no recursion.** Only the six registered models are - checked; nested models (``AgentConfig``, ``SandboxConfig``, criteria, …) are NOT - walked. Walking them would silently expand the documentation commitment to - dozens of models nobody signed up for. ``CliMatch`` is deliberately absent for - that reason: its fields are documented in the ``cli_called`` reference, and - registering a third nested model under ``SandboxConfig`` would start exactly - the tree-walk this bullet exists to prevent. A new field on a registered model - fails ``make lint`` until it is documented or exempted -- that is the intent, - not a bug in the rule. -* **Inline-code match, deliberately simple.** A field counts as documented when - its bare name appears wrapped in Markdown inline-code backticks anywhere in the - doc. This is a floor, not a proof — a field name that appears in an unrelated - context (e.g. a - common word like ``rows``) can pass spuriously. Accepted: the rule exists to - catch *entirely undocumented* fields, and a fuzzier "documented in the right - section" rule invites false passes that erode trust in the gate. - - A corollary for models that share a vocabulary: ``RecordedCli`` and - ``CliResponse`` both declare ``exit_code`` / ``stdout`` / ``stderr``, so - registering the second only newly guards ``when``, and a FUTURE field on either - that reuses a name the other already documents passes without its own doc line. - Still net-positive, but do not read a green gate here as per-model coverage. - -Like CE027/CE029, this is intentionally NOT a ``BaseRule`` registered in -``tests/lint/runner.py`` (that runner is AST-only over ``.py`` files); it reasons -over Markdown and is wired as ``tests/test_custom_lint.py::TestCE030DocSchemaParity``. +``DOCUMENTED_MODELS`` is the registry of tracked models, each paired with its doc +page. Every field of a registered model must appear in that page as Markdown inline +code, or be listed in ``EXEMPT`` with a reason it is not user-authored. A new field +that is neither fails ``make lint``; that is the intent. + +Nested models (``SandboxConfig``, ``CliMatch``, criteria) are NOT walked; each +registration is a standing documentation obligation. + +BLIND SPOT: a field name used as inline code in an unrelated context (e.g. ``rows``) +passes. Models sharing field names (``RecordedCli`` / ``CliResponse``) share coverage, +so a green gate is not per-model coverage. + +Wired as ``tests/test_custom_lint.py::TestCE030DocSchemaParity``, not the AST runner: +it reasons over Markdown. + +Rationale: .claude/notes/lint-rules.md § CE030 """ from __future__ import annotations diff --git a/tests/lint/live_verdict_contract.py b/tests/lint/live_verdict_contract.py index 75ab8c5ef..c56e4340e 100644 --- a/tests/lint/live_verdict_contract.py +++ b/tests/lint/live_verdict_contract.py @@ -1,72 +1,24 @@ """CE036 — every live-observable criterion must honor the ``live_verdict`` contract. -``EarlyStopWatcher``'s deferred fail-stop, verdict latching, and ``_prev_verdicts`` -flip-attribution (``orchestration/early_stop.py``) are correct ONLY because every -armed criterion's ``live_verdict`` is: - -* **deterministic** — a pure function of the ``turn_records`` prefix handed in, with - no wall-clock, randomness, or hidden instance state; and -* **monotonic** — once it returns ``"pass"``/``"fail"`` for some trajectory prefix it - returns that SAME verdict for every longer prefix. ``"undecided"`` is the only - verdict allowed to change. - -That contract is documented on ``LiveVerdict`` / ``BaseCriterion.live_verdict`` -(``criteria/base.py``) but, until this rule, nothing enforced it: a third criterion -(in-tree or third-party plugin) implementing ``live_verdict`` non-monotonically would -type-check, pass CE025, and silently corrupt the stop logic — latching a verdict the -run then contradicts. See GitHub issue #61 item 2. - -Design choices, each load-bearing: - -* **Replay, not static analysis.** Monotonicity over arbitrary Python is undecidable, - so there is no sound *static* check to write. What IS mechanical is replaying a - criterion against every prefix of a recorded trajectory and asserting the property - directly. That is what ``contract_violations`` does. -* **Seeded permutations widen the walk.** ``permuted_violations`` re-runs the - determinism + monotonicity walk over seeded reorderings of each case's commands — - an order-sensitive bug (verdict read off the *latest* command instead of the - accumulated set) can look perfectly monotone on the one ordering the author wrote - and flip on a reordering. Each shuffle is RENUMBERED (``sequence_number`` reassigned - in the new order) so it stays a trajectory the watcher could actually hand over — it - sorts by that field before calling ``live_verdict`` — which also keeps the layer - effective for a checker that sorts by it too. The terminal-verdict and polarity - checks stay authored-ordering-only, where they are sound. -* **Fixtures are mandatory, and the registry says so.** A property test over random - trajectories would return ``"undecided"`` almost always and pass *vacuously*, - proving nothing. So each live criterion type must supply cases in ``CASES``, and - ``missing_case_types`` — driven by the ``SuccessCriterion`` union, exactly like - CE025 — fails when a newly added ``LiveSuccessCriterion`` has none. Adding a live - criterion now forces the author to demonstrate the contract in the same change. -* **Each case declares what it reaches.** ``ContractCase.reaches`` pins the verdict on - the FULL trajectory, so a fixture that quietly stops exercising its decision path - (a renamed tool, a changed regex) fails loudly instead of degrading into another - vacuous all-``undecided`` replay. -* **Polarity honesty is checked too.** ``live_decidable_polarities`` (on the model) is - documented as a subset of what the checker's ``live_verdict`` can emit for that - instance. A case that terminally decides a polarity the instance does NOT claim is a - real bug — the watcher would treat that trigger as inert while the checker decides - it — so ``contract_violations`` reports it. - -**Honest limits.** (1) This proves the contract holds *on the trajectories the author -supplied*, not in general. A careless implementation with an agreeable fixture still -passes. The rule raises the cost of the bug and puts the contract in front of the next -implementer; it does not close the hole. Nothing short of a proof would. (2) It covers -the in-tree ``SuccessCriterion`` union only — an out-of-tree plugin criterion never -appears in ``live_criterion_types``, and this module lives under ``tests/`` (not shipped -in the wheel), so a plugin shipping a live criterion should copy the replay pattern — -a ``ContractCase``-style fixture plus the prefix walk — into its own test suite, with -this module as the reference implementation (docs/EXTENDING.md says so where plugin -authors will read it). (3) The determinism probe is two -back-to-back calls on identical input: it catches RNG and per-call mutable state, but -two calls microseconds apart will rarely disagree on a *wall-clock* read, so a -slowly-varying ``datetime.now()`` dependency largely escapes it (the monotonicity -replay is the likelier tripwire for one, and only if the fixture happens to straddle -the flip). - -Like CE025/CE030, this is intentionally NOT a ``BaseRule`` registered in -``tests/lint/runner.py`` (that runner is AST-only, one ``.py`` file at a time); it -reasons over the criteria registry and executes checkers, and is wired as -``tests/test_custom_lint.py::TestCE036LiveVerdictContract``. +``EarlyStopWatcher`` (``orchestration/early_stop.py``) is correct only if every armed +criterion's ``live_verdict`` is: + +* **deterministic** — a pure function of the ``turn_records`` prefix handed in; and +* **monotonic** — once it returns ``"pass"``/``"fail"`` for a prefix, it returns that + verdict for every longer prefix. Only ``"undecided"`` may change. + +``contract_violations`` replays each ``CASES`` fixture prefix by prefix; +``permuted_violations`` repeats that over seeded reorderings; ``missing_case_types`` and +``polarity_gaps`` fail a live type with no cases, or a claimed polarity no case reaches. + +**Honest limits.** (1) Proves the contract only on the supplied trajectories. +(2) In-tree ``SuccessCriterion`` union only; plugins copy the replay pattern +(docs/EXTENDING.md). (3) The determinism probe is two back-to-back calls, so it rarely +catches a slowly-varying wall-clock read. + +Wired as ``tests/test_custom_lint.py::TestCE036LiveVerdictContract``. + +Rationale: .claude/notes/lint-rules.md § CE036 """ from __future__ import annotations @@ -356,26 +308,18 @@ def _walk_prefixes( ) -> tuple[list[str], LiveVerdict | None]: """Prefix-by-prefix determinism + monotonicity walk over ONE command ordering. - The shared core of both replay modes: ``contract_violations`` walks the - fixture's authored ordering (and layers the terminal-verdict/polarity checks - on top), ``permuted_violations`` walks seeded reorderings (where those extra - checks would be unsound — see its docstring). Returns the breach list and the - full-trajectory verdict — or ``None`` for that verdict when the TERMINAL prefix - raised, since there is then no verdict to compare against and the stale value - from the previous prefix would stack a bogus breach on the real one. - - 1. **Determinism** — ``live_verdict`` called twice on an identical prefix must - agree. Catches RNG and per-call mutable state; NOT a reliable wall-clock - tripwire — the two calls land microseconds apart (module docstring, honest - limit 3). - 2. **Monotonicity** — once a prefix decides, every longer prefix returns that - same verdict. - 3. **No raising** — an exception from ``live_verdict`` is reported as a labeled - violation (case + prefix length) rather than crashing the walk; the remaining - prefixes still replay so one bad prefix does not mask breaches elsewhere. The - watcher runs mid-turn where a raise would take down the stop logic, and the - shape ``command_executed`` pins for a malformed regex — degrade to - ``"undecided"``, never raise — is the contract for every implementation. + Shared core of ``contract_violations`` (authored ordering) and + ``permuted_violations`` (seeded reorderings). Per prefix it reports: + + 1. **Determinism** — two calls on an identical prefix disagree. NOT a reliable + wall-clock tripwire (module docstring, honest limit 3). + 2. **Monotonicity** — a decided verdict changes on a longer prefix. + 3. **No raising** — any exception, as a labeled violation; the walk continues. + + Returns the breach list and the full-trajectory verdict, or ``None`` for that + verdict when the terminal prefix raised. + + Rationale: .claude/notes/lint-rules.md § CE036 """ violations: list[str] = [] decided: LiveVerdict | None = None @@ -468,27 +412,16 @@ def permuted_violations( ) -> list[str]: """Determinism + monotonicity under seeded reorderings of the case's commands. - ``contract_violations`` walks ONE ordering — the one the fixture author wrote. - But the contract quantifies over ANY trajectory, and the orderings an author - does not think of are exactly where an order-sensitive bug (e.g. a verdict - computed from the *latest* command instead of the accumulated set) hides: - such a checker can look perfectly monotone on the authored ordering and flip - on a reordering. Seeded shuffles probe those orderings essentially for free. - - Each shuffle is RENUMBERED (``sequence_number`` reassigned 0..N-1 in the new - order) so the permuted trajectory is one the runtime could actually produce: - ``EarlyStopWatcher._collect_verdicts`` keeps its partial trajectory sorted by - ``sequence_number``, so ``live_verdict`` never sees a list whose order - contradicts those numbers. Without the renumber this layer would (a) report - breaches on inputs the watcher cannot construct, and (b) degrade to a silent - no-op for any future checker that sorts by ``sequence_number`` itself — the - shuffle would just sort straight back to the authored ordering. - - Deliberately NOT checked here: ``case.reaches`` and polarity honesty. A - reordering may legitimately change the terminal verdict for a criterion whose - semantics are order-sensitive, so pinning either would make this layer - unsound for exactly the criteria it exists to probe. Both stay enforced on - the authored ordering by ``contract_violations``. + Runs ``_walk_prefixes`` over ``shuffles`` seeded shuffles, each RENUMBERED + (``sequence_number`` reassigned 0..N-1 in the new order) so it is a trajectory + the watcher could produce. Do not drop the renumber: without it, a checker that + sorts by ``sequence_number`` silently replays the authored ordering. + + Does NOT check ``case.reaches`` or polarity honesty: a reordering may + legitimately change the terminal verdict, so both are unsound here and stay + enforced on the authored ordering by ``contract_violations``. + + Rationale: .claude/notes/lint-rules.md § CE036 """ rng = random.Random(seed) violations: list[str] = [] diff --git a/tests/lint/plugin_manifest_parity.py b/tests/lint/plugin_manifest_parity.py index ba0076930..55d40ea3b 100644 --- a/tests/lint/plugin_manifest_parity.py +++ b/tests/lint/plugin_manifest_parity.py @@ -1,30 +1,18 @@ """CE044 — the marketplace entry and the plugin manifest are one metadata surface. -``.claude-plugin/marketplace.json`` is what the ``/plugin`` browser and the -plugin directories show *before* install; ``plugins/coder-eval/.claude-plugin/ -plugin.json`` is what an installed user's copy carries *after*. Six fields are -byte-identical duplicates across the two files (``description``, ``keywords``, -``author``, ``homepage``, ``repository``, ``license``, plus ``name`` and -``displayName``), and nothing compared them — the only test that reads -``plugin.json`` at all is ``tests/test_action_version_pin.py``, and only its -``version``. A one-sided edit — retitling the plugin in the marketplace but not -the manifest — would ship silently and show two different one-liners in the wild. - -The second half of the rule is the one that has already bitten: the marketplace -entry must not carry a *discovery* field the plugin manifest cannot mirror. The -marketplace schema allows both ``keywords`` ("Tags for plugin discovery and -categorization") and ``tags`` ("Tags for searchability and discovery"); the -plugin-manifest schema has no ``tags`` property at all. Splitting discovery -strings across the two therefore drops half of them from the installed copy, and -leaves a future editor with no rule for which list a new term belongs in. So an -extra key on the entry is a lint failure unless it is listed in -``MARKETPLACE_ONLY`` with a written reason — the allowlist *is* the rule, kept in -code rather than in tribal knowledge. - -Like CE026-CE031 and CE033 this reasons over whole files (JSON, plus resolving a -``source`` path to a directory) rather than one ``.py`` AST, so it is not a -``BaseRule`` in the runner; it is wired as a dedicated ``@pytest.mark.lint`` test -class in ``tests/test_custom_lint.py``. +``.claude-plugin/marketplace.json`` is what the ``/plugin`` browser shows before +install; ``plugins/coder-eval/.claude-plugin/plugin.json`` is what an installed copy +carries. For each marketplace entry: + +- its ``source`` must resolve to a directory that holds a plugin manifest; +- every key in ``SHARED_KEYS`` must be equal on both sides; +- any other key on the entry fails unless ``MARKETPLACE_ONLY`` lists it with a + reason. The allowlist is the rule: a new key needs a written reason. + +Not a ``BaseRule``: it reasons over whole JSON files, and is wired as a +``@pytest.mark.lint`` class in ``tests/test_custom_lint.py``. + +Rationale: .claude/notes/lint-rules.md § CE044 """ from __future__ import annotations diff --git a/tests/lint/plugin_reference.py b/tests/lint/plugin_reference.py index d4b77b87f..ecf862bfd 100644 --- a/tests/lint/plugin_reference.py +++ b/tests/lint/plugin_reference.py @@ -1,44 +1,23 @@ """CE033 — the plugin's bundled criteria reference is generated from the models. -An installed Claude Code plugin is copied to ``~/.claude/plugins/cache/`` without -its parent directories, so a skill cannot read ``docs/TASK_DEFINITION_GUIDE.md`` -from this repository at runtime — every reference a skill needs has to ship -*inside* ``plugins/coder-eval/``. A bundled copy of the criterion vocabulary is -exactly the kind of file that drifts: a criterion gains a field, or a whole 15th -criterion lands, and the copy quietly keeps teaching the old schema to every -plugin user. - -So the copy is not written by hand. The ``SuccessCriterion`` discriminated union -in ``coder_eval.models`` is the single source of truth; ``render_criteria()`` -renders ``plugins/coder-eval/reference/criteria.md`` from it, ``make -plugin-reference`` calls ``write()``, and CE033 (``check()``) re-renders and diffs -against disk. There is deliberately **no ``--check`` mode and no arg parser** — -CE033 *is* the checker; a second entry point would be untested duplication. - -Two rendering rules keep this small and are load-bearing: - -- Fields inherited from ``BaseSuccessCriterion`` / ``LiveSuccessCriterion`` are - documented once, in their own section, and **computed** — never a hardcoded - name list, which would be a second declaration of the base schema. (When - ``stop_early:`` replaced ``stop_when`` + ``max_steps_to_decide``, a hardcoded - list would have started rendering the new field into all 14 per-criterion - sections and leaked two dead names; the computed set absorbed it with no edit.) -- **Every** field gets its model description, required and optional alike, each in - a table of its own, the second group under an ``Optional:`` label. What a - field *means* is the half of the schema an authoring agent gets wrong (that - ``min_count: 0`` lets a criterion pass when nothing matched, that ``weight: 0`` - makes a criterion informational), so it is rendered in full — never truncated, - never a curated subset, which would need a hardcoded name list and so a second - declaration of the schema. Defaults and types are still deliberately absent: - rendering defaults would mean handling ``default_factory`` (whose - ``FieldInfo.default`` is ``PydanticUndefined``) and rendering types would mean - normalizing ``X | None`` annotations — two helpers serving the half of the - reference an authoring agent needs least. ``coder-eval plan`` and the model - docstrings cover the rest. - -Like CE026-CE031 this is not a ``BaseRule`` in the AST runner; it reasons over -Markdown and pydantic metadata, and is wired as +A skill in an installed plugin cannot read ``docs/``, so the criterion vocabulary ships +inside the plugin. ``render_criteria()`` renders +``plugins/coder-eval/reference/criteria.md`` from the ``SuccessCriterion`` union; +``make plugin-reference`` calls ``write()``; CE033 (``check()``) re-renders and diffs +against disk. There is no ``--check`` mode: CE033 is the checker. + +Rendering rules: + +- Inherited fields are rendered once, in their own section, from a COMPUTED set — + never a hardcoded name list. +- Every field gets its full model description; required and optional fields go in + separate tables. Defaults and types are not rendered. + +Each criterion's summary is the first line of its ``src/`` class docstring, so editing +that line changes the generated file. Wired as ``tests/test_custom_lint.py::TestCE033PluginReferenceParity``. + +Rationale: .claude/notes/lint-rules.md § CE033 """ from __future__ import annotations diff --git a/tests/lint/pricing_mirror.py b/tests/lint/pricing_mirror.py index 430799b77..059308038 100644 --- a/tests/lint/pricing_mirror.py +++ b/tests/lint/pricing_mirror.py @@ -1,36 +1,19 @@ """CE065 — the evalboard's rate table is generated from ``coder_eval.pricing``. -``evalboard/lib/pricing.ts`` used to carry a hand-copied mirror of the Python -rate card. Keeping a hand-copy honest needed five layers of bookkeeping: a -regex parser that re-read ``pricing.py`` at test time, a meta-guard against that -regex silently narrowing, a ``DELIBERATELY_UNMIRRORED`` exemption set, a -staleness guard for the exemption set, and a comment begging the next reader to -keep the set honest. It still shipped a real bug — ``claude-sonnet-5``, -``gpt-5.6-sol``, ``gpt-5.6-terra`` and ``gpt-5.6-luna`` sat in the exemption set -under "the evalboard never runs them" while appearing tens of thousands of times -in the run corpus, so every one of those runs rendered "—" for cost with nothing -failing. - -If a *test* can read the table, a *generator* can emit it. So the table is no -longer copied: ``render_pricing()`` renders -``evalboard/lib/pricing.generated.ts`` from ``pricing.builtin_rates()``, ``make -pricing-mirror`` calls ``write()``, and CE065 (``check()``) re-renders and diffs -against disk. There is deliberately **no ``--check`` mode and no arg parser** — -CE065 *is* the checker, the same rule ``plugin_reference.py`` states. - -The exemption set encoded TWO different things, and they survive differently. -That three OpenRouter models must stay unpriced so ``runs.ts``'s apportionment of -the provider's real bill still fires is a property of the RATE, so it is now data -on the rate itself (``ModelPricing.per_request_billing``), beside the rate it -qualifies; nothing has to remember it. That four heavy frontier variants are not -priced on the frontend is a property of the FRONTEND, not of the rate, so it -stays here as ``DELIBERATELY_UNMIRRORED`` — an explicit list with the same -stale-membership guard the deleted test carried, because an exemption nobody -re-reads is what shipped the bug above. - -Like CE028 and CE033 this is not a ``BaseRule`` in the AST runner: it reasons -over generated text rather than one Python AST, so it is wired as a -``@pytest.mark.lint`` class in ``tests/test_custom_lint.py``. +``render_pricing()`` renders ``evalboard/lib/pricing.generated.ts`` from +``pricing.builtin_rates()``; ``make pricing-mirror`` calls ``write()``, and CE065 +(``check()``) re-renders and diffs against disk. There is deliberately **no ``--check`` mode +and no arg parser** — CE065 *is* the checker, the rule ``plugin_reference.py`` states. + +Two sets stay off the board. A rate flagged ``ModelPricing.per_request_billing`` is omitted +because the provider bills per request, so the board shows the captured actual cost. +``DELIBERATELY_UNMIRRORED`` is a property of the FRONTEND, not of the rate, and carries a +stale-membership guard. + +Like CE028 and CE033 this is not a ``BaseRule`` in the AST runner: it reasons over generated +text, so it is wired as a ``@pytest.mark.lint`` class in ``tests/test_custom_lint.py``. + +Rationale: .claude/notes/lint-rules.md § CE065 """ from __future__ import annotations @@ -47,19 +30,12 @@ _MIRROR_REL = "evalboard/lib/pricing.generated.ts" _GENERATED_HEADER = "// generated by `make pricing-mirror` — do not edit" -# Priced in Python, deliberately NOT priced on the frontend: heavy frontier variants -# no harness runs, so a rate here buys nothing. Unlike ``per_request_billing`` this is -# not a fact about the rate — the rate is correct, the evalboard just has no use for -# it — which is why it lives beside the generator rather than on ``ModelPricing``. -# -# KEEP THIS SET HONEST. Membership silences the mirror for one id indefinitely, so a -# stale entry hides a live bug rather than a non-issue: `claude-sonnet-5`, -# `gpt-5.6-sol`, `gpt-5.6-terra` and `gpt-5.6-luna` sat in the predecessor of this set -# under "the evalboard never runs them" while appearing tens of thousands of times in -# the run corpus, so every one of those runs rendered "—" for cost with nothing -# failing. Before adding an id, grep the corpus for it — absence from run data is the -# ONLY justification, and it expires the moment a harness adopts the model. -# ``_assert_exemptions_are_live`` fails the build once an id leaves ``pricing.py``. +# Priced in Python, deliberately NOT priced on the frontend: heavy frontier variants no +# harness runs, so a rate here buys nothing. Unlike ``per_request_billing`` this is not +# a fact about the rate — the rate is correct, the evalboard just has no use for it — +# which is why it lives beside the generator rather than on ``ModelPricing``. +# KEEP THIS SET HONEST: grep the run corpus before adding an id. +# Rationale: .claude/notes/lint-rules.md § Keeping DELIBERATELY_UNMIRRORED honest DELIBERATELY_UNMIRRORED = frozenset( { "gpt-5.4-mini", diff --git a/tests/lint/prose_budget.py b/tests/lint/prose_budget.py index 518f5efe7..1226699fc 100644 --- a/tests/lint/prose_budget.py +++ b/tests/lint/prose_budget.py @@ -1,9 +1,9 @@ -"""Measure and ratchet the essay-shaped prose in ``src/coder_eval``. +"""Check the docstring and comment prose rules in ``src/coder_eval`` and ``tests`` (``_ROOTS``). -One gated number: ``essay_words`` — words in docstrings over 150 words (Typer command -docstrings exempt, they render as ``--help``) plus words in comment runs of three or -more consecutive lines. Each file's comments are capped as a SHARE OF ITS LENGTH, so the house style -is *no new essays*, not *no new documentation*. +Two per-file rules, with no baseline: no docstring over 150 prose words (Typer commands +and ``@abstractmethod`` exempt), and own-line comments within a SHARE OF THE FILE'S +LENGTH. The house style is *no new essays*, not *no new documentation*. The report also +lists words in comment runs of three or more lines, for orientation only. Also resolves every ``Rationale: § `` pointer, and — under ``--assert-code-unchanged `` — proves a commit moved prose only, by comparing the @@ -28,11 +28,18 @@ _DOCSTRING_ESSAY_WORDS = 150 _COMMENT_BLOCK_LINES = 3 -# Own-line comments a file may carry: a FLOOR for small files, then a share of its -# length. Proportional on purpose — there is no tree-wide total to hand-maintain, a -# file that loses code loses budget with it, and a NEW file is governed from its -# first commit. The tree's natural maximum sits just under this (a constants module -# at one comment per constant); the floor is what protects those. +# The longest own-line comment RUN a file may carry, and the blank lines a run +# reads through. A cap on the block: the shape a comment may not take is a +# PARAGRAPH. ONE blank line, not two: one is how a paragraph is split to duck the +# cap, two is the separation PEP 8 already puts between a banner and its section. +_COMMENT_RUN_LINES = 8 +_RUN_BLANK_BRIDGE = 1 + +# Own-line comments a file may carry in TOTAL: a FLOOR for small files, then a +# share of its length. The run cap governs the shape of any one comment; this is +# the outlier backstop for a file that is mostly commentary however it is broken +# up. Proportional on purpose — no tree-wide total to hand-maintain, a file that +# loses code loses budget with it, and a NEW file is governed from its first commit. _COMMENT_LINE_FLOOR = 20 _COMMENT_LINE_RATIO = 0.15 @@ -51,22 +58,22 @@ "Examples:", ) -_SRC = Path("src/coder_eval") +_ROOTS: tuple[Path, ...] = (Path("src/coder_eval"), Path("tests")) -# Exempt by (path relative to src/coder_eval, function name) pair, and only for a -# function at module level: `Sandbox.run_command` is a method and a bare-name exemption -# would silently excuse it. Registered in src/coder_eval/cli/__init__.py. +# Exempt by (repo-relative path, function name) pair, and only for a function at module +# level: `Sandbox.run_command` is a method and a bare-name exemption would silently +# excuse it. Registered in src/coder_eval/cli/__init__.py. _TYPER_COMMANDS = frozenset( { - ("cli/run_command.py", "run_command"), - ("cli/execute_command.py", "execute_command"), - ("cli/plan_command.py", "plan_command"), - ("cli/evaluate_command.py", "evaluate_command"), - ("cli/report_command.py", "report_command"), - ("cli/aggregate_command.py", "aggregate_command"), - ("cli/export_command.py", "export_command"), - ("cli/harbor_command.py", "reward_command"), - ("cli/run_task_internal_command.py", "run_task_internal_command"), + ("src/coder_eval/cli/run_command.py", "run_command"), + ("src/coder_eval/cli/execute_command.py", "execute_command"), + ("src/coder_eval/cli/plan_command.py", "plan_command"), + ("src/coder_eval/cli/evaluate_command.py", "evaluate_command"), + ("src/coder_eval/cli/report_command.py", "report_command"), + ("src/coder_eval/cli/aggregate_command.py", "aggregate_command"), + ("src/coder_eval/cli/export_command.py", "export_command"), + ("src/coder_eval/cli/harbor_command.py", "reward_command"), + ("src/coder_eval/cli/run_task_internal_command.py", "run_task_internal_command"), } ) @@ -202,12 +209,31 @@ def measure_source(source: str, rel: str) -> FileProse | None: ) +def _roots(repo_root: Path) -> tuple[Path, ...]: + """``_ROOTS``, after checking each one exists. + + Raises: + FileNotFoundError: a root is not a directory — a renamed root must fail the gate, + never measure zero files. + """ + for root in _ROOTS: + if not (repo_root / root).is_dir(): + raise FileNotFoundError(f"prose budget root does not exist: {root}") + return _ROOTS + + +def _python_files(repo_root: Path) -> list[tuple[Path, Path]]: + """``(absolute, repo-relative)`` for every ``*.py`` under each root, sorted.""" + return sorted( + (path, path.relative_to(repo_root)) for root in _roots(repo_root) for path in (repo_root / root).rglob("*.py") + ) + + def measure(repo_root: Path) -> Measurement: - """Scan ``src/coder_eval``. Files that do not parse are skipped, never fatal.""" + """Scan the configured roots. Files that do not parse are skipped, never fatal.""" files: dict[Path, FileProse] = {} skipped: list[Path] = [] - for path in sorted((repo_root / _SRC).rglob("*.py")): - rel = path.relative_to(repo_root / _SRC) + for path, rel in _python_files(repo_root): prose = measure_source(path.read_text(encoding="utf-8"), rel.as_posix()) if prose is None: skipped.append(rel) @@ -222,7 +248,9 @@ def total_words(files: dict[Path, FileProse]) -> int: def _subsystem(rel: Path) -> str: - return rel.parts[0] if len(rel.parts) > 1 else "top-level" + root = next(root for root in _ROOTS if rel.is_relative_to(root)) + below = rel.relative_to(root) + return f"{root.as_posix()}/{below.parts[0]}" if len(below.parts) > 1 else root.as_posix() def render_report(measurement: Measurement) -> str: @@ -236,9 +264,9 @@ def render_report(measurement: Measurement) -> str: lines.append(subsystem) for rel, prose in rows: lines.append( - f" {rel.as_posix():<48}{prose.docstring_words:>7} doc{prose.comment_words:>7} cmt{prose.total:>8}" + f" {rel.as_posix():<64}{prose.docstring_words:>7} doc{prose.comment_words:>7} cmt{prose.total:>8}" ) - lines.append(f" {'subtotal':<48}{'':>7} {'':>7} {sum(p.total for _, p in rows):>8}") + lines.append(f" {'subtotal':<64}{'':>7} {'':>7} {sum(p.total for _, p in rows):>8}") lines.append("") files = measurement.files @@ -260,7 +288,7 @@ def render_report(measurement: Measurement) -> str: key=lambda row: (-row[2], row[0].as_posix(), row[1]), ) lines += ["", f"ESSAYS ({len(roster)} docstrings over {_DOCSTRING_ESSAY_WORDS} words)"] - lines += [f" {f'{rel.as_posix()}::{name}':<68}{words:>6}" for rel, name, words in roster] + lines += [f" {f'{rel.as_posix()}::{name}':<84}{words:>6}" for rel, name, words in roster] return "\n".join(lines) + "\n" @@ -290,8 +318,7 @@ def check_pointers(repo_root: Path) -> list[str]: """Every ``Rationale: § `` must resolve. Returns the failures.""" failures: list[str] = [] headings: dict[Path, set[str]] = {} - for path in sorted((repo_root / _SRC).rglob("*.py")): - rel = path.relative_to(repo_root / _SRC) + for path, rel in _python_files(repo_root): for line in _prose_lines(path.read_text(encoding="utf-8")): match = _POINTER.search(line.strip()) if not match: @@ -323,8 +350,8 @@ def check_pointer_placement(repo_root: Path) -> list[str]: file unparseable, which the measurement silently reports as zero words. """ failures: list[str] = [] - for path in sorted((repo_root / _SRC).rglob("*.py")): - rel = path.relative_to(repo_root / _SRC).as_posix() + for path, rel_path in _python_files(repo_root): + rel = rel_path.as_posix() source = path.read_text(encoding="utf-8") lines = source.split("\n") @@ -370,6 +397,38 @@ def check_pointer_placement(repo_root: Path) -> list[str]: return failures +def own_comment_runs(source: str) -> list[tuple[int, int]]: + """``(first line, length)`` for every own-line comment run, in file order. + + A run is consecutive own-line comments, reading through up to + ``_RUN_BLANK_BRIDGE`` blank lines so that splitting a paragraph on whitespace + does not read as several short comments. Code between two comments always ends + the run. A trailing ``# noqa`` never starts one: it is a directive, not + commentary. + + Blind spot: at module scope ruff format keeps two blank lines, so a paragraph split + on two blank lines there reads as two runs. + """ + lines = source.split("\n") + try: + tokens = list(tokenize.generate_tokens(io.StringIO(source).readline)) + except (SyntaxError, tokenize.TokenError, ValueError): + return [] + own = sorted( + token.start[0] + for token in tokens + if token.type == tokenize.COMMENT and lines[token.start[0] - 1].strip().startswith("#") + ) + runs: list[list[int]] = [] + for line in own: + bridged = runs and line - runs[-1][-1] <= _RUN_BLANK_BRIDGE + 1 + if bridged and all(not lines[between - 1].strip() for between in range(runs[-1][-1] + 1, line)): + runs[-1].append(line) + else: + runs.append([line]) + return [(run[0], len(run)) for run in runs] + + def comment_line_budget(total_lines: int) -> int: """A file's own-line comment allowance.""" return max(_COMMENT_LINE_FLOOR, round(_COMMENT_LINE_RATIO * total_lines)) @@ -378,12 +437,14 @@ def comment_line_budget(total_lines: int) -> int: def check_comment_density(repo_root: Path) -> list[str]: """Every file's own-line comments must fit :func:`comment_line_budget`. - Own-line only. A trailing ``# noqa`` is a directive, not commentary, and a - per-member annotation on an enum is the contract a dispatcher reads — counting - either would push against documenting them. + The backstop under :func:`check_comment_runs`: a file may pass the run cap with + every block short and still be mostly commentary. Own-line only. A trailing + ``# noqa`` is a directive, not commentary, and a per-member annotation on an enum + is the contract a dispatcher reads — counting either would push against + documenting them. """ failures: list[str] = [] - for path in sorted((repo_root / _SRC).rglob("*.py")): + for path, rel in _python_files(repo_root): source = path.read_text(encoding="utf-8") try: tokens = list(tokenize.generate_tokens(io.StringIO(source).readline)) @@ -397,14 +458,29 @@ def check_comment_density(repo_root: Path) -> list[str]: } budget = comment_line_budget(len(lines)) if len(own) > budget: - rel = path.relative_to(repo_root / _SRC).as_posix() failures.append( - f"{rel}: {len(own)} own-line comments against a budget of {budget} " + f"{rel.as_posix()}: {len(own)} own-line comments against a budget of {budget} " f"({len(lines)} lines). Move rationale to .claude/notes/." ) return failures +def check_comment_runs(repo_root: Path) -> list[str]: + """No own-line comment run may exceed :data:`_COMMENT_RUN_LINES`. + + A cap on the BLOCK, with no per-file allowance: a file may carry any number of + one-line notes, and none of them may grow into a paragraph. The essay bar in + prose, one granularity down. + """ + return [ + f"{rel.as_posix()}:{line}: comment run of {length} lines " + f"(bar is {_COMMENT_RUN_LINES}). Move the narrative to .claude/notes/." + for path, rel in _python_files(repo_root) + for line, length in own_comment_runs(path.read_text(encoding="utf-8")) + if length > _COMMENT_RUN_LINES + ] + + def check_essays(repo_root: Path) -> list[str]: """No docstring may exceed the prose bar unless it is an interface contract. @@ -420,6 +496,17 @@ def check_essays(repo_root: Path) -> list[str]: ] +def collect_failures(repo_root: Path) -> list[str]: + """Every check's failures, each prefixed with the check that raised it.""" + return ( + [f"unresolved pointer: {failure}" for failure in check_pointers(repo_root)] + + [f"misplaced pointer: {failure}" for failure in check_pointer_placement(repo_root)] + + [f"comment run: {failure}" for failure in check_comment_runs(repo_root)] + + [f"comment budget: {failure}" for failure in check_comment_density(repo_root)] + + [f"docstring essay: {failure}" for failure in check_essays(repo_root)] + ) + + def code_shape(source: str) -> str: """``ast.dump`` of the module with every docstring filtered out of its body. @@ -463,13 +550,17 @@ def _git(repo_root: Path, *args: str) -> tuple[int, str]: def assert_code_unchanged(repo_root: Path, ref: str) -> list[str]: - """Report every ``src/coder_eval`` file whose code — not prose — differs from ``ref``.""" - code, listing = _git(repo_root, "diff", "--name-only", ref, "--", _SRC.as_posix()) + """Report every file under the configured roots whose code — not prose — differs from ``ref``.""" + roots = [root.as_posix() for root in _roots(repo_root)] + code, listing = _git(repo_root, "diff", "--name-only", ref, "--", *roots) if code != 0: return [f"git diff against {ref!r} failed"] + code, untracked = _git(repo_root, "ls-files", "--others", "--exclude-standard", "--", *roots) + if code != 0: + return ["git ls-files for untracked files failed"] findings: list[str] = [] - for name in sorted(filter(None, listing.splitlines())): + for name in sorted(set(filter(None, (listing + untracked).splitlines()))): if not name.endswith(".py"): continue shown, before = _git(repo_root, "show", f"{ref}:{name}") @@ -482,16 +573,27 @@ def assert_code_unchanged(repo_root: Path, ref: str) -> list[str]: except SyntaxError: findings.append(f"{name}: could not parse both revisions") continue - dropped = directive_comments(before) - directive_comments(after) - findings += [ - f"{name}: dropped directive comment {comment!r} x{count}" for comment, count in sorted(dropped.items()) - ] + before_directives, after_directives = directive_comments(before), directive_comments(after) + for verb, changed in ( + ("dropped", before_directives - after_directives), + ("added", after_directives - before_directives), + ): + findings += [ + f"{name}: {verb} directive comment {comment!r} x{count}" for comment, count in sorted(changed.items()) + ] return findings def main(argv: list[str]) -> int: repo_root = Path(__file__).resolve().parents[2] + try: + return _run(repo_root, argv) + except FileNotFoundError as error: + print(error, file=sys.stderr) + return 1 + +def _run(repo_root: Path, argv: list[str]) -> int: if argv[:1] == ["--assert-code-unchanged"]: if len(argv) != 2: print("usage: --assert-code-unchanged ", file=sys.stderr) @@ -500,23 +602,16 @@ def main(argv: list[str]) -> int: for finding in findings: print(finding, file=sys.stderr) return 1 if findings else 0 + if argv: + print("usage: prose_budget.py [--assert-code-unchanged ]", file=sys.stderr) + return 2 print(render_report(measure(repo_root)), end="") - failed = False - for failure in check_pointers(repo_root): - print(f"unresolved pointer: {failure}", file=sys.stderr) - failed = True - for failure in check_pointer_placement(repo_root): - print(f"misplaced pointer: {failure}", file=sys.stderr) - failed = True - for failure in check_comment_density(repo_root): - print(f"comment budget: {failure}", file=sys.stderr) - failed = True - for failure in check_essays(repo_root): - print(f"docstring essay: {failure}", file=sys.stderr) - failed = True - return 1 if failed else 0 + failures = collect_failures(repo_root) + for failure in failures: + print(failure, file=sys.stderr) + return 1 if failures else 0 if __name__ == "__main__": diff --git a/tests/lint/rules/_layers.py b/tests/lint/rules/_layers.py index 034336775..e638ad646 100644 --- a/tests/lint/rules/_layers.py +++ b/tests/lint/rules/_layers.py @@ -1,49 +1,22 @@ """The package-layer predicates, declared once and shared by CE004 and CE066. -Both rules ask where a file sits in ``src/coder_eval/``, and a second copy of -the answer is how a package added to one regex silently escapes the other. So -the package anchor and the ``cli/`` boundary are each spelled once, here. -``_model_ctor.py`` is the in-tree precedent for a ``_``-prefixed shared rule -helper. - -The two rules do NOT share an exemption set, because they do not ask the same -question. CE004 bans ``cli`` imports from everything that must run without the -CLI, which is the whole package except ``cli/`` itself. CE066 bans reaching into -the reports layer, which ``reports/`` may obviously do to itself, so its "core" -also excludes ``reports/``. When CE004 borrowed CE066's predicate wholesale it -inherited the ``reports/`` exemption, and a ``cli`` import added inside the -reports package — which the orchestrator imports mid-run, closing a -cli -> orchestration -> reports -> cli cycle — would have passed silently. - -Both scopes are ALLOWLISTS of what is exempt, so a new subpackage is in scope by -default rather than exempt until someone notices. The denylist form is what -leaves holes, twice over. An earlier draft named only -``orchestrator.py`` as the top-level core module, which exempted -``result_metrics.py`` — the very module CE066's fix message tells a violator to -move their metric into — along with ``run_record.py``, ``stats.py`` and -``timing.py``. Its successor listed ten core directories and ``isolation/`` was -not one of them, so ``isolation/docker_runner.py`` — the ``driver: docker`` -evaluation path, which imports ``models``, ``orchestration`` and ``streaming`` — -could import anything with both rules silent. - -A relative import is RESOLVED against the importing file rather than pattern- -matched: ``from .reports import x`` means ``coder_eval.reports`` in a top-level -module and ``coder_eval.orchestration.reports`` inside ``orchestration/``, so the -dots have to be counted against the file's own package. See ``_absolute_module``. - -The package regex is anchored on ``src/`` because the unanchored form made a -repo-root file core: this project's own checkout directory is named -``coder_eval``, so ``…/coder_eval/conftest.py`` matched the package. - -Blind spot: anchoring narrows that trap without closing it. A clone whose parent -directory is literally named ``src`` — ``~/src/coder_eval/conftest.py`` — still -matches, and so now does that clone's ``tests/`` tree. No path substring can -separate the package from a checkout laid out like it; closing it properly means -relativising every rule's path against the repo root. It is unreachable today: -CE004 and CE066 are only ever handed paths under the runner's ``SRC``, never the -repo root, and ``_ALSO_SCAN_TESTS`` is ``{"CE048"}``, which uses neither -predicates. ``TestCoreLayerMembership`` pins the residual so nobody reads the -anchoring as a complete fix. +Both rules ask where a file sits in ``src/coder_eval/``, so the package anchor and the +``cli/`` boundary are each spelled once, here. ``_model_ctor.py`` is the precedent for a +``_``-prefixed shared rule helper. + +The two do NOT share an exemption set. CE004's scope is the package minus ``cli/``; CE066 +also excludes ``reports/``, which may reach into itself. Both scopes are ALLOWLISTS, so a +new subpackage is in scope by default. + +A relative import is RESOLVED against the importing file, not pattern-matched: +``from .reports import x`` means ``coder_eval.reports`` at top level and +``coder_eval.orchestration.reports`` inside ``orchestration/``. See ``_absolute_module``. + +BLIND SPOT: the package regex is anchored on ``src/``, which narrows but does not close the +trap of a checkout laid out like the package. Unreachable today; +``TestCoreLayerMembership`` pins the residual. + +Rationale: .claude/notes/lint-rules.md § _layers """ import ast diff --git a/tests/lint/rules/_model_ctor.py b/tests/lint/rules/_model_ctor.py index da355c9d0..297c7502f 100644 --- a/tests/lint/rules/_model_ctor.py +++ b/tests/lint/rules/_model_ctor.py @@ -1,21 +1,16 @@ """Resolve `coder_eval.models` constructor calls inside one module's AST. -CE060 and CE061 ask the same first question — *is this call building an -`AssistantMessage`?* — and answering it takes more than matching a name: a -module may bind the class under any alias, reach it through a relative import, -or never bind it at all and spell it `models.AssistantMessage(...)`. CE060 -worked that out once; duplicating it into CE061 would mean a model rename or a -new import spelling needs two fixes in two rules, and the second one is the one -that gets missed. So it lives here and both rules consume it. - -The class name is taken from the model itself rather than written as a string, -the way CE056 imports `IN_CONTAINER_ENV` and CE057 derives its target set from -`SIDECAR_MODULES`: renaming the model moves both rules with it. +CE060, CE061 and CE064 consume it to ask whether a call builds a given class +under any alias, a relative import, or the `models.AssistantMessage(...)` +spelling. A new import spelling is fixed here, once. The class name comes from +the model itself, so renaming the model moves every consumer. BLIND SPOT, inherited by every consumer: a re-export through an intermediate module (`from .sibling import AssistantMessage`) is invisible, because resolving it means following imports across files and no rule in this package does that. + +Rationale: .claude/notes/lint-rules.md § _model_ctor """ import ast @@ -45,14 +40,12 @@ def reaches_module(node: ast.ImportFrom, module_path: str) -> bool: return True if not node.level: return False - # A relative spelling carries only a TRAILING SLICE of the absolute path, - # and how much of it depends on the dot count: `from ..timing import` gives - # "timing", `from ..streaming.events import` gives "streaming.events". So - # match any suffix of the target, segment-wise, allowing the import to - # continue on into a submodule below it (`..models.criteria`). Comparing a - # single `rpartition` tail was right only while every target was - # one segment deep; it silently missed `coder_eval.streaming.events` - # entirely, which is a rule blind for a whole file rather than a near miss. + # A relative spelling carries only a TRAILING SLICE of the absolute path, and its + # length depends on the dot count: `from ..timing import` gives "timing", + # `from ..streaming.events import` gives "streaming.events". So match any suffix of + # the target, segment-wise, allowing the import to continue into a submodule below + # it (`..models.criteria`). A single `rpartition` tail missed a multi-segment target + # such as `coder_eval.streaming.events` entirely. segments = module_path.split(".") spelled = module.split(".") return any(spelled[: len(segments) - i] == segments[i:] for i in range(1, len(segments))) @@ -89,7 +82,7 @@ def local_bindings(tree: ast.AST, class_name: str) -> set[str]: def constructor_name(func: ast.expr, names: set[str], class_name: str) -> str | None: - """The spelling this call used to name the model, or None if it did not. + """The spelling this call names the model with, or None if it does not name it. A bare name has to be bound in this module to be ours; the attribute spelling is matched on the attribute alone, since the module binding it diff --git a/tests/lint/rules/ce014_merge_strategy_declared.py b/tests/lint/rules/ce014_merge_strategy_declared.py index 28b37d3ee..0673e5e39 100644 --- a/tests/lint/rules/ce014_merge_strategy_declared.py +++ b/tests/lint/rules/ce014_merge_strategy_declared.py @@ -1,29 +1,18 @@ """CE014: every ``list``-typed field on a config-merge root model must declare an explicit ``MergeField(strategy=...)``. -The declarative merge engine reads a per-field strategy off the Pydantic -``FieldInfo`` (``coder_eval.models.merge_strategy.merge_strategy_of``), falling -back to a type-aware default: nested ``BaseModel`` / free-form ``dict`` -> ``deep``; -``list`` / scalar -> ``replace``. A ``list`` is the one type whose default -(``replace``) is easy to mean-otherwise (``append``) — so a list field that -silently keeps the ``replace`` default when it meant ``append`` is a latent -resolution bug. This rule forces the choice to be explicit. - -Scope is the set of model classes the engine actually feeds through -``merge_layers`` / ``resolve_root`` — the three ``-D``-reachable roots, the -sandbox sub-models reached by deep merge, AND the two models merged outside the -``-D`` roots (``TaskDefinition`` for ``pre_run``/``post_run``, ``SimulationConfig`` -for ``constraints``). Scoping by class name (not file) keeps the rule pinned to -the engine's real roots and avoids flagging unrelated list fields that happen to -share a file (e.g. ``PreRunCommand`` in ``tasks.py``). - -Nested-``BaseModel`` and free-form ``dict`` fields are allowed to rely on the -type-aware ``deep`` default (a plain ``Field(...)`` is fine) — the nested-replace -regression is structurally impossible for them. An explicit ``replace`` override -on a list is permitted and visible; this rule only requires the annotation, not -a particular strategy. +The rule requires the annotation, not a particular strategy: an explicit +``replace`` on a list is fine. Nested-``BaseModel`` and free-form ``dict`` +fields may use a plain ``Field(...)``. Default strategies: +``coder_eval.models.merge_strategy.merge_strategy_of``. + +Scope: classes named in ``_MERGE_ROOT_CLASSES``, inside ``_SCOPED_PATHS``, +matched by class name. A model newly fed through ``merge_layers`` / +``resolve_root`` goes unchecked until it is added there. Add ``# noqa: CE014`` on the field line for an intentional exception. + +Rationale: .claude/notes/lint-rules.md § CE014 """ import ast diff --git a/tests/lint/rules/ce019_telemetry_non_fatal.py b/tests/lint/rules/ce019_telemetry_non_fatal.py index baa35442c..49f21fe7c 100644 --- a/tests/lint/rules/ce019_telemetry_non_fatal.py +++ b/tests/lint/rules/ce019_telemetry_non_fatal.py @@ -1,22 +1,18 @@ """CE019: telemetry public functions must wrap their body in try/except Exception. -Telemetry is a side-channel that must NEVER raise into a run. Each public -function in ``src/coder_eval/telemetry.py`` therefore has to guard its whole -body with a broad ``try/except Exception`` (or a bare ``except``). This rule -turns that design invariant into mechanical enforcement. - -Scope: only files ending in ``telemetry.py`` under ``src/coder_eval/``, and only -the four core public functions (the explicit allowlist below). Private helpers -(``_coerce_props``) and the ``track_command`` decorator are out of scope — their -only side-effecting call is ``track_event``, which is itself guarded. - -A function body is "guarded" when, after skipping a leading docstring, ``global`` -declaration, and any leading no-op guard clauses (``if ...: return``/``pass``), -the remaining body is a single ``try`` whose handlers include a broad -``except Exception`` or a bare ``except``. - -Allowlist caveat (mirrors CE002): the function-name set is explicit — extend it -when a new public telemetry function is added. +Telemetry is a side-channel that must NEVER raise into a run. + +Scope: files ending in ``telemetry.py`` under ``src/coder_eval/``, and only the +functions in ``_GUARDED_FUNCTIONS``. Excluding the rest is safe: ``_coerce_props`` +and the ``track_command`` decorator reach side effects only through ``track_event``, +and the other private helpers run only inside ``init_telemetry``'s guard. + +A body is "guarded" when, after a leading docstring, ``global`` declaration and +no-op guard clauses (``if ...: return``/``pass``), what remains is a single +``try`` with a bare ``except`` or ``except Exception``. + +Allowlist caveat (mirrors CE002): extend ``_GUARDED_FUNCTIONS`` when a new public +telemetry function is added. """ import ast diff --git a/tests/lint/rules/ce020_no_sdk_typed_base_agent_fields.py b/tests/lint/rules/ce020_no_sdk_typed_base_agent_fields.py index e044c5096..12e049e99 100644 --- a/tests/lint/rules/ce020_no_sdk_typed_base_agent_fields.py +++ b/tests/lint/rules/ce020_no_sdk_typed_base_agent_fields.py @@ -1,33 +1,25 @@ """CE020: No ``BaseAgentConfig`` field may be typed against ``claude_agent_sdk``. -``BaseAgentConfig`` is the vendor-neutral Pydantic base shared by every agent kind -(Claude Code, Codex, NoOp, and third-party BYOA configs). A field on it whose -annotation references a ``claude_agent_sdk`` type leaks a Claude-Code-specific type -onto agents that have nothing to do with the Claude SDK — the leaky-abstraction this -refactor removed (``plugins: list[SdkPluginConfig]`` → local ``LocalPluginConfig``; -``setting_sources`` moved down to ``ClaudeCodeAgentConfig``). - -The boundary is mechanically detectable, so this rule guards it: any ``AnnAssign`` -field inside the ``BaseAgentConfig`` class body whose annotation references a name -imported from ``claude_agent_sdk`` is flagged. Fix by defining a local vendor-neutral -type (alias / TypedDict mirror) or moving the field down to the concrete subclass that -actually needs the SDK type. - -Scope: only ``models/agent_config.py`` is inspected. The rule is deliberately narrow: -- A module-level use of an SDK type (e.g. ``dataclasses.fields(ClaudeAgentOptions)``) - is NOT flagged — only ``AnnAssign`` annotations inside ``BaseAgentConfig`` are. -- SDK-typed fields on subclasses (``ClaudeCodeAgentConfig`` etc.) are allowed. - -Import forms caught (so the leak can't sneak back via a different import style): +``BaseAgentConfig`` is shared by every agent kind, so an ``AnnAssign`` in its class +body whose annotation references a name from ``claude_agent_sdk`` is flagged. Fix +with a local vendor-neutral type (alias / TypedDict mirror), or move the field to +the subclass that needs the SDK type. + +Scope: only ``models/agent_config.py``. Module-level SDK uses and SDK-typed fields +on subclasses (``ClaudeCodeAgentConfig`` etc.) are allowed. + +Import forms caught: - ``from claude_agent_sdk import SettingSource`` (and aliased ``... as SS``). - ``from claude_agent_sdk.types import SettingSource`` (submodule path). -- ``import claude_agent_sdk`` / ``import claude_agent_sdk as sdk`` used in attribute - form (``list[sdk.SettingSource]``). +- ``import claude_agent_sdk [as sdk]`` used in attribute form + (``list[sdk.SettingSource]``). -Not caught: ``from claude_agent_sdk import *`` (a wildcard hides the names) — but -ruff F403/F405 already bans star imports, so that hole is covered upstream. +BLIND SPOT: ``from claude_agent_sdk import *`` hides the names; ruff F403/F405 +(``pyproject.toml``) bans star imports. Add ``# noqa: CE020`` on the offending field for a deliberate exception. + +Rationale: .claude/notes/lint-rules.md § CE020 """ import ast diff --git a/tests/lint/rules/ce021_guarded_evaluationresult_parse.py b/tests/lint/rules/ce021_guarded_evaluationresult_parse.py index 82f793c08..10678e81d 100644 --- a/tests/lint/rules/ce021_guarded_evaluationresult_parse.py +++ b/tests/lint/rules/ce021_guarded_evaluationresult_parse.py @@ -1,30 +1,20 @@ """CE021: ``EvaluationResult.model_validate_json(...)`` must be inside a guarding try. -``task.json`` is the harness's always-produce/always-consume artifact: the only -thing that crosses the container boundary, and the per-task record every -dashboard/timeline reads. Parsing it with a bare -``EvaluationResult.model_validate_json(text)`` means a present-but-malformed file -— a schema skew between a stale ``:latest`` image and the host (the docker -version checks only warn), or a truncated/torn write — surfaces as an uncaught -``pydantic.ValidationError`` / ``json.JSONDecodeError`` (both subclass -``ValueError``) that crashes the run. That was a real incident at -``docker_runner.py`` (the parse re-bucketed the task to a non-persisted in-memory -ERROR with no per-task report). The fix is to degrade: catch ``ValueError`` and -persist a synthetic ERROR record — see ``batch.py::_load_completed_result`` / -``recover_task_results`` and ``docker_runner.py::_handle_malformed_task_json``. - -This rule mechanically enforces that: every ``EvaluationResult.model_validate_json`` -call must be lexically nested inside a ``try`` whose ``except`` handlers catch -``ValueError`` (or the broader ``Exception`` / ``BaseException``, or a bare -``except``). A handler catching only an unrelated type (e.g. ``except OSError``) -does NOT guard. Note ``recover_task_results`` uses ``except (OSError, ValueError)`` -— the ``ValueError`` tuple member counts as guarding. - -Scope is intentionally narrow to ``EvaluationResult`` (the always-produce/consume -contract). Other models' ``model_validate_json`` calls are not matched. - -Add ``# noqa: CE021`` on the call line only if a caller genuinely must not catch -(honored automatically by the runner's suppression logic — no rule-side work). +A malformed ``task.json`` raises ``ValidationError`` / ``JSONDecodeError`` (both +``ValueError``); the caller must degrade to a persisted synthetic ERROR record, +not crash. Pattern: ``batch.py::_load_completed_result``. + +The call must sit lexically inside a ``try`` BODY whose handlers catch +``ValueError``, ``Exception``, ``BaseException``, or bare ``except``; a tuple +naming one of them counts. An unrelated type (``except OSError``) does not +guard, and neither do the ``else`` / ``except`` / ``finally`` bodies. + +Scope: ``EvaluationResult.model_validate_json`` spelled on that bare name only; +other models are not matched. + +Add ``# noqa: CE021`` on the call line only if a caller genuinely must not catch. + +Rationale: .claude/notes/lint-rules.md § CE021 """ import ast diff --git a/tests/lint/rules/ce022_dialog_loop_statement_cap.py b/tests/lint/rules/ce022_dialog_loop_statement_cap.py index f91a20d1b..921b03018 100644 --- a/tests/lint/rules/ce022_dialog_loop_statement_cap.py +++ b/tests/lint/rules/ce022_dialog_loop_statement_cap.py @@ -1,22 +1,16 @@ """CE022: ``Orchestrator._simulation_dialog_loop`` must stay under its statement cap. -``_simulation_dialog_loop`` is the one function in the tree that keeps a -``# noqa: PLR0915``: it is a sequential dialog driver whose residual length is -irreducible without a state-object rewrite (see the 2026-06-23 -decompose-god-functions plan, Phase 5). But ``# noqa: PLR0915`` *disables ruff's -statement check entirely* — without a guard the function could silently regrow -back toward its pre-decomposition size and nothing would fail. - -This rule re-imposes a bound: it counts the statements in -``_simulation_dialog_loop`` (every statement node in the body, recursively — the -same notion ruff PLR0915 bounds) and fires if the count exceeds ``_CAP``. The cap -is the measured post-decomposition size plus a small headroom, so ordinary edits -don't trip it but a real regrowth does. Deliberately narrow: it targets the one -named function in ``orchestrator.py``, not a general size rule (ruff's 80/25 -ceiling already covers every other function). - -If a future decomposition legitimately brings the function under ruff's ceiling, -remove the ``# noqa: PLR0915`` AND this rule together. +The rule counts every statement node in ``_simulation_dialog_loop`` in +``orchestrator.py`` (recursively, the notion ruff PLR0915 bounds; ``def`` or +``async def``) and fires when the count exceeds ``_CAP``. It targets only that one +function; ruff's ceiling covers every other function. + +HAZARD: the function keeps ``# noqa: PLR0915``, which disables ruff's statement +check for it, so this rule is its only bound. Bump ``_CAP`` only with a reviewed +change to the dialog driver. If a decomposition brings the function under ruff's +ceiling, remove the ``# noqa: PLR0915`` AND this rule together. + +Rationale: .claude/notes/lint-rules.md § CE022 """ import ast diff --git a/tests/lint/rules/ce023_no_proxy_shim_import.py b/tests/lint/rules/ce023_no_proxy_shim_import.py deleted file mode 100644 index 743b69ab5..000000000 --- a/tests/lint/rules/ce023_no_proxy_shim_import.py +++ /dev/null @@ -1,52 +0,0 @@ -"""CE023: Don't import from the deprecated coder_eval.proxy.* shim in runtime code. - -PR #463 removed the LLM Gateway proxy subsystem and relocated pricing to the -top-level coder_eval.pricing module. What remains under coder_eval/proxy/ is a -backwards-compatibility shim: proxy/pricing.py re-exports coder_eval.pricing -behind a DeprecationWarning, kept only so out-of-tree consumers' existing -`import coder_eval.proxy.pricing` keep working. In-tree runtime/agent code must -import the authoritative location directly (coder_eval.pricing) so a new agent -can't silently couple to the relocated/shimmed path — e.g. adding model rates to -a dict the proxy removal deleted, leaving them unpriced after merge. - -Both `from coder_eval.proxy... import ...` and bare `import coder_eval.proxy...` -forms are checked. Skipped for files inside coder_eval/proxy/ (the shim itself). -Use `# noqa: CE023` for a deliberate backwards-compatibility reference. -""" - -import ast -import re - -from tests.lint.rules.base import BaseRule - - -class NoProxyShimImports(BaseRule): - id = "CE023" - - _PROXY_IMPORT = re.compile(r"^coder_eval\.proxy\b") - _SKIP_PATH = re.compile(r"[/\\]coder_eval[/\\]proxy[/\\]") - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._skip_file = bool(self._SKIP_PATH.search(filepath)) - - def visit_ImportFrom(self, node: ast.ImportFrom) -> None: - if not self._skip_file and node.module and self._PROXY_IMPORT.match(node.module): - names = ", ".join(a.name for a in node.names) - self.violation( - node, - f"import from deprecated shim '{node.module}'; " - f"import directly from the authoritative module, e.g. 'from coder_eval.pricing import {names}'", - ) - self.generic_visit(node) - - def visit_Import(self, node: ast.Import) -> None: - if not self._skip_file: - for alias in node.names: - if self._PROXY_IMPORT.match(alias.name): - self.violation( - node, - f"import of deprecated shim '{alias.name}'; " - f"import directly from the authoritative module (e.g. coder_eval.pricing)", - ) - self.generic_visit(node) diff --git a/tests/lint/rules/ce024_discriminated_unions.py b/tests/lint/rules/ce024_discriminated_unions.py index a0fde0126..bcac9b6eb 100644 --- a/tests/lint/rules/ce024_discriminated_unions.py +++ b/tests/lint/rules/ce024_discriminated_unions.py @@ -1,27 +1,19 @@ """CE024: unions of ``type: Literal``-tagged models in ``coder_eval/models/`` must declare a discriminator. -A bare ``A | B | C`` union of tagged Pydantic models validates via smart-union: -input missing its ``type`` tag (or carrying a typo'd one) silently coerces to -whichever variant happens to fit, instead of raising a crisp discriminator -error. That is exactly how the ``SuccessCriterion`` union shipped — tag-less -criterion dicts coerced to the structurally-nearest variant — until it was -wrapped in ``Annotated[..., Field(discriminator="type")]`` -(``models/mutations.py:48`` is the canonical compliant shape). - -Flagged: a module-level assignment (plain, PEP 695 ``type X = ...``, or -annotated) whose value is a ``|``-chain (or a ``Union[...]`` subscript) of -two or more names that ALL refer to classes -defined in the same file with a ``type: Literal[...]`` field — including such -a union inside an ``Annotated[...]`` that carries no discriminator metadata. - -Compliant (not flagged): the union as the first element of ``Annotated[...]`` -whose metadata contains ``Field(discriminator=...)`` or a ``Discriminator(...)`` -call (the callable form used by ``CriterionResultUnion`` in -``models/results.py``). Unions with any untagged or imported member are out of -scope (same-file conservatism, mirroring CE009's documented trade). - -Add ``# noqa: CE024`` on the assignment line for a deliberate exception. +Flagged: a module-level assignment (plain, PEP 695 ``type X = ...``, or annotated) +whose value is a ``|``-chain or a ``Union[...]`` of two or more names that ALL refer +to same-file classes with a ``type: Literal[...]`` field. The union is flagged bare +or inside an ``Annotated[...]`` with no discriminator metadata. + +Compliant: the union as the first element of ``Annotated[...]`` whose metadata holds +``Field(discriminator=...)`` or a ``Discriminator(...)`` call; ``models/mutations.py`` +has the canonical shape. A union with any untagged or imported member is out of +scope (same-file conservatism, as in CE009). + +``# noqa: CE024`` on the assignment line for a deliberate exception. + +Rationale: .claude/notes/lint-rules.md § CE024 """ import ast diff --git a/tests/lint/rules/ce032_criteria_path_seam.py b/tests/lint/rules/ce032_criteria_path_seam.py index 60a78ca1d..cfd4aabad 100644 --- a/tests/lint/rules/ce032_criteria_path_seam.py +++ b/tests/lint/rules/ce032_criteria_path_seam.py @@ -1,22 +1,14 @@ """CE032: Criterion checkers must resolve sandbox paths through the Sandbox seam. -`Sandbox.resolve_files` is the single place criterion `path` semantics live: -literal-first resolution (so a real file named `report[2024].json` is not -reinterpreted as a character class), glob expansion for artifacts whose location -the prompt does not pin, ignore-pattern filtering (so `.venv` / `node_modules` / -`dist` cannot be graded as agent output), and exactly-one enforcement on content -reads. A checker that builds its own path with `sandbox.sandbox_dir / ` -and reads it directly silently opts out of all of that, so path semantics differ -per criterion — which is exactly how `reference_comparison.agent_file` drifted -from every other path field. - -Use `sandbox.file_exists` / `sandbox.get_file_content` / `sandbox.resolve_files` -instead. Only files under `coder_eval/criteria/` are checked; reading -`sandbox.sandbox_dir` on its own (an initialization guard, or passing the root -to a sub-agent) is fine — the rule fires on joining a path onto it. - -Use `# noqa: CE032` for a checker that genuinely needs the raw root (e.g. it -walks a directory tree rather than addressing a file). +Under `coder_eval/criteria/`, a `/` join onto `.sandbox_dir` fires. Use +`sandbox.file_exists` / `sandbox.get_file_content` / `sandbox.resolve_files`, the +single place criterion `path` semantics live. Reading `sandbox.sandbox_dir` on its +own (an initialization guard, or passing the root to a sub-agent) is fine. + +Use `# noqa: CE032` for a checker that genuinely needs the raw root (e.g. it walks +a directory tree rather than addressing a file). + +Rationale: .claude/notes/lint-rules.md § CE032 """ import ast diff --git a/tests/lint/rules/ce037_no_dead_private_helper.py b/tests/lint/rules/ce037_no_dead_private_helper.py index 54bb14061..4e0de532e 100644 --- a/tests/lint/rules/ce037_no_dead_private_helper.py +++ b/tests/lint/rules/ce037_no_dead_private_helper.py @@ -1,31 +1,22 @@ """CE037: a module-level private helper in ``src/`` must have a caller. -A ``def _helper(...)`` that nothing in ``src/`` references is not merely dead -weight — it actively misleads. The bug that motivated this rule shipped an -``_rmtree_restrictive`` whose docstring explained, correctly and in detail, why -``rmtree(..., ignore_errors=True)`` orphans a mode-000 reference tree… while -both live cleanup sites went on calling exactly that. A reader auditing the -cleanup path found a function asserting the shipped code was broken, and a test -that called the helper directly made the real path read as covered. - -The rule turns "helper written, never wired" into a ``make lint`` failure at the -commit that introduces it, which is the only moment anyone knows where it was -supposed to be called from. - -Scope is deliberately narrow so it stays a bug detector rather than a style -nag: - -* module-level ``def`` / ``async def`` only (methods are found via ``self``, - which this cannot see), -* names starting with a single underscore only (public API has out-of-tree - callers, dunders are protocol), -* decorated functions are skipped (a decorator is a registration — - ``@register_criterion``, ``@field_validator``, ``@app.command`` — so the - reference is the decorator, not a call), -* a name re-exported in ``__all__`` is skipped. +Fires when the name of such a helper occurs only once (its definition) in the +concatenated text of ``src/coder_eval``. Any other occurrence counts as a caller. +Scope: + +* module-level ``def`` / ``async def`` only (not methods, not closures), +* a single leading underscore only (not public names, not dunders), +* undecorated only (a decorator is a registration), +* not re-exported in ``__all__``. + +Motivating case: ``_rmtree_restrictive`` documented why +``rmtree(..., ignore_errors=True)`` orphans a mode-000 reference tree, while both +live cleanup sites still called exactly that. Use ``# noqa: CE037`` for a deliberate SPI hook that genuinely has no in-tree caller, with a comment naming who calls it. + +Rationale: .claude/notes/lint-rules.md § CE037 """ import ast diff --git a/tests/lint/rules/ce038_acquire_inside_try.py b/tests/lint/rules/ce038_acquire_inside_try.py index 662cf7ca5..6c1223964 100644 --- a/tests/lint/rules/ce038_acquire_inside_try.py +++ b/tests/lint/rules/ce038_acquire_inside_try.py @@ -1,24 +1,10 @@ """CE038: in an async context manager, the acquire must sit INSIDE the try. -An ``@contextlib.asynccontextmanager`` whose shape is:: +Fires only when all four hold: the function is decorated ``asynccontextmanager``; +a name is bound by `` = await ...`` in the statement immediately before a +``try``; that ``try`` has a ``finally``; the ``finally`` references the name. - held = await acquire() # <-- outside - try: - yield - finally: - release(held) - -leaks whenever a cancellation lands on that ``await``. This is not hypothetical -and ``asyncio.shield`` does not fix it: shield protects the *inner* task, so the -awaiting coroutine still receives ``CancelledError``, propagates it out of -``__aenter__``, and never reaches the ``finally`` — while the shielded work goes -right on completing. The motivating bug held a reference directory at mode 000 -with no matching restore: unreadable for the rest of the run, plus a stale -registry entry that poisoned the next window on the same path. The comment above -it claimed shielding prevented exactly that. - -The fix is mechanical — move the acquire inside the ``try`` and initialise the -name to an empty value before it:: +Fix: initialise the name to an empty value and move the acquire inside:: held = [] try: @@ -27,11 +13,13 @@ finally: release(held) -Fires only when all four conditions hold, so it stays specific: the function is -an async context manager, a name is bound by an ``await`` in the statement -immediately preceding a ``try``, that ``try`` has a ``finally``, and the -``finally`` references the bound name. ``# noqa: CE038`` if the acquire genuinely -cannot fail partway. +HAZARD: ``asyncio.shield`` does not fix it. It protects the inner task, not the +await, so ``CancelledError`` still leaves ``__aenter__`` before the ``finally`` +while the acquire completes. + +``# noqa: CE038`` if the acquire genuinely cannot fail partway. + +Rationale: .claude/notes/lint-rules.md § CE038 """ import ast diff --git a/tests/lint/rules/ce039_config_error_escalates.py b/tests/lint/rules/ce039_config_error_escalates.py index 9b59f0e9d..41156e2f0 100644 --- a/tests/lint/rules/ce039_config_error_escalates.py +++ b/tests/lint/rules/ce039_config_error_escalates.py @@ -1,27 +1,18 @@ """CE039: a criterion checker must not book an IO/config error as score 0.0. -``CriterionResult(score=0.0)`` means *the agent did the work and it was wrong*. -It is gating (``all_criteria_passed`` is a strict AND) and it flows into every -downstream count that consumes criterion scores: ``CriterionAggregate`` -mean/median, ``suite_thresholds`` gates on dataset-fanned suites, run and -experiment pass rates, the JUnit report, the evalboard. - -An ``except OSError`` around a file the TASK AUTHOR named is not that. The -motivating case: a typo in ``reference_comparison.reference_file`` raised -``FileNotFoundError`` (an ``OSError``), got turned into a gating 0.0, and was -counted against the agent's pass rate — silently zeroing every row of a -dataset-fanned suite while looking like a genuine similarity failure. - -Raise ``CheckerMisuseError`` instead. ``criteria/base.py``'s -``_ESCALATING_EXCEPTIONS`` routes it to ``FinalStatus.ERROR``, which is what an -eval-config error is. - -Fires only on a ``return CriterionResult(...)`` with a literal ``score=0.0`` -lexically inside an ``except`` handler for ``OSError`` / ``FileNotFoundError`` / -``PermissionError`` / ``IsADirectoryError``, in ``coder_eval/criteria/``. A -failure attributable to the AGENT's own output (its file is missing, its JSON is -malformed) is legitimately 0.0 — mark those ``# noqa: CE039`` with a one-line -reason. +``CriterionResult(score=0.0)`` means *the agent did the work and it was wrong*, +and it gates. An IO failure on a path the TASK AUTHOR named is an eval-config +error: raise ``CheckerMisuseError``, which ``criteria/base.py``'s +``_ESCALATING_EXCEPTIONS`` routes to ``FinalStatus.ERROR``. + +Fires, in ``coder_eval/criteria/``, on a ``return CriterionResult(...)`` with a +literal ``score=0.0`` lexically inside an ``except`` handler for ``OSError`` / +``IOError`` / ``FileNotFoundError`` / ``PermissionError`` / ``IsADirectoryError``. + +A failure caused by the AGENT's own output (its file is missing, its JSON is +malformed) is legitimately 0.0: mark it ``# noqa: CE039`` with a one-line reason. + +Rationale: .claude/notes/lint-rules.md § CE039 """ import ast diff --git a/tests/lint/rules/ce043_no_command_output_truncation.py b/tests/lint/rules/ce043_no_command_output_truncation.py index fa1e5084f..e26de8c27 100644 --- a/tests/lint/rules/ce043_no_command_output_truncation.py +++ b/tests/lint/rules/ce043_no_command_output_truncation.py @@ -1,26 +1,20 @@ """CE043: Agents must not truncate a command's output when recording it. -``CommandTelemetry.result_summary`` is contractually the *untruncated* tool-result -body, and its length drives the ``result_tokens`` computed field (the cost -simulator's cache-independent measure of tool-output size). An agent that clips a -command's captured output before storing it silently under-reports every tool -result — exactly the bug the Codex agent shipped with (``f"Output: {output[:100]}"``), -which pinned ~77% of its Bash results at ~31 tokens and skewed the cost model. - -This rule flags, inside ``src/coder_eval/agents/``, a constant-upper-bound slice -(``x[:N]``) applied to a value that denotes captured command output: +``CommandTelemetry.result_summary`` is the *untruncated* tool-result body, and +its length drives the ``result_tokens`` computed field. Inside +``src/coder_eval/agents/``, the rule flags a constant-upper-bound slice +(``x[:N]``) of a value that denotes captured command output: * a name whose id is/ends with ``output`` / ``stdout`` / ``stderr`` (e.g. ``output[:100]``, ``aggregated_output[:512]``, ``proc_stdout[:80]``) * an attribute access ``.aggregated_output`` / ``.output`` / ``.stdout`` / ``.stderr`` - (e.g. ``command_item.aggregated_output[:100]``) -Store the output whole (it is already bounded by the harness's own exec-output -truncation) and trim for DISPLAY in the renderers/reports instead. +Store the output whole and trim for DISPLAY in the renderers/reports instead. + +Add ``# noqa: CE043``, with a comment explaining why, only for a slice that never +becomes a ``result_summary`` (e.g. a short crash/log message). -Add ``# noqa: CE043`` on the offending line for a genuinely non-recorded use -(e.g. slicing stdout only to build a short crash/log message that never becomes a -``result_summary``), with a comment explaining why. +Rationale: .claude/notes/lint-rules.md § CE043 """ import ast diff --git a/tests/lint/rules/ce046_env_info_spreads_super.py b/tests/lint/rules/ce046_env_info_spreads_super.py index a1807a78c..5a33089be 100644 --- a/tests/lint/rules/ce046_env_info_spreads_super.py +++ b/tests/lint/rules/ce046_env_info_spreads_super.py @@ -1,30 +1,19 @@ """CE046: a ``get_environment_info`` override must spread the base result. -``Agent.get_environment_info`` (agent.py) emits the ``system_prompt_semantics`` -run marker from the ClassVar of the same name, so EVERY run — including -out-of-tree SPI agents — records which system-prompt regime built its prompts. -Dashboards read an ABSENT marker as "a run from before the marker existed" and -pool it into a legacy bucket, so an override that returns a bare dict does not -merely omit a key: it silently mis-buckets every one of that agent's runs. - -The motivating bug: ``OpenCodeAgent.get_environment_info`` returned -``{"opencode_model": ..., "opencode_pure": ...}`` with no ``super()`` spread, so -no OpenCode run ever carried the marker and no test caught it. Every other agent -(codex/antigravity/claude_code) spreads the base correctly. - -The base's docstring already states the contract ("Overrides should spread -``super().get_environment_info()`` rather than returning a bare dict"); this rule -makes it mechanical. +``Agent.get_environment_info`` emits the ``system_prompt_semantics`` run marker. +An override that returns a bare dict drops it, and every run of that agent is +silently mis-bucketed as pre-marker. Fires on any method named ``get_environment_info`` defined directly in a class body that neither * calls ``super().get_environment_info()`` (the override contract), nor * references ``self.system_prompt_semantics`` (the base itself, which emits - the marker directly — exempt so the rule does not flag its own source). + the marker directly). -``# noqa: CE046`` if an agent genuinely must not record the marker (there is no -such case today). +``# noqa: CE046`` only if an agent genuinely must not record the marker. + +Rationale: .claude/notes/lint-rules.md § CE046 """ import ast diff --git a/tests/lint/rules/ce048_no_in_process_typer_command_call.py b/tests/lint/rules/ce048_no_in_process_typer_command_call.py index 37c10a2f0..dae922d9e 100644 --- a/tests/lint/rules/ce048_no_in_process_typer_command_call.py +++ b/tests/lint/rules/ce048_no_in_process_typer_command_call.py @@ -1,23 +1,21 @@ """CE048: never call a Typer command function in process. -Typer builds a command's parser from its signature, so every parameter's default -is an ``OptionInfo`` / ``ArgumentInfo`` sentinel, not the value it stands for. -Click substitutes the real defaults when it *invokes* the command; a direct -Python call does not — every unspecified argument arrives as a truthy sentinel -object. - -The failure is silent, which is what makes it worth a rule. ``evaluate``'s -``in_place: bool | None = typer.Option(None, "--in-place/--copy")`` reads as "no -preference" and selects copy-vs-in-place from the target shape; called -in-process, ``in_place`` was an ``OptionInfo``, which is truthy, so the tests -silently graded in place and the default they meant to cover was never -exercised. Nothing failed — the wrong branch simply ran. - -The fix is the one already applied to ``run`` / ``execute`` / ``evaluate``: keep -the Typer signature as a thin wrapper and put the body in a plain function with -real Python defaults (``run_pipeline``, ``run_evaluation``). Call THAT. +A Typer command's parameter defaults are ``OptionInfo`` / ``ArgumentInfo`` +sentinels. Click substitutes the real defaults only when it *invokes* the +command; a direct Python call passes a truthy sentinel for every unspecified +argument, and the wrong branch runs silently. + +Call the plain-function body with real Python defaults (``run_pipeline``, +``run_evaluation``), or drive the command through ``typer.testing.CliRunner``. + +Scope: ``src/`` and ALSO ``tests/``, where this defect occurs; +``cli/__init__.py`` is exempt. A call fires only on a name imported from a +``coder_eval.cli`` module whose parameters carry a ``typer.Option`` / +``typer.Argument`` default. Use ``# noqa: CE048`` only where the sentinel behavior is itself under test. + +Rationale: .claude/notes/lint-rules.md § CE048 """ import ast diff --git a/tests/lint/rules/ce049_no_score_or_zero.py b/tests/lint/rules/ce049_no_score_or_zero.py index 491536172..b9f0d40af 100644 --- a/tests/lint/rules/ce049_no_score_or_zero.py +++ b/tests/lint/rules/ce049_no_score_or_zero.py @@ -1,28 +1,18 @@ """CE049: never coalesce a possibly-unmeasured score to a numeric literal. -``weighted_score is None`` means *nothing measured this row*, and it is a -different fact from ``weighted_score == 0.0``, which means *this row was measured -and scored nothing*. ``score or 0.0`` erases that difference — and it does it -silently, producing a real-looking number that every downstream consumer treats -as a genuine miss. - -The motivating bug: ``build_task_event`` published ``Score = float( -result.weighted_score or 0.0)`` on every ``CoderEval.Task.End``. Four shipped -App Insights tiles compute ``avg(todouble(customDimensions.Score))`` with no -status filter, so one ``coder-eval execute`` night dragged every score tile -toward zero, indistinguishable from a genuinely bad night. The hazard was -already documented in prose in ``orchestrator.py`` ("every downstream -`score or 0.0` would launder it into a real-looking failure") — this rule makes -it mechanical. +``weighted_score is None`` means *nothing measured this row*; ``0.0`` means *this +row was measured and scored nothing*. ``score or 0.0`` silently erases that +difference. Fires on `` or `` where the left operand's trailing name looks like a score or a rate. The fix is to omit the value, keep it ``None``, or branch explicitly on ``is None``. -``# noqa: CE049`` for a genuinely aggregate-internal use where a missing value -really is a miss — e.g. summing a variant's scores where an errored row must -count as 0.0 (see ``orchestration/experiment._measured_scores``, which makes that -decision explicitly and states why). +``# noqa: CE049`` for an aggregate-internal use where a missing value really is +a miss, e.g. an errored row that must count as 0.0 (see +``orchestration/experiment._measured_scores``). + +Rationale: .claude/notes/lint-rules.md § CE049 """ import ast diff --git a/tests/lint/rules/ce050_no_union_getattr_probe.py b/tests/lint/rules/ce050_no_union_getattr_probe.py index 2f7920857..5cb3e9877 100644 --- a/tests/lint/rules/ce050_no_union_getattr_probe.py +++ b/tests/lint/rules/ce050_no_union_getattr_probe.py @@ -1,38 +1,20 @@ """CE050: no untyped ``getattr`` probe for a field of a discriminated union. -``getattr(criterion, "command", None)`` reads as "the members that have a -command". It is not: it is a string the type checker cannot see. Rename -``RunCommandCriterion.command`` and pyright reports nothing, ruff reports -nothing, and the probe silently returns ``None`` forever — the guard it powers -becomes a permanent no-op with every gate green. - -``models/tasks.py`` already states the rule in prose, verbatim: "isinstance -narrowing, NOT getattr(c, 'files'/'command'): with an untyped string probe, -renaming ... turns this load-time guard into a silent no-op that pyright cannot -see." This promotes that convention to a gate. - -The motivating bug: ``orchestration/regrade.warn_on_embedded_commands`` — the -only disclosure of what shell a rebuilt, untrusted run config would execute on -the grader's host — probed with ``getattr(c, "command", None)``. Besides being -rename-fragile it structurally could not name ``agent_judge``, the criterion -that spawns a tool-using agent and therefore has the widest blast radius of all. - Fires on ``getattr(, "", ...)`` in ``src/coder_eval/`` where the -literal is a field name declared by a member of one of the tracked discriminated -unions AND ```` is named like a criterion / template source / route. The -field list is derived from the models at collection time, so it tracks renames -instead of going stale. - -The receiver-name filter is deliberate, and it is the rule's known limit. Field -names like ``command``, ``tool`` and ``prompt`` are far too common to flag on -their own — the agents legitimately probe raw SDK event objects for exactly those -— so a name-only rule would fire a dozen times on code that has nothing to do -with these unions and would be turned off within a week. Scoping to the -receiver's name catches the real shape (``for c in task.success_criteria: ... -getattr(c, "command", None)``) and leaves an unusual receiver name uncovered. - -The fix is ``isinstance`` narrowing. ``# noqa: CE050`` for a probe that really is +literal is a field declared by a member of ``SuccessCriterion``, +``TemplateSource`` or ``ApiRoute`` (derived from the models at collection time, +minus common names such as ``type``) AND ```` is named like a criterion, +template source or route. Pyright cannot see the string, so a rename turns the +probe into a permanent ``None``. + +Known limit, deliberate: ``command``/``tool``/``prompt`` are too common to flag +alone (agents probe SDK event objects for them), so the rule scopes to +criterion-shaped receiver names and an unusual receiver name is uncovered. + +Fix with ``isinstance`` narrowing; ``# noqa: CE050`` for a probe that really is duck-typed across unrelated objects. + +Rationale: .claude/notes/lint-rules.md § CE050 """ import ast diff --git a/tests/lint/rules/ce051_no_driver_override.py b/tests/lint/rules/ce051_no_driver_override.py index f6943139f..029e72326 100644 --- a/tests/lint/rules/ce051_no_driver_override.py +++ b/tests/lint/rules/ce051_no_driver_override.py @@ -1,29 +1,19 @@ """CE051: a sandbox driver may not be rewritten silently. -The driver IS the isolation boundary. Rewriting ``docker`` to ``tempdir`` behind -the caller's back does not degrade gracefully — it moves execution from a -container onto the operator's own machine, where the task's criteria address -paths and toolchains that do not exist. They score 0.0 and the row is written -back FAILURE for a trajectory that passed, and the same commands (``rm -rf -/verifier``, ``mkdir -p /logs/verifier``) run unsandboxed on the grading host. - -The motivating bug: ``regrade.grading_sandbox_config`` rewrote the driver -unconditionally on BOTH new grading entry points, which also neutralized the -``driver: docker`` refusal in ``Sandbox.adopt`` — a guard added in the same -change specifically to catch this. - -A driver downgrade must be an explicit, logged, operator-visible decision. Fires -on any construction that carries an existing sandbox config forward while -replacing ``driver``: - - * ``SandboxConfig.model_validate({**cfg.model_dump(), "driver": ...})`` +In ``src/coder_eval/``, fires on any construction that carries an existing +sandbox config forward while replacing ``driver``: + + * ``SandboxConfig.model_validate({**cfg.model_dump(), "driver": ...})`` (a + spread dict only; a dict built from scratch is an ordinary construction) * ``cfg.model_copy(update={"driver": ...})`` * ``setattr(cfg, "driver", ...)`` / ``cfg.driver = ...`` -Exempt: ``models/sandbox.py`` (the model's own construction), and any site -carrying ``# noqa: CE051`` with a reason — today the two legitimate ones are the -in-container rewrite in ``run_task_internal_command`` and the opt-in host-grading -branch, which refuses by default and stamps ``graded_on_host`` on the row. +A driver downgrade must be an explicit, logged, operator-visible decision. +Exempt: ``models/sandbox.py`` (the model's own construction). Elsewhere, +``# noqa: CE051`` must name the reason, for example an opt-in that refuses by +default and stamps the row. + +Rationale: .claude/notes/lint-rules.md § CE051 """ import ast diff --git a/tests/lint/rules/ce052_process_lethal_must_be_container_gated.py b/tests/lint/rules/ce052_process_lethal_must_be_container_gated.py index 406bbe090..e7b6365a0 100644 --- a/tests/lint/rules/ce052_process_lethal_must_be_container_gated.py +++ b/tests/lint/rules/ce052_process_lethal_must_be_container_gated.py @@ -1,42 +1,19 @@ """CE052: a process-lethal call must be gated on actually being in the container. -``os._exit`` bypasses ``atexit``, buffered IO, ``finally`` blocks and every -exception handler: the process is simply gone. That is the correct primitive for -exactly one thing in this codebase — reaping the container's own disposable main -process when the host that started it has died — and it is safe there only -because that process is *ours to destroy*. In any other process it is not a -degraded outcome, it is an unattributable one. - -The motivating bug: ``run_task_internal_command`` armed its host-heartbeat -watchdog — a daemon thread whose whole authority is ``os._exit(137)`` — as an -unconditional side effect of the command body. A test invoked that command -in-process (legitimately: the command must refuse a malformed ``context.json``, -and asserting that means calling it), and the pytest worker inherited the -thread. Forty seconds later — 20s grace plus the 20s stale window — it found no -heartbeat and exited the worker, mid-way through whatever unrelated test file -that worker had since moved on to. - -Every property of that failure is the one this rule exists to prevent: - - * it named the wrong test — a different one on each run, on each platform, - with no traceback, because there is no exception to raise; - * it was invisible at low load — with 14 local workers the file finished and - the run ended before the timer fired, so it reproduced only on CI's 2; - * and it took the coverage gate with it. A dead worker returns no coverage - data, so a single killed process reported as "total of 65.13 is less than - fail-under=80.00" — a failure naming neither the test nor the cause. - -Fires on ``os._exit(...)`` anywhere in ``src/coder_eval/`` that is not lexically -inside a branch testing ``CODER_EVAL_IN_CONTAINER``. That env var is the repo's -established in-container predicate (``Sandbox.enforces_permission_windows``, -``orchestration/evaluation.resolve_reference_dir``) and is deliberately NOT -``sandbox.driver`` — ``run_task_internal_command`` rewrites the driver to -``tempdir`` before building the in-container Orchestrator, so a driver-based gate -disables itself on precisely the path that needs it. - -The check is lexical (an enclosing ``if``/``elif`` whose test mentions the var), -not a data-flow proof. That is enough to force the guard to be written down at -the site, which is the property that was missing. +Fires on ``os._exit(...)`` (any module alias) anywhere in ``src/coder_eval/`` that is +not lexically inside the body of an ``if``/``elif`` whose test mentions +``CODER_EVAL_IN_CONTAINER`` or ``IN_CONTAINER_ENV``. An ``else`` arm is not gated. +``os._exit`` skips ``atexit``, ``finally`` and every handler, so it is correct only for +reaping the container's own disposable main process. + +HAZARD: do not gate on ``sandbox.driver``. ``run_task_internal_command`` rewrites the +driver to ``tempdir`` before it builds the in-container Orchestrator, so a driver gate +disables itself on the one path that needs it. + +The check is lexical, not a data-flow proof: it forces the guard to be written at the +site. Add ``# noqa: CE052`` with a reason for an intentional exception. + +Rationale: .claude/notes/lint-rules.md § CE052 """ import ast diff --git a/tests/lint/rules/ce053_run_record_filename_literal.py b/tests/lint/rules/ce053_run_record_filename_literal.py index 6727f592f..afbf5a986 100644 --- a/tests/lint/rules/ce053_run_record_filename_literal.py +++ b/tests/lint/rules/ce053_run_record_filename_literal.py @@ -1,26 +1,17 @@ """CE053: no bare run-record or run-log filename literal outside ``path_utils``. -``path_utils`` defines ``TASK_JSON_FILENAME`` / ``PRE_GRADE_JSON_FILENAME`` and -its comment states why: "~12 sites name them — including three that ``rglob`` for -the first — and two half-copies of the same string in different packages is how a -rename becomes a silent no-op on the sites it missed." +Fires on any string constant in ``src/coder_eval/`` (outside ``path_utils.py``) +that equals a filename ``path_utils`` exports — ``TASK_JSON_FILENAME``, +``PRE_GRADE_JSON_FILENAME`` and the four per-run log names (``task.log`` / +``grade.log`` / ``docker.log`` / ``grade.docker.log``) — or has one as its last +``/``-separated segment (``"*/task.json"``). Any string containing ``/`` whose +text after the last ``/`` is a name fires, prose included; a name followed by +more text (``"task.json is missing"``) or after a backslash does not. The set is +read from ``path_utils``, never retyped. -The constant shipped with that rationale and the twelve pre-existing literals -were not converted, so it created exactly the second source of truth it argues -against and delivered zero rename safety: the new modules used the constant, and -``orchestrator.py``, ``batch.py``, ``docker_runner.py``, ``reports/markdown.py``, -``reports/junit.py``, ``reports/helpers.py`` and ``report_command.py`` kept the -string — the three ``rglob("task.json")`` calls the comment specifically cites -among them. - -A rationale that only a human remembers is not a rule. Fires on any string -constant in ``src/coder_eval/`` (outside ``path_utils.py``) that equals one of -those filenames, or embeds it as a trailing path segment (``"*/task.json"``). Import the constant instead; ``# noqa: CE053`` for a genuinely unrelated string. -Covers the per-run LOG names too (``task.log`` / ``grade.log`` / ``docker.log`` / -``grade.docker.log``) — same shape, one release later, and the ``docker.log`` -case was worse because its consumer skips silently when the file is absent. +Rationale: .claude/notes/lint-rules.md § CE053 """ import ast diff --git a/tests/lint/rules/ce054_env_info_key_round_trip.py b/tests/lint/rules/ce054_env_info_key_round_trip.py index 94e0919f0..254500f4c 100644 --- a/tests/lint/rules/ce054_env_info_key_round_trip.py +++ b/tests/lint/rules/ce054_env_info_key_round_trip.py @@ -1,22 +1,19 @@ """CE054: every ``environment_info`` key that is READ must also be WRITTEN. -``EvaluationResult.environment_info`` is a ``dict[str, Any]`` bag, so nothing — -not pydantic, not pyright — connects the site that writes a key to the site that -reads it back. A reader whose writer was never added (or was later removed) is -silently inert: ``.get("k")`` returns ``None``, the guard takes its early return, -and the feature reports success while doing nothing. - -The motivating case: ``verify_reference_unchanged`` read -``environment_info.get("reference_digest")`` to refuse a re-grade whose answer key -had changed. Nothing anywhere wrote that key — a whole-tree grep found exactly one -occurrence, the read itself. The anti-cheat guard shipped, was documented in -CLAUDE.md and the user guide as protection, and never fired once. Every automated -gate in the repo was green. - -This is deliberately a one-way check. An unread key is ordinary (recorded for a -human or a downstream consumer); an unwritten key is always a bug. +``EvaluationResult.environment_info`` is ``dict[str, Any]``, so nothing connects +a writer to a reader; a reader with no writer silently gets ``None``. In +``src/coder_eval/``, a literal key read via ``.environment_info.get("k")`` or +``.environment_info["k"]`` must appear in an ``environment_info["k"] = ...`` +write somewhere in the tree. Computed keys, ``graded_by_*`` keys and +``_EXTERNALLY_WRITTEN`` entries are skipped. + +One-way on purpose: an unread key is ordinary; an unwritten key is always a bug. +The anti-cheat guard that shipped this way read ``reference_digest``, which +nothing wrote, so it never fired. Use ``# noqa: CE054`` for a key genuinely supplied from outside this repo. + +Rationale: .claude/notes/lint-rules.md § CE054 """ import ast @@ -44,9 +41,9 @@ def _written_keys() -> set[str]: """ written: set[str] = set() pattern = re.compile(r"""environment_info\[\s*["']([\w.-]+)["']\s*\]\s*=""") - # Also count keys named in a dict literal that becomes environment_info, and - # the f-string-built provenance keys (`f"graded_by_{key}"`), which no literal - # scan can resolve — those are covered by the prefix allowance below. + # Only subscript assignments count; a key set in a dict literal does not. The + # f-string-built provenance keys (`f"graded_by_{key}"`), which no literal scan + # can resolve, are covered by the prefix allowance in `_require_writer`. for path in sorted(_SRC_ROOT.rglob("*.py")): try: text = path.read_text(encoding="utf-8") diff --git a/tests/lint/rules/ce056_no_container_env_literal.py b/tests/lint/rules/ce056_no_container_env_literal.py index 59ced6eac..b8785928d 100644 --- a/tests/lint/rules/ce056_no_container_env_literal.py +++ b/tests/lint/rules/ce056_no_container_env_literal.py @@ -1,33 +1,16 @@ """CE056: no bare ``CODER_EVAL_IN_CONTAINER`` literal outside ``container_paths``. -``models/container_paths.py`` defines ``IN_CONTAINER_ENV`` and its comment states -why: the string is the predicate for four separate gates, and "two half-copies of -the same string in different packages is how a rename becomes a silent no-op". - -The constant shipped with that rationale, every READER was migrated to it -- and -the single WRITER was not. ``docker_runner`` kept emitting -``--env CODER_EVAL_IN_CONTAINER=1``, which is the one site that produces the -value all four gates consume. Changing the constant would therefore have updated -every consumer and left the container exporting the old name, so all four gates -would read "not in a container" at once: - - * ``Sandbox.enforces_permission_windows`` -- the reference-solution anti-cheat - window silently stops being applied, and a run that is NOT protected scores - like one that is; - * ``resolve_reference_dir`` -- the ``/work/references`` branch is skipped; - * ``_should_grade_in_container`` -- a grading container dispatches another - grading container; - * the orphan-container heartbeat watchdog's ``os._exit(137)`` gate. +Fires on any string constant in ``src/coder_eval/``, outside +``models/container_paths.py``, that equals the env-var name or starts with ``NAME=`` +(the child-process assignment form). Prose that only mentions the name does not fire. +Import ``IN_CONTAINER_ENV`` from ``coder_eval.models`` instead; add ``# noqa: CE056`` +for a genuinely unrelated string. -None of those fail loudly. This is the CE053 shape exactly (a rename-safety -constant that shipped beside the literals it was meant to replace), and CE052 -cannot catch it -- that rule inspects ``if`` guards, so it never looks at the -writer at all. +HAZARD: several gates read this one variable (see ``models/container_paths.py``). A +WRITER left on the literal disarms all of them silently after a rename. CE052 does not +cover this: it inspects ``if`` guards, never the writer. -Fires on any string constant in ``src/coder_eval/`` (outside the defining module) -that equals the env-var name or embeds it as an ``NAME=value`` assignment. -Import ``IN_CONTAINER_ENV`` from ``coder_eval.models`` instead; ``# noqa: CE056`` -for a genuinely unrelated string. +Rationale: .claude/notes/lint-rules.md § CE056 """ import ast diff --git a/tests/lint/rules/ce057_sidecar_shim_stdlib_only.py b/tests/lint/rules/ce057_sidecar_shim_stdlib_only.py index 55ffe9f32..f213d8559 100644 --- a/tests/lint/rules/ce057_sidecar_shim_stdlib_only.py +++ b/tests/lint/rules/ce057_sidecar_shim_stdlib_only.py @@ -1,27 +1,17 @@ """CE057: a sidecar module copied beside a generated sandbox shim stays stdlib-only. ``Sandbox._generate_cli_recorders`` writes every module in -``models.sandbox.SIDECAR_MODULES`` into the recorder directory beside each -``record_cli`` shim that declares response rules, and the shim imports it as a -sibling. That sidecar runs inside the sandbox, where ``coder_eval`` is not -installed and no project dependency is guaranteed, so one -``from coder_eval.models import ...`` or ``import pydantic`` makes every shadowed -CLI die with an ImportError the moment the agent runs it. It surfaces as "the -tool is broken", never as "the harness wrote an unimportable sidecar", and it -costs a whole run to diagnose. +``models.sandbox.SIDECAR_MODULES`` beside each ``record_cli`` shim that declares +response rules, and the shim imports it as a sibling where ``coder_eval`` is not +installed. Such a module may import only roots in ``STDLIB_ALLOWED``, an allowlist, +so growing it is a deliberate edit. A relative import always fires. +``from __future__`` gets its own message: drop the line, do not widen the allowlist. -Import-time enforcement (a test that renders and executes a shim) only catches it -when a test happens to declare a response rule; this rule catches it the moment -the import is written. +HAZARD: the target set derives from ``SIDECAR_MODULES``, and +``tests/test_custom_lint.py`` asserts it matches a file that exists, so the rule +cannot silently guard zero files. -A stdlib module that is genuinely needed is added to ``STDLIB_ALLOWED`` below -- -deliberately an allowlist rather than a check against ``sys.stdlib_module_names``, -so growing the sidecar's surface is a decision someone makes on purpose. - -``from __future__ import ...`` falls out of that allowlist too, and is reported -separately: it is not an import hazard (every interpreter that can run the shim -supports it), so the fix is to drop the line rather than widen the allowlist -- -which is what the generic message would otherwise suggest. +Rationale: .claude/notes/lint-rules.md § CE057 """ import ast diff --git a/tests/lint/rules/ce058_no_timing_literal.py b/tests/lint/rules/ce058_no_timing_literal.py index d1364c5ed..390373211 100644 --- a/tests/lint/rules/ce058_no_timing_literal.py +++ b/tests/lint/rules/ce058_no_timing_literal.py @@ -1,84 +1,27 @@ """CE058: an unknown timing value may not become a numeric literal. -``duration_ms is None`` means *this was never timed*, and it is a different -fact from ``duration_ms == 0.0``, which means *it was timed and took no -measurable time*. Writing the literal publishes the second while meaning the -first, and every consumer downstream — an average, a breakdown percentage, a -timeline cell — then treats the invention as a measurement. Same reasoning as -CE049 on the score side. - -Two shipped defects motivate it. Antigravity constructed every -``AssistantMessage`` with ``generation_duration_ms=0.0``, so the task page's -Generation cell read ``0ms`` and its thinking/tool/text breakdown rendered -``0%`` for months with nothing failing. And Codex published the SDK item's own -``duration_ms`` straight through, so ``avg_command_time_ms`` divided real -milliseconds by a command count of which 70 of 211 in one nightly had never -been timed at all. - -A third field family joined the first two: ``TurnRecord.harness_startup_ms`` -and ``harness_teardown_ms``, the turn's head and tail buckets. They are the -same invariant one level up — a turn whose stream carried no assistant message -was never timed at either end, and a ``0.0`` there would claim the harness -started instantly, which is exactly the reading that sends a real gap into the -evalboard's ``Unaccounted`` cell while a named bucket says it was measured at -zero. A measured ``0.0`` remains a legitimate answer — a window subtracted -down to nothing by the tool execution inside it, or a clamped inversion where -both ends really were observed — so the two values must stay distinguishable. - -Six syntactic forms, one invariant, one id — the shapes the codebase actually -produced: - -1. a ``0`` / ``0.0`` constructor keyword on one of the telemetry constructors - that carry these fields; +In ``src/coder_eval/``, ``None`` means *never timed*; ``0.0`` means *timed, +instant*. Timing fields match ``_TIMING_NAME`` (``duration_ms``, command-time +pair, ``*_duration_ms``, ``*_startup_ms``/``*_teardown_ms``, ``*_union_ms``); +constructors, ``_TIMING_CONSTRUCTORS``. Six forms: + +1. a zero keyword on a timing constructor; 2. ``duration_ms or 0``; -3. ``x if x is not None else 0.0`` (and the ``is None`` mirror); -4. ``if x.duration_ms is None: x.duration_ms = 0.0`` — the form no existing - rule shape covers, and where the live Claude instance was hiding - (``_finalize_commands`` set it on every command force-closed without a - tool result, in the one harness a timing audit had called healthy); -5. ``model_copy(update={"duration_ms": 0.0})`` — a keyword rule is blind to a - dict, and the dict is how ``CommandTelemetry.duration_ms`` is actually - written on the Antigravity DONE path, so forms 1-4 alone would have left - the next author's ``"duration_ms": 0.0`` in that idiom unguarded; -6. ``cmd.duration_ms = 0.0`` as a PLAIN assignment — form 4 without the - ``is None`` guard, or under a guard that tests something else. - -Form 6 exists because form 4 was passing the live defect by coincidence. Form 4 -keys on the ``if`` test naming a timing attribute, and the shipped -``_finalize_commands`` bug happened to spell it that way -(``if cmd.duration_ms is None:``) — but the assignment sat inside an outer -``if cmd.result_status is None:`` block, and rewriting it to set the literal -under THAT guard instead, which reads just as naturally and books the identical -lie, was invisible to all five earlier forms. The rule was one plausible -refactor away from silent. A guard is only evidence about the value when the -guard names the value; without one there is no evidence at all, which is -strictly worse and must not be the case the rule misses. - -It uses ``_zero_literal``, not form 4's broader ``_numeric_literal``, and the -asymmetry is the point. Under ``if x is None`` the guard PROVES the value was -never measured, so any invented number is a defect. A bare assignment proves -nothing: ``cmd.duration_ms = elapsed_ms`` is how a measured value is written, -and a literal ``1234.0`` is a legitimate test factory or replay. Only the -placeholder zero is the tell — the same narrowing form 1 already makes, and for -the same reason. - -Forms 4 and 6 overlap on the zero case, so form 4 records the statements it -flags and form 6 skips them. The visit order makes that sound rather than -lucky: ``visit_If`` runs its own check BEFORE ``generic_visit`` descends into -the body, so the assignment is always registered before ``visit_Assign`` sees -it. Form 4 keeps its wider literal set, so a guarded ``= 1234.0`` still fires -exactly once, from form 4. - -BLIND SPOTS worth knowing. Form 1 keys on the callee's spelling, so -``AssistantMessageTelemetry`` (an import alias for ``AssistantMessage`` in -``claude_code_agent``) is matched by name only; renaming that alias silently -disarms form 1 for that module. And form 6 keys on the TARGET's spelling, so it -sees ``cmd.duration_ms = 0.0`` but not a write through a rebound local or -``setattr(cmd, field, 0.0)`` — the same limit every AST rule here has without -type inference. - -``# noqa: CE058`` for a genuinely aggregate-internal use where a missing value -really is a zero, with a comment saying so. +3. ``x if x is not None else 0.0`` (and the mirror); +4. ``if x.duration_ms is None: x.duration_ms = ``; +5. ``model_copy(update={"duration_ms": 0.0})``; +6. ``cmd.duration_ms = 0.0`` with no ``is None`` guard naming it. + +Forms 2-4 flag any number (each replaces an unmeasured value); forms 1, 5, 6 only +zero, since those shapes also write a real ``1234.0``. Form 6 skips form 4's hits: +``visit_If`` records them before ``generic_visit`` descends. + +BLIND SPOTS: forms 1 and 6 key on spelling; renaming the +``AssistantMessageTelemetry`` alias, a rebound local, or ``setattr`` escapes. + +``# noqa: CE058`` where missing genuinely means zero. + +Rationale: .claude/notes/lint-rules.md § CE058 """ import ast @@ -87,20 +30,10 @@ from tests.lint.rules.base import BaseRule -# Trailing-segment match, so `cmd.duration_ms` and `generation_duration_ms` -# fire while `duration_ms_limit` does not. The `_startup_ms` / `_teardown_ms` -# arms need a leading segment for the same reason the `_duration_ms` arm does: -# the shipped fields are `harness_*`, and a bare `startup_ms` is more likely a -# budget than a measurement. -# -# THREE field families, not two. `tool_union_ms` is the turn's third wall-clock -# bucket, on the same model and under the same None-vs-0.0 contract as the -# `harness_*` pair — and it matched NO arm above, so `TurnRecord(tool_union_ms=0.0)` -# would have been invisible even though `TurnRecord` is already in -# `_TIMING_CONSTRUCTORS`. Naming the field `tool_union_duration_ms` to inherit -# the generic `_duration_ms` arm for free was considered and rejected: the two -# fields beside it needed their own arm for exactly this reason, and one -# spelling across the four buckets is worth two lines of regex. +# Trailing-segment match, so `cmd.duration_ms` and `generation_duration_ms` fire while +# `duration_ms_limit` does not. THREE field families, not two: `tool_union_ms` is the +# turn's third wall-clock bucket and matches no other arm. +# Rationale: .claude/notes/lint-rules.md § CE058 field families _TIMING_NAME = re.compile( r"^(duration_ms|generation_duration_ms|total_command_time_ms|avg_command_time_ms" r"|[a-z_]*_duration_ms|[a-z_]*_(?:startup|teardown)_ms|[a-z_]*_union_ms)$" diff --git a/tests/lint/rules/ce059_generation_window_is_two_reads.py b/tests/lint/rules/ce059_generation_window_is_two_reads.py index 254de2b34..4f78b9a5b 100644 --- a/tests/lint/rules/ce059_generation_window_is_two_reads.py +++ b/tests/lint/rules/ce059_generation_window_is_two_reads.py @@ -1,39 +1,19 @@ """CE059: one clock read cannot measure a window. -An ``AssistantMessage`` that receives the SAME name for both ``started_at`` and -``completed_at`` records a zero-length generation window — whatever -``generation_duration_ms`` happens to say beside it. The Antigravity reducer -read ``datetime.now()`` once and passed it as both bounds, so -``started_at == completed_at`` on 368 of 368 sampled messages and every -consumer that derives a window from the two stamps saw nothing at all. A window -needs two reads at two moments. +In ``src/coder_eval/agents/``, an ``AssistantMessage`` (or +``AssistantMessageTelemetry``) call may not pass the same name for both ``started_at`` +and ``completed_at``: that records a zero-length window, whatever +``generation_duration_ms`` says. The check is skipped unless BOTH bounds are a bare +``ast.Name``; attribute and call expressions are not compared. -Separate id from CE058 deliberately: this is a different invariant (a -zero-length window, regardless of what the duration field says), and one -invariant per id is what makes a ``# noqa`` mean one thing. +It does NOT fire when the same call passes ``generation_duration_ms=None``. That call +states that no window was measurable, so it claims none. Keep this exemption; do not +replace it with ``# noqa`` lines at those sites. -WHAT IT DOES NOT FIRE ON, and why that is the rule rather than a stack of -suppressions: a call that passes ``generation_duration_ms=None`` in the same -breath is not claiming a window — it is saying, in the field built to say it, -that none was measurable. Three sites are legitimately like that (Codex's -rollout rebuild, and both sub-agent syntheses on Codex and Claude: the -generation arrives as a tool result and is never streamed), and collapsing -their bounds to one ``now()`` is then a formatting choice, not a false -measurement. Exempting them here — rather than through four permanent -``# noqa`` lines — keeps the rule pointed at the case that actually misleads: -a duration asserted beside two stamps that cannot support it. +BLIND SPOT: two DIFFERENT names that hold the same instant at runtime. No AST rule can +see it; the replay-based ``assert_timing_captured`` golden invariant catches it. -Scoped to ``src/coder_eval/agents/``, the layer that measures. The check is -skipped unless BOTH bounds are a bare ``ast.Name`` — comparing attribute or -call expressions (``self.a`` vs ``self.b``) would be guesswork. - -BLIND SPOT: two DIFFERENT names that hold the same instant at runtime. Codex -already produces that shape — ``started = _ms_to_dt(self.open_start_ms)`` and -``completed = _ms_to_dt(self.open_end_ms if ... is not None else -self.open_start_ms)`` collapse to one instant whenever ``open_end_ms`` is -None. No AST rule can see it. The catch for that case is the replay-based -``assert_timing_captured`` golden invariant, which runs the real reducer and -asserts a non-zero window actually came out. +Rationale: .claude/notes/lint-rules.md § CE059 """ import ast diff --git a/tests/lint/rules/ce060_message_id_declared.py b/tests/lint/rules/ce060_message_id_declared.py index fbdb48200..a2d3337e9 100644 --- a/tests/lint/rules/ce060_message_id_declared.py +++ b/tests/lint/rules/ce060_message_id_declared.py @@ -1,68 +1,23 @@ """CE060: an assistant message must declare its identity. -``AssistantMessage.message_id`` is what lets a consumer tell two generations -apart. Antigravity simply omitted the kwarg, so the field defaulted to ``None`` -on every message it ever recorded, and the evalboard — which groups assistant -emissions by ``message_id`` and falls back to a wall-clock ``SAME_EMISSION_GAP_MS`` -threshold when either side lacks one — folded a whole turn's generations into a -single timeline row once the harness's windows became contiguous. Nothing -failed: the consumer sums a group, so every total came out right, and the -golden snapshots had ratified the ``null`` the day they were written. It was -not confined to the timeline either — a grouped emission is one API call to the -thinking-cost simulator, so its whole cache cascade was computed from one call -per turn. The mechanism and the blast radius live in -``docs/agents/HARNESS_PARITY.md`` § Timing capture; neither is restated here. +In ``src/coder_eval/agents/``, every ``AssistantMessage(...)`` must pass +``message_id=``, not as a literal ``None``; ``**`` expansion does not count. -Separate id from CE058 and CE059 deliberately: those two are about *timing* -(an unknown duration published as a literal, a window built from one clock -read), this one is about *identity*. One invariant per id is what makes a -``# noqa`` mean one thing. +The rule derives its constructor set from each module's own ``coder_eval.models`` +imports (an absolute or relative ``from`` import of ``AssistantMessage`` under +any alias, and +``.AssistantMessage(...)``); the class name comes from +``AssistantMessage.__name__``. That resolution lives in ``_model_ctor.py``, shared +with CE061: fix a new import spelling there, once. -WHY IT RESOLVES ALIASES where CE058 and CE059 hardcode constructor names: -CE058's own docstring already concedes that spelling-based matching dies on a -rename, and the weakness is live — ``claude_code_agent.py`` binds *only* -``AssistantMessage as AssistantMessageTelemetry`` and never the bare name, so a -name list guards that file's two construction sites purely because somebody -wrote the current alias into a different rule. CE060 instead derives its -constructor set from each module's own ``coder_eval.models`` imports, which -removes the gap rather than documenting it and catches an arbitrary -``AssistantMessage as Msg`` besides. Widening the other two rules the same way -is recorded in ``.claude/harness-candidates.md``; it is a change to two shipped -rules and needs its own mutation checks. +BLIND SPOT 1: the runtime ``None``. The kwarg must be PRESENT, not statically +non-``None``. A fallback expression (OpenCode ``messageID``, Pi ``responseId``) is a +decision and can still yield ``None``; the golden corpus catches only a later change. -What it removes is the *local binding* spelling, not every rename: the class's -own name still has to be known, so it is taken from the model itself -(``AssistantMessage.__name__``) rather than written here as a string, the way -CE056 imports ``IN_CONTAINER_ENV`` and CE057 derives its target set from -``SIDECAR_MODULES``. Renaming the model therefore moves this rule with it. +BLIND SPOT 2: a re-export through an intermediate module +(``from .sibling import AssistantMessage``); see ``_model_ctor.py``. -That resolution lives in ``_model_ctor.py`` and is shared with CE061, which -needs the identical answer to a different question. Keeping two copies would -mean a new import spelling needs two fixes in two rules. - -BLIND SPOT 1: the runtime ``None``. The rule requires the kwarg to be -*present*, not non-``None`` when it runs. ``opencode_agent.py`` passes -``str(part.get("messageID") or "") or None`` and ``pi_agent.py`` the same shape -for ``responseId``, so either records ``None`` whenever the id is missing from -the payload (`pi_a_single_text_turn.json` is a snapshot of that shape, though -its null comes from a fixture that emits no ``responseId`` rather than from a -live CLI omission). No AST rule can see it, and demanding a statically -non-``None`` value would be wrong: passing a fallback expression *is* deciding -what the id is. The sensor for that case is the golden corpus, and only -partially — a snapshot is written from whatever the code currently does, so it -catches a later change, never an initial omission. - -BLIND SPOT 2: a binding the resolver cannot follow. It reads one module's own -imports, so it sees the direct forms — absolute or relative ``from ... import -AssistantMessage``, under any alias — and the attribute spelling -``.AssistantMessage(...)``. What remains invisible is a re-export -through an intermediate module (``from .sibling import AssistantMessage``); see -``_model_ctor.py``. - -A ``**``-expanded call fires: such a call has not declared the field at the -site. There is no carve-out because no site in ``src/coder_eval/agents/`` uses -``**`` expansion for these constructors; if one is ever added, pass -``message_id=`` explicitly beside it. +Rationale: .claude/notes/lint-rules.md § CE060 """ import ast diff --git a/tests/lint/rules/ce061_window_via_close_window.py b/tests/lint/rules/ce061_window_via_close_window.py index 47e49a3e3..edd18c48b 100644 --- a/tests/lint/rules/ce061_window_via_close_window.py +++ b/tests/lint/rules/ce061_window_via_close_window.py @@ -1,59 +1,24 @@ """CE061: a generation window must come from the shared helper. -Pi shipped measuring its window from its own ``turn_start`` while four sibling -reducers tiled from a mark, so the wall clock between one turn's end and the -next turn's start — the model time that PRODUCED that turn — fell into no -bucket at all. Nothing failed. ``docs/agents/HARNESS_PARITY.md`` asserted the -four-bucket identity, and the only sensor for it -(``tests/_fixtures/golden_streams/_scrub.py``) checks ONE side: it catches a -bucket claiming more time than the turn contains and says nothing about one -claiming less. Pi's own tests passed because they were written against Pi's -own arithmetic. - -That is the shape this rule guards against: not a reducer that computes the -window wrongly, but a reducer that computes it AT ALL instead of asking -``coder_eval.timing.close_window``. A new harness whose author reimplements the -arithmetic inline arrives with a green test suite by construction. - -Separate id from CE058, CE059 and CE060 deliberately. Those three are about the -VALUES a message carries — an unknown duration published as a literal, a window -built from one clock read, a missing identity. This one is about PROVENANCE: -where the arithmetic came from. One invariant per id is what makes a ``# noqa`` -mean one thing. - -NOTE what this rule no longer covers, and deliberately: the tool SUBTRACTION is -not part of a window's geometry any more, so "did this reducer subtract -correctly" is not a question here. CE063 owns it — no module in ``agents/`` may -import ``busy_ms`` at all. - -BLIND SPOT, and it is the whole weakness of the chosen shape: this proves the -module IMPORTS the helper, never that any particular call used it. The value -passed to ``generation_duration_ms=`` is always a local (``generation_ms``, -``gen_parts[idx]``), so no AST rule can trace it back to a call. The sensors for -the arithmetic itself are ``tests/test_timing_close_window.py`` and the -per-reducer window tests; this rule adds only the cheap structural half that -neither can reach — a sixth harness rolling its own. - -It costs NO suppression. It used to cost exactly one: ``claude_code_agent.py`` -computed its window from a monotonic delta and subtracted tool time once at -finalization, because a call issued by an earlier emission is still running when -the next window closes — and forcing that into ``close_window`` would have meant -a mode flag on a helper whose whole value is having one shape. Moving the -subtraction to ``timing.subtract_tool_time`` dissolved the exception: -the collector is already the place where every span is known, so claude-code -needs no separate pass and calls the same shrunken helper as the other four. -``tests/test_custom_lint.py::TestCE061WindowViaCloseWindow::test_the_rule_is_now_exemption_free`` -pins the suppression set EMPTY, so a new exemption has to be argued for. - -EXEMPT, because both are honest claims that no window was measured: an explicit -``generation_duration_ms=None`` (codex's rollout rebuild, claude-code's -sub-agent synthesis) and the kwarg absent altogether, which defaults to -``None``. Not matched: ``**``-expansion and ``model_copy(update={...})`` — CE058 -already covers the ``model_copy`` dict shape for timing literals. - -Alias resolution, and its blind spot, live in ``_model_ctor.py``, shared with -CE060. The helper's own name is taken from the function object rather than -written here as a string, so renaming it moves this rule too. +In ``src/coder_eval/agents/``, a module that passes a non-``None`` +``generation_duration_ms=`` to an ``AssistantMessage`` must import +``coder_eval.timing.close_window`` (a from-import under any alias, or the +``timing`` module itself). The invariant is PROVENANCE of the window arithmetic; +tool-time subtraction is CE063's. + +EXEMPT, as honest claims that no window was measured: an explicit +``generation_duration_ms=None`` and the kwarg absent. Not matched: +``**``-expansion and ``model_copy(update={...})`` (CE058 covers that dict shape). + +BLIND SPOT: this proves the module IMPORTS the helper, never that a particular +call used it; the published value is always a local. The arithmetic's sensors +are ``tests/test_timing_close_window.py`` and the per-reducer window tests. + +``TestCE061WindowViaCloseWindow::test_the_rule_is_now_exemption_free`` pins the +suppression set EMPTY, so a new exemption must be argued for. Alias resolution, +and its blind spot, live in ``_model_ctor.py``, shared with CE060. + +Rationale: .claude/notes/lint-rules.md § CE061 """ import ast diff --git a/tests/lint/rules/ce063_no_busy_ms_in_agents.py b/tests/lint/rules/ce063_no_busy_ms_in_agents.py index 18d0e1b69..52e2a54cb 100644 --- a/tests/lint/rules/ce063_no_busy_ms_in_agents.py +++ b/tests/lint/rules/ce063_no_busy_ms_in_agents.py @@ -1,51 +1,22 @@ """CE063: a reducer may not compute its own tool subtraction. Tool execution comes out of a generation window in exactly ONE place: -``coder_eval.timing.subtract_tool_time``. Before that, five -reducers each did it themselves — four through ``close_window`` as they -flushed, claude-code once at finalization — while the head and the tail were -already computed centrally at the collector seam. That asymmetry is where every -timing defect on this branch actually lived, and none of them was in the -arithmetic: they were in the bookkeeping AROUND it. When to reset a per-step -span list (clearing it at ``step_start`` wiped a span before the flush could -subtract it — a 100% overstatement of that window). When to clear a spent start -stamp (a second flush with no intervening start republished the previous span — -3000 ms of generation for a 2000 ms turn). When to advance the mark. - -A sixth harness whose author reaches for ``busy_ms`` is rebuilding exactly that -bookkeeping, and its tool time would then be subtracted TWICE: once by the -reducer and once by the collector, which subtracts from every window it is -handed. The result is a silently under-reported generation figure on one -harness only — the shape that takes a corpus comparison to notice. - -Separate id from CE061 deliberately, and CE061 is NOT rebodied into this. -CE061 asks where a window's ARITHMETIC came from, and four reducers still call -``close_window``, so its property is still live and still worth guarding — it -is not superseded. This one asks a different question: whether a reducer -subtracts tool time at all. One invariant per id is what makes a ``# noqa`` -mean one thing. (Phase 5 did make CE061 exemption-free: claude-code now calls -the shrunken ``close_window`` like the other four, so its one permanent -suppression is gone.) - -WHY NOT ``_imports_the_helper``, which CE061 uses. That function deliberately -returns True for a bare module import (``from coder_eval import timing``), so -that ``timing.close_window(...)`` counts as reaching the helper — its own -comment says a rule that missed it "would tell an author to change a working -call site." Inverted into a BAN that branch flags any reducer importing the -module and calling ``timing.close_window(...)``, which after Phase 5 is four of -them. So this rule keys on the ``busy_ms`` NAME binding plus an -``ast.Attribute`` match for the ``timing.busy_ms`` spelling, and leaves the -module import alone. - -The name is taken from the function object rather than written here as a -string, the way CE061 takes ``close_window``: renaming it moves this rule too. - -BLIND SPOT: a reducer that re-implements the union inline, without importing -anything, is invisible — as is one reaching ``busy_ms`` through a re-export. -The sensor for the arithmetic itself is -``tests/test_timing_identity_contract.py``, which drives every harness off a -scripted clock and asserts the four buckets tile the turn to the millisecond; -this rule adds only the cheap structural half that a static check can reach. +``coder_eval.timing.subtract_tool_time``. In ``src/coder_eval/agents/`` this fires on +a ``from`` import of ``busy_ms`` (any alias, relative forms included) and on the +``timing.busy_ms`` spelling. Every timing defect in per-reducer subtraction lived +not in the arithmetic but in the bookkeeping AROUND it: when to reset a span list, +clear a spent start stamp, advance the mark. Separately, a reducer that also +subtracts takes tool time out twice. + +HAZARD: do not reuse CE061's ``_imports_the_helper``: inverted into a ban, its +bare-module-import branch flags any reducer calling ``timing.close_window``. The banned +name comes from the function object, so a rename moves the rule. + +BLIND SPOT: a reducer that re-implements the union inline, or reaches ``busy_ms`` +through a re-export, is invisible. ``tests/test_timing_identity_contract.py`` is the +sensor for the arithmetic. + +Rationale: .claude/notes/lint-rules.md § CE063 """ import ast diff --git a/tests/lint/rules/ce064_turn_bracket_on_the_clock.py b/tests/lint/rules/ce064_turn_bracket_on_the_clock.py index 713f823f9..d7214bbf7 100644 --- a/tests/lint/rules/ce064_turn_bracket_on_the_clock.py +++ b/tests/lint/rules/ce064_turn_bracket_on_the_clock.py @@ -1,63 +1,20 @@ """CE064: a clocked harness must stamp its turn BRACKET off that same clock. -``decompose_turn`` computes the head and the tail by subtracting a generation -window bound from an ``AgentStartEvent`` / ``AgentEndEvent`` timestamp. Those -two stamps therefore have to share a basis, and a reducer that derives its -window bounds from a ``TurnClock`` while letting the bracket fall back to -``StreamEvent.timestamp``'s ``default_factory=datetime.now`` puts a -monotonic-derived stamp and a raw wall stamp inside one subtraction — the exact -split ``timing.TurnClock`` exists to remove, reintroduced at the one seam the -clock does not own. +In ``src/coder_eval/agents/``, a module that imports ``TurnClock`` must pass an +explicit ``timestamp=`` to every ``AgentStartEvent`` and ``AgentEndEvent``. +``decompose_turn`` subtracts a generation-window bound from a bracket stamp, so +both must share a basis; ``StreamEvent.timestamp`` defaults to a raw +``datetime.now()``. -MEASURED, not hypothetical. Instrumenting ``decompose_turn`` on a live -antigravity turn printed:: +SCOPE IS DERIVED, never a harness list: a module is in scope because it imports +``TurnClock``, so a harness that adopts a clock comes into scope with no edit +here. Which harnesses have a clock: ``docs/agents/HARNESS_PARITY.md``. - PROBE tail: elapsed=-0.017000ms busy=0.000000ms raw=-0.017000ms - last_completed = 09:05:22.033099 - agent_end = 09:05:22.033082 +BLIND SPOT: presence, not correctness. The rule cannot tell +``self.clock.now()`` from ``datetime.now()`` written at the call site. The guard +for the source is behavioural: ``tests/_bracket_clock.py``. -an ``AgentEndEvent`` stamped 17 us BEFORE its own last message finished, which -cannot happen: the event is constructed strictly after the final flush. -``decompose_turn`` then clamps the negative to ``0.0`` and publishes it, which -is "measured, and instant" — the CE058 confusion, arrived at from the other -direction. The published ``harness_teardown_ms`` was ``0.0`` for a harness -whose real tail is ~0.1 ms. - -WHY IT ONLY SHOWED ON ONE HARNESS, and why the rule is not scoped to that one: -the drift between the two clocks is tens of microseconds, so it can only flip a -sign where the true interval is itself that small. Antigravity is the only -harness that spawns its process ONCE in ``start()`` and holds it across turns, -so nothing happens between its last flush and its ``AgentEndEvent``; every -other harness books a head of 0.2-6 s and a tail of 7-543 ms, where the drift -is invisible. Invisible is not absent. The fix belongs at every clocked site -because that is what makes the subtraction single-basis rather than -usually-close, and "usually-close" is not a property a millisecond field can -rest on. - -SCOPE IS DERIVED, never listed. The rule applies to a module under -``agents/`` that imports ``TurnClock`` — antigravity, pi and claude-code today. -Codex and OpenCode take their spans from the CLI's own epoch stamps and -deliberately have no ``TurnClock`` (see that class's docstring), so a raw -``datetime.now()`` bracket is CONSISTENT with their bounds and the rule must -not fire on them; the noop agent has no windows at all. The day one of them -adopts a clock, this rule starts applying to it with no edit here — which is -the half a hardcoded harness list would get wrong. - -Separate id from CE058/CE059/CE060/CE061 for the reason CE060 states: one -invariant per id, so a ``# noqa`` means one thing. CE058 is about publishing a -literal for an unknown duration, CE059 about a window built from a single clock -read, CE060 about identity, CE061 about where a window's arithmetic comes from. -This one is about the turn's OUTER bounds, which no other rule looks at — they -all scope to ``AssistantMessage``, and the bracket is not one. - -BLIND SPOT: presence, not correctness. The rule requires ``timestamp=`` to be -passed; it cannot tell ``self.clock.now()`` from ``datetime.now()`` written out -at the call site, because an agent may legitimately reach its clock through any -expression (a local ``clock`` in ``communicate``, ``state.clock`` from the -caller, ``self.clock`` inside the state). Demanding a specific spelling would -make the rule a syntax check on three harnesses' internal structure. What it -removes is the SILENT case — a default nobody chose — which is the one that -shipped. +Rationale: .claude/notes/lint-rules.md § CE064 """ import ast diff --git a/tests/lint/rules/ce066_no_report_imports_in_core.py b/tests/lint/rules/ce066_no_report_imports_in_core.py index e740cb986..55da3c2fb 100644 --- a/tests/lint/rules/ce066_no_report_imports_in_core.py +++ b/tests/lint/rules/ce066_no_report_imports_in_core.py @@ -1,37 +1,20 @@ """CE066: core may import only the reports package's public WRITERS. -The invariant is not "core must not import reports" — core legitimately *writes* -reports: ``orchestrator.py`` writes the per-task HTML and ``orchestration/batch.py`` -drives ``ReportGenerator``. What must not happen is core reaching into the reports -layer for a **metric, a statistic, a serializer or a formatter**, because that is -how a number the evaluation loop needs comes to live in a rendering module. +The invariant is not "core must not import reports" — core legitimately *writes* reports. +What must not happen is core reaching into the reports layer for a **metric, a statistic, a +serializer or a formatter**, because that is how a number the evaluation loop needs comes to +live in a rendering module. Those names live in ``result_metrics.py``, ``stats.py`` and +``run_record.py``. -Before the split that was the actual shape of the code: the orchestrator imported -``turn_time_buckets`` and ``visible_turn_count`` from ``reports_stats``, and -``orchestration/batch.py`` imported the run.json row serializer from -``reports_experiment``. Those names now live in ``result_metrics.py``, ``stats.py`` -and ``run_record.py``, and this rule is what stops the next one drifting back. +Scope: ``_layers.is_core_path`` (the package, minus ``cli/`` and ``reports/``). The +permitted set is an ALLOWLIST of writers, so a newly added report helper is banned from core +by default. Both the ABSOLUTE and the RELATIVE spelling are checked; the relative form is +the local idiom. -An ALLOWLIST, not a denylist — the CE018 rationale. A newly added report helper is -banned from core by default rather than after someone notices. The list is purely -writers; ``eval_result_to_task_dict`` is deliberately absent, because carrying a -serializer on it would be the rule documenting a wart instead of the wart being -removed. +BLIND SPOT: the rule checks the imported NAME, not what is done with it. Importing +``ReportGenerator`` and then reaching through the class for a private helper is invisible. -Both the ABSOLUTE and the RELATIVE spelling are checked. That is not a detail: -the relative form is the local idiom — both surviving edges in the tree are -``from .reports import write_task_html`` (orchestrator.py) and ``from ..reports -import ReportGenerator`` (orchestration/batch.py) — and an earlier draft of this -rule matched only ``node.module``, which for a relative import holds -``"reports"`` with the dots in ``node.level``. It therefore fired on nothing the -codebase actually writes, and its own tests passed because they used the -absolute form. An unrun assertion is documentation, not enforcement. - -**Blind spot, stated deliberately:** the rule checks the imported NAME, not what -is done with it. ``from coder_eval.reports import ReportGenerator`` followed by -reaching through the class for a private helper is invisible here. That is the -cheap version, consistent with CE004's own "catches the one mistake we have -actually seen" note. +Rationale: .claude/notes/lint-rules.md § CE066 """ import ast diff --git a/tests/lint/rules/no_agent_timing_access.py b/tests/lint/rules/no_agent_timing_access.py index 6d1e5ea9d..d40033ce5 100644 --- a/tests/lint/rules/no_agent_timing_access.py +++ b/tests/lint/rules/no_agent_timing_access.py @@ -1,8 +1,7 @@ -"""CE006: ``.agent.max_turns`` and ``.agent.turn_timeout`` are no longer fields. +"""CE006: ``.agent.max_turns`` and ``.agent.turn_timeout`` are not ``AgentConfig`` fields. -Phase 2 of the agent-timing refactor (2026-05-07) deleted these from -``AgentConfig``. They live on ``TaskDefinition`` (top-level) and as a per-call -argument on ``Agent.communicate(..., max_turns=...)``. +For where they live, see CE007; ``max_turns`` is also a per-call argument on +``Agent.communicate(..., max_turns=...)``. This rule blocks reintroduction. Pattern matched: any AST node of shape ``.agent.`` — both reads and writes. @@ -11,8 +10,8 @@ - ``criterion.max_turns`` / ``criterion.turn_timeout`` — those are real fields on ``LLMJudgeCriterion`` / ``AgentJudgeCriterion`` and are not nested under ``.agent``, so the AST shape doesn't match. -- ``task.max_turns`` / ``task.turn_timeout`` — top-level fields on - ``TaskDefinition``; not nested under ``.agent``, doesn't match. +- ``task.max_turns`` / ``task.turn_timeout`` — not nested under ``.agent``, + doesn't match (see CE007). If a legitimate exception arises, suppress with ``# noqa: CE006`` and a comment explaining why. diff --git a/tests/lint/rules/no_cli_imports_in_core.py b/tests/lint/rules/no_cli_imports_in_core.py index b4f7ba1d2..b67223877 100644 --- a/tests/lint/rules/no_cli_imports_in_core.py +++ b/tests/lint/rules/no_cli_imports_in_core.py @@ -1,30 +1,18 @@ -"""CE004: core layers must not import from coder_eval.cli. - -The rule's scope is everything under src/coder_eval/ except the cli/ package -itself. Importing from coder_eval.cli creates an upward dependency that breaks -testability in isolation. The package anchor and the cli/ boundary live in -``_layers`` so CE004 and CE066 cannot drift apart about where either is; -re-enumerating the packages here is how that list rots. - -``reports/`` is in scope, unlike under CE066. The reports package runs without -the CLI — the orchestrator writes a task report mid-run — so a ``cli`` import -there closes a cli -> orchestration -> reports -> cli cycle. CE004 once borrowed -CE066's core predicate whole and inherited its ``reports/`` exemption; nothing -had imported ``cli`` from there yet, so the hole was latent rather than live. - -``harbor/`` is in scope for the same reason ``orchestration/`` is: -its reward writer wants to raise a plain exception (``RewardWriteSkippedError``, -or the re-exported ``RegradeError``) and let the CLI wrap it into an exit -code — exactly the ``orchestration/regrade.py`` -> ``evaluate`` shape. - -Both the absolute and the RELATIVE spelling are checked — see -``_layers.imports_package`` for why that distinction is load-bearing rather than -pedantic. - -Note: this is a single, narrow rule (no upward imports into cli). For a -fully layered import graph (no upward imports between any layers), evaluate -import-linter / grimp — purpose-built for that. CE004 is the cheap version -that catches the one mistake we have actually seen. +"""CE004: core layers must not import from ``coder_eval.cli``. + +An upward dependency on ``coder_eval.cli`` breaks testability in isolation. Scope is +everything under ``src/coder_eval/`` except ``cli/`` itself, via ``_layers`` so CE004 and +CE066 cannot drift apart about where the boundary is. + +``reports/`` is in scope, unlike under CE066: it runs without the CLI, so a ``cli`` import +there closes a ``cli -> orchestration -> reports -> cli`` cycle. ``harbor/`` is in scope for +the same reason ``orchestration/`` is. Both the absolute and the RELATIVE spelling are +checked; see ``_layers.imports_package``. + +BLIND SPOT: one narrow rule, not a layered import graph. For that, evaluate import-linter / +grimp. + +Rationale: .claude/notes/lint-rules.md § CE004 """ import ast diff --git a/tests/lint/rules/no_top_level_run_limits_access.py b/tests/lint/rules/no_top_level_run_limits_access.py index 02ce61aca..71cea1638 100644 --- a/tests/lint/rules/no_top_level_run_limits_access.py +++ b/tests/lint/rules/no_top_level_run_limits_access.py @@ -1,9 +1,8 @@ -"""CE007: ``.max_turns`` / ``.task_timeout`` / ``.turn_timeout`` are no longer top-level fields. +"""CE007: ``.max_turns`` / ``.task_timeout`` / ``.turn_timeout`` are not top-level; they live under ``run_limits``. -Phase 1 of the unify-run-limits refactor (2026-05-12) removed these from -``TaskDefinition``, ``ExperimentDefaults``, and ``ExperimentVariant``. They -live under ``run_limits`` now. This rule blocks reintroduction by flagging -``.`` attribute access in core code. +This holds for ``TaskDefinition``, ``ExperimentDefaults``, and +``ExperimentVariant``. The rule flags ``.`` attribute access +in core code. Pattern matched: an ``Attribute`` read or write whose attribute name is one of the three banned names AND whose immediate prefix matches a known task-config diff --git a/tests/lint/rules/no_transcript_regex_in_eval.py b/tests/lint/rules/no_transcript_regex_in_eval.py index c3eaa7690..06bc15376 100644 --- a/tests/lint/rules/no_transcript_regex_in_eval.py +++ b/tests/lint/rules/no_transcript_regex_in_eval.py @@ -1,26 +1,18 @@ """CE013: Don't regex-parse agent transcripts inside ``evaluation/`` or ``criteria/``. -Replacing text-with-JSON verdict parsing with a typed tool channel was the -whole point of the 2026-05-20 judge refactor. The legacy parser used regex -against structural-tag patterns (``[ASSISTANT]``, ``[RESULT - …]``) and -JSON-shape literals (``\\{``, ``"score"``, ``"rationale"``) — both highly -coupled to ``ClaudeCodeAgent._format_messages`` rendering choices that -have no business being a correctness contract. - -This rule prevents that pattern from coming back. Inside ``evaluation/`` -or ``criteria/``, any call to ``re.compile`` / ``re.search`` / ``re.match`` / -``re.findall`` / ``re.finditer`` / ``re.fullmatch`` whose first positional -argument is a string literal containing one of: +Inside ``evaluation/`` or ``criteria/``, flags any ``re.compile`` / +``re.search`` / ``re.match`` / ``re.findall`` / ``re.finditer`` / +``re.fullmatch`` call whose first positional argument is a string literal +containing one of: * ``[ASSISTANT]``, ``[RESULT``, ``[TOOL USE]`` (structural transcript tags) - * ``"score"``, ``"rationale"`` (JSON-shape verdict fields) - * ``\\{`` followed by a verdict-shape hint (loose: literal ``\\{`` with - one of the above markers anywhere in the pattern) + * ``"score"``, ``"rationale"``, in either quote style (JSON-shape verdict fields) -…is flagged. Use the verdict tool channel instead. +Use the verdict tool channel instead. Add ``# noqa: CE013 -- `` for +legitimate non-correctness-critical sites (e.g. log scrubbing, telemetry-only +utterance extraction). -Add ``# noqa: CE013 -- `` for legitimate non-correctness-critical -sites (e.g. log scrubbing, telemetry-only utterance extraction). +Rationale: .claude/notes/lint-rules.md § CE013 """ import ast diff --git a/tests/lint/rules/yaml_models_forbid_extras.py b/tests/lint/rules/yaml_models_forbid_extras.py index 781efb548..f28e4dd0c 100644 --- a/tests/lint/rules/yaml_models_forbid_extras.py +++ b/tests/lint/rules/yaml_models_forbid_extras.py @@ -1,42 +1,23 @@ """CE009: Pydantic input-config models must declare ``extra='forbid'``. -Without ``extra='forbid'`` a misspelled YAML key (e.g. ``directry: foo/`` vs -``directory: foo/`` on ``ReferenceSource``, or a typo on a criterion field) is -silently dropped on the floor — the user gets no signal that the key was -ignored. ``extra='forbid'`` rejects unknown keys at load time, surfacing the -typo with a concrete field name in the error message. - -Scope (hard-coded path filter) — every model module that parses user-facing -YAML (task or experiment definitions): - - ``src/coder_eval/models/tasks.py`` - - ``src/coder_eval/models/criteria.py`` - - ``src/coder_eval/models/mutations.py`` - - ``src/coder_eval/models/experiment.py`` - - ``src/coder_eval/models/templates.py`` - - ``src/coder_eval/models/sandbox.py`` - - ``src/coder_eval/models/agent_config.py`` - - ``src/coder_eval/models/limits.py`` - -Result/persistence models are EXEMPT — deliberate round-trip leniency: -``models/results.py`` and ``models/telemetry.py`` are out of scope entirely -(they preserve forward-compat fields through ``model_dump_json`` → -``model_validate_json`` round-trips of task.json records), and the -result-shaped classes inside ``models/experiment.py`` (``VariantResult``, -``ExperimentResult``, …) carry per-class ``# noqa: CE009`` suppressions for -the same reason. - -A class is considered compliant when ANY of the following holds: - 1. Its own body declares ``model_config = ConfigDict(..., extra='forbid', ...)``. - 2. It inherits from a base defined in the SAME file whose body declares the - above (pydantic inherits ``model_config`` from the parent). - 3. It does NOT directly extend ``BaseModel`` in its ``bases`` clause — only - direct ``BaseModel`` subclasses are flagged. A class extending a non-compliant - same-file base that itself extends ``BaseModel`` slips through this rule; - compliance is treated as the base's responsibility (and the base is itself - flagged). This trade keeps the rule cheap to implement without walking the - transitive base chain across files. +Without it a misspelled YAML key is silently dropped; ``extra='forbid'`` rejects the +key at load time and names the field. + +Scope: the model modules that parse task or experiment YAML, listed in +``_SCOPED_PATHS``. ``models/results.py`` and ``models/telemetry.py`` are out of scope +(task.json round-trips keep forward-compat fields); result-shaped classes in +``models/experiment.py`` carry per-class ``# noqa: CE009`` for the same reason. + +A class is compliant when its own body declares +``model_config = ConfigDict(..., extra='forbid', ...)``, when a base defined in the +SAME file declares it, or when it does not directly extend ``BaseModel``. + +BLIND SPOT: a class that extends a non-compliant same-file base passes; only that base +is flagged. Base chains are not followed across files. Add ``# noqa: CE009`` on the class statement line for an intentional exception. + +Rationale: .claude/notes/lint-rules.md § CE009 """ import ast @@ -44,10 +25,7 @@ from tests.lint.rules.base import BaseRule -# Path suffixes are matched against the runner's filepath after normalizing -# backslashes (Windows). Keeping the canonical form forward-slash means the -# rule definition stays platform-neutral while still matching on Windows where -# pathlib hands us native-separator strings. +# Forward-slash suffixes; the filepath is normalized first, so Windows paths match too. _SCOPED_PATHS = ( "src/coder_eval/models/tasks.py", "src/coder_eval/models/criteria.py", @@ -71,7 +49,6 @@ def _is_basemodel_base(b: ast.expr) -> bool: def _declares_extra_forbid(class_body: list[ast.stmt]) -> bool: """True iff the class body assigns ``model_config = ConfigDict(..., extra='forbid', ...)``.""" for stmt in class_body: - # Match: ``model_config = ConfigDict(...)`` if not isinstance(stmt, ast.Assign): continue if not ( @@ -92,49 +69,33 @@ class YamlModelsForbidExtras(BaseRule): def __init__(self, filepath: str) -> None: super().__init__(filepath) - # Map of class name → whether it declares extra="forbid". Filled by visit_ClassDef - # on the first pass; used to satisfy descendants in the same file. + # Class name → declares extra="forbid"; filled before the visit, read by same-file descendants. self._extra_forbid_by_class: dict[str, bool] = {} - # Path filter: only run on the scoped files. Normalize backslashes - # to forward slashes so the suffix match works on Windows runners where - # pathlib hands the rule native-separator strings. normalized = filepath.replace("\\", "/") self._active = any(normalized.endswith(p) for p in _SCOPED_PATHS) def check(self, tree: ast.AST) -> list: # type: ignore[override] if not self._active: return [] - # First pass: record which classes declare extra="forbid" so descendants - # in the same file can be considered compliant via inheritance. + # First pass: record which classes declare it, so same-file descendants inherit compliance. for node in ast.walk(tree): if isinstance(node, ast.ClassDef): self._extra_forbid_by_class[node.name] = _declares_extra_forbid(node.body) - # Second pass: flag classes that neither declare it themselves nor have - # a base in the same file that does. + # Second pass: flag classes with neither their own declaration nor a compliant same-file base. self.visit(tree) return self.violations def visit_ClassDef(self, node: ast.ClassDef) -> None: - # Only flag classes that descend from BaseModel (directly or via a base - # defined elsewhere in the file). descends_from_basemodel = any(_is_basemodel_base(b) for b in node.bases) - # Or transitively: any base name we've seen with extra="forbid" counts. - # We don't track transitive base→BaseModel chains; we trust the AST + the - # convention that all model classes in these files derive (eventually) - # from BaseModel. - # If this class declares it directly → compliant. if _declares_extra_forbid(node.body): self.generic_visit(node) return - # If any base is a known-compliant class in the same file → compliant - # (pydantic inherits model_config). + # A compliant same-file base is enough: pydantic inherits model_config. for b in node.bases: if isinstance(b, ast.Name) and self._extra_forbid_by_class.get(b.id, False): self.generic_visit(node) return - # Otherwise: only flag if this class directly extends BaseModel — leaves - # of the class hierarchy that descend from a known model base are out of - # scope (their compliance is the base's responsibility). + # Only direct BaseModel subclasses are flagged; see the module BLIND SPOT. if descends_from_basemodel: self.violation( node, diff --git a/tests/lint/runner.py b/tests/lint/runner.py index 81ddadc6d..51aeebaae 100644 --- a/tests/lint/runner.py +++ b/tests/lint/runner.py @@ -18,7 +18,6 @@ from tests.lint.rules.ce020_no_sdk_typed_base_agent_fields import NoSdkTypedBaseAgentFields from tests.lint.rules.ce021_guarded_evaluationresult_parse import GuardedEvaluationResultParse from tests.lint.rules.ce022_dialog_loop_statement_cap import SimulationDialogLoopStatementCap -from tests.lint.rules.ce023_no_proxy_shim_import import NoProxyShimImports from tests.lint.rules.ce024_discriminated_unions import DiscriminatedUnions from tests.lint.rules.ce032_criteria_path_seam import CriteriaPathSeam from tests.lint.rules.ce037_no_dead_private_helper import NoDeadPrivateHelper @@ -58,20 +57,12 @@ from tests.lint.violation import Violation -# CE062 IS DELIBERATELY UNUSED and must stay that way — the ids above jump 061 -# to 063. It was claimed during the turn-timing work and then folded into CE063 -# rather than shipped. An id is a permanent documentation anchor: a suppression -# comment carrying 062 in an older branch, review or commit message must never -# start meaning something new. -# -# Claim 068 next. NOTE 065 IS TAKEN and is not in ALL_RULES: doc-surface and -# whole-tree rules are `@pytest.mark.lint` classes in tests/test_custom_lint.py -# rather than BaseRules, so the `_rule_ids` uniqueness assert below cannot see -# them. Enumerating them here is how this note fell behind CE044, so grep -# instead: `grep -E '^class Test(CE[0-9]{3})' tests/test_custom_lint.py`. Spell -# it `[0-9]`, not `\d` — GNU and BSD `grep -E` read `\d` as a literal `d` and -# report zero hits, which reads as "no ids taken". The whole id space is -# unioned in one place by TestRuffExternalCoversEveryRule._known(). +# CE062 and CE023 are RETIRED ids and must stay unused — the ids above jump 061 to +# 063. An id is a permanent documentation anchor: a suppression comment carrying one +# in an older branch or commit message must never start meaning something new. +# Claim 068 next; 065 is taken without being in ALL_RULES. To see the whole space: +# `grep -E '^class Test(CE[0-9]{3})' tests/test_custom_lint.py`. +# Rationale: .claude/notes/lint-rules.md § The CE id space type RuleClass = type[BaseRule] ALL_RULES: list[RuleClass] = [ @@ -97,7 +88,6 @@ NoSdkTypedBaseAgentFields, GuardedEvaluationResultParse, SimulationDialogLoopStatementCap, - NoProxyShimImports, DiscriminatedUnions, CriteriaPathSeam, NoDeadPrivateHelper, @@ -142,8 +132,8 @@ def _is_suppressed(source_lines: list[str], v: Violation) -> bool: """Honor `# noqa` placed on any line spanned by the offending AST node. AST nodes for multi-line statements report `lineno` at the start, so a - `# noqa: CE002` placed on a closing paren or inner argument line was - previously missed. We scan every physical line from `line` through + `# noqa: CE002` may sit on a closing paren or inner argument line. + We scan every physical line from `line` through `end_line` (inclusive) for a matching suppression marker. """ if v.line == 0 or v.line > len(source_lines): diff --git a/tests/lint/workflow_outputs.py b/tests/lint/workflow_outputs.py index 7a9012869..74d300526 100644 --- a/tests/lint/workflow_outputs.py +++ b/tests/lint/workflow_outputs.py @@ -1,36 +1,20 @@ """CE035 — every ``steps..outputs.`` / ``needs..outputs.`` reference in a workflow must resolve to a key its writer actually produces. -The motivating bug shipped in ``verify-published-action.yml``: two steps read -``steps.parity.outputs.version``, but the ``parity`` step writes only ``pin`` / -``newest`` / ``lagging`` (the *shell variable* was ``VERSION``, the *output key* was -``newest``). GitHub expands an unwritten output to the empty string, so -``TAG_REF: v${{ steps.parity.outputs.version }}`` became the bare string ``v``, -``git show "v:action.yml"`` exited 128 under ``set -euo pipefail``, and the preflight -job was red on 100% of triggers — which, via ``needs: preflight``, meant the paid -end-to-end tier could never run at all. - -Nothing caught it: the workflow is invisible to ruff, pyright, pytest and the AST lint -runner, and ``actionlint`` models ``steps.*.outputs`` as an open string map, so an -unwritten shell key is untyped and unflagged there too. - -**Writers are mechanically enumerable, and this rule only reasons about the ones that -are.** For a referenced step id: - -* ``run:`` step → the keys it echoes/prints into ``$GITHUB_OUTPUT``. Writers are - collected by an over-approximating scan (any ``key=`` / ``key<<`` in an ``echo`` or - ``printf`` in the body), because over-approximating *writers* can only make the rule - quieter, never produce a false failure. If a body touches ``$GITHUB_OUTPUT`` in a way - the scan cannot read (no key found at all), the step is skipped rather than guessed at. -* local composite (``uses: ./``) → the ``outputs:`` block of the repo's ``action.yml``. -* third-party ``uses:`` → **skipped**. Resolving those needs the action's own metadata, - which is not on disk; pretending otherwise would fail on every pinned action. -* a missing step id, or a ``needs`` output absent from that job's ``outputs:`` map, is - always a finding — those are fully enumerable from the file. - -Like CE026-CE031 this is deliberately NOT a ``BaseRule`` in ``tests/lint/runner.py``: -that runner is AST-only over ``.py`` files, whereas this rule reasons over workflow YAML -plus embedded shell. It is wired as ``tests/test_custom_lint.py::TestCE035WorkflowOutputParity``. +Writers, per referenced step id: + +* ``run:`` step → any ``key=`` / ``key<<`` in an ``echo`` or ``printf``. + Over-approximating writers is safe; INVENTING one is a false failure. A body that + touches ``$GITHUB_OUTPUT`` with no readable key is skipped. +* local composite (``uses: ./``) → the ``outputs:`` block of ``action.yml``. +* BLIND SPOT: third-party ``uses:`` → **skipped**; its metadata is not on disk. +* a missing step id, or a ``needs`` output absent from that job's ``outputs:``, is + always a finding. + +Wired as ``tests/test_custom_lint.py::TestCE035WorkflowOutputParity``, not the AST +runner: it reads workflow YAML plus shell. + +Rationale: .claude/notes/lint-rules.md § CE035 """ from __future__ import annotations @@ -48,13 +32,11 @@ NEEDS_OUTPUT_REF = re.compile(r"needs\.(?P[A-Za-z_][A-Za-z0-9_-]*)\.outputs\.(?P[A-Za-z_][A-Za-z0-9_.-]*)") # `echo "key=value"` / `printf 'key=%s' …` / `echo "key< skip the step" contract in `_written_keys` and turns every real -# reference to that step into a false CE035 failure. See the docstring: over-approximating -# writers is safe, INVENTING one is not. +# Loose about what precedes the key, but the key must start at a TOKEN boundary: the start +# of the arguments, whitespace, or an opening quote. Without that anchor +# `printf "%s=%s\n" "$KEY" "$VAL"` yielded the key `{'s'}`, which defeats the "no readable +# key => skip the step" contract in `_written_keys` and turns every real reference to that +# step into a false CE035 failure. Over-approximating writers is safe; INVENTING one is not. OUTPUT_WRITE = re.compile(r"""(?:echo|printf)\s+(?:[^\n]*?["'\s])?(?P[A-Za-z_][A-Za-z0-9_.-]*)(?:=|<<)""") diff --git a/tests/test_action_inputs.py b/tests/test_action_inputs.py index a5db4cd1a..331c186bc 100644 --- a/tests/test_action_inputs.py +++ b/tests/test_action_inputs.py @@ -1,25 +1,16 @@ """Executable contract for the argv ``action.yml`` builds from its inputs. -The composite action's two bash steps assemble two command lines — a ``uv tool -install`` and a ``coder-eval run`` — out of eight string inputs. Everything that -can go wrong there goes wrong *silently*: an extra dropped from the requirement -string installs a working CLI that is missing an agent, and a value mangled by -word splitting or pathname expansion reaches the CLI as a different value than -the workflow wrote, so the run measures something else and still exits 0. - -These tests therefore execute the shipped script rather than reimplementing it. -Each step's ``run:`` body is pulled straight out of ``action.yml`` and run under -bash with ``uv`` / ``coder-eval`` replaced by stubs that record their argv, so the -assertions are about the real text that ships to consumers. A rewrite of the -script that changes the resulting command line fails here even if it looks -equivalent. - -The design these tests pin: the action promotes NONE of ``coder-eval run``'s 21 -flags to a named input. Everything goes through ``args``, one argv entry per -line, appended verbatim. That is what makes a ``-D`` override whose value is a -bracketed list (``key=[A,B,C]`` — a bash character class) survive; the earlier -whitespace-split input silently rewrote it to one name whenever a file in the -working directory happened to match. +Pins the ``uv tool install`` and ``coder-eval run`` command lines that the +composite action's two bash steps assemble from its inputs. Each step's ``run:`` +body is read from ``action.yml`` and run under bash with ``uv`` / ``coder-eval`` +replaced by argv-recording stubs, so a rewrite that changes the resulting command +line fails here. + +The action promotes no ``coder-eval run`` flag to a named input: ``args`` takes one +argv entry per line, appended verbatim, so a bracketed ``-D key=[A,B,C]`` value +reaches the CLI unexpanded. + +Rationale: .claude/notes/reporting.md § Why the action argv tests run the shipped script """ from __future__ import annotations @@ -73,21 +64,13 @@ def _step_script(step_name: str) -> str: def _stub(dir_: Path, name: str) -> Path: """A fake executable recording its argv and its inherited ``CE_PROBE``, exit 0. - Written in bash rather than Python on purpose. ``shell: bash`` on a Windows - runner is Git Bash, which rewrites arguments that look like absolute POSIX - paths on the way to a *native* Windows binary: a python-shebang stub gets - ``/action-checkout`` as ``C:/Program Files/Git/action-checkout``, and - switching that conversion off only moves the failure, because the shebang - launcher then cannot hand python its own script path either. A bash stub - never crosses that boundary, so argv arrives byte-for-byte everywhere. - - argv is recorded NUL-delimited instead of as JSON so a value carrying a - quote, a backslash or a space needs no escaping on the way out of bash. Each - invocation truncates the file; no test invokes the stub twice. - - ``CE_PROBE`` is how the env-passthrough test observes what the child actually - received: the passthrough exports into the step's own shell, so only a process - the script itself launches can report it. + argv goes NUL-delimited to ``argv.bin`` and ``CE_PROBE`` to ``probe.txt``. Each + invocation truncates the record, so a test must invoke the stub at most once. + + Keep the stub a bash script: Git Bash on a Windows runner rewrites POSIX-path + arguments passed to a native binary, such as a Python-shebang stub. + + Rationale: .claude/notes/reporting.md § Why the action argv tests run the shipped script """ record = dir_ / "argv.bin" # Forward slashes, not the native separator: the consumer is MSYS bash, which @@ -391,7 +374,7 @@ def test_outputs_are_written_before_a_failing_exit(self, run_script, tmp_path): assert proc.returncode == 3, "the step must exit with coder-eval's own code" assert _outputs(tmp_path)["run-dir"] == "runs/ci" - # The action no longer appends run.md to the job summary: a consumer that has + # The action does not append run.md to the job summary: a consumer that has # to redact the report first cannot undo a write that already happened. def test_nothing_is_written_to_the_job_summary(self, run_script, tmp_path): (tmp_path / "runs" / "ci").mkdir(parents=True) @@ -445,13 +428,9 @@ def test_a_malformed_entry_is_reported_by_position_not_by_value(self, run_script assert "s3cr3t-token-value" not in out assert "entry #1" in out - # An `env` value carrying a newline splits into a second entry, because the - # loop is line-based. That used to be an argv-rewrite: the pairs were - # `export`ed into the step's own shell, which is where CE_ARGS and - # CE_RUN_DIR are read from AFTER the loop. They are collected and handed to - # `env` now, so the injected entry reaches the child as data and nothing - # else. Reachable without a hostile author: any interpolated value or a - # rotated multi-line secret. + # A newline in an `env` value splits into a second entry; it must reach the child + # as data only, never rewrite the step. + # Rationale: .claude/notes/reporting.md § Why the action argv tests run the shipped script @pytest.mark.parametrize("hijack", ["CE_ARGS", "CE_RUN_DIR", "GITHUB_OUTPUT"]) def test_a_newline_in_a_value_cannot_rewrite_the_step(self, run_script, tmp_path, hijack): rc, argv, out = _coder_eval( diff --git a/tests/test_action_version_pin.py b/tests/test_action_version_pin.py index dc8be8916..501c856a0 100644 --- a/tests/test_action_version_pin.py +++ b/tests/test_action_version_pin.py @@ -1,29 +1,14 @@ """The two derived version pins must equal ``pyproject.toml``'s version. -``pyproject.toml`` is the single version source; two files carry a *derived* pin -of it, and one ``release.yml`` step bumps both inside the release commit: - -- ``action.yml``'s ``version:`` default — the published composite action installs - ``coder-eval==``, so a consumer pinning - ``UiPath/coder_eval@vX.Y.Z`` (or the moving ``@v0``) must get X.Y.Z and not - some other release. -- ``plugins/coder-eval/.claude-plugin/plugin.json``'s ``version`` — the Claude - Code plugin manifest. ``claude plugin validate --strict`` rejects a manifest - with no version, and a pinned-but-stale one strands users on a cached copy - because Claude Code keys plugin updates off it. - -The release-time seds make both invariants mechanically true *at rest* — every -commit on main has the pins in agreement with ``pyproject.toml``. - -Nothing asserted it, which is how ``action.yml`` shipped pinned to 0.8.6 while -main was already 0.8.9: the seds live on the release path only, so a hand-edit -(or a release whose amend step was skipped) drifts silently and ``@v0`` -consumers install a version other than the tag they pinned. - -This also guards the ``# <-- kept in sync`` anchor itself: ``release.yml``'s sed -is keyed on that exact trailing comment, so a reformat that detaches it turns -the release-time bump into a no-op (caught there by a ``grep -q`` guard, but -only after the tag exists). +Pins, on every commit, ``action.yml``'s ``version:`` default and +``plugins/coder-eval/.claude-plugin/plugin.json``'s ``version`` to +``pyproject.toml``. Also pins the line shapes that ``release.yml``'s sed is keyed +on: the ``# <-- kept in sync`` anchor in ``action.yml`` and the comma-terminated +``"version"`` line in ``plugin.json``. + +A reformat that detaches either anchor turns the release-time bump into a no-op. + +Rationale: .claude/notes/reporting.md § Why the derived version pins are tested on every commit """ from __future__ import annotations diff --git a/tests/test_agent.py b/tests/test_agent.py index 9636b1082..07a5b5a5a 100644 --- a/tests/test_agent.py +++ b/tests/test_agent.py @@ -86,9 +86,9 @@ async def test_discard_pending_turn_rolls_back_when_partial_build_failed(): """If _set_pending swallowed an exception and left pending_turn=None, discard must still roll back the iteration counter. - Regression: previously the rollback gated on (pending_turn is not None), so - a swallowed partial-build exception caused _iteration to drift permanently - higher on every double-failure. + Pins: the rollback fires on ``_iteration_was_incremented`` even when + ``pending_turn`` is None (the two signals are OR'd), so a swallowed + partial-build exception cannot drift ``_iteration`` higher on every double-failure. """ config = parse_agent_config(type=AgentKind.CLAUDE_CODE, permission_mode="acceptEdits") agent = ClaudeCodeAgent(config) @@ -781,8 +781,8 @@ class _ToolUseEvent: assert "[TOOL USE] Read" in formatted # Test 5: Non-tool_use event of the same shape — falls through to the - # unknown-tag branch (was previously filtered; now we surface "an - # unknown message type appeared" via its class name). + # unknown-tag branch, which surfaces "an unknown message type appeared" + # via its class name. class _ThinkingEvent: type = "thinking" @@ -857,35 +857,15 @@ class RateLimitEvent: def test_format_messages_system_message_subclasses_are_filtered(): - """Regression: SystemMessage SUBCLASSES (TaskStartedMessage, etc.) must - be filtered out the same way SystemMessage itself is. - - claude-agent-sdk 0.1.x added ``TaskStartedMessage``, - ``TaskNotificationMessage``, and ``TaskProgressMessage`` for sub-agent - lifecycle reporting. Each is declared as a subclass of - ``SystemMessage`` with an explicit drop-in contract: - - "Subclass of SystemMessage: existing ``isinstance(msg, - SystemMessage)`` and ``case SystemMessage()`` checks continue to - match." - - An earlier version of ``_format_messages`` compared the exact - ``type(msg).__name__`` string against ``"SystemMessage"``, which - defeated the SDK's drop-in design — the subclasses fell through to - an "unknown message type" branch that ran ``str(msg)[:100]`` and - emitted a truncated Python-repr containing nested ``data={...}`` - dict literals. Even though the typed verdict tool channel has - since obviated the brace-walking verdict parser that originally - motivated this fix, the underlying ``isinstance``-vs-name-equality - contract is still worth pinning. - - This test exercises the real SDK ``TaskStartedMessage`` instance - (not a name-collision mock) and asserts: - - 1. The lifecycle message is silently filtered (not emitted as a - tag, exactly as ``SystemMessage`` itself would be). - 2. A verdict-shaped JSON literal in a sibling ``AssistantMessage`` - survives intact in the formatter output. + """SystemMessage SUBCLASSES (TaskStartedMessage, etc.) are filtered like SystemMessage itself. + + Pins: ``_format_messages`` drops a real SDK ``TaskStartedMessage`` (not a + name-collision mock) without emitting a tag, and a verdict-shaped JSON + literal in a sibling ``AssistantMessage`` survives intact. + + HAZARD: the SDK declares its task lifecycle messages as drop-in + ``SystemMessage`` subclasses, so the filter must use ``isinstance``; a + ``type(msg).__name__`` comparison sends them to the unknown-type branch. """ from claude_agent_sdk import ( AssistantMessage, @@ -957,8 +937,8 @@ def test_format_messages_system_message_subclasses_are_filtered(): assert formatted.count("{") == formatted.count("}") # Formatter contract: verdict JSON survives intact in the textual transcript - # used for log auditing. The judge no longer parses this output — it's - # purely a human-readable artifact now — but a regression that drops or + # used for log auditing. The judge does not parse this output — it is + # purely a human-readable artifact — but a regression that drops or # truncates the verdict text would still mask debugging signal. assert verdict_json in formatted diff --git a/tests/test_agent_config_no_timing_fields.py b/tests/test_agent_config_no_timing_fields.py index 88c7f2621..ce3c4d7e8 100644 --- a/tests/test_agent_config_no_timing_fields.py +++ b/tests/test_agent_config_no_timing_fields.py @@ -1,5 +1,6 @@ -"""Phase-2 regression: ``max_turns`` and ``turn_timeout`` are no longer fields on -``AgentConfig``. They live on ``TaskDefinition`` (top-level) and on +"""``AgentConfig`` has no ``max_turns`` or ``turn_timeout`` field. + +They live under ``TaskDefinition.run_limits`` and, for ``max_turns``, on ``Agent.communicate(max_turns=...)`` per-call. """ diff --git a/tests/test_agent_config_sdk_decoupling.py b/tests/test_agent_config_sdk_decoupling.py index f31af717b..48960c6b8 100644 --- a/tests/test_agent_config_sdk_decoupling.py +++ b/tests/test_agent_config_sdk_decoupling.py @@ -1,7 +1,7 @@ -"""Regression: BaseAgentConfig no longer leaks claude_agent_sdk types. +"""BaseAgentConfig exposes no claude_agent_sdk types. -``plugins`` stays on the vendor-neutral base (Codex + docker_runner read it) but is -retyped to the local ``LocalPluginConfig`` TypedDict; ``setting_sources`` moves down to +``plugins`` is on the vendor-neutral base (Codex + docker_runner read it), typed as the +local ``LocalPluginConfig`` TypedDict; ``setting_sources`` is on ``ClaudeCodeAgentConfig`` (the only consumer). """ diff --git a/tests/test_agent_golden_master.py b/tests/test_agent_golden_master.py index 216bbe80f..0438f10ab 100644 --- a/tests/test_agent_golden_master.py +++ b/tests/test_agent_golden_master.py @@ -31,9 +31,8 @@ from tests._fixtures.golden_streams.pi_fixtures import PI_SCENARIOS, run_pi_scenario -# Codex is an optional extra (mirrors test_codex_agent's guard). Import its -# fixtures only when present so the Claude golden tests in this module still -# collect and run in a base (no-codex) environment instead of erroring at import. +# Codex is an optional extra (mirrors test_codex_agent's guard): import its fixtures only when present, so +# this module still collects in a base (no-codex) environment instead of erroring at import. _HAS_CODEX = importlib.util.find_spec("openai_codex") is not None if _HAS_CODEX: from tests._fixtures.golden_streams.codex_fixtures import CODEX_SCENARIOS, run_codex_scenario @@ -42,45 +41,23 @@ run_codex_scenario = None -# Scenarios that legitimately produce no measurable generation window. Strict -# is the default: a new scenario is asserted to have one until it is named -# here, so a harness that silently stops recording windows fails instead of -# passing. Each entry carries the reason it cannot have one. +# Scenarios with no measurable generation window. Strict by default: a scenario not named here must have one, so +# a harness that silently stops recording windows fails. Each entry carries the reason it cannot have one. NO_GENERATION_WINDOW: frozenset[str] = frozenset( { "claude_g_crash_format_placeholder", # crash partial: 0 assistant messages "claude_h1_timeout_process_error", # timeout partial: 0 assistant messages "claude_h2_process_error_crash", # crash partial: 0 assistant messages - # Drives a SCRIPTED monotonic clock (a constant 1000.0 until the - # deadline flips) so the deadline break is deterministic. The window - # is zero by fixture construction, not by anything the harness did. + # A SCRIPTED monotonic clock makes the deadline break deterministic; the window is zero by construction. "claude_i_in_loop_deadline_break", "codex_g_items_rebuild", # rollout rebuild: Turn items carry no timestamps - # Codex emissions whose ENTIRE measurable window was tool execution. - # The window is subtracted down to 0 because that is the honest - # answer, not because nothing was recorded — see the generation-window - # subtraction in codex_agent._flush_message. + # Codex emissions whose ENTIRE measurable window was tool execution, subtracted down to an honest 0. "codex_d_cross_flush_is_error", # flush lands before the tool completes: zero-width window "codex_e_orphan_tool", # the tool never completes, so the window never opens - # Same shape, reached from the opposite direction. This scenario injects - # a 5 ms CLI tool interval into a replay whose whole turn is well under - # one millisecond, so the tool spans BOTH windows entirely and the - # central subtraction takes each down to a measured 0.0. It is the tool - # interval that is fictional, not the subtraction — which is why the - # scenario is in FICTIONAL_DURATIONS too. - # - # BE HONEST ABOUT WHAT IS LEFT. With both exemptions on, this snapshot - # asserts neither the identity nor a positive window, and it does NOT - # record the tiling the scenario is named for — `SCRUB_KEYS` masks - # `started_at`, `completed_at` and `generation_duration_ms`, so nothing - # about where a window opened survives into the JSON. What it still - # pins is the STRUCTURE: two assistant messages, their content blocks, - # their token buckets, and one resolved command. OpenCode's tiling is - # asserted where it can be — `tests/test_timing_identity_contract.py` - # (scripted clock, ms-exact) and - # `tests/test_opencode_agent.py::TestGenerationWindowsTileTheTurn`. - # `pi_c_multi_turn_tiling` is the same scenario shape on a harness whose - # stamps come from its own clock, and it needs neither exemption. + # An injected 5 ms CLI tool interval spans both sub-millisecond windows, so each subtracts to 0.0. With + # FICTIONAL_DURATIONS too, this snapshot pins only STRUCTURE (SCRUB_KEYS masks where a window opened); + # its tiling is asserted in tests/test_timing_identity_contract.py and + # tests/test_opencode_agent.py::TestGenerationWindowsTileTheTurn. "opencode_c_multi_step_tiling", } ) @@ -90,41 +67,23 @@ def _expect_window(harness: str, scenario_name: str) -> bool: return f"{harness}_{scenario_name}" not in NO_GENERATION_WINDOW -# Scenarios that inject their own SDK timestamps, so their recorded durations -# are FICTIONAL and cannot be reconciled against the replay's real wall clock. -# `_rebase_notifications` / `_rebase_lines` put those stamps on the replay's -# clock, which fixes the era — but the SDK's stamps are integer MILLISECONDS -# and these scenarios declare 17-900 ms of item time, while the replay itself -# runs in well under one. No rebasing closes that; the agent's own clock would -# have to be faked too. Everything else — every claude, antigravity and pi -# scenario, and the codex/opencode ones that inject nothing — is checked. -# -# The last two entries were ADDED to buy stability, and the trade is worth -# stating. They previously injected NO stamps at all, so `_flush_message` took -# `_ms_to_dt(None)` for both window bounds — two adjacent `datetime.now()` -# reads, which collide at microsecond resolution often enough that -# `assert_timing_captured`'s `completed_at > started_at` failed roughly one run -# in twenty under parallel load, naming a different scenario each time. Their -# identity check was near-vacuous anyway (a window of width zero reconciles -# trivially), so giving them real bounds trades that for a stable, meaningful -# bounds-span assertion. +# Scenarios that inject their own integer-MILLISECOND SDK stamps (17-900 ms of item time) into a replay that runs in +# well under one, so their durations are FICTIONAL and the identity is not checked; rebasing +# (`_rebase_notifications` / `_rebase_lines`) fixes the era, not the width. Every other scenario is checked. +# HAZARD: codex_c_reasoning_placeholder and codex_h_no_turn_completed_crash inject stamps for STABILITY. Without them +# both window bounds are adjacent `datetime.now()` reads that collide at microsecond resolution and flake +# `completed_at > started_at`. FICTIONAL_DURATIONS: frozenset[str] = frozenset( { "codex_b_command_execution", # 250 ms command + 150 ms generation - "codex_c_reasoning_placeholder", # 300 ms of item time — see below + "codex_c_reasoning_placeholder", # 300 ms of item time — see above "codex_d_cross_flush_is_error", # 400 ms command "codex_e_orphan_tool", # command started, never completed "codex_f_collab_fallback", # 900 ms collab wait - "codex_h_no_turn_completed_crash", # 200 ms of item time — see below + "codex_h_no_turn_completed_crash", # 200 ms of item time — see above "opencode_b_tool_call_resolved", # 17 ms tool interval - # 5 ms tool interval, injected as CLI epoch stamps. OpenCode takes its - # tool bounds from the CLI payload rather than from its own clock, so - # every scenario of this harness that resolves a tool injects them — - # there is no version of this scenario that stays commensurable with a - # sub-millisecond replay. Its TILING property (the second window opens - # at the first `step_finish`) is what the scenario is for, and that is - # still snapshotted; the identity is asserted for this harness by - # tests/test_timing_identity_contract.py, on a scripted clock. + # 5 ms tool interval as CLI epoch stamps: OpenCode takes tool bounds from the CLI payload, so no + # tool-resolving scenario of it stays commensurable with a sub-millisecond replay. "opencode_c_multi_step_tiling", } ) @@ -162,8 +121,7 @@ def _compare_or_regen(name: str, actual_scrubbed: dict[str, Any]) -> None: @pytest.mark.parametrize("scenario", CLAUDE_SCENARIOS, ids=lambda s: s.name) async def test_claude_golden(scenario, tmp_path): raw = await run_claude_scenario(scenario, str(tmp_path)) - # Reconciliation is asserted on the UNscrubbed dump (token buckets are never - # scrubbed, but cost/timestamps are — assert before masking to be explicit). + # Reconciliation is asserted on the UNscrubbed dump, before cost and timestamps are masked. assert_reconciliation(raw) assert_timing_captured( raw, @@ -264,9 +222,8 @@ async def test_pi_reconciliation_invariant(scenario, tmp_path): assert_reconciliation(await run_pi_scenario(scenario, str(tmp_path))) -# The ONE place a harness is listed for golden coverage. Derived from AgentKind -# rather than from register_builtins, whose built-in list is a hardcoded tuple -# inside the function body that returns nothing and exposes no set. +# The ONE place a harness is listed for golden coverage. Keyed on AgentKind, not register_builtins, which +# exposes no set of built-ins. SCENARIOS_BY_AGENT: dict[AgentKind, list[Any]] = { AgentKind.CLAUDE_CODE: CLAUDE_SCENARIOS, AgentKind.CODEX: CODEX_SCENARIOS, @@ -275,13 +232,9 @@ async def test_pi_reconciliation_invariant(scenario, tmp_path): AgentKind.PI: PI_SCENARIOS, } -# An ALLOWLIST of exclusions, not a denylist of inclusions: a new AgentKind -# member fails the coverage test until someone decides which it is. +# An ALLOWLIST of exclusions: a new AgentKind member fails the coverage test until someone decides which it is. _NO_GOLDEN_COVERAGE: dict[AgentKind, str] = { - # The agentless backend (NoOpAgent): it runs no model and streams nothing, - # so there is no event stream to record. AgentKind.NONE: "agentless backend — runs no model, streams nothing", - # A sentinel for "agent type could not be determined". Never registered. AgentKind.UNKNOWN: "sentinel for an undeterminable type — never registered", } @@ -325,8 +278,7 @@ def test_every_builtin_agent_kind_is_accounted_for(self): def test_every_covered_harness_has_scenarios(self): empty = harnesses_without_scenarios(SCENARIOS_BY_AGENT) if not _HAS_CODEX: - # The optional extra is absent, so CODEX_SCENARIOS is [] by - # construction — not by anyone forgetting to record a stream. + # The optional extra is absent, so CODEX_SCENARIOS is [] by construction, not by omission. empty -= {AgentKind.CODEX} assert not empty, f"listed as covered but has NO scenarios: {sorted(k.value for k in empty)}" @@ -363,9 +315,7 @@ def _record( identity is trivially satisfied, so these cases constrain only what each is about; the identity has its own cases below. """ - # MODEL-VALID, not merely shaped like a record. `assert_timing_captured` - # validates the dump into a `TurnRecord` so it can call production's own - # span selector instead of re-deriving one, and a fixture missing the + # MODEL-VALID: `assert_timing_captured` validates the dump into a `TurnRecord`, so a fixture missing # required fields would fail there rather than on the thing it is about. return { "iteration": 1, @@ -402,11 +352,8 @@ def test_exactly_zero_raises_too(self): assert_timing_captured(self._record(windows=[0.0]), expect_generation_window=True) def test_collapsed_bounds_raise_even_with_a_healthy_duration(self): - # Two harnesses take the duration from a MONOTONIC clock and the - # bounds from the wall clock, so a reducer can report a real duration - # beside two stamps that collapsed to one instant. CE059 sees that - # statically only when both bounds are the same ast.Name; this is the - # check for when they are two different names holding one value. + # A real duration beside two bounds collapsed to one instant. CE059 sees that statically only when both + # bounds are the same ast.Name; this catches two different names holding one value. with pytest.raises(AssertionError, match="bounds that span it"): assert_timing_captured(self._record(windows=[500.0], bounds_collapse=True), expect_generation_window=True) @@ -489,11 +436,8 @@ def test_a_turn_with_no_generation_must_report_neither(self): assert_timing_captured(self._record(windows=[], overhead=(None, None)), expect_generation_window=False) def test_an_unmeasurable_window_is_not_something_to_measure_against(self): - # codex_g_items_rebuild's shape: an assistant message exists, but it was - # rebuilt after the turn ended with placeholder now() bounds and says so - # via generation_duration_ms=None. Those stamps are not window bounds, so - # the honest head and tail are None — keying on "any assistant message" - # would have demanded a number derived from a placeholder. + # codex_g_items_rebuild's shape: a rebuilt message with placeholder now() bounds and + # generation_duration_ms=None. Those stamps are not window bounds, so the honest head and tail are None. with pytest.raises(AssertionError, match=r"harness_startup_ms is 0\.0"): assert_timing_captured(self._record(windows=[None], overhead=(0.0, 3.5)), expect_generation_window=False) diff --git a/tests/test_agent_telemetry.py b/tests/test_agent_telemetry.py index adcf59108..2b1415e07 100644 --- a/tests/test_agent_telemetry.py +++ b/tests/test_agent_telemetry.py @@ -1134,8 +1134,10 @@ async def mock_query(prompt, options): @pytest.mark.asyncio async def test_result_fallback_backfills_when_no_message_id(self, tmp_path): - """Legacy SDK / mock-stream path: AssistantMessages without message_id carry zeros, - and the ResultMessage usage backfills the last AssistantMessage (pre-fix behavior).""" + """With no message_id, the ResultMessage usage backfills the last AssistantMessage. + + Legacy SDK / mock-stream path: AssistantMessages without message_id carry zeros. + """ _tool_use_block_cls, assistant_message_cls, _user_message_cls, text_block_cls, _, result_message_cls = ( create_mock_sdk_messages() ) @@ -1286,33 +1288,16 @@ async def mock_query(prompt, options): class TestClaudeHeadIsMeasuredAtFirstOutput: """claude-code's head is the wall clock up to the first observed model output. - `_ClaudeTurnState.__init__` stamps `last_event_wall`, and - `_seed_first_generation_window` re-stamps it at the first `message_start`. - So the first window opens where the model first spoke, and the CLI spawn, - provider resolution and time to first token before it are the head. - - `_build_claude_query` is NOT in the head: it runs at `communicate`'s - `:1095`, before `AgentStartEvent` is emitted at `:1106`, so it precedes the - head's own start stamp. It used to sit inside msg0's generation window - (`last_event_wall` was stamped at state construction, ahead of the build); - it now sits inside `duration_seconds` but outside all four buckets, as - unexplained residual. That is why the budget below still matters and why it - is not the same guard it was: at 0.03-0.10 ms the residual is noise, and - the two tests keep it that way. - - It used to be `0.0`, and that was a CLAMPED NEGATIVE rather than a - measurement: both marks were stamped before `AgentStartEvent` was emitted, - so `decompose_turn`'s `max(..., 0.0)` produced it. The rejection rested on - claude-code running the model in-process. It does not — `claude-agent-sdk` - spawns the `claude` CLI over `anyio.open_process` and `_pump_messages` - calls `query()` once per `communicate()`, a fresh CLI per turn. - - The two budget tests below survive the rewrite with their meaning INVERTED. - `_build_claude_query`'s cost now lands in the head rather than inside msg0's - generation, so they no longer guard "the build is cheap enough to leave - hidden by the clamp" — they guard "our own setup is a negligible part of a - head that is now published", which is what makes the head readable as the - harness's latency rather than as ours. + `_seed_first_generation_window` re-stamps `last_event_wall` at the first + `message_start`, so the CLI spawn, provider resolution and time to first + token before it are the head. + + Pins: `_build_claude_query` stays cheap. It runs in `communicate` before + `AgentStartEvent` is emitted, so its cost is inside `duration_seconds` but + outside all four buckets, as unexplained residual; these budget tests keep + that residual noise. + + Rationale: .claude/notes/timing.md § Why the query build sits outside all four buckets """ # Measured at 0.03 ms bare and 0.10 ms with four plugin roots. The bound is diff --git a/tests/test_aggregate.py b/tests/test_aggregate.py index b0d53e327..4ce332a84 100644 --- a/tests/test_aggregate.py +++ b/tests/test_aggregate.py @@ -85,7 +85,7 @@ def test_build_run_summary_buckets_every_status_by_category() -> None: assert summary.tasks_succeeded == sum(1 for s in statuses if s.category == "succeeded") assert summary.tasks_error == sum(1 for s in statuses if s.category == "error") assert summary.tasks_failed == sum(1 for s in statuses if s.category == "failed") - # Budget sub-counters are a subset of "failed" — both previously unexercised. + # Budget sub-counters are a subset of "failed". assert summary.tasks_token_budget_exceeded == 1 assert summary.tasks_cost_budget_exceeded == 1 # The two the hand-picked test omitted both classify as "failed". diff --git a/tests/test_antigravity_agent.py b/tests/test_antigravity_agent.py index 117c8f892..681985e3b 100644 --- a/tests/test_antigravity_agent.py +++ b/tests/test_antigravity_agent.py @@ -169,8 +169,8 @@ def read_denied(workspaces) -> bool: tc = ag_types.ToolCall(name="read_file", canonical_path=str(skill_md)) return any(p.when(tc) for p in policies if p.when is not None) - assert read_denied([workdir]) is True # pre-fix: out-of-workspace → denied - assert read_denied(agent._resolve_workspaces(skills_paths)) is False # fix permits it + assert read_denied([workdir]) is True # workdir only: out-of-workspace → denied + assert read_denied(agent._resolve_workspaces(skills_paths)) is False # resolved workspaces permit it def test_to_token_usage_maps_gemini_buckets(): @@ -837,20 +837,14 @@ async def _record_sleep(seconds: float) -> None: async def test_communicate_finalizes_gracefully_under_a_realistic_turn_timeout(monkeypatch): - """A never-resolving orphan under a REALISTIC configured timeout (300s, the - framework's own experiments/default.yaml turn_timeout) must finalize through - the poll loop's own graceful path -- force-close the orphan, grade normally - -- instead of the ThreadedWatchdog cutting the whole turn at `timeout` first. - - Pre-fix, `_MAX_BACKGROUND_POLLS * _BACKGROUND_POLL_INTERVAL_SECONDS` (120 * - 5s = 600s) was DOUBLE the 300s default, so the watchdog always won that race - and this exact scenario -- a tool call spuriously left ACTIVE with no real - background job behind it, confirmed live in the final validation run -- burned - the full turn timeout and crashed as TurnTimeoutError with zero criteria - graded, a strict regression versus the pre-fix immediate finalize. Deriving - the poll deadline from a FRACTION of the real `timeout` (not a disconnected - cycle count) fixes it: the loop now exits through its own graceful path with - room to spare before the watchdog's harder cutoff would ever fire.""" + """A never-resolving orphan under a realistic 300s timeout finalizes gracefully. + + Pins: the poll deadline is a FRACTION of the configured `timeout` (300s is + experiments/default.yaml's own turn_timeout), so the poll loop force-closes + the orphan and grades normally before the ThreadedWatchdog cuts the turn. + + Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll + """ from coder_eval.agents import antigravity_agent monkeypatch.setattr(antigravity_agent.asyncio, "sleep", _no_sleep) @@ -994,24 +988,17 @@ def should_stop() -> bool: class _TwoLayerReentrancyGuardedConversation: - """Faithfully mirrors the REAL SDK's two-generator-layer shape: - ``Conversation.receive_steps()`` (the public method ``_drain()`` calls) is - ITSELF an async generator that delegates to - ``LocalConnection.receive_steps()`` (``async for step in - self._connection.receive_steps(): yield step``, verified against the - installed SDK) -- and the ``_is_receiving`` re-entrancy flag lives on that - INNER, connection-layer generator, not the outer one. A single-layer fake - (putting the flag directly on the generator ``_drain()`` iterates) cannot - catch a bug in how the outer/inner boundary is handled, since aclose()-ing - a generator always closes ITSELF -- the question this fake exists to probe - is whether that also reaches the inner one, and (confirmed live against - real asyncio semantics) it does NOT do so synchronously: a `GeneratorExit` - thrown into a delegating generator's frame does not immediately run the - generator it was mid-iterating -- that's deferred to the event loop's - async-gen finalizer, exactly like the original single-layer bug, just one - level down. ``_drain()``'s fix is therefore a bounded retry (yielding via - ``asyncio.sleep(0)`` for that already-scheduled finalizer to land), not a - claim that the inner generator closes synchronously.""" + """A fake Conversation that reproduces the real SDK's two async-generator layers. + + ``receive_steps()`` (the Conversation layer ``_drain()`` iterates) delegates + to a connection-layer generator, and the ``_is_receiving`` re-entrancy flag + lives on that INNER generator, cleared only in its own ``finally``. + + HAZARD: do not collapse this to one layer. Closing the outer generator does + not synchronously close the inner one, and only the two-layer shape + reproduces the transient ``RuntimeError`` that ``_drain()`` retries past. + + Rationale: .claude/notes/agents.md § The receive_steps re-entrancy window""" last_response = "" @@ -1491,11 +1478,9 @@ async def test_cap_reached_on_a_poll_redrain_stops_polling(monkeypatch): # --------------------------------------------------------------------------- # Generation window # -# Antigravity used to read datetime.now() ONCE per flush and pass it as both -# bounds with generation_duration_ms=0.0, so every task page reported 0ms of -# generation. The reducer now measures a real window and subtracts the tool -# executions that closed inside it — this harness interleaves tool calls into -# one generation, so a window legitimately contains time that is not model time. +# Each flush publishes a real window. This harness interleaves tool calls into +# one generation, so a window legitimately contains tool time, which the +# central `subtract_tool_time` takes back out. # --------------------------------------------------------------------------- _CLOCK_BASE = datetime(2026, 1, 1, 12, 0, 0) @@ -1505,23 +1490,13 @@ class _Clock: """Controlled stand-in for the reducer's clocks — a `TurnClock` and `time`. ONE monotonically advancing counter, read by both: every read — the turn - clock's `now()` or `time.monotonic()` — costs TICK_MS. So the fixture's - timeline is driven by read ORDER, not by elapsed time, and the two are - deliberately coupled rather than independent. That is enough to pin the - arithmetic exactly. - - Every WALL stamp the reducer records now derives from its per-turn - `TurnClock`, so this stands in for that object rather than for the - module's `datetime`. That distinction is load-bearing, not cosmetic: a - derived stamp does not read `datetime.now()`, so the old patch would no - longer reach it and these tests would quietly measure the real clock and - pass by accident. `time` is still patched because `duration_seconds` and - the poll deadlines read `time.monotonic()` directly, and must — a deadline - may not move when the wall clock steps. - - What it still does NOT prove is that the reducer keeps the two in their - proper roles; with one basis for every wall stamp there is no longer a - second role to confuse it with. + clock's `now()` or `time.monotonic()` — costs TICK_MS. The timeline is + driven by read ORDER, not by elapsed time, which pins the arithmetic exactly. + + HAZARD: every wall stamp the reducer records derives from its `TurnClock`, + so patching the module's `datetime` instead reaches nothing and the tests + silently measure the real clock. `time` stays patched because + `duration_seconds` and the poll deadlines read `time.monotonic()` directly. """ TICK_MS = 100.0 @@ -1731,12 +1706,9 @@ async def test_tool_execution_is_subtracted_from_the_window(monkeypatch): assert second.generation_duration_ms == pytest.approx(span_ms - bash.duration_ms) # The absolute figures are artifacts of `_Clock`, which charges one TICK_MS - # per clock READ. They moved from 400/300 to 300/200 when the reducer - # stopped taking a monotonic reading it no longer needs: a flush now reads - # the turn clock once where it used to read two clocks, so each window is - # one tick shorter on this fixture's read-driven timeline. Nothing about - # real elapsed time changed — the 100ms tool, which is still two reads - # apart, is unmoved. + # per clock READ: a flush reads the turn clock once, so on this fixture's + # read-driven timeline the window is 300 and the 100ms tool, two reads + # apart, leaves 200. assert span_ms == pytest.approx(300.0) assert second.generation_duration_ms == pytest.approx(200.0) @@ -1798,14 +1770,12 @@ async def test_a_straddling_tool_is_charged_only_for_its_in_window_part(monkeypa async def test_a_tool_still_open_at_the_flush_is_not_generation_time(monkeypatch): """The sibling of the straddle test above, for the window the tool opened IN. - Subtracting only CLOSED intervals published the part of a still-running - call that had already elapsed as model time, while the call's own - `duration_ms` counted it again. These windows tile the turn, so there is no - slack to absorb that: measured on tasks/hello_date with a live - gemini-3.1-pro-preview, a Bash opening 1.7 ms before the flush drove - Sum(generation) + Sum(command) 0.26 ms PAST the turn's own - `duration_seconds`, on a turn whose entire headroom was 1.4 ms. Four - sibling runs passed by 1.2-8.7 ms out of ~12 s, so it was a coin flip. + Pins: the part of a still-running call that has already elapsed at the flush + is subtracted from that window, so generation plus that part equals the + span. Counting it as model time books it twice (its own `duration_ms` counts + it too), and the windows cover the turn end to end with no slack to absorb that. + + Rationale: .claude/notes/agents.md § Per-harness generation marks """ _install_clock(monkeypatch, _Clock()) steps = [ @@ -1871,14 +1841,11 @@ async def test_generation_and_tool_time_account_for_the_turn(): """Σ generation + Σ tool + head + tail lands inside the turn's own duration. Bounds, not equality: the fake conversation's own overhead sits in the - residual. Before the window existed the generation half was identically 0. - - The HEAD is part of the sum, and has to be: the first window now opens at - the first observed `Step` rather than at turn entry, so the dispatch before - it is a measured bucket instead of time hidden inside msg0's generation. - Asserting `generation + tool` alone against a share of the turn was an - assertion that the head stays empty — which is what this phase deliberately - stopped being true. + residual. The HEAD is part of the sum because the first window opens at the + first observed `Step`, not at turn entry, so the dispatch before it is a + measured bucket rather than time inside msg0's generation. + + Rationale: .claude/notes/agents.md § First-generation window seeding """ steps = [ _step("THINKING", "DONE", thinking="plan", usage=_usage(100, 0, 5, 5)), @@ -1910,20 +1877,12 @@ async def test_generation_and_tool_time_account_for_the_turn(): assert head_ms > 0, "the dispatch before the first Step is now a measured bucket, not 0.0" assert gen_ms + tool_ms + head_ms + tail_ms <= turn_ms - # NO relative LOWER bound. This case runs on the REAL clock, and the fake - # conversation's own overhead is the residual — under parallel load the - # denominator (`duration_seconds`, the agent's monotonic span) inflates - # while the measured buckets do not, so any `>= share * turn_ms` assertion - # is a scheduler-noise detector. It was one: a `>= 0.5 *` bound survived - # here only while the sum excluded the head, and failed under `-n auto` - # once the head joined it. - # - # The share this test was reaching for IS asserted, exactly, in - # tests/test_timing_identity_contract.py — on a scripted clock, where the - # magnitudes are real and the identity closes to the millisecond. What is - # left here is what an end-to-end run can honestly claim: the buckets are - # measured, the head is no longer the clamped 0.0, and nothing overflows - # the turn. + # NO relative LOWER bound. This case runs on the REAL clock, so under parallel load + # the denominator (`duration_seconds`) inflates while the measured buckets do not, + # and any `>= share * turn_ms` assertion is a scheduler-noise detector. The exact + # share is asserted in tests/test_timing_identity_contract.py on a scripted clock. + # What an end-to-end run can honestly claim is left here. + # Rationale: .claude/notes/timing.md § decompose_turn async def test_timing_change_moves_no_token_bucket(): @@ -2007,12 +1966,11 @@ async def test_the_published_window_reconciles_to_its_own_bounds(monkeypatch): async def test_the_window_is_measured_without_relying_on_the_negative_clamp(monkeypatch): """A positive window, and no clamp underneath it. - The span used to be read off `time.monotonic()` while the tool intervals - were wall, so the two could disagree and drive the result negative; the - clamp that caught it published a `0.0` indistinguishable from a real - instant generation, and a debug line was the only trace. One basis makes - that unrepresentable: `busy_ms` clips to the window and unions overlaps, so - it cannot exceed a span derived from the same clock. + Pins: the span and the tool intervals share one clock basis, so `busy_ms` + (which clips to the window and unions overlaps) cannot exceed the span, and + no clamp publishes a `0.0` indistinguishable from a real instant generation. + + Rationale: .claude/notes/timing.md § TurnClock """ _install_clock(monkeypatch, _Clock()) steps = [ diff --git a/tests/test_cli_called_criterion.py b/tests/test_cli_called_criterion.py index ce50cee15..8667b423e 100644 --- a/tests/test_cli_called_criterion.py +++ b/tests/test_cli_called_criterion.py @@ -281,10 +281,9 @@ class TestLogHandling: escalate: a `FinalStatus.ERROR` reads as "harness broken, discard this data point", which is a strictly better outcome for a failing agent than FAILED. - An earlier revision raised `CheckerMisuseError` on `rule_error`, believing only a - task author could produce it. `test_a_crafted_rule_error_cannot_launder_a_failure` - is the regression test for that. The authoring concern it was addressing is - handled at LOAD time instead, by `RecordedCli._validate_responses_are_evaluable`. + `test_a_crafted_rule_error_cannot_launder_a_failure` pins this for `rule_error`. + Authoring faults are rejected at LOAD time, by + `RecordedCli._validate_responses_are_evaluable`. """ def test_a_shim_rule_fault_scores_zero_without_escalating(self, sandbox_with_log): @@ -317,9 +316,10 @@ def test_a_non_string_rule_error_still_scores_zero(self, sandbox_with_log): assert "42" in (result.error or "") def test_a_crafted_rule_error_cannot_launder_a_failure(self, sandbox_with_log): - """Regression: `rule_error` used to RAISE, on the premise that only a task - author could cause it. The log is agent-writable, so one appended line turned - an honest FAILED into a FinalStatus.ERROR -- i.e. "discard this data point". + """An appended `rule_error` line scores zero and does not raise. + + The log is agent-writable, so raising would let one appended line turn an + honest FAILED into a FinalStatus.ERROR -- i.e. "discard this data point". """ sandbox, sandbox_dir = sandbox_with_log # The agent never ran the required command, so this must fail. @@ -434,9 +434,10 @@ def test_empty_log_is_zero_calls_not_an_error(self, sandbox_with_log): assert result.error is None def test_malformed_line_now_fails_instead_of_being_skipped(self, sandbox_with_log): - """Superseded behaviour: an unparseable line used to be skipped with the - score untouched, which let a max_count: 0 guard pass on a truncated record - of the forbidden call. It is now a harness fault, like a missing log.""" + """An unparseable line is a harness fault, like a missing log. + + Skipping it would let a max_count: 0 guard pass on a truncated record of the + forbidden call.""" sandbox, sandbox_dir = sandbox_with_log log_path = sandbox_dir / LOG log_path.parent.mkdir(parents=True, exist_ok=True) @@ -642,9 +643,10 @@ def test_failure_details_show_what_was_actually_recorded(self, sandbox_with_log) assert "(+1 more)" in details def test_clustered_short_flags_are_split(self, sandbox_with_log): - """`-yf` used to parse as one flag named `yf`, so an aliases: [y] predicate - missed it -- leaving the `-y` escape one keystroke away from the hole - `aliases` exists to close.""" + """`-yf` splits into `-y` and `-f`, so an aliases: [y] predicate sees it. + + Parsed as one flag named `yf`, the `-y` escape would be one keystroke away + from the hole `aliases` exists to close.""" sandbox, sandbox_dir = sandbox_with_log _write_log(sandbox_dir, [_call(["ixp", "fields", "delete", "-yf", "proj-1"])]) guard = CliCalledCriterion( @@ -700,7 +702,7 @@ def test_undeclared_flag_leaves_its_neighbour_positional(self): assert flags == {"yes": [""]} def test_equals_form_keeps_a_dash_leading_value_and_invents_no_flag(self): - """`--offset=-1` used to drop the value AND invent a flag named `1`.""" + """`--offset=-1` keeps the value `-1` and invents no flag named `1`.""" positional, flags = split_flags(["get", "--offset=-1"], frozenset(), frozenset()) assert positional == ["get"] assert flags == {"offset": ["-1"]} @@ -740,7 +742,7 @@ def test_argv_not_a_list_of_strings_fails_loudly(self, sandbox_with_log): assert "unusable record" in (result.error or "") def test_required_flag_missing_entirely_scores_zero(self, sandbox_with_log): - """The branch separating `equals` from `absent`, previously uncovered.""" + """Pins the branch separating `equals` from `absent`.""" sandbox, sandbox_dir = sandbox_with_log _write_log(sandbox_dir, [_call(["ixp", "projects", "configure-model", "proj-1"])]) criterion = CliCalledCriterion( diff --git a/tests/test_codex_agent.py b/tests/test_codex_agent.py index 5f9654f05..8f35de229 100644 --- a/tests/test_codex_agent.py +++ b/tests/test_codex_agent.py @@ -314,7 +314,7 @@ def test_wire_api_fixed_to_responses(self, monkeypatch): """wire_api is always 'responses' — the pinned codex binary dropped 'chat' support, so it's a fixed constant, not an operator knob.""" monkeypatch.setenv("CODEX_BASE_URL", "https://my-res.openai.azure.com/openai") - monkeypatch.setenv("CODEX_WIRE_API", "chat") # ignored — no longer a knob + monkeypatch.setenv("CODEX_WIRE_API", "chat") # ignored — not a knob agent = CodexAgent(parse_agent_config(type=AgentKind.CODEX, model="dep")) provider = agent._build_thread_options()["config"]["model_providers"]["custom"] assert provider["wire_api"] == "responses" @@ -1523,9 +1523,8 @@ def __exit__(self, *exc): class TestCommunicatePostWatchdogTimeoutRace: """Regression for the post-watchdog timeout race: when the watchdog fires but the pump completes before the cancel lands, the trailing `if timeout_hit` - block must set _state=ERROR (consistent with every other timeout/crash path). - Previously this path left _state unchanged — a latent inconsistency now fixed - by routing it through the shared _finalize_and_raise_timeout kernel.""" + block must set _state=ERROR (consistent with every other timeout/crash path), + via the shared _finalize_and_raise_timeout kernel.""" async def test_post_watchdog_timeout_sets_error_state_and_partial(self, monkeypatch): notifications = [_delta("done"), _turn_completed()] @@ -2411,8 +2410,7 @@ class TestFlushMessageWindowBounds: The end-to-end cases above all describe a stream whose stamps advance, so they cannot reach the awkward case the reducer still hands `close_window`: the emission's own first stamp (`item_start`), whose `min()` against the - mark is the backwards-clock defence. The tool-span arguments this class - also used to cover are gone — the subtraction moved to + mark is the backwards-clock defence. Tool-span subtraction belongs to `timing.subtract_tool_time`, and `tests/test_event_collector.py::TestSubtractToolTime` pins it there. """ @@ -2470,8 +2468,7 @@ def test_a_mark_later_than_the_first_item_does_not_invert_the_window(self): def test_the_published_window_is_raw_and_ignores_a_call_still_open(self): """The reducer publishes the RAW span; the collector subtracts. - It used to bound a still-open call at the window's end and take that - slice out here. `timing.subtract_tool_time` sees every span at + `timing.subtract_tool_time` sees every span at once, so a call is subtracted from the windows its REAL interval overlaps once it resolves — no boundary approximation, and nothing for this reducer to remember. A call that never resolves has no @@ -2499,10 +2496,9 @@ def test_a_call_opening_after_the_window_closes_is_ignored(self): class TestFlushMessageGenTimeSplit: """`gen_ms` is apportioned across sub-messages by their output share. - It used to land entirely on the FIRST spec, so a thinking+action - generation reported the thinking row as the whole generation and the - action row as instant — 98.5% of Codex generation time booked to - thinking. Billing tokens still travel with the first spec only; time is a + Booking it all on the FIRST spec would report a thinking+action + generation's thinking row as the whole generation and its action row as + instant. Billing tokens still travel with the first spec only; time is a property of the content, not of the call. """ @@ -2562,8 +2558,6 @@ def test_no_output_anywhere_splits_evenly(self): def test_billing_tokens_stay_on_the_first_sub_message_only(self): # Time is split; input/cache are per-CALL figures and must not be. - # The comment this phase edited previously claimed the two travelled - # together, so assert them apart explicitly. msgs = self._flush(gen_ms=1000, think_out=800, action_out=200) assert len(msgs) == 2 assert msgs[0].input_tokens == 500 - 200 # fresh slice diff --git a/tests/test_codex_agent_unit.py b/tests/test_codex_agent_unit.py index ee480f638..3d144c655 100644 --- a/tests/test_codex_agent_unit.py +++ b/tests/test_codex_agent_unit.py @@ -134,19 +134,13 @@ def test_command_output_recorded_whole_not_truncated(self): assert cmd.result_tokens > 100 -# --------------------------------------------------------------------------- -# Execution bounds -# -# The SDK delivers `started_at_ms` / `completed_at_ms` on the item notification -# and the agent discarded both, publishing the item's own `duration_ms` -# instead — 0.0 for 70 of 211 commands in one nightly, absent for 25 more, and -# no execution bounds at all, so no Codex tool call could be placed on a -# timeline. All three telemetry builders now derive their timing identically. -# -# Pure logic: the builders take plain SimpleNamespace roots, so these live -# here rather than behind test_codex_agent.py's importorskip — otherwise a -# clean `make test` (which syncs no codex extra) skips them entirely. -# --------------------------------------------------------------------------- +# Execution bounds. The SDK delivers `started_at_ms` / `completed_at_ms` on the item +# notification and the agent discarded both, publishing the item's own `duration_ms` +# instead — 0.0 for 70 of 211 commands in one nightly, absent for 25 more. All three +# telemetry builders now derive their timing identically. +# Pure logic: the builders take plain SimpleNamespace roots, so these live here rather +# than behind test_codex_agent.py's importorskip — otherwise a clean `make test` +# (which syncs no codex extra) skips them entirely. _EPOCH_MS = 1_800_000_000_000 @@ -225,8 +219,8 @@ def test_both_stamps_give_bounds_and_an_exact_duration(self, factory, root_type) @pytest.mark.parametrize(("factory", "root_type"), _BUILDERS) def test_timestamp_is_the_tools_own_start(self, factory, root_type): - # It used to be datetime.now() at COMPLETION, which places the call - # after its own execution. + # datetime.now() at COMPLETION would place the call after its own + # execution. tel = self._build(factory(), root_type, started_ms=_EPOCH_MS, completed_ms=_EPOCH_MS + 250) assert tel.timestamp == datetime.fromtimestamp(_EPOCH_MS / 1000) diff --git a/tests/test_command_executed.py b/tests/test_command_executed.py index 9c6a37113..6462390a1 100644 --- a/tests/test_command_executed.py +++ b/tests/test_command_executed.py @@ -124,11 +124,10 @@ def test_no_turn_records(self): def test_empty_commands(self): """Turns exist but have no commands ⇒ score by the same ``min_count`` math. - Used to short-circuit on a separate ``"No commands found"`` branch, but - that branch returned ``0.0`` even when ``min_count=0`` (the negative- - assertion pattern), which was wrong. Now the empty case falls through - to the normal scoring math: with ``min_count=1`` and zero matches, the - score is ``0/1 = 0.0`` and the details mirror the positive shape. + The empty case has no separate branch, so ``min_count=0`` (the + negative-assertion pattern) still passes. With ``min_count=1`` and zero + matches, the score is ``0/1 = 0.0`` and the details mirror the positive + shape. """ sandbox = MockSandbox() turn_records = [_make_turn(commands=[])] @@ -842,10 +841,9 @@ def test_argv_joined_short_command(self): def test_every_wrapper_form_is_unwrapped(self): """One case per shell/flag shape the agents emit — the allowlist can't rot. - The predicate replaced an enumerated allowlist that omitted ``zsh`` - (Codex's shell on macOS, codex_agent.py) and ``-ic`` while listing the - exotic ``-lic``; on those hosts the normalization silently reverted to - the pre-fix false-negative behaviour. Each entry must strip the wrapper. + Covers ``zsh`` (Codex's shell on macOS, codex_agent.py) and ``-ic``; a + shape the predicate misses silently turns normalization into a false + negative. Each entry must strip the wrapper. """ cases = { # zsh — Codex's default login shell on macOS @@ -1077,11 +1075,10 @@ def test_bash_record_without_command_is_not_shell_normalized(self): def test_normalized_haystack_shares_the_raw_truncation_window(self): """Both haystacks describe the same <=2000-char window (no past-cap leak). - Previously the raw haystack was truncated at 2000 chars but normalization - ran over the FULL command, so quote-stripping could slide content from - past the cap into the normalized haystack — a task relying on the - 2000-char bound changed verdict. Normalization now runs over the - already-truncated window. + Normalization runs over the already-truncated window. Normalizing the + FULL command would let quote-stripping slide content from past the cap + into the normalized haystack and change the verdict of a task relying on + the 2000-char bound. """ cmd = "bash -lc " + ("word " * 600) + "TARGET" # TARGET sits well past 2000 chars haystacks = _match_haystacks(cmd, is_shell=True) diff --git a/tests/test_command_statistics.py b/tests/test_command_statistics.py index bab27b601..eed953b05 100644 --- a/tests/test_command_statistics.py +++ b/tests/test_command_statistics.py @@ -50,9 +50,9 @@ def test_nothing_timed_reports_no_average_rather_than_zero(self): def test_an_untimed_command_is_not_ranked_among_the_slowest(self): """The slowest list is built from timed commands only. - It used to coalesce a missing duration to 0.0, which put an untimed - command in a "slowest" ranking at the bottom — a measurement it never - had. The pre-filter and the reported value are now the same fact. + Coalescing a missing duration to 0.0 would rank an untimed command in + the "slowest" list with a measurement it never had. The pre-filter and + the reported value are the same fact. """ now = datetime.now() commands = [ diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index 4c1429104..be986fa87 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -631,36 +631,6 @@ def test_flags_oversized_sync_dialog_loop(self): assert len(self._run(src)) == 1 -@pytest.mark.lint -class TestCE023NoProxyShimImports: - """CE023 flags imports of the deprecated coder_eval.proxy.* shim outside it.""" - - @staticmethod - def _run(src: str, *, path: str = "src/coder_eval/agents/antigravity_agent.py"): - import ast - - from tests.lint.rules.ce023_no_proxy_shim_import import NoProxyShimImports - - return NoProxyShimImports(path).check(ast.parse(src)) - - def test_flags_from_proxy_pricing_import(self): - assert self._run("from coder_eval.proxy.pricing import calculate_cost") - - def test_flags_bare_proxy_import(self): - assert self._run("import coder_eval.proxy.pricing") - - def test_allows_canonical_pricing_import(self): - assert not self._run("from coder_eval.pricing import calculate_cost") - - def test_does_not_match_lookalike_module(self): - # `coder_eval.proxything` is a different package, not the proxy shim. - assert not self._run("from coder_eval.proxything import x") - - def test_skips_shim_package_itself(self): - src = "from coder_eval.proxy.pricing import calculate_cost" - assert not self._run(src, path="src/coder_eval/proxy/__init__.py") - - @pytest.mark.lint class TestCE024DiscriminatedUnions: """CE024 flags bare module-level unions of same-file `type: Literal`-tagged models.""" @@ -975,10 +945,9 @@ def test_src_scan_requires_a_real_consumer_not_any_literal(self, tmp_path: Path) class TestCE029DocYamlExamples: """CE029 — self-contained YAML examples in the docs must validate. - A published snippet that raises when copy-pasted reads as a broken feature. - The motivating bug: the `prompt_mutations` recipe used `text:` where the - field is `content:`, and every mutation model sets `extra="forbid"`. Scans - real Markdown, so it lives here rather than in the AST-only runner. + Scans real Markdown, so it lives here rather than in the AST-only runner. + + Rationale: .claude/notes/lint-rules.md § CE029 """ REPO_ROOT = Path(__file__).parent.parent @@ -1401,18 +1370,11 @@ def test_drift_is_detected(self, tmp_path: Path): RUBRIC_READERS = {"task", "lint-tasks", "init"} # Whether each skill must locate a repository's eval tree before it can do anything. -# All six currently must, and each for its own reason: `analyze` needs the run store, -# `init` and `check-skill` must know where tasks already live before writing beside -# them, `lint-tasks` and `task` glob the task tree, and `ci` writes the resolved glob -# into the workflow it emits. Every one of them used to carry its own hardcoded guess -# (`runs/latest`, `tasks/`), which is wrong in any repository that names the tree -# something else or nests it — so the policy is declared once in -# reference/repo-layout.md and a reader that stops pointing at it has forked it. -# -# A mapping rather than a set, mirroring SKILL_DISABLE_MODEL_INVOCATION: a SEVENTH skill -# then has to state whether it needs discovery instead of silently defaulting to "no" -# and quietly reintroducing a hardcoded path. `False` is a legitimate answer — a skill -# that touches no task or run tree — but it has to be written down. +# A hardcoded guess (`runs/latest`, `tasks/`) is wrong in any repository that names or +# nests the tree differently, so the policy lives once in reference/repo-layout.md. A +# mapping rather than a set, mirroring SKILL_DISABLE_MODEL_INVOCATION: a new skill has +# to state whether it needs discovery instead of defaulting to "no". `False` is a +# legitimate answer, but it has to be written down. SKILL_NEEDS_EVAL_ROOT_DISCOVERY = { "analyze": True, "ci": True, @@ -1448,15 +1410,13 @@ def test_drift_is_detected(self, tmp_path: Path): # undocumented. Adding a surface is one edit here. SKILL_DOC_SURFACES = ("plugins/coder-eval/README.md", "docs/PLUGIN.md", "README.md", "CLAUDE.md") -# Claude Code loads a listing of every skill's name and description into context. -# The listing's character budget scales at ~1% of the model's context window and is -# SHARED with every other skill the user has installed; when it overflows, -# descriptions are dropped starting with the least-invoked skills. So a plugin that -# grows its descriptions without bound quietly evicts the user's own skills. This -# ceiling makes growth a reviewed decision: raising it is allowed, in a commit that -# says why — which is exactly what a silent drift would not be. Asserted on the SUM, -# not per skill: the longest single description is ~300 against a 1,536 per-entry -# truncation limit, so a per-skill cap would guard nothing. +# Claude Code loads a listing of every skill's name and description into context. The +# budget scales at ~1% of the model's context window and is SHARED with every other +# skill the user has installed; on overflow, descriptions are dropped starting with +# the least-invoked. A plugin that grows its descriptions without bound quietly evicts +# the user's own skills, so this ceiling makes growth a reviewed decision. Asserted on +# the SUM, not per skill: the longest single description is ~300 against a 1,536 +# per-entry truncation limit, so a per-skill cap would guard nothing. SKILL_LISTING_BUDGET_CHARS = 1_600 # Tokens that name THIS repository's files. An installed plugin is copied to @@ -1676,16 +1636,13 @@ def test_bundled_files_reference_no_repo_paths(self, skill: Path): ) def test_lint_tasks_skill_is_read_only(self): - # Assert BOTH keys, because neither alone carries the contract: `allowed-tools` names - # the tools this skill expects to use, `disallowed-tools` removes the write tools from - # the pool. Assert only the allowlist and a denylist regression passes; assert only the - # denylist and a widened allowlist (say `Bash`) passes. - # - # Neither key is the real guarantee, which is why the skill body carries a STANDING - # prohibition too: per the skills spec, `disallowed-tools` "clears when you send your - # next message", and this skill's step 1 deliberately asks the user one before linting a - # whole directory. So the frontmatter covers the first turn and the prose covers the - # rest — `test_lint_tasks_read_only_rule_survives_the_next_turn` guards that half. + # Assert BOTH keys: assert only the allowlist and a denylist regression passes; + # assert only the denylist and a widened allowlist (say `Bash`) passes. Neither + # is the real guarantee, which is why the skill body carries a STANDING + # prohibition too — per the skills spec `disallowed-tools` "clears when you send + # your next message", and this skill's step 1 asks the user one. The frontmatter + # covers the first turn, the prose the rest; + # `test_lint_tasks_read_only_rule_survives_the_next_turn` guards that half. meta = _skill_frontmatter(PLUGIN_ROOT / "skills" / "lint-tasks" / "SKILL.md") # `and allowed` first: an ABSENT allowed-tools is the weakest state, not the @@ -1774,16 +1731,13 @@ def test_skill_docs_surfaces_list_every_skill(self, skill: Path): assert not missing, f"{name} is not documented in {missing} — a shipped skill nobody can discover" def test_skill_docs_surfaces_state_the_right_count(self): - # The companion to the test above, which only checks that each NAME appears. These - # surfaces also state the count in prose, and adding the sixth skill meant hand-editing - # seven such sites across four files. Without this, a seventh ships with every count - # silently wrong — the exact drift that repair was. Derived from disk: no count is - # written down here. - # - # Three phrasings are in use and all three are covered: " skills" / " slash - # commands" (both READMEs, docs/PLUGIN.md), "x " (CLAUDE.md's `SKILL.md` x 6), - # and "The other " (the model-invokable subset, which is the skill count minus - # the explicit-invocation-only ones). + # The companion to the test above, which only checks that each NAME appears. + # These surfaces also state the count in prose. Derived from disk: no count is + # written down here. Three phrasings are in use and all three are covered: + # " skills" / " slash commands" (both READMEs, docs/PLUGIN.md), + # "x " (CLAUDE.md's `SKILL.md` x 6), and "The other " (the + # model-invokable subset, the skill count minus the explicit-invocation-only + # ones). words = {2: "two", 3: "three", 4: "four", 5: "five", 6: "six", 7: "seven", 8: "eight"} count = len(PLUGIN_SKILLS) assert count in words, f"{count} skills — extend `words` to cover the new count" @@ -1838,16 +1792,12 @@ def test_analyze_routes_fixes_to_the_right_layer(self): ids=[str(p.relative_to(PLUGIN_ROOT)) for p in PLUGIN_TEXT_FILES if p.suffix == ".md"], ) def test_bundled_markdown_fences_balance(self, doc: Path): - # A skill body is an instruction document; an unbalanced fence silently swallows - # everything after it. `analyze` shipped a ```markdown block containing a ```diff - # block, and because a closing fence may not carry an info string, the inner - # opener closed the outer block early and the next bare ``` opened one that never - # closed — burying 32 lines including the whole Principles section. Nothing caught - # it, because it is still valid YAML frontmatter and valid-ish Markdown. - # - # CommonMark rule applied here: a fence closes only on a run of backticks at least - # as long as the opener AND carrying no info string. Nesting therefore requires the - # OUTER fence to be longer (````markdown wrapping ```diff). + # An unbalanced fence silently swallows everything after it. `analyze` shipped a + # ```markdown block containing a ```diff block; a closing fence may not carry an + # info string, so the inner opener closed the outer block early and buried 32 + # lines. CommonMark: a fence closes only on a backtick run at least as long as the + # opener AND with no info string, so nesting needs a longer OUTER fence + # (````markdown wrapping ```diff). open_len = 0 for n, raw in enumerate(doc.read_text(encoding="utf-8").splitlines(), 1): line = raw.strip() @@ -2014,9 +1964,9 @@ def test_cli_setup_declares_pin_resolution(self): ) def test_cli_setup_conditions_the_upgrade_suggestion_on_a_pin(self): - # "Version skew" used to terminate in "suggest upgrading", full stop. Against a - # pinned repository that is the single most destructive thing these skills could - # recommend, so the upgrade advice must now sit BEHIND the pin question. + # An unconditional "suggest upgrading" is, against a pinned repository, the single + # most destructive thing these skills could recommend, so the upgrade advice in + # "Version skew" must sit BEHIND the pin question. section = ( (PLUGIN_ROOT / "reference" / "cli-setup.md").read_text(encoding="utf-8").partition("## Version skew")[2] ) @@ -2699,12 +2649,10 @@ def test_registered_fields_are_actually_consumed_on_the_real_tree(self): class TestCE026ActionDocSurfaces: """CE026 — the Action's onboarding surfaces must be truthful and self-sufficient. - The motivating bug: docs/CI_GATE.md said "there is nothing to install" above a - copy-pasteable `uses:` step with no agent runtime, while the correcting - prerequisite note sat 11 lines below and the tutorial's sibling snippet *did* - show the steps. An integrator who copied it got a run that dies on a missing - `claude` binary. Reasons over Markdown + YAML, so it lives here rather than in - the AST-only runner (precedent: CE027-CE031). + Reasons over Markdown + YAML, so it lives here rather than in the AST-only runner + (precedent: CE027-CE031). + + Rationale: .claude/notes/lint-rules.md § CE026 """ REPO_ROOT = Path(__file__).parent.parent @@ -2979,30 +2927,16 @@ def test_scoped_to_the_criteria_package(self): class TestCE034ArmedPositiveRequiresSuccess: """CE034 — an armed, live-passable `command_executed` must require success. - `require_success` defaults to False, so a criterion counts an invocation that - CRASHED. On an unarmed criterion that is merely generous. On an armed one it - corrupts the run's verdict, because three behaviours compose: - - 1. `live_verdict` and `_check_impl` share `_matching_commands`, so a failed - invocation live-PASSES a positive criterion (`min_count > 0`, no - `max_count`) the moment it is observed; - 2. `stop_early.on_pass: stop` ends the run on that pass — and - `decide_within` latches it, so the timeout never fires either; - 3. gating is FIRED-ONLY: a run the watcher cut gates on the ARMED SUBSET - (`armed_criteria_passed`), so unarmed criteria are never consulted. - - Net effect on `tasks/early_stop_weighted_low_weight_absorbed.yaml` before this - rule existed: an agent that ran `python app.py` BEFORE creating app.py scored a - weighted 1.0 over the armed subset and reported SUCCESS — with no app.py and a - crashed script — because the unarmed `file_exists` was bypassed. Found by - running the plugin's own `lint-tasks` skill against this repository's tasks. - - Only *pass-capable* instances are constrained, read off the model's own - `live_decidable_polarities()` rather than re-deriving the shape here. A - negative assertion (`min_count: 0, max_count: 0`, i.e. "must NOT call curl") - is fail-only and must NOT set `require_success`: a curl that failed is still a - curl that was called, and requiring success there would blind the criterion to - exactly the calls it exists to forbid. + `require_success` defaults to False, so a crashed invocation counts. On an armed + positive criterion (`min_count > 0`, no `max_count`) that call live-PASSES, + `stop_early.on_pass: stop` ends the run, and FIRED-ONLY gating consults only the + armed subset — so the run reports SUCCESS past every unarmed criterion. + + Scope: pass-capable instances only, read off the model's own + `live_decidable_polarities()`. A fail-only negative (`min_count: 0, max_count: 0`) + must NOT set `require_success`: a forbidden call that failed is still a call. + + Rationale: .claude/notes/lint-rules.md § CE034 """ ROOT = Path(__file__).parent.parent @@ -3125,12 +3059,10 @@ def test_unarmed_positive_is_not_constrained(self): # `analyze` carries a `| Current runs | Older runs | Where |` table, because a run written # before the rename spells two of these differently. The third cell is load-bearing: only -# a TOP-LEVEL key is a model field with a `validation_alias`, so only those rows can be -# checked against the schema. `max_iterations` is a key inside the free-form `task_config` -# dict and appears in no `AliasChoices` at all — a guard that swept the whole table would -# be unsatisfiable against the very prose it guards, and would get "fixed" by deleting the -# row. Keeping the scoping visible in the shipped table rather than hidden in this file is -# the point: an author adding a row has to say which kind of key it is. +# a TOP-LEVEL key is a model field with a `validation_alias`. `max_iterations` is a key in +# the free-form `task_config` dict and in no `AliasChoices`, so a guard over the whole table +# would be unsatisfiable and get "fixed" by deleting the row. The shipped table keeps the +# scoping visible: an author adding a row has to say which kind of key it is. _TOP_LEVEL_CELL = "top-level record key" @@ -3166,21 +3098,16 @@ def _record_fields_referenced(block: str) -> set[str]: class TestRunRecordFieldVocabulary: """Every task.json field the run-analysis surfaces name must exist on the models. - `jq` returns `null` for a key that does not exist instead of failing, so a wrong - field name does not surface as an error — it produces a table of nulls that reads - like a run with nothing in it. Both surfaces shipped six such names at once - (`turns`, `total_tokens`, `assistant_turn_count`, `max_turns`, `criteria_count`, - `all_criteria_perfect`), and the failure is worst exactly where the instruction - applies: the >20-task path, where the agent is explicitly told NOT to fall back to - reading whole files. - - Scoped deliberately: only the fenced blocks that mention `success_criteria_results` - (the summary-extraction programs), and only the HEAD of each dotted path. Deeper - segments are not checked because `task_config` is a free-form dict, so - `.task_config.resolved.run_limits.max_turns` is unverifiable from the schema. The - allowed set unions the run-level and criterion-level models rather than tracking - which scope each expression sits in — a weakening that still catches every name - above, since none of them exists on either model. + `jq` yields `null` for a missing key instead of failing, so a wrong field name + ships as a table of nulls, not an error. + + Scope: only fenced blocks that mention `success_criteria_results`, and only the + HEAD of each dotted path — `task_config` is a free-form dict, so deeper segments + are unverifiable. The allowed set unions the run-level and criterion-level models + instead of tracking each expression's scope, so a name valid on only one of those + models passes in any scope. + + Rationale: .claude/notes/lint-rules.md § TestRunRecordFieldVocabulary """ @staticmethod @@ -3258,11 +3185,11 @@ def test_legacy_keys_named_by_analyze_are_real_model_aliases(self): ) def test_analyze_does_not_deny_the_legacy_key_absolutely(self): - # The skill used to say flatly "There is no top-level `turns`", which is true of - # current runs and false of anything written before the rename — so an agent - # reading a real older run was told its correct extraction was wrong. The four - # OTHER names in that sentence were never top-level in any generation and were - # denied on purpose; rewriting the sentence must not take them with it. + # A flat "There is no top-level `turns`" is true of current runs and false of + # anything written before the rename — it tells an agent reading a real older run + # that its correct extraction is wrong. The four OTHER names in that sentence were + # never top-level in any generation and are denied on purpose; rewriting the + # sentence must not take them with it. text = " ".join((PLUGIN_ROOT / "skills" / "analyze" / "SKILL.md").read_text(encoding="utf-8").split()) assert "There is no top-level `turns`" not in text, ( "analyze denies the legacy `turns` key absolutely again — it is what runs " @@ -3338,14 +3265,9 @@ class TestCE035WorkflowOutputParity: """CE035 — a `steps..outputs.` / `needs..outputs.` reference must resolve to a key its writer actually produces. - The motivating bug: `verify-published-action.yml` read - `steps.parity.outputs.version` twice, but that step writes `pin`/`newest`/`lagging` - (the shell *variable* was `VERSION`, the output *key* was `newest`). GitHub expands an - unwritten output to '', so `TAG_REF: v${{ … }}` became the bare `v`, `git show - "v:action.yml"` exited 128 under `set -euo pipefail`, and the preflight job was red on - 100% of triggers — taking the paid e2e tier (`needs: preflight`) with it. Invisible to - ruff/pyright/pytest, and actionlint models `steps.*.outputs` as an open string map. Reasons over workflow YAML + embedded shell, so it lives here, not in the AST runner. + + Rationale: .claude/notes/lint-rules.md § CE035 """ REPO_ROOT = Path(__file__).parent.parent @@ -3494,10 +3416,10 @@ def test_catches_a_needs_reference_to_a_nonexistent_job(self, tmp_path: Path): assert "['promote', 'release']" in findings[0].message def test_a_dynamic_printf_writer_is_unreadable_not_a_bogus_key(self, tmp_path: Path): - """Regression: the writer scan used to capture the conversion letter out of a - format string (`printf "%s=%s\\n"` -> the key `s`). That non-empty-but-wrong set - defeats the "no readable key => skip" contract and false-FAILS a correct workflow, - which is the one direction the docstring promises the rule can never take.""" + """The writer scan reads no key from a format string, never its conversion letter. + + `printf "%s=%s\\n"` must not yield the key `s`: a wrong non-empty set defeats "no readable key => skip" + and false-FAILS a correct workflow, the one direction the docstring promises the rule never takes.""" from tests.lint.workflow_outputs import _written_keys, find_unresolved_output_refs assert _written_keys({"run": 'printf "%s=%s\\n" "$K" "$V" >> "$GITHUB_OUTPUT"'}) is None @@ -3540,14 +3462,11 @@ def test_finding_reports_the_line_of_the_offending_reference(self, tmp_path: Pat @pytest.mark.lint class TestCE036LiveVerdictContract: """CE036 — every live-observable criterion's `live_verdict` must be deterministic - and monotonic (GitHub issue #61 item 2). + and monotonic. `EarlyStopWatcher` latches verdicts, defers the fail-stop, and attributes pass-stop flips against the previous round — all correct only while `live_verdict` never - contradicts an earlier decision and never varies for identical input. That contract - was documented on `LiveVerdict`/`BaseCriterion.live_verdict` but unenforced: a third - criterion implementing it non-monotonically would type-check, pass CE025, and - silently corrupt the stop logic. + contradicts an earlier decision and never varies for identical input. Monotonicity over arbitrary Python is undecidable, so there is no sound static rule to write. This replays each criterion against every prefix of a recorded trajectory @@ -3557,6 +3476,8 @@ class TestCE036LiveVerdictContract: Honest limit (documented on the helper module too): this proves the contract on the trajectories an author supplied, not in general. + + Rationale: .claude/notes/lint-rules.md § CE036 """ def test_real_criteria_honor_the_contract(self): @@ -3682,7 +3603,7 @@ def raises_at_the_end(records): assert "RAISED" in violations[0] and "prefix length 2" in violations[0], violations def test_detects_a_fixture_that_stopped_exercising_its_decision_path(self): - """Fixture rot: the case claims a decision the trajectory no longer reaches.""" + """Fixture rot: the case claims a decision the trajectory does not reach.""" from tests.lint.live_verdict_contract import contract_violations checker = self._checker(lambda _records: "undecided") @@ -3808,19 +3729,13 @@ def sorted_recency_verdict(records): class TestCE044PluginManifestParity: """CE044 — the marketplace entry and the plugin manifest it points at are one surface. - Eight fields are byte-identical duplicates across the two manifests and nothing - compared them: the only test that read ``plugin.json`` at all was - ``test_action_version_pin.py``, and only its ``version``. A one-sided edit ships - two different one-liners — one in the ``/plugin`` browser, one in the installed copy. - - The second half is the motivating defect: the marketplace schema allows both - ``keywords`` and a near-synonymous ``tags``, while the plugin-manifest schema has no - ``tags`` property at all, so discovery strings parked there are dropped from an - installed user's manifest and a future editor has no rule for where a new term goes. - An extra key on the entry now fails unless ``MARKETPLACE_ONLY`` records why. + Every field the two manifests share must be identical, and an extra key on the + marketplace entry fails unless ``MARKETPLACE_ONLY`` records why. Reasons over JSON files and a ``source`` path, so it is wired here rather than as a ``BaseRule`` in the AST runner. + + Rationale: .claude/notes/lint-rules.md § CE044 """ REPO_ROOT = Path(__file__).parent.parent @@ -3884,45 +3799,27 @@ def _write_pair(root: Path, entry_extra: dict | None = None) -> None: class TestCE045PluginPathIsAPluginRoot: """CE045 — a claude-code local plugin path must name a plugin ROOT, not a skills dir. - `agent.plugins: [{type: local, path: X}]` reaches the Claude Code SDK as a plugin - directory, so a skill is found at `X/skills//SKILL.md`. Point X one level - deeper — at the directory that holds the skill directories — and NOTHING loads. - Probed against the real CLI, from a cwd that is not the skill's own repo (project - discovery would otherwise find it regardless of `--plugin-dir`, and the namespace - prefix is the real signal): - - claude --plugin-dir /skills -> nothing - claude --plugin-dir -> `root:probe-beta` - - The cost is invisible and total: every activation suite the plugin generated - reported recall 0.0, which the bundled template's own comment calls "reads exactly - like a broken skill", and `ci` wrote the same path into users' SCHEDULED workflows, - where it renders as a permanent red indistinguishable from the drift the schedule - exists to detect. - - INCIDENT RECORD — the corpus below is that record, not this prose. Six wrong-value - lines across five files shipped at once: docs/PLUGIN.md, tutorial 07, - activation.yaml (comment and example), check-skill, and ci. Nothing held them in - agreement, which is why they drifted together. - - The unit under test is the VALUE, not the sentence around it: a path whose last - segment is `skills` cannot be a plugin root, whatever the prose claims. - - SCOPE. The rule keys on `SKILL_SOURCE_PATH`, the variable the plugin emits. That is - a limit, NOT a statement that other variables may use the deeper form — `$PLUGIN_PATH` - feeds `experiments/plugin-comparison.yaml`, whose default agent is claude-code, and - is unlinted. The guard that reaches every user, including the repos where - `/coder-eval:check-skill` actually writes suites, is the runtime warning in - `utils.process_plugins`; this rule only keeps THIS repo's shipped strings honest. + `agent.plugins: [{type: local, path: X}]` reaches the SDK as a plugin directory, so + a skill resolves at `X/skills//SKILL.md`; one level deeper loads nothing and + every activation suite reports recall 0.0. The unit under test is the VALUE: a path + whose last segment is `skills` cannot be a plugin root. `KNOWN_BAD_LINES` is the + incident record. + + SCOPE: `SKILL_SOURCE_PATH` assignments, plus literal local `path:` values in + `tasks/` and `experiments/`. That is a limit, not a license: `$PLUGIN_PATH` (feeds + `experiments/plugin-comparison.yaml`) is unlinted. The guard that reaches users is + the runtime warning in `utils.process_plugins`; this rule keeps only this repo's + shipped strings honest. + + Rationale: .claude/notes/lint-rules.md § CE045 """ REPO_ROOT = Path(__file__).parent.parent - # Verbatim pre-fix lines, one per surface that shipped the wrong value. The mutation - # guard replays these through the FULL extract-then-predicate pipeline. An earlier - # revision asserted the predicate against hand-written strings the matcher could - # never produce, which is how the Actions form below stayed unreachable while the - # rule looked covered. + # Verbatim lines that shipped the wrong value, one per surface. The mutation guard + # replays these through the FULL extract-then-predicate pipeline: a predicate checked + # against hand-written strings the matcher can never produce leaves a form (like the + # Actions one below) unreachable while the rule looks covered. KNOWN_BAD_LINES = ( 'export SKILL_SOURCE_PATH="$(pwd)/.claude/skills"', "# export SKILL_SOURCE_PATH=/abs/path/to/.claude/skills", @@ -4463,30 +4360,16 @@ def test_the_rule_is_now_exemption_free(self): class TestRuffExternalCoversEveryRule: """Every CE rule's documented `# noqa` must be accepted by ruff. - `[tool.ruff.lint] external` is what stops ruff reporting RUF102 "Invalid - rule code" for a suppression it does not own. It was hand-maintained and had - fallen ~14 ids behind — including CE054 and CE048, whose own docstrings - advertise `# noqa: CE054` / `# noqa: CE048` as the supported escape hatch. So - the first person to use the documented exemption got a red `make check` - instead, for doing exactly what the rule told them to. - - The list then drifted a SECOND time, and this class is why it drifted - quietly: it read `ALL_RULES` alone, so it could not see a rule that is a - `@pytest.mark.lint` class here rather than a `BaseRule`. CE044 and CE065 are - both such rules, both were missing, and only CE065 was noticed — by a human - reading a diff. `_known()` now unions both registries. - - Both directions are asserted. A declared id for a deleted rule is the - exemption-set rot that the generated pricing table exists to remove. + `[tool.ruff.lint] external` is what stops ruff reporting RUF102 "Invalid rule + code" for a suppression it does not own. `_known()` unions both registries — + `ALL_RULES` and the `@pytest.mark.lint` classes here — and both directions are + asserted: a missing id, and a declared id for a rule that no longer exists. Blind spot: the `@pytest.mark.lint` half of `_known()` discovers ids by the - `class TestCE\\d{3}` naming convention, which every such class follows today - but nothing enforces. A class named otherwise is invisible here, and its id - can go undeclared exactly as CE044 did. + `class TestCE\\d{3}` naming convention, which every such class follows + today but nothing enforces. A class named otherwise is invisible here. - Nothing is red today for want of these two entries — no `# noqa: CE044` or - `# noqa: CE065` exists in the tree — so this is pre-emptive rather than the - fix for a broken build. + Rationale: .claude/notes/lint-rules.md § TestRuffExternalCoversEveryRule """ @staticmethod @@ -4520,11 +4403,10 @@ def test_every_listed_id_is_well_formed(self): class TestCE056NoContainerEnvLiteral: """CE056 flags a bare `CODER_EVAL_IN_CONTAINER` outside container_paths. - The motivating miss: every READER of the gate was migrated to - `IN_CONTAINER_ENV` and the single WRITER (`docker_runner`'s - `--env CODER_EVAL_IN_CONTAINER=1`) was not, so a rename would have disarmed - four gates at once, all silently. CE052 cannot see it -- that rule inspects - `if` guards, and the writer is not one. + Readers and writers alike must use `IN_CONTAINER_ENV`. CE052 does not overlap: + that rule inspects `if` guards, and a writer is not one. + + Rationale: .claude/notes/lint-rules.md § CE056 """ @staticmethod @@ -4704,31 +4586,16 @@ def test_extractors_narrow_to_the_marketing_region(self, tmp_path: Path): class TestCE055NoAbsoluteCriterionPath: """CE055 — a criterion `path:` in `tasks/` must be sandbox-relative. - Criterion paths are joined onto the sandbox root, and joining an ABSOLUTE - path discards that root: `Path(sandbox) / "/opt/marker"` is `/opt/marker`. - Containment then refuses it, so the criterion can never match no matter what - the agent does. - - Two in-tree tasks were broken this way, and the failure mode is why a static - rule earns its place on top of the runtime guard: - - * `tasks/byod_smoke_test.yaml` checked `/opt/byod_marker`. It IS in a CI - bucket, and CI reported `Results: 7/8 succeeded` with a gating 0.0 reading - "file does not exist" for a file that plainly existed. The real cause sat - in a warning inside a task log. - * `tasks/dockerfile_build_example/dockerfile_build_example.yaml` checked - `/opt/greeting.txt` and `/opt/secret_check.txt`. It is in NO bucket, so - nothing ran it at all — the runtime guard, however loud, is never reached. - - That second case is the argument: a runtime error only fires for tasks - somebody runs, and this repo ships example tasks that CI does not. This rule - reads the YAML. - - The fix is never "make containment allow it". An absolute path here is a - claim about the container IMAGE rather than about anything the agent produced - in its workspace, and `run_command` (`test -f /opt/marker`) states that - directly — while staying inside the trust gate that governs recorded shell on - the detached grading path. + Joining an ABSOLUTE path onto the sandbox root discards the root + (`Path(sandbox) / "/opt/marker"` is `/opt/marker`), so containment refuses it and + the criterion can never match. This rule reads the YAML, so it also covers tasks + no CI bucket runs, where the runtime guard is never reached. + + Never fix a violation by relaxing containment. An absolute path is a claim about + the container IMAGE: use `run_command` (`test -f /opt/marker`), which stays inside + the trust gate for recorded shell on the detached grading path. + + Rationale: .claude/notes/lint-rules.md § CE055 """ ROOT = Path(__file__).parent.parent diff --git a/tests/test_detached_grading_boundaries.py b/tests/test_detached_grading_boundaries.py index 8f8b327eb..8f0608331 100644 --- a/tests/test_detached_grading_boundaries.py +++ b/tests/test_detached_grading_boundaries.py @@ -226,17 +226,13 @@ def _run_with_context(tmp_path: Path, grade: object = True, **extra: object): ) def test_invoking_the_command_here_arms_no_process_lethal_watchdog(self, tmp_path: Path) -> None: - """The command's heartbeat watchdog reaps an ORPHANED CONTAINER by calling - `os._exit(137)` on itself. This suite invokes the command in-process, so an - unconditionally-armed thread exits the pytest WORKER instead — 40s later - (20s grace + 20s stale), inside whatever unrelated test that worker has - moved on to. It shipped that way: it killed a different test on each run - and on each platform, with no traceback, and the dead worker's lost - coverage data then failed the gate as `65.13 < 80.00`. - - Asserted on the live thread list rather than by patching `threading`, so - the guard is proven at the only place that matters — whether a thread now - exists in this process. + """Pins: invoking the command outside a container starts no daemon thread. + + The heartbeat watchdog calls `os._exit(137)`; armed in-process, it exits + the pytest WORKER 40s later, inside an unrelated test. Assert on the live + thread list, not by patching `threading`. + + Rationale: .claude/notes/isolation.md § The heartbeat watchdog is armed only inside a container """ import threading @@ -285,9 +281,8 @@ def test_an_unreadable_prior_degrades_to_a_message_not_a_traceback(self, tmp_pat # The in-container default is asserted BEHAVIOURALLY by # `TestGradePlumbedIntoTheContainerOrchestrator::test_an_absent_key_still_grades`. - # It used to be a `assert 'context.get("grade", True)' in source` grep, which - # is the same static check that already failed here once: it passes happily - # while the line it describes is never executed. + # A `assert 'context.get("grade", True)' in source` grep is no substitute: it + # passes while the line it describes is never executed. class TestInContainerRegradeBranch: @@ -882,13 +877,12 @@ def _sandbox(tmp_path: Path): def test_an_absolute_path_is_refused_as_a_config_error(self, tmp_path: Path) -> None: """Refused, not silently scored 0.0. - This used to return `[]`, which the checker reports as "file does not - exist" — a gating verdict about the AGENT for a file that plainly does - exist and that no agent behaviour could ever put inside the sandbox. - `tasks/byod_smoke_test.yaml` was broken exactly that way for several - commits (it checks `/opt/byod_marker`, baked into the BYOD image) and - the only signal was a warning in the task log. CE039 names this - distinction; `CheckerMisuseError` is its signal. + Returning `[]` would report "file does not exist" — a gating verdict + about the AGENT for a file that exists and that no agent behaviour could + put inside the sandbox. CE039 names this distinction; + `CheckerMisuseError` is its signal. + + Rationale: .claude/notes/isolation.md § Criterion paths are contained, quietly """ sandbox, _ = self._sandbox(tmp_path) outside = tmp_path / "outside.txt" diff --git a/tests/test_detached_grading_guards.py b/tests/test_detached_grading_guards.py index f3939774a..e07cfad3b 100644 --- a/tests/test_detached_grading_guards.py +++ b/tests/test_detached_grading_guards.py @@ -247,8 +247,7 @@ def test_a_restored_path_drops_entries_inside_the_graded_run(tmp_path: Path) -> run's own task.json — a shareable artifact. Every entry an attacker could have placed there must be dropped; only the run's real toolchain survives. - The run-directory SIBLING case is the one this test used to pin the wrong way - round: it asserted such an entry was kept. The workspace is only part of the + A run-directory SIBLING is dropped too: the workspace is only part of the run dir, and ``artifacts/`` and the run root travel in the same archive. """ run_dir = tmp_path / "run" @@ -329,19 +328,12 @@ def test_the_actual_cost_join_is_skipped_on_a_re_grade(tmp_path: Path) -> None: async def test_a_container_run_records_the_driver_it_was_authored_with(tmp_path: Path, monkeypatch) -> None: """`task_config.resolved` must describe the task as AUTHORED, not as rewritten. - `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before - building the in-container Orchestrator — the one legitimate rewrite, since we - are already inside the container the driver asked for. But the Orchestrator - then recorded the REWRITTEN copy, so a docker run's own `task.json` claimed - `driver: tempdir`. + Pins: an Orchestrator built with `recorded_task` records that task's + `driver: docker`, not the in-container `tempdir` rewrite it runs. A recorded + `tempdir` lets `evaluate ` skip the host-grading refusal and the + `graded_on_host` stamp. `recorded_task` is the seam, exercised without docker. - That fed straight into the gate that reads the driver back out of the record: - `evaluate ` on a container row skipped the host-grading refusal AND - the `graded_on_host` stamp, and graded a container task against the host - filesystem silently. Verified against a real docker run before the fix: a - `driver: docker` task re-graded on the host, unprompted and unstamped. - - `recorded_task` is the seam, exercised here without needing docker. + Rationale: .claude/notes/orchestration.md § Recording the task as authored """ from coder_eval.config import settings from coder_eval.models import ( diff --git a/tests/test_docker_build_failure.py b/tests/test_docker_build_failure.py index 32f28e0d7..f5e8464b5 100644 --- a/tests/test_docker_build_failure.py +++ b/tests/test_docker_build_failure.py @@ -1,8 +1,8 @@ """Build-failure observability: a failed `docker build` must not vanish. -A task image is built before run_dir / docker.log / task.json exist, so a build -failure used to leave an empty result directory with no status and no log. These -tests assert the fix: the build log is captured to docker.log and a synthetic +A task image is built before run_dir / docker.log / task.json exist. These tests +pin that a build failure still leaves a status and a log: the build log is captured +to docker.log and a synthetic ``BUILD_FAILED`` task.json is written, and the batch layer records BUILD_FAILED (not generic ERROR) at the run level. Hermetic — docker is never invoked. """ diff --git a/tests/test_docker_runner_mounts.py b/tests/test_docker_runner_mounts.py index dae7d3449..ff1662790 100644 --- a/tests/test_docker_runner_mounts.py +++ b/tests/test_docker_runner_mounts.py @@ -587,7 +587,7 @@ def test_docker_isolation_doc_names_only_real_denylist_entries(): Drift-guard for the doc finding: the prose may enumerate a subset, but every directory it names in the *drops* clause must be a real denylist member, and the - headline ``security/`` (the largest, previously-omitted drop) must be named — so + headline ``security/`` (the largest drop) must be named — so the user-facing contract can't silently diverge from the source-of-truth constant. """ doc = (Path(__file__).parent.parent / "docs" / "DOCKER_ISOLATION.md").read_text(encoding="utf-8") @@ -620,7 +620,7 @@ def test_copy_claude_home_tolerates_self_referential_symlink_loop(tmp_path: Path (host / "settings.json").write_text('{"ok": true}') dest = tmp_path / "copy" - _copy_claude_home(host, dest) # must not raise (pre-fix: "too many levels of symbolic links") + _copy_claude_home(host, dest) # must not raise ("too many levels of symbolic links") assert (dest / "settings.json").read_text() == '{"ok": true}' looped = dest / "plugins" / "marketplaces" / "uipath-marketplace" / "plugins" / "uipath" @@ -779,17 +779,12 @@ def test_dac_capabilities_are_dropped(self, tmp_path): def test_fowner_and_chown_are_deliberately_kept(self, tmp_path): """Dropping FOWNER would disable the harness's OWN chmod. - chmod(2) is gated on owner-or-CAP_FOWNER, and the in-container - orchestrator that applies the mode-000 window is the same root process - with the same capability set as the agent. On native Linux the bind - mount preserves the host uid that ran coder-eval, so with FOWNER dropped - `chmod 000 /work/references` fails with EPERM and the run completes - UNPROTECTED while still looking protected. Verified in a container: - root + uid-1000-owned dir + FOWNER dropped -> "Operation not permitted". - - So the drop only ever bites on the hosts where it also disables the - control. Closing the re-chmod hole needs a different uid, not a smaller - capability set -- see docs/DOCKER_ISOLATION.md. + Pins: `--cap-drop` names neither FOWNER nor CHOWN. The orchestrator that + applies the mode-000 window has the agent's capability set, so on native + Linux a dropped FOWNER makes `chmod 000 /work/references` fail with EPERM + and the run completes UNPROTECTED -- see docs/DOCKER_ISOLATION.md. + + Rationale: .claude/notes/isolation.md § Capability drops and the anti-cheat window """ argv = self._argv(self._make_runner(tmp_path, reference=None), tmp_path) diff --git a/tests/test_early_stop.py b/tests/test_early_stop.py index 791098bd3..ad75bc27d 100644 --- a/tests/test_early_stop.py +++ b/tests/test_early_stop.py @@ -94,13 +94,8 @@ from tests._fixtures.live_criteria import FROZEN_TS, make_command, make_turn -# --------------------------------------------------------------------------- # -# Helpers -# --------------------------------------------------------------------------- # - -# Telemetry/turn primitives are shared with the CE036 contract-replay fixtures -# (tests/lint/live_verdict_contract.py); the thin wrappers below keep this file's -# historical call shape (tool- ids, no sequence numbers) at every call site. +# Primitives shared with the CE036 contract-replay fixtures (tests/lint/live_verdict_contract.py); +# the wrappers below fix tool- ids and omit sequence numbers. _TS = FROZEN_TS @@ -153,7 +148,7 @@ class _DummyNoStopAgent: ``validate_early_stop`` only reads the flag off the registered class, so no ``Agent`` machinery is needed. Guardrail 1 must keep rejecting agents that - have not opted into the cooperative interrupt (all built-ins now support it). + have not opted into the cooperative interrupt. """ supports_cooperative_stop = False @@ -233,9 +228,6 @@ def _cmd_crit( ) -# --- Phase-3 helpers: results / info / events ------------------------------ # - - def _crit_result(ctype: str, score: float) -> CriterionResult: return CriterionResult(criterion_type=ctype, description=f"{ctype} result", score=score) @@ -307,11 +299,6 @@ def _unresolved_skill_end(skill: str, *, tool_id: str = "orphan-1") -> ToolEndEv return ToolEndEvent(task_id="t", tool=_skill_cmd(skill, tool_id=tool_id), status=ToolEndStatus.UNRESOLVED) -# --------------------------------------------------------------------------- # -# Config surface -# --------------------------------------------------------------------------- # - - class TestConfigSurface: def test_stop_early_defaults_none(self) -> None: # None = no run-level opinion: the criteria's own blocks decide arming. @@ -447,11 +434,6 @@ def test_block_arms_command_executed(self) -> None: assert _cmd_crit(max_steps_to_decide=5).is_stop_armed is True -# --------------------------------------------------------------------------- # -# skill_triggered live verdict -# --------------------------------------------------------------------------- # - - class TestEngagedSkillNames: def test_claude_skill_tool_namespaced(self) -> None: assert _engaged_skill_names(_cmd("Skill", {"skill": "plugin:date-teller"})) == {"date-teller"} @@ -535,11 +517,6 @@ def test_decidable_narrows_per_instance(self) -> None: assert _skill_crit("date-teller", "").live_decidable_polarities() == frozenset({"fail"}) -# --------------------------------------------------------------------------- # -# command_executed live verdict -# --------------------------------------------------------------------------- # - - class TestCommandExecutedLiveVerdict: checker = CommandExecutedChecker() @@ -618,11 +595,6 @@ def test_decidable_is_subset_of_type_universe(self) -> None: assert crit.live_decidable_polarities() <= frozenset({"pass", "fail"}) -# --------------------------------------------------------------------------- # -# Base default: unobservable criteria -# --------------------------------------------------------------------------- # - - class TestBaseLiveVerdictDefault: def test_unobservable_criterion_is_not_a_live_success_criterion(self) -> None: # file_exists is not observable mid-run: its model is plain @@ -644,11 +616,6 @@ def test_skill_triggered_decidable_is_subset_of_type_universe(self) -> None: assert crit.live_decidable_polarities() <= frozenset({"pass", "fail"}) -# --------------------------------------------------------------------------- # -# Resolution-time guardrails -# --------------------------------------------------------------------------- # - - class TestValidateEarlyStop: def test_unarmed_is_noop_even_with_bad_shape(self) -> None: # No blocks anywhere → validator never inspects anything. @@ -792,8 +759,7 @@ def test_guardrail5_simulation_rejected(self) -> None: validate_early_stop(task) def test_guardrail1_non_supporting_agent_rejected(self, dummy_no_stop_kind: str) -> None: - # Codex/antigravity now support the cooperative interrupt, so guardrail 1 - # is exercised with a dummy agent that leaves the flag at False. + # Guardrail 1 needs an agent kind that leaves supports_cooperative_stop False, so a dummy kind is registered. task = _task(criteria=[_skill_crit("s", "s", stop_on_pass=True)], agent_type=dummy_no_stop_kind) with pytest.raises(EarlyStopConfigError, match="cooperative stopping"): validate_early_stop(task) @@ -826,19 +792,14 @@ def test_guardrail1_armed_antigravity_accepts(self) -> None: validate_early_stop(task) # no raise def test_unarmed_task_is_plain_noop(self) -> None: - # No blocks -> no watcher, byte-for-byte default behavior. The old - # "at least one criterion" guard is gone with the master arm: there is - # nothing left to arm a task that has no blocks. + # No blocks -> no watcher, byte-for-byte default behavior. task = _task(criteria=[_skill_crit("s", "s")]) validate_early_stop(task) # no raise assert early_stop_active(task) is False def test_guardrail3_unobservable_criterion_unrepresentable(self) -> None: - # The stop_early block exists only on LiveSuccessCriterion, so an armed - # unobservable criterion cannot even be constructed (extra='forbid') — - # the old runtime "observable" guard is now a schema property. Match on - # the field name so the rejection is provably about stop_early, not - # some other typo'd kwarg. + # The block exists only on LiveSuccessCriterion (extra='forbid'); matching the field name proves + # the rejection is about stop_early, not another typo'd kwarg. with pytest.raises(ValueError, match="stop_early"): FileExistsCriterion( type="file_exists", @@ -898,13 +859,6 @@ def test_fanned_stacked_activation_accepts(self) -> None: validate_early_stop(task) # no raise -# --------------------------------------------------------------------------- # -# Guardrail integration: the plan and run resolution surfaces actually invoke -# validate_early_stop (not just the helper in isolation). Real task YAMLs go -# through the real load + 5-layer merge; a bad arming must surface as a clean -# CLI-level error on BOTH surfaces, never a silent no-op. -# --------------------------------------------------------------------------- # - _UNARMED_CRITERION = """\ - type: skill_triggered description: date-teller activation @@ -957,12 +911,8 @@ class TestGuardrailResolutionSurfaces: """A bad arming is rejected by the real plan/run wiring, not only the helper.""" def test_run_surface_rejects_master_arm(self, tmp_path: Path) -> None: - # run_limits.stop_early: true (the removed master arm) propagates out - # of resolve_all_tasks as EarlyStopConfigError (a ValueError, so the - # run CLI converts it to a clean BadParameter) instead of being - # demoted to a skipped task. (An armed UNOBSERVABLE criterion no - # longer reaches this validator at all — the block exists only on - # LiveSuccessCriterion, so it is a pydantic schema error at load.) + # The removed master arm escapes resolve_all_tasks as EarlyStopConfigError (a ValueError, so the + # run CLI converts it to a clean BadParameter), not a skipped task. task_file = _write_task_yaml(tmp_path, criterion_yaml=_UNARMED_CRITERION, stop_early=True) with pytest.raises(EarlyStopConfigError, match="has been removed"): _resolve_surface(task_file, tmp_path) @@ -1101,12 +1051,6 @@ def test_early_stop_ab_variants_disarm_and_arm(self, tmp_path: Path) -> None: assert early_stop_active(by_variant["smoke"]) is True # the task's block alone arms -# --------------------------------------------------------------------------- # -# Cooperative should_stop seam on ClaudeCodeAgent — still UNWIRED: the -# orchestrator does not pass should_stop yet, so these drive the agent directly. -# --------------------------------------------------------------------------- # - - class _DummyMsg: """Minimal SDK-message stand-in. @@ -1240,6 +1184,8 @@ def test_fixture_resolves_without_error(self, task_file: Path, tmp_path: Path) - class TestCooperativeStopSeam: + """Pins the should_stop seam on ClaudeCodeAgent, driven directly rather than through the orchestrator.""" + def test_stopped_early_member_on_both_enums(self) -> None: assert AgentEndStatus.STOPPED_EARLY.value == "stopped_early" assert TurnEndStatus.STOPPED_EARLY.value == "stopped_early" @@ -1294,11 +1240,6 @@ async def test_timeout_beats_stop_precedence(self) -> None: assert AgentEndStatus.STOPPED_EARLY not in {e.status for e in ends} -# --------------------------------------------------------------------------- # -# Phase 3: EarlyStopReason / EarlyStopInfo / armed_criteria_passed -# --------------------------------------------------------------------------- # - - class TestEarlyStopModels: def test_reason_values(self) -> None: assert EarlyStopReason.CRITERION_PASSED.value == "criterion_passed" @@ -1390,9 +1331,8 @@ def test_armed_criteria_passed_zero_total_weight_fails_closed(self) -> None: assert result.armed_criteria_passed([crit]) is False def test_armed_criteria_passed_default_threshold_still_requires_all(self) -> None: - # gate_threshold=1.0 (the default) must reproduce the old all()-must-pass - # rule exactly: one armed criterion at 0.0 fails the gate regardless of - # the other armed criterion's weight. + # gate_threshold=1.0 (the default) is strict AND: one armed criterion at 0.0 fails the gate + # regardless of the other armed criterion's weight. criteria = [ _skill_crit("date-teller", "date-teller", stop_on_pass=True, weight=0.8), _skill_crit("weather-teller", "date-teller", stop_on_fail=True, weight=0.2), @@ -1475,11 +1415,6 @@ def test_armed_criteria_passed_weighted_gate_with_command_executed(self) -> None assert high_weight_fails.armed_criteria_passed(criteria, gate_threshold=0.7) is False -# --------------------------------------------------------------------------- # -# Phase 3: EarlyStopWatcher -# --------------------------------------------------------------------------- # - - def _watcher(criteria: list[Any], *, max_turns: int | None = 20, gate_threshold: float = 1.0) -> EarlyStopWatcher: task = _task(criteria=criteria) assert task.run_limits is not None @@ -1621,11 +1556,8 @@ def test_auto_positive_pass_stops(self) -> None: assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED def test_auto_mixed_pass_stops_ignoring_undecided_distractors(self) -> None: - # THE mixed-arming fix: one positive + two distractors, all armed `auto`. - # Engaging ONLY the expected skill pass-stops on turn 1 even though the two - # distractors are still "undecided" — fail-armed criteria are not required - # to live-pass. (Under the old "every armed must pass" rule this could never - # fire, since a distractor can never live-pass.) + # Mixed arming: one positive + two distractors, all with both triggers. Engaging ONLY the expected + # skill pass-stops on turn 1 — undecided fail-armed distractors are not required to live-pass. watcher = _watcher( [ _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True), # positive -> pass @@ -1720,8 +1652,7 @@ def test_ceiling_bound_fires_fail_stop_when_high_weight_criterion_fails(self) -> assert watcher.info.reason == EarlyStopReason.CRITERION_FAILED def test_default_gate_threshold_fires_fail_stop_on_any_weight(self) -> None: - # At the default gate_threshold=1.0, even the low-weight criterion's - # failure alone must still fire — byte-for-byte the pre-weighting rule. + # At the default gate_threshold=1.0 even the low-weight criterion's failure alone fires. watcher = _watcher( [ _skill_crit("date-teller", "date-teller", stop_on_pass=True, stop_on_fail=True, weight=0.8), @@ -1840,7 +1771,7 @@ def test_timeout_only_arming_pass_within_budget_never_stops(self) -> None: # (no stop_on_pass). The skill engages on call 1 — well within the # budget of 3 — so the verdict latches pass and the run continues # untouched: no pass-stop (not armed for one), and the timeout can - # never fire again (the verdict is no longer undecided). Extra calls + # never fire again (the verdict is already decided). Extra calls # beyond the budget prove the latch holds. watcher = _watcher([_skill_crit("date-teller", "date-teller", max_steps_to_decide=3)]) _feed(watcher, _skill_events("date-teller")) @@ -2113,15 +2044,8 @@ def test_budget_timeout_not_latched_when_orphan_already_decided(self) -> None: assert watcher.info is None def test_pass_stop_cuts_undecided_fail_only_sibling_documented_gap(self) -> None: - # KNOWN one-sided trade, pinned so a future deferral redesign flips it - # consciously: the pass-stop deferral holds only for PASS-CAPABLE - # siblings. An armed fail-only-decidable criterion that still needs - # evidence (command_executed with min_count>=1 AND max_count set — - # polarities == {"fail"} but the frozen score needs the command run) - # is NOT deferred on, so an on_pass=stop sibling can cut before its - # minimum count is reached and the armed gate scores it 0. Documented - # in TASK_DEFINITION_GUIDE.md § stop_early: authoritative scoring for - # such combinations belongs on the kill-switched run. + # Pinned one-sided trade: the pass-stop defers only on PASS-CAPABLE siblings, so a fail-only + # sibling still needing evidence is cut and scores 0 — see TASK_DEFINITION_GUIDE.md § stop_early. watcher = _watcher( [ _skill_crit("date-teller", "date-teller", stop_on_pass=True), @@ -2255,11 +2179,6 @@ def test_decision_budget_accumulates_across_retry_attempts(self) -> None: assert watcher.info.tool_call_index == 2 -# --------------------------------------------------------------------------- # -# Phase 3: Orchestrator wiring -# --------------------------------------------------------------------------- # - - class _ScriptedAgent: """Duck-typed agent: replays scripted events through the callback, polling ``should_stop`` after each and breaking when it flips (mirrors the real @@ -2422,14 +2341,12 @@ async def test_advisory_not_gated_on_early_stop(self, tmp_path) -> None: assert result.armed_criteria_passed(self._criteria()) is True # armed gate passes async def test_decision_budget_exceeded_gates_through_armed_gate(self, tmp_path) -> None: - # A criterion capped at max_steps_to_decide=1 that never engages its - # skill fires a timeout fail-stop — and then gates through the SAME - # weighted armed gate as any other stop (no force-fail bypass). The - # mocked checker deliberately scores everything 1.0, so the armed gate - # passes: the stop truncates the run, the gate decides the verdict. - # (In a real run the frozen trajectory would score the undecided - # criterion 0.0 and the gate would fail — asserted separately below in - # test_decision_budget_exceeded_real_scores_fail_the_gate.) + # A criterion capped at max_steps_to_decide=1 that never engages its skill fires + # a timeout fail-stop, then gates through the SAME weighted armed gate as any + # other stop (no force-fail bypass). The mocked checker scores everything 1.0, + # so the gate passes: the stop truncates the run, the gate decides the verdict. + # With real scores the undecided criterion scores 0.0 and the gate fails: + # test_decision_budget_exceeded_real_scores_fail_the_gate. criteria = [ _skill_crit(self._SKILL, self._SKILL, stop_on_pass=True, max_steps_to_decide=1), FileExistsCriterion(path="artifact.txt", description="artifact must exist"), @@ -2685,11 +2602,6 @@ async def test_setup_kill_switch_leaves_watcher_none(self, tmp_path: Path, monke assert orch._early_stop_watcher is None -# --------------------------------------------------------------------------- # -# Report / telemetry surfaces -# --------------------------------------------------------------------------- # - - def _stopped_result( *, reason: EarlyStopReason = EarlyStopReason.CRITERION_PASSED, @@ -2740,11 +2652,9 @@ def test_task_dict_reflects_decision_budget_exceeded(self) -> None: def test_runtime_note_omits_the_turns_avoided_claim(self) -> None: """The note states the reason and the gate, and claims no turn saving. - It used to render ``<= N turn(s) avoided`` from ``max_turns - sdk_turn_index``. - On Codex and Antigravity one ``communicate()`` is a single SDK turn, so that - subtraction advertised the entire max_turns budget as saved when all that was - actually cut was a tool-call tail. ``turns_remaining_at_stop`` is still - persisted on ``EarlyStopInfo``, where its docstring calls it an upper bound. + ``turns_remaining_at_stop`` stays persisted on ``EarlyStopInfo`` as an upper bound. + + Rationale: .claude/notes/reporting.md § The claims the reports do NOT make """ lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(_stopped_result())])) blob = "\n".join(lines) @@ -2814,13 +2724,6 @@ def test_telemetry_dims_reflect_decision_budget_exceeded(self) -> None: assert props["EarlyStopReason"] == "decision_budget_exceeded" -# --------------------------------------------------------------------------- # -# Cooperative should_stop seam on CodexAgent — mirrors TestCooperativeStopSeam. -# SDK-independent: the pump is driven over fake notifications (agentMessage -# deltas need no openai_codex types) and turn/completed handling is stubbed. -# --------------------------------------------------------------------------- # - - class _CodexNotifIter: """Counting iterator over fake notifications (the pump pulls via ``next``).""" @@ -2918,6 +2821,9 @@ async def _run_codex_communicate( class TestCodexCooperativeStopSeam: + """Pins the should_stop seam on CodexAgent without the SDK: fake agentMessage notifications, + turn/completed handling stubbed.""" + async def test_stop_after_first_dispatched_notification(self) -> None: notifications = [_codex_delta(0), _codex_delta(1), _codex_delta(2), _codex_completed()] agent, record, sink, stream, handle = await _run_codex_communicate(notifications=notifications, stop_after=1) @@ -3058,12 +2964,6 @@ def _capturing_init(self: Any, *args: Any, **kwargs: Any) -> None: recover.assert_not_awaited() -# --------------------------------------------------------------------------- # -# Cooperative should_stop seam on AntigravityAgent — same contract, driven over -# a fake step stream (mirrors tests/test_antigravity_agent.py's conventions). -# --------------------------------------------------------------------------- # - - def _ag_step(i: int) -> SimpleNamespace: """A minimal streamed text step (plain strings stand in for the SDK enums).""" return SimpleNamespace( @@ -3138,6 +3038,8 @@ async def _run_antigravity_communicate( class TestAntigravityCooperativeStopSeam: + """Pins the same should_stop seam on AntigravityAgent, driven over a fake step stream.""" + async def test_stop_after_first_processed_step(self) -> None: agent, record, sink, conversation = await _run_antigravity_communicate(stop_after=1, n_steps=3) # The deciding step is kept; the next is never pulled. @@ -3250,13 +3152,9 @@ def __exit__(self, exc_type: Any, *_exc: Any) -> bool: await agent.discard_pending_turn() -# --------------------------------------------------------------------------- # -# Orchestrator-level wiring on a non-Claude agent type: the watcher, gating and -# report row are agent-agnostic — an armed codex task flows end to end. -# --------------------------------------------------------------------------- # - - class TestOrchestratorEarlyStopWiringCodex: + """Pins that the watcher, gating and report row are agent-agnostic: an armed codex task runs end to end.""" + _SKILL = "date-teller" def _criteria(self) -> list[Any]: diff --git a/tests/test_error_handling.py b/tests/test_error_handling.py index 1bb90addf..b3b6e711e 100644 --- a/tests/test_error_handling.py +++ b/tests/test_error_handling.py @@ -686,7 +686,7 @@ def test_truncate_log_custom_max_chars(self): class TestAgentCrashErrorNoPartialField: - """AgentCrashError no longer carries partial_turn_record.""" + """AgentCrashError has no partial_turn_record attribute.""" def test_constructs_without_kwargs(self): err = AgentCrashError("boom") diff --git a/tests/test_event_collector.py b/tests/test_event_collector.py index 9755120a5..a73c82cd4 100644 --- a/tests/test_event_collector.py +++ b/tests/test_event_collector.py @@ -1104,10 +1104,9 @@ def test_a_mid_stream_record_leaves_it_unset(self): def test_the_span_set_is_computed_exactly_once(self): """All four buckets must be measured against ONE selection. - `_overhead_ms` used to accept `tool_spans=None` and fall back to - building its own set — a second selection, which is what the comment at - the single call site says must never happen. The parameter is now - required, so the fallback is unrepresentable rather than merely unused. + `_overhead_ms` takes `tool_spans` as a required parameter, so a fallback + that builds its own set — a second selection, which the comment at the + single call site forbids — is unrepresentable. """ import inspect diff --git a/tests/test_execute_command.py b/tests/test_execute_command.py index 6e3775ad5..d90545c92 100644 --- a/tests/test_execute_command.py +++ b/tests/test_execute_command.py @@ -288,9 +288,8 @@ async def test_docker_forwards_grade_to_the_container(tmp_path: Path, grade: boo # The in-container grading default is asserted behaviourally in # tests/test_detached_grading_boundaries.py # (`TestGradePlumbedIntoTheContainerOrchestrator`), which drives the real command -# with `Orchestrator` patched and reads the captured `grade` kwarg. The source-text -# grep that used to live here was duplicated verbatim in that file and proved -# nothing: deleting `grade=grade` at the call site left both greps green. +# with `Orchestrator` patched and reads the captured `grade` kwarg. A source-text +# grep proves nothing: deleting `grade=grade` at the call site leaves it green. def test_execute_help_explains_the_refused_flags() -> None: diff --git a/tests/test_execute_evaluate_loop.py b/tests/test_execute_evaluate_loop.py index 88a8f59ef..5ed16398d 100644 --- a/tests/test_execute_evaluate_loop.py +++ b/tests/test_execute_evaluate_loop.py @@ -422,12 +422,11 @@ def test_grading_the_same_run_twice_reaches_the_same_verdict(tmp_path: Path) -> def test_execute_records_max_turns_exhausted_exactly_as_run_does(tmp_path: Path) -> None: """`max_turns_exhausted` is a fact about the RUN, not a verdict. - It used to be captured AFTER the grading switch's early return, so under - `execute` it was never recorded at all: the row finalized NOT_GRADED and the - command exited 0 where `run` reported MAX_TURNS_EXHAUSTED and exited 1 — for - identical agent output. `_seed_from_prior_result` cannot restore a fact the - execute phase never captured, so a later `evaluate` inherited the wrong - terminal status too. + Pins: `execute` records it before the grading switch's early return, so a + later `evaluate` inherits it. `_seed_from_prior_result` cannot restore a fact + the execute phase never captured. + + Rationale: .claude/notes/isolation.md § Detached grading and `Sandbox.adopt` """ from coder_eval.streaming.collector import EventCollector @@ -575,16 +574,12 @@ def test_a_copy_grade_leaves_the_runs_artifacts_pointer_alone(tmp_path: Path) -> assert _row(task_dir)["final_status"] == FinalStatus.SUCCESS.value -# --------------------------------------------------------------------------- -# post_run belongs to the GRADING phase -# -# `post_run` is defined as running "after the evaluation verdict is finalized", -# and it may mutate the workspace. Under `execute` there is no verdict to come -# after, so running it there inverted its own contract AND broke the round-trip -# guarantee: the criteria had not read the tree yet, so `evaluate` graded a -# workspace `post_run` had already modified. `execute` now DEFERS it to whichever -# command grades. -# --------------------------------------------------------------------------- +# post_run belongs to the GRADING phase. It is defined as running "after the +# evaluation verdict is finalized" and it may mutate the workspace, so under +# `execute` — where no verdict is coming — running it inverted its own contract and +# broke the round-trip guarantee. `execute` now DEFERS it to whichever command grades. +# Rationale: .claude/notes/orchestration.md § Execute vs. run: the grading switch + # Deliberately destructive, and destructive of the exact file the criteria read. # A `post_run` that mutates something no criterion observes (`rm -rf @@ -748,15 +743,12 @@ def test_an_in_place_grade_refuses_a_recorded_post_run_without_consent(tmp_path: def test_the_baseline_post_run_alone_does_not_prompt(tmp_path: Path) -> None: """The whole point of the exemption, on the shape that is 100% of real runs. - `experiments/default.yaml` appends `rm -rf node_modules .npm-prefix` to every - task, so once post_run began running on the in-place path, scanning it - naively made EVERY `evaluate ` demand --allow-recorded-commands. - A refusal that always fires is read as a formality and waved through, which - is how the gate would have stopped protecting the authored commands that DO - represent a choice by whoever wrote the run directory. + Pins: a run directory whose recorded post_run holds only the grader's own + `experiments/default.yaml` baseline grades in place without + --allow-recorded-commands, and the baseline command still runs. The agentless + task authors no post_run of its own. - The agentless task authors no post_run of its own, so the recorded list holds - only the grader's own baseline — nothing the record chose. + Rationale: .claude/notes/orchestration.md § What the gate covers, and why each part is in scope """ run_dir = tmp_path / "r" _invoke(["execute", str(AGENTLESS_TASK), "--run-dir", str(run_dir)]) diff --git a/tests/test_experiment_reports.py b/tests/test_experiment_reports.py index a6e72271b..cde2567f6 100644 --- a/tests/test_experiment_reports.py +++ b/tests/test_experiment_reports.py @@ -540,7 +540,7 @@ def test_variant_html_task_links_include_replicate_segment(self, tmp_path, sampl # Every per-task link in the variant HTML must traverse the replicate dir. assert 'href="task-a/00/task.html"' in html assert 'href="task-b/00/task.html"' in html - # Guard against the pre-fix flat shape slipping back in. + # The flat (replicate-less) link shape must not appear. assert 'href="task-a/task.html"' not in html assert 'href="task-b/task.html"' not in html @@ -1174,7 +1174,7 @@ def test_paired_diff_line_for_two_variants_equal_counts(self): assert "Cohen's d" in md def test_paired_diff_needs_two_common_tasks(self): - # Unequal replicate counts no longer exclude a task, but one task is still + # Unequal replicate counts do not exclude a task, but one task is still # a single pair — too few for a paired comparison. per_rep = { "a": {"task-1": [0.9, 0.85, 0.95]}, diff --git a/tests/test_experiment_resolver.py b/tests/test_experiment_resolver.py index ec82aadb1..c37f7e2ae 100644 --- a/tests/test_experiment_resolver.py +++ b/tests/test_experiment_resolver.py @@ -344,8 +344,8 @@ class TestTurnTimeoutResolution: """Regression tests for turn_timeout flowing through the field-merge resolver.""" def test_turn_timeout_in_agent_dict_now_rejected(self): - """Legacy turn_timeout under agent: is no longer hoisted — it fails loudly - via the agent model's extra='forbid' (the hoist shim was removed).""" + """turn_timeout under agent: is not hoisted — it fails loudly via the + agent model's extra='forbid'.""" default_exp = ExperimentDefinition( experiment_id="default", defaults=ExperimentDefaults(agent={"type": "claude-code", "turn_timeout": 300}), diff --git a/tests/test_experiment_runner.py b/tests/test_experiment_runner.py index a9259fa53..254055b79 100644 --- a/tests/test_experiment_runner.py +++ b/tests/test_experiment_runner.py @@ -216,12 +216,9 @@ def test_skips_invalid_yaml_continues_with_rest(self, tmp_path, run_dir, default def test_skip_true_excludes_task_from_resolution(self, tmp_path, run_dir, default_experiment): """`skip: true` in the YAML excludes the task and records it in `skipped`. - Regression guard for MST-9675: TaskDefinition previously had no ``skip`` - field, so authors writing ``skip: true`` to quarantine a known-blocked - task got a silent no-op — the task still ran (and often errored). Honor - the field at resolve_all_tasks time and report it via SkippedTask with - a ``"skip: true"`` reason prefix so consumers can distinguish opt-outs - from load failures. + Pins: resolve_all_tasks drops the task and reports it via SkippedTask + with a ``"skip: true"`` reason prefix, so consumers can distinguish + opt-outs from load failures. """ blocked = _write_task_yaml(tmp_path, "task-blocked", agent={"type": "claude-code"}) # Mutate the loaded YAML to add `skip: true` while keeping every other diff --git a/tests/test_harbor_agent.py b/tests/test_harbor_agent.py index d012533b3..dff56ff18 100644 --- a/tests/test_harbor_agent.py +++ b/tests/test_harbor_agent.py @@ -1,7 +1,7 @@ """``coder_eval.harbor.agent.CoderEvalAgent`` — the Harbor-agent extension point. ``harbor`` is not a project dependency (it only needs to be present INSIDE a -Harbor trial container, see ``harbor/agent.py``'s module docstring), so this +Harbor trial container, see the ``ImportError`` that ``harbor/agent.py`` raises), so this module cannot simply be imported in a normal test run. Rather than skip it entirely (leaving `run()`'s command construction and `populate_context_post_run()`'s trajectory parsing with zero coverage), stub diff --git a/tests/test_harbor_export_golden.py b/tests/test_harbor_export_golden.py index 340f8ac36..9fea140a8 100644 --- a/tests/test_harbor_export_golden.py +++ b/tests/test_harbor_export_golden.py @@ -11,14 +11,15 @@ class of bug directly, cheaper than enumerating every field in prose. no ``WORKDIR`` (exercises the append-a-WORKDIR path), ``reference:``, ``run_limits.task_timeout``, resource limits, ``network: none``, and both a filesystem criterion and a ``reference_comparison`` criterion (proving the -placeholder-agent design from the packager's own module docstring survives -end to end, not just in an isolated unit test). +placeholder-agent design survives end to end, not just in an isolated unit test). Regenerate after an INTENTIONAL mapping change with:: GOLDEN_REGEN=1 uv run pytest tests/test_harbor_export_golden.py and review the resulting diff before committing. + +Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml """ from __future__ import annotations diff --git a/tests/test_harbor_packager.py b/tests/test_harbor_packager.py index 22038a52a..f75fb931f 100644 --- a/tests/test_harbor_packager.py +++ b/tests/test_harbor_packager.py @@ -163,7 +163,10 @@ def test_network_bridge_maps_to_public(self, tmp_path: Path) -> None: class TestVerifierTaskYaml: - """tests/task.yaml must never set agent: {type: none} — see packager.py's module docstring.""" + """tests/task.yaml must never set agent: {type: none}. + + Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml + """ def test_never_sets_agent_type_none(self, tmp_path: Path) -> None: task_file = _write_task(tmp_path) @@ -184,10 +187,12 @@ def test_reloads_as_a_valid_task_definition(self, tmp_path: Path) -> None: assert len(reloaded.success_criteria) == 2 def test_reference_comparison_survives_the_none_agent_trap(self, tmp_path: Path) -> None: - """The corrected design: a placeholder real agent type unblocks reference_comparison. + """A placeholder real agent type unblocks reference_comparison. If tests/task.yaml set agent: {type: none} instead, this would raise at - TaskDefinition.model_validate — see the module docstring for why. + TaskDefinition.model_validate. + + Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml """ task_file = _write_task( tmp_path, diff --git a/tests/test_image_from_dockerfiles.py b/tests/test_image_from_dockerfiles.py index 2400aa393..0e197fdd7 100644 --- a/tests/test_image_from_dockerfiles.py +++ b/tests/test_image_from_dockerfiles.py @@ -235,8 +235,8 @@ def test_accepts_runtime_image_with_version_label(self, tmp_path: Path, mocker) def test_rejects_image_without_version_label(self, tmp_path: Path, mocker) -> None: """A non-framework image (no org.coder-eval.version label) -> actionable DockerRunError. - The host pins --entrypoint, so the build is no longer gated on the baked - ENTRYPOINT; the runtime-image check uses the version label instead. + The host pins --entrypoint, so the build is not gated on the baked + ENTRYPOINT; the runtime-image check uses the version label. """ dockerfile = tmp_path / "Dockerfile" dockerfile.write_text("FROM ubuntu:24.04\n") diff --git a/tests/test_judge_anthropic.py b/tests/test_judge_anthropic.py index 4747867ec..dc214715e 100644 --- a/tests/test_judge_anthropic.py +++ b/tests/test_judge_anthropic.py @@ -31,7 +31,7 @@ def _make_response(*, score: float = 0.5, rationale: str = "ok") -> MagicMock: def _make_client(response: MagicMock | None = None) -> MagicMock: client = MagicMock() # spec-bound to the real ``AsyncMessages.create`` signature so a kwarg the - # installed SDK no longer accepts (e.g. a removed ``temperature``) fails + # installed SDK does not accept (e.g. a removed ``temperature``) fails # here instead of silently passing against an unconstrained MagicMock. client.messages = MagicMock(spec=AsyncMessages) client.messages.create = AsyncMock( @@ -108,9 +108,9 @@ async def test_invoke_anthropic_judge_passes_temperature_and_max_tokens() -> Non async def test_invoke_anthropic_judge_escalates_on_signature_break() -> None: - """A kwarg the installed SDK no longer accepts must escalate as infra, not + """A kwarg the installed SDK does not accept must escalate as infra, not silently score the row 0.0 (see judge_bedrock.py's parallel retry/escalation - contract and CLAUDE.md's CE039 rationale).""" + contract and CE039).""" from coder_eval.errors import JudgeInfrastructureError client = _make_client() diff --git a/tests/test_judge_context_builder.py b/tests/test_judge_context_builder.py index cf42decc5..0b145d6d8 100644 --- a/tests/test_judge_context_builder.py +++ b/tests/test_judge_context_builder.py @@ -394,18 +394,14 @@ def test_scrub_reference_skips_secrets_below_min_length() -> None: def test_scrub_runs_before_clip_so_partial_secrets_dont_survive() -> None: - """SECURITY regression for bug_001: scrub must run BEFORE clipping, not after. - - scrub_reference uses str.replace which only matches the secret as a contiguous - whole string. If the budget clips the prompt mid-secret, the surviving prefix - no longer matches the full secret string — replace finds nothing — and a - partial reference fragment is persisted unsanitized. - - Concrete trigger: a multi-KB reference is inlined into the prompt envelope - by ``include_reference=True``. The transcript budget forces clipping. The - surviving prefix of the prompt contains the leading portion of the reference - content. Scrub-before-clip ensures the secret is redacted while still - present in full, so the post-clip prompt cannot leak any portion. + """SECURITY: scrub must run BEFORE clipping, not after. + + Pins: a multi-KB reference inlined by ``include_reference=True`` and clipped + by the transcript budget leaves no portion of the secret in the persisted + prompt. Clipping first leaves a prefix that exact-substring ``scrub_reference`` + cannot match. + + Rationale: .claude/notes/contracts.md § Scrub before truncate """ from coder_eval.evaluation.judge_context import build_judge_transcript diff --git a/tests/test_judge_models.py b/tests/test_judge_models.py index 311f8a358..87da6dc8a 100644 --- a/tests/test_judge_models.py +++ b/tests/test_judge_models.py @@ -10,8 +10,7 @@ # Representative vendor-prefixed model names (the Bedrock AWS-id dialect) the judge -# model-name translators must normalize. Formerly sourced from the proxy's -# DEFAULT_MODEL_MAP; kept inline here now that the proxy has been removed. +# model-name translators must normalize. _VENDOR_MODEL_NAMES = [ "anthropic.claude-sonnet-4-6", "anthropic.claude-opus-4-6-v1", diff --git a/tests/test_judge_persistence.py b/tests/test_judge_persistence.py index e5bf0da5a..0c627d962 100644 --- a/tests/test_judge_persistence.py +++ b/tests/test_judge_persistence.py @@ -514,7 +514,7 @@ def test_load_attaches_dict_when_schema_mismatch(tmp_path: Path) -> None: def test_load_reads_legacy_json_sibling(tmp_path: Path) -> None: """Sibling files spilled before the YAML switch are ``judge-.json``. - The loader must still read them so previously-archived runs render.""" + The loader must still read them so archived runs render.""" judge = _make_judge_result(transcript=None) judge.transcript_path = "judge-0.json" legacy = ( diff --git a/tests/test_litellm_route.py b/tests/test_litellm_route.py index 087d6dc33..3055ae44a 100644 --- a/tests/test_litellm_route.py +++ b/tests/test_litellm_route.py @@ -244,10 +244,9 @@ def test_unknown_backend_raises(self, monkeypatch): class TestCheckerContextModel: - """CheckerContext/ApiRouteContext — the typed replacement for the old - hand-validated open dict (previously 0% covered, per the PR #137 review). - ``extra="forbid"`` + real field types now do what the hand-rolled - ``validate_checker_context_shape`` used to.""" + """CheckerContext/ApiRouteContext validate the checker context shape. + + ``extra="forbid"`` plus real field types reject a malformed context.""" @staticmethod def _validate(value): @@ -720,8 +719,8 @@ def test_none_when_no_base_url(self): assert _litellm_preflight_error(s) is None def test_scheme_less_base_url_returns_clean_error(self): - # Regression: a scheme-less URL used to make urlopen raise a bare - # ValueError that escaped as a traceback. Now it returns a clean message. + # A scheme-less URL makes urlopen raise a bare ValueError; the preflight + # returns a clean message rather than a traceback. s = Settings(api_backend=ApiBackend.LITELLM, litellm_base_url="localhost:4000", litellm_model="m") err = _litellm_preflight_error(s) assert err is not None and "http(s)" in err diff --git a/tests/test_llm_judge_criterion.py b/tests/test_llm_judge_criterion.py index ca0a1450e..4ef78caca 100644 --- a/tests/test_llm_judge_criterion.py +++ b/tests/test_llm_judge_criterion.py @@ -807,8 +807,8 @@ def test_judge_bedrock_route_threads_model_unchanged(sandbox: Sandbox) -> None: def test_judge_empty_rationale_returns_judge_result_with_error(sandbox: Sandbox) -> None: """A model that emits whitespace-only rationale → JudgeCriterionResult(error=...). - Locks in the ValidationError → JudgeCriterionResult(score=0.0) chain — - the pre-fix code would silently emit ``rationale: `` (blank) in details. + Pins the ValidationError → JudgeCriterionResult(score=0.0) chain, so a + blank ``rationale: `` never reaches details silently. """ from coder_eval.models import JudgeCriterionResult diff --git a/tests/test_new_criteria.py b/tests/test_new_criteria.py index 1fede7659..872c08d98 100644 --- a/tests/test_new_criteria.py +++ b/tests/test_new_criteria.py @@ -28,7 +28,6 @@ def test_regex_match_found(self, tmp_path): pattern=r"async def \w+\(", ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -37,7 +36,6 @@ def test_regex_match_found(self, tmp_path): assert "async def my_function(" in result.details assert result.error is None - # Cleanup sandbox.cleanup(preserve=False) def test_regex_match_not_found(self, tmp_path): @@ -58,7 +56,6 @@ def test_regex_match_not_found(self, tmp_path): pattern=r"async def \w+\(", ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -67,7 +64,6 @@ def test_regex_match_not_found(self, tmp_path): assert "not found" in result.details assert result.error is None - # Cleanup sandbox.cleanup(preserve=False) def test_regex_must_not_match(self, tmp_path): @@ -89,7 +85,6 @@ def test_regex_must_not_match(self, tmp_path): must_match=False, ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -97,7 +92,6 @@ def test_regex_must_not_match(self, tmp_path): assert result.score == 1.0 assert "correctly absent" in result.details - # Cleanup sandbox.cleanup(preserve=False) def test_regex_must_not_match_fails(self, tmp_path): @@ -119,7 +113,6 @@ def test_regex_must_not_match_fails(self, tmp_path): must_match=False, ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -128,7 +121,6 @@ def test_regex_must_not_match_fails(self, tmp_path): assert "should not be present" in result.details assert "TODO" in result.details - # Cleanup sandbox.cleanup(preserve=False) def test_regex_with_flags(self, tmp_path): @@ -150,7 +142,6 @@ def test_regex_with_flags(self, tmp_path): flags=re.IGNORECASE, ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -158,7 +149,6 @@ def test_regex_with_flags(self, tmp_path): assert result.score == 1.0 assert "hello" in result.details - # Cleanup sandbox.cleanup(preserve=False) def test_regex_file_not_found(self, tmp_path): @@ -175,7 +165,6 @@ def test_regex_file_not_found(self, tmp_path): pattern=r"test", ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -183,7 +172,6 @@ def test_regex_file_not_found(self, tmp_path): assert result.score == 0.0 assert "does not exist" in result.error - # Cleanup sandbox.cleanup(preserve=False) def test_regex_invalid_pattern(self, tmp_path): @@ -204,7 +192,6 @@ def test_regex_invalid_pattern(self, tmp_path): pattern=r"[invalid(regex", # Invalid pattern ) - # Execute check checker = SuccessChecker(sandbox) result = checker.check(criterion) @@ -212,5 +199,4 @@ def test_regex_invalid_pattern(self, tmp_path): assert result.score == 0.0 assert "Invalid regex pattern" in result.error - # Cleanup sandbox.cleanup(preserve=False) diff --git a/tests/test_opencode_agent.py b/tests/test_opencode_agent.py index 7b7b679ea..229ba1d90 100644 --- a/tests/test_opencode_agent.py +++ b/tests/test_opencode_agent.py @@ -9,8 +9,10 @@ --format json`` — the CLI's own compact vocabulary (``step_start`` / ``step_finish`` / ``text`` / ``tool_use``, payload under ``part``). Do NOT "correct" them toward the ``session.next.*`` names in the server's OpenAPI -schema: those describe `opencode serve`'s SSE surface, and an earlier version of -this harness parsed them and silently captured zero telemetry on a real run. +schema: those describe `opencode serve`'s SSE surface, and parsing them captures +zero telemetry on a real run. + +Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash """ from __future__ import annotations @@ -1079,10 +1081,10 @@ async def test_missing_cli_is_actionable(self, monkeypatch, tmp_path): class TestZeroTelemetryIsLoud: """A clean exit that captured no token telemetry must crash, not score. - An earlier version of this harness parsed the `session.next.*` server - vocabulary instead of the CLI's and reported SUCCESS 1.0 with zero turns, - zero tokens and zero cost — indistinguishable from a real pass in every - aggregate. Drift must be an ERROR, not a quiet empty success. + A harness parsing the `session.next.*` server vocabulary instead of the + CLI's would report SUCCESS 1.0 with zero turns, zero tokens and zero cost — + indistinguishable from a real pass in every aggregate. Drift must be an + ERROR, not a quiet empty success. The guard keys on the TELEMETRY, not the event vocabulary: recognizing the event names is not the property worth protecting, and checking them alone @@ -1817,13 +1819,12 @@ def test_orphan_result_is_never_dropped(self): class TestGenerationWindowExcludesToolExecution: - """A tool running inside a step is not model time — asserted where it is now DECIDED. + """A tool running inside a step is not model time — asserted where it is DECIDED. - The reducer no longer subtracts anything. It publishes the RAW window, and - `timing.subtract_tool_time` takes the tool union back out of it - once, for all five harnesses. So these cases drive the reducer and then a - real collector, and assert the PUBLISHED number — the one that reaches - `task.json` — rather than an intermediate the reducer used to own. + The reducer publishes the RAW window, and `timing.subtract_tool_time` takes + the tool union back out of it once, for all five harnesses. So these cases + drive the reducer and then a real collector, and assert the PUBLISHED + number — the one that reaches `task.json`. They are not duplicates of `tests/test_event_collector.py::TestSubtractToolTime`: those pin the @@ -1838,12 +1839,10 @@ def _finish_step(self, monkeypatch, spans, open_starts=()): """Drive the reducer, then publish through a real collector. `spans` are RESOLVED calls (both bounds); `open_starts` are calls that - never returned. An unresolved call now contributes NO span — it has no + never returned. An unresolved call contributes NO span — it has no `execution_completed_at`, and inventing one is what `None` exists to - prevent — where the reducer used to bound it at the window's end. That - is a real change and a better one: the collector sees every span at - once, so a call straddling a boundary is clipped to each window it - actually overlapped instead of approximated at the boundary. + prevent. The collector sees every span at once, so a call straddling a + boundary is clipped to each window it actually overlapped. """ class _Clock(datetime): @@ -1920,12 +1919,11 @@ def test_the_window_never_goes_negative(self, monkeypatch): assert message.generation_duration_ms == 0.0 def test_a_tool_still_open_at_the_boundary_contributes_no_span(self, monkeypatch): - """The behaviour that CHANGED with the move, stated rather than implied. + """A call still open at the step boundary subtracts nothing from that window. - The reducer used to bound a still-open call at the window's end and - subtract that slice. The collector cannot: a call with no - `execution_completed_at` was never timed. Its time is subtracted when it - RESOLVES, from whichever windows its real interval overlaps. + A call with no `execution_completed_at` was never timed, so nothing + bounds it at the window's end. Its time is subtracted when it RESOLVES, + from whichever windows its real interval overlaps. """ message = self._finish_step(monkeypatch, [], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) assert message.generation_duration_ms == pytest.approx(1000.0) @@ -2074,20 +2072,14 @@ def now(tz=None): class TestToolSpansSurviveTheStepBoundary: """A tool that closes BETWEEN two steps still belongs to the next window. - This used to be a bookkeeping problem: a per-step span list, cleared at - `step_start` — after the window it feeds had already opened at `gen_mark` — - so a call closing in the gap had its span wiped before the next - `step_finish` could subtract it. That list is gone. - `timing.subtract_tool_time` sees every span at once and clips each - to the windows it overlaps, so the property now holds by construction - rather than by a reset rule. Kept, and re-pointed at the collector, because - the property is what matters: a future reducer change could still break it - by moving a mark or failing to emit the ToolEnd the collector reduces. - - It needs the NON-TERMINAL tool path to reach: the CLI normally emits one - already-`completed` event per call, which closes inside the step that - opened it. That is why the measured corpus reads 0.00% and a reproduction - has to drive the state object. + Pins: a non-terminal tool that completes in the gap between `step_finish` + and the next `step_start` is subtracted from the next generation window, + through the ToolEnd the collector reduces. + + HAZARD: moving a generation mark or dropping that ToolEnd breaks it. The + CLI rarely takes this path, so the test drives the state object directly. + + Rationale: .claude/notes/agents.md § Per-harness generation marks """ def _run(self, monkeypatch): @@ -2173,14 +2165,13 @@ def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self, monke def test_a_duplicate_step_finish_does_not_republish_the_previous_window(self, monkeypatch): """A spent `step_started_at` must not seed the next window. - `close_window`'s `min(mark, item_start)` pulls the window open to cover - the item's own start. That is the backwards-clock defence — which this - reducer genuinely needs, since its stamps are raw `datetime.now()` and - not on a `TurnClock`. But a start stamp left in place after its step was - published is not a backwards clock: it is a stale value BEFORE the mark, - so the guard reopens the next window at the previous step's start and - publishes that whole span again. Reproduced on Pi's identical twin - before the fix: 3000 ms of generation for a 2000 ms turn. + Pins: a flush clears its spent start stamp, so a duplicate `step_finish` + with no intervening `step_start` does not reopen the next window at the + previous step's start through `close_window`'s `min(mark, item_start)`. + Keep that `min()`: this reducer's stamps are raw `datetime.now()`, not on + a `TurnClock`, so it needs the backwards-clock defence. + + Rationale: .claude/notes/agents.md § Per-harness generation marks """ monkeypatch.setattr(agent_module, "datetime", _SteppedClock) state = _OpenCodeTurnState(task_id="t1", iteration=1, user_input="go", model="m") diff --git a/tests/test_orchestrator.py b/tests/test_orchestrator.py index 00a4c79f9..273a8ad8d 100644 --- a/tests/test_orchestrator.py +++ b/tests/test_orchestrator.py @@ -177,7 +177,7 @@ def test_enabled_simulation_never_gets_litellm(self, tmp_path, monkeypatch): simulation=SimulationConfig(enabled=True, persona="p", goal="g"), api_route=self._litellm_api_route(), ) - orchestrator._resolve_routes() # no raise -- this used to be rejected + orchestrator._resolve_routes() # no raise assert isinstance(orchestrator.eval_route, LiteLLMRoute) assert not isinstance(orchestrator.simulator_route, LiteLLMRoute) @@ -2343,9 +2343,8 @@ async def test_evaluation_loop_evaluate_only_loads_reference(tmp_path): """Evaluate-only branch (agent is None) must still stage the reference and forward it to SuccessChecker.check_all_async. - Regression: previously this branch called check_all without the reference, - so judge-style criteria (llm_judge / agent_judge) silently saw no - reference even when task.reference was set — surfaced as + Without it, judge-style criteria (llm_judge / agent_judge) silently see no + reference even when task.reference is set, logged as "include_reference=True but reference not set" in the judge_context log. """ from datetime import datetime diff --git a/tests/test_pi_agent.py b/tests/test_pi_agent.py index f15e14457..f74783766 100644 --- a/tests/test_pi_agent.py +++ b/tests/test_pi_agent.py @@ -382,7 +382,7 @@ async def test_plugins_that_do_not_resolve_warn_loudly(self, patch_exec, tmp_pat with caplog.at_level("WARNING"): await _agent(plugins=[{"type": "local", "path": "/no/such/dir"}]).start(str(tmp_path)) assert "0 skill dir(s) resolved" in caplog.text or "did not resolve" in caplog.text - # plugins is no longer named in the "NOT enforced" warning. + # plugins is not named in the "NOT enforced" warning. assert "plugins" not in "".join(r.message for r in caplog.records if "NOT enforced" in r.message) async def test_unenforced_fields_warn_but_system_prompt_does_not(self, patch_exec, tmp_path, caplog): @@ -1106,13 +1106,13 @@ def now(self) -> datetime: class TestGenerationWindowExcludesToolExecution: - """A tool running inside a turn is not model time — asserted where it is now DECIDED. + """A tool running inside a turn is not model time — asserted where it is DECIDED. - The reducer no longer subtracts anything. It publishes the RAW window, and + The reducer does not subtract anything. It publishes the RAW window, and `timing.subtract_tool_time` takes the tool union back out of it once, for all five harnesses. So these cases drive the reducer and then a real collector, and assert the PUBLISHED number — the one that reaches - `task.json` — rather than an intermediate the reducer used to own. + `task.json`. They are not duplicates of `tests/test_event_collector.py::TestSubtractToolTime`: those pin the @@ -1127,12 +1127,10 @@ def _finish_turn(self, spans, open_starts=()): """Drive the reducer, then publish through a real collector. `spans` are RESOLVED calls (both bounds); `open_starts` are calls that - never returned. An unresolved call now contributes NO span — it has no + never returned. An unresolved call contributes NO span — it has no `execution_completed_at`, and inventing one is what `None` exists to - prevent — where the reducer used to bound it at the window's end. That - is a real change and a better one: the collector sees every span at - once, so a call straddling a boundary is clipped to each window it - actually overlapped instead of approximated at the boundary. + prevent. The collector sees every span at once, so a call straddling a + boundary is clipped to each window it actually overlapped. """ state = _PiTurnState(task_id="t", iteration=1, user_input="x", model="m", clock=_FixedClock(self.WINDOW_END)) state.turn_started_at = self.WINDOW_START @@ -1201,12 +1199,11 @@ def test_the_window_never_goes_negative(self): assert message.generation_duration_ms == 0.0 def test_a_tool_still_open_at_the_boundary_contributes_no_span(self): - """The behaviour that CHANGED with the move, stated rather than implied. + """A call still open at the boundary takes nothing out of the window. - The reducer used to bound a still-open call at the window's end and - subtract that slice. The collector cannot: a call with no - `execution_completed_at` was never timed. Its time is subtracted when it - RESOLVES, from whichever windows its real interval overlaps. + A call with no `execution_completed_at` was never timed, so the collector + cannot bound it. Its time is subtracted when it RESOLVES, from whichever + windows its real interval overlaps. """ message = self._finish_turn([], open_starts=[self.WINDOW_START + timedelta(milliseconds=600)]) assert message.generation_duration_ms == pytest.approx(1000.0) @@ -1243,9 +1240,9 @@ class _SteppedClock: """A `TurnClock` stand-in the test moves by hand, in ms from `_SPAN_BASE`. INJECTED, never monkeypatched onto the module. Pi derives every wall stamp - from its turn clock now, so patching `agent_module.datetime` would no - longer reach it: the tests would quietly start measuring the real clock and - pass by accident instead of failing. Injection also puts the "one clock per + from its turn clock, so patching `agent_module.datetime` does not reach it: + the tests would quietly measure the real clock and pass by accident instead + of failing. Injection also puts the "one clock per turn" lifetime in the constructor signature where it can be read. """ @@ -1263,15 +1260,11 @@ def _turn_end_payload(): class TestGenerationWindowsTileTheTurn: """Each window runs from the PREVIOUS `turn_end`, not from its own `turn_start`. - Pi was the only harness measuring from its own turn start, so the wall - clock between one `turn_end` and the next `turn_start` — the model time - that PRODUCED the next turn — fell into no bucket at all. The four-bucket - identity is asserted only as an upper bound, so nothing failed. + Pins: the wall clock between one `turn_end` and the next `turn_start` — the + model time that PRODUCED the next turn — lands inside a window. The tool-span + half is TestToolSpansSurviveTheTurnBoundary. - The gap is small in practice (measured across 25 real window pairs: median - 0.25 ms, max 0.75 ms). The value here is that it closes, and that the tool - spans keep working once it does — see TestToolSpansSurviveTheTurnBoundary, - which is the half that carries the weight. + Rationale: .claude/notes/agents.md § Per-harness generation marks """ def _two_turns(self): @@ -1294,23 +1287,17 @@ def test_the_second_window_abuts_the_first(self): def test_the_inter_turn_gap_is_inside_a_window_rather_than_unaccounted(self): messages = self._two_turns() # 1000 -> 2000, which includes the 600ms between `turn_end` and the - # next `turn_start`. Untiled this reported 400ms and lost the 600. + # next `turn_start`. Untiled, this reads 400ms and loses the 600. assert messages[1].generation_duration_ms == pytest.approx(1000.0) class TestToolSpansSurviveTheTurnBoundary: """A tool that closes BETWEEN two turns still belongs to the next window. - This used to be a bookkeeping problem: a per-turn span list, cleared at - `turn_start` — after the window it feeds had already opened at the mark — - so a call closing in the gap had its span wiped before the flush could - subtract it. That list is gone. `timing.subtract_tool_time` sees - every span at once and clips each to the windows it overlaps, so the - property now holds by construction rather than by a reset rule. - - Kept, and re-pointed at the collector, because the property itself is what - matters and a future reducer change could still break it — by moving a - mark, or by failing to emit the ToolEnd the collector reduces. + `timing.subtract_tool_time` sees every span at once and clips each to the + windows it overlaps, so the property holds by construction. Hazard: a + reducer change can still break it — by moving a mark, or by failing to emit + the ToolEnd the collector reduces. """ def _run(self): @@ -1371,7 +1358,7 @@ def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self): `generation_duration_ms` and both bounds to a placeholder, so a snapshot records that a window was measured and never what it measured. Its identity check (`_scrub.py`) is an upper bound besides, so - under-accounting — the defect this phase fixes — passes it silently. + under-accounting passes it silently. `scripts/timing/decompose_run.py --max-residual-pct` is the two-sided check on live runs; this is the committed one. """ @@ -1388,15 +1375,13 @@ def test_the_four_bucket_identity_closes_exactly_across_the_boundary(self): def test_a_duplicate_turn_end_does_not_republish_the_previous_window(self): """A spent `turn_started_at` must not seed the next window. - `close_window`'s `min(mark, item_start)` pulls the window open to cover - the item's own start. That is the backwards-clock defence, but a start - stamp left in place after its turn was published is not a backwards - clock — it is a stale value BEFORE the mark, so the guard reopens the - next window at the previous turn's start and publishes that whole span - again. Reproduced before the fix: 3000 ms of generation for a 2000 ms - turn. This reducer promises to survive a malformed stream, and Pi's CLI - retries internally, so a duplicate or replayed `turn_end` is a transport - hiccup rather than a hypothetical. + Pins: a second `turn_end` with no intervening `turn_start` opens its + window at the first turn's end, and the two windows sum to the 2000 ms + wall clock. Hazard: `close_window`'s `min(mark, item_start)` reopens a + window at any start stamp left before the mark, so the reducer must + clear its start once the turn is published. + + Rationale: .claude/notes/agents.md § Per-harness generation marks """ clock = _SteppedClock() state = _PiTurnState(task_id="t", iteration=1, user_input="go", model="m", clock=clock) diff --git a/tests/test_pr_review_workflow.py b/tests/test_pr_review_workflow.py index c791d9231..218aee489 100644 --- a/tests/test_pr_review_workflow.py +++ b/tests/test_pr_review_workflow.py @@ -1,21 +1,15 @@ """Mechanical guards for the untrusted-input ``claude-pr-review`` workflow. -This workflow runs privileged (write-scoped ``GITHUB_TOKEN``, repo secrets) over -attacker-controlled PR content, so three invariants are locked in here so a -future edit fails loudly instead of silently regressing the hardening: - -1. ``include_comments_by_actor`` stays in sync with the CODEOWNERS ``*`` owners — - the comment-source allowlist is a hand-maintained duplicate of that list, and - drift silently drops a maintainer's review guidance from Claude's context. -2. The ``--allowedTools`` list contains no tool that can read on-disk secrets, - re-ingest unfiltered untrusted content, or reach the network — e.g. a shell - ``cat`` (reads a persisted ``.git/config`` token) or ``gh pr view`` (reads - every comment verbatim, bypassing the actor allowlist). -3. ``persist-credentials: false`` removes the on-disk token the action's own - ``git fetch`` needs, so auth is restored via an env-only credential helper. - Both halves are locked in so the fetch path can't be re-broken (helper - dropped → "could not read Username") nor the token re-persisted to disk (a - literal ``secrets.*`` baked into the helper instead of an env reference). +The workflow runs with a write-scoped ``GITHUB_TOKEN`` and repo secrets over +attacker-controlled PR content. Pins three invariants: + +1. ``include_comments_by_actor`` equals the CODEOWNERS ``*`` owners. +2. ``--allowedTools`` holds no tool that can read on-disk secrets, re-ingest + unfiltered untrusted content, or reach the network. +3. Checkout sets ``persist-credentials: false``, and ``git fetch`` authenticates + through an env-only credential helper with no literal ``secrets.*`` in it. + +Rationale: .claude/notes/reporting.md § The claude-pr-review hardening invariants """ from __future__ import annotations @@ -115,18 +109,15 @@ def test_checkout_does_not_persist_credentials() -> None: def test_git_fetch_auth_is_env_only() -> None: """The action's internal ``git fetch`` must authenticate via an env-only helper. - ``persist-credentials: false`` (above) removes the on-disk token that the - claude-code-action's ``git fetch origin `` relies on, so a step must - reconfigure git auth. This locks in the full linkage so a rename or reorder - can't silently re-break the "could not read Username" regression: - - * a run-step configures a git credential helper (host-scoped ``credential..helper``); - * the helper reads the token from an env var, never a baked-in ``secrets.*`` - literal (which would re-persist it to ~/.gitconfig and reopen the exfil surface); - * the action step supplies *that exact* env var, sourced from - ``secrets.GITHUB_TOKEN`` (not only the ``with.github_token`` input, which - octokit uses but the raw ``git fetch`` does not); - * the helper step runs *before* the action step (else the fetch precedes the config). + Pins the full linkage, so a rename or reorder fails here: + + * a run-step configures a host-scoped ``credential..helper``; + * the helper reads the token from an env var, never a ``secrets.*`` literal; + * the action step supplies *that exact* env var from ``secrets.GITHUB_TOKEN`` + (the ``with.github_token`` input alone does not reach the raw ``git fetch``); + * the helper step runs *before* the action step. + + Rationale: .claude/notes/reporting.md § The claude-pr-review hardening invariants """ workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8")) steps = workflow["jobs"]["claude-review"]["steps"] diff --git a/tests/test_prose_budget.py b/tests/test_prose_budget.py index eee26ff6f..ae78252aa 100644 --- a/tests/test_prose_budget.py +++ b/tests/test_prose_budget.py @@ -1,12 +1,13 @@ """Unit tests for the prose budget measurement (``tests/lint/prose_budget.py``). Every case runs against a synthetic tree in ``tmp_path`` or a plain string, never -against the real ``src/coder_eval`` — its word counts change with every prose commit, -so asserting on them here would make this file a second, drifting baseline. +against the real tree: its word counts change with every prose commit, so a test +that asserted on them would fail on unrelated edits. """ from __future__ import annotations +import subprocess import textwrap from pathlib import Path @@ -19,14 +20,39 @@ def _words(count: int) -> str: return " ".join(f"w{index}" for index in range(count)) -def _tree(tmp_path: Path, files: dict[str, str]) -> Path: +def _write(tmp_path: Path, files: dict[str, str]) -> Path: for rel, text in files.items(): - path = tmp_path / "src" / "coder_eval" / rel + path = tmp_path / rel path.parent.mkdir(parents=True, exist_ok=True) path.write_text(textwrap.dedent(text), encoding="utf-8") return tmp_path +def _tree(tmp_path: Path, files: dict[str, str]) -> Path: + (tmp_path / "tests").mkdir(exist_ok=True) + return _write(tmp_path, {f"src/coder_eval/{rel}": text for rel, text in files.items()}) + + +def _git(root: Path, *args: str) -> None: + subprocess.run( + [ + "git", + "-c", + "user.name=t", + "-c", + "user.email=t@t", + "-c", + "commit.gpgsign=false", + "-c", + "core.hooksPath=/dev/null", + *args, + ], + cwd=root, + check=True, + capture_output=True, + ) + + class TestDocstringMeasurement: def test_long_function_docstring_is_counted(self) -> None: prose = prose_budget.measure_source(f'def f():\n """{_words(200)}"""\n', "m.py") @@ -54,7 +80,7 @@ def test_module_docstring_is_named_module(self) -> None: class TestTyperExemption: def test_module_level_typer_command_is_exempt(self) -> None: source = f'def run_command():\n """{_words(400)}"""\n' - prose = prose_budget.measure_source(source, "cli/run_command.py") + prose = prose_budget.measure_source(source, "src/coder_eval/cli/run_command.py") assert prose is not None assert prose.docstring_words == 0 @@ -67,7 +93,7 @@ def test_same_name_as_a_method_elsewhere_is_counted(self) -> None: def test_same_name_nested_inside_the_exempt_module_is_counted(self) -> None: source = f'class Helper:\n def run_command(self):\n """{_words(400)}"""\n' - prose = prose_budget.measure_source(source, "cli/run_command.py") + prose = prose_budget.measure_source(source, "src/coder_eval/cli/run_command.py") assert prose is not None assert prose.docstring_words == 400 @@ -77,9 +103,9 @@ def test_every_exempt_pair_still_exists(self) -> None: failure CE057's membership test exists to catch.""" import ast - package = Path(__file__).resolve().parents[1] / "src" / "coder_eval" + repo_root = Path(__file__).resolve().parents[1] for rel, name in sorted(prose_budget._TYPER_COMMANDS): - path = package / rel + path = repo_root / rel assert path.is_file(), f"_TYPER_COMMANDS names {rel}, which does not exist" tree = ast.parse(path.read_text(encoding="utf-8")) top_level = {node.name for node in tree.body if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef)} @@ -130,8 +156,8 @@ def test_a_syntax_error_is_skipped_not_fatal(self, tmp_path: Path) -> None: }, ) measurement = prose_budget.measure(root) - assert measurement.skipped == [Path("broken.py")] - assert measurement.files[Path("good.py")].docstring_words == 200 + assert measurement.skipped == [Path("src/coder_eval/broken.py")] + assert measurement.files[Path("src/coder_eval/good.py")].docstring_words == 200 def test_total_words_sums_docstrings_and_comments(self, tmp_path: Path) -> None: root = _tree( @@ -145,7 +171,7 @@ def test_total_words_sums_docstrings_and_comments(self, tmp_path: Path) -> None: class TestCommentBudget: - """The per-file comment budget that replaced the hand-maintained baseline.""" + """The per-file total: a floor, then a share of the file's length.""" def test_the_floor_applies_to_a_small_file(self) -> None: assert prose_budget.comment_line_budget(10) == 20 @@ -174,6 +200,51 @@ def test_the_budget_shrinks_with_the_file(self) -> None: """The point of a ratio: deleting code takes its comment budget with it.""" assert prose_budget.comment_line_budget(2000) > prose_budget.comment_line_budget(1000) + def test_it_catches_what_the_run_cap_cannot(self, tmp_path: Path) -> None: + """Short blocks all the way down: every run passes, the file is still mostly prose.""" + body = ("# one\n# two\n# three\nx = 1\n") * 30 + root = _tree(tmp_path, {"a.py": body}) + assert prose_budget.check_comment_runs(root) == [] + assert len(prose_budget.check_comment_density(root)) == 1 + + +class TestCommentRuns: + """The comment-run cap: a bar on the BLOCK, with no per-file allowance.""" + + def test_a_run_at_the_bar_passes(self, tmp_path: Path) -> None: + root = _tree(tmp_path, {"a.py": "\n".join(["# pad"] * 8) + "\nx = 1\n"}) + assert prose_budget.check_comment_runs(root) == [] + + def test_a_run_over_the_bar_fails(self, tmp_path: Path) -> None: + root = _tree(tmp_path, {"a.py": "\n".join(["# pad"] * 9) + "\nx = 1\n"}) + failures = prose_budget.check_comment_runs(root) + assert len(failures) == 1 + assert "comment run of 9 lines" in failures[0] + + def test_many_short_runs_cost_nothing(self, tmp_path: Path) -> None: + """The inversion the ratio had: 200 one-line notes are not an essay.""" + root = _tree(tmp_path, {"a.py": "\n".join(["# note", "x = 1"] * 200) + "\n"}) + assert prose_budget.check_comment_runs(root) == [] + + def test_one_blank_line_does_not_split_a_paragraph(self, tmp_path: Path) -> None: + body = "\n".join(["# pad"] * 5) + "\n\n" + "\n".join(["# pad"] * 5) + "\nx = 1\n" + failures = prose_budget.check_comment_runs(_tree(tmp_path, {"a.py": body})) + assert len(failures) == 1 + assert "comment run of 10 lines" in failures[0] + + def test_two_blank_lines_are_a_section_break(self, tmp_path: Path) -> None: + body = "\n".join(["# pad"] * 5) + "\n\n\n" + "\n".join(["# pad"] * 5) + "\nx = 1\n" + assert prose_budget.check_comment_runs(_tree(tmp_path, {"a.py": body})) == [] + + def test_code_between_comments_ends_a_run(self, tmp_path: Path) -> None: + body = "\n".join(["# pad"] * 5) + "\nx = 1\n" + "\n".join(["# pad"] * 5) + "\ny = 2\n" + assert prose_budget.check_comment_runs(_tree(tmp_path, {"a.py": body})) == [] + + def test_a_trailing_comment_is_a_directive_not_commentary(self, tmp_path: Path) -> None: + """A stack of trailing `# noqa` is not a paragraph and starts no run.""" + root = _tree(tmp_path, {"a.py": "\n".join(["x = 1 # noqa: E501"] * 40) + "\n"}) + assert prose_budget.check_comment_runs(root) == [] + class TestProseWords: def test_an_args_block_does_not_count(self) -> None: @@ -337,7 +408,8 @@ def test_the_report_names_files_subsystems_the_total_and_the_essays(self, tmp_pa ) report = prose_budget.render_report(prose_budget.measure(root)) assert "a.py" in report - assert "top-level" in report and "agents" in report + assert "src/coder_eval/agents\n" in report + assert "src/coder_eval\n" in report assert "subtotal" in report assert "TOTAL 206" in report assert "ESSAYS" in report @@ -351,7 +423,7 @@ def test_the_report_names_files_subsystems_the_total_and_the_essays(self, tmp_pa class TestPointerPlacement: - """The guard promoted after three phases shipped this defect shape to review.""" + """A ``Rationale:`` pointer must be the last prose line of its block.""" def _root(self, tmp_path: Path, source: str) -> Path: root = _tree(tmp_path, {"a.py": source}) @@ -404,3 +476,140 @@ def test_a_file_with_no_pointer_is_not_flagged(self, tmp_path: Path) -> None: def test_a_syntax_error_is_skipped_not_fatal(self, tmp_path: Path) -> None: assert prose_budget.check_pointer_placement(self._root(tmp_path, "def f(:\n")) == [] + + +_ESSAY = f'def f():\n """{_words(200)}"""\n' +_DENSE = "\n".join(["# pad"] * 40) + "\nx = 1\n" +_UNRESOLVED = '"""Rationale: .claude/notes/absent.md § nowhere"""\n' + + +class TestRoots: + def test_a_missing_root_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("nope"),)) + with pytest.raises(FileNotFoundError, match="nope"): + prose_budget.measure(tmp_path) + + def test_a_missing_root_fails_the_gate_without_a_traceback( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("nope"),)) + assert prose_budget.main([]) == 1 + assert capsys.readouterr().err == "prose budget root does not exist: nope\n" + + def test_two_roots_are_both_measured(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = _write(tmp_path, {"src/coder_eval/a.py": _ESSAY, "tests/b.py": _ESSAY}) + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("src/coder_eval"), Path("tests"))) + assert set(prose_budget.measure(root).files) == {Path("src/coder_eval/a.py"), Path("tests/b.py")} + + def test_default_roots_include_tests(self, tmp_path: Path) -> None: + root = _write(tmp_path, {"src/coder_eval/a.py": _ESSAY, "tests/b.py": _ESSAY}) + assert set(prose_budget.measure(root).files) == {Path("src/coder_eval/a.py"), Path("tests/b.py")} + + def test_runs_and_essays_follow_roots(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = _write(tmp_path, {"src/coder_eval/a.py": "x = 1\n", "tests/c.py": _DENSE, "tests/e.py": _ESSAY}) + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("src/coder_eval"),)) + assert prose_budget.check_comment_runs(root) == [] + assert prose_budget.check_comment_density(root) == [] + assert prose_budget.check_essays(root) == [] + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("tests"),)) + runs = prose_budget.check_comment_runs(root) + assert len(runs) == 1 + assert runs[0].startswith("tests/c.py:1: ") + density = prose_budget.check_comment_density(root) + assert len(density) == 1 + assert density[0].startswith("tests/c.py: ") + essays = prose_budget.check_essays(root) + assert len(essays) == 1 + assert essays[0].startswith("tests/e.py::f: ") + + def test_pointer_checks_follow_roots(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + severed = "# One.\n# Rationale: .claude/notes/timing.md § close_window\n# severed tail.\nx = 1\n" + root = _write( + tmp_path, + { + "src/coder_eval/a.py": "x = 1\n", + "tests/d.py": _UNRESOLVED, + "tests/s.py": severed, + ".claude/notes/timing.md": "# Timing\n\n## close_window\n", + }, + ) + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("src/coder_eval"),)) + assert prose_budget.check_pointers(root) == [] + assert prose_budget.check_pointer_placement(root) == [] + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("tests"),)) + pointers = prose_budget.check_pointers(root) + assert len(pointers) == 1 + assert pointers[0].startswith("tests/d.py: ") + placement = prose_budget.check_pointer_placement(root) + assert len(placement) == 1 + assert placement[0].startswith("tests/s.py:") + + def test_subsystem_names_the_root(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = _write(tmp_path, {"tests/x.py": _ESSAY, "tests/lint/y.py": _ESSAY}) + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("tests"),)) + report = prose_budget.render_report(prose_budget.measure(root)) + assert "\ntests/lint\n" in report + assert "\ntests\n" in report + + +class TestCollectFailures: + def test_every_check_is_prefixed(self, tmp_path: Path) -> None: + misplaced = f"def g():\n {Q}Do it.\n\n Rationale: .claude/notes/absent.md § x\n tail.\n {Q}\n" + root = _tree(tmp_path, {"essay.py": _ESSAY, "dense.py": _DENSE, "misplaced.py": misplaced}) + prefixes = {failure.split(": ", 1)[0] for failure in prose_budget.collect_failures(root)} + assert prefixes == { + "unresolved pointer", + "misplaced pointer", + "comment run", + "comment budget", + "docstring essay", + } + + def test_a_clean_tree_has_no_failures(self, tmp_path: Path) -> None: + assert prose_budget.collect_failures(_tree(tmp_path, {"a.py": "x = 1\n"})) == [] + + +class TestAssertCodeUnchanged: + def _repo(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + root = _write(tmp_path, {"tests/x.py": 'def f():\n """One."""\n return 1\n'}) + _git(root, "init", "-q") + _git(root, "add", "-A") + _git(root, "commit", "-qm", "init") + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("tests"),)) + return root + + def test_a_test_file_code_change_is_reported(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = self._repo(tmp_path, monkeypatch) + _write(root, {"tests/x.py": 'def f():\n """One."""\n return 2\n'}) + findings = prose_budget.assert_code_unchanged(root, "HEAD") + assert len(findings) == 1 + assert findings[0].startswith("tests/x.py: code changed") + + def test_an_added_directive_comment_is_reported(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = self._repo(tmp_path, monkeypatch) + _write(root, {"tests/x.py": 'def f():\n """One."""\n return 1 # noqa: E501\n'}) + findings = prose_budget.assert_code_unchanged(root, "HEAD") + assert findings == ["tests/x.py: added directive comment '# noqa: E501' x1"] + + def test_a_new_untracked_file_with_code_is_reported(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = self._repo(tmp_path, monkeypatch) + _write(root, {"tests/new.py": "x = 1\n"}) + findings = prose_budget.assert_code_unchanged(root, "HEAD") + assert findings == ["tests/new.py: code changed (AST differs after stripping docstrings)"] + + def test_a_missing_root_raises(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = self._repo(tmp_path, monkeypatch) + monkeypatch.setattr(prose_budget, "_ROOTS", (Path("nope"),)) + with pytest.raises(FileNotFoundError, match="nope"): + prose_budget.assert_code_unchanged(root, "HEAD") + + def test_a_docstring_only_change_is_not_reported(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = self._repo(tmp_path, monkeypatch) + _write(root, {"tests/x.py": 'def f():\n """Two, longer."""\n return 1\n'}) + assert prose_budget.assert_code_unchanged(root, "HEAD") == [] + + +class TestMain: + def test_an_unknown_argument_is_a_usage_error(self, capsys: pytest.CaptureFixture[str]) -> None: + assert prose_budget.main(["--assert-code-unchanged-typo", "HEAD"]) == 2 + assert "usage:" in capsys.readouterr().err diff --git a/tests/test_reference_permissions.py b/tests/test_reference_permissions.py index 733889ee4..d49dc4ec5 100644 --- a/tests/test_reference_permissions.py +++ b/tests/test_reference_permissions.py @@ -32,18 +32,11 @@ from coder_eval.path_utils import digest_tree -# These tests drive `chmod` against the HOST filesystem. Windows `chmod` honours -# only the read-only bit, so mode 000 never takes and every assertion reads back -# 0o555/0o777. -# -# This is NOT a coverage gap for Windows users. The window is enforced only when -# CODER_EVAL_IN_CONTAINER=1, which only DockerRunner sets — and Docker Desktop on -# Windows runs LINUX containers (WSL2), so the in-container orchestrator that -# performs the chmod is on Linux and behaves exactly as these tests assert. A -# Windows host only ever sees the window under `driver: tempdir`, where it is a -# deliberate no-op regardless of platform. So there is nothing here for a Windows -# runner to exercise — the real behaviour is covered by the Linux CI jobs and by -# `tasks/anti_cheat_reference`, which runs in the container. +# These tests drive `chmod` against the HOST filesystem, and Windows `chmod` honours +# only the read-only bit, so mode 000 never takes. Not a coverage gap for Windows +# users: the window is enforced only in the container, which is Linux even under +# Docker Desktop on Windows. +# Rationale: .claude/notes/permissions.md § Why the chmod tests are Linux-only pytestmark = pytest.mark.skipif( sys.platform == "win32", reason="host-side POSIX mode semantics; the window runs in a Linux container on every host OS", @@ -196,8 +189,8 @@ def _boom(self, *a, **k): async def test_overlapping_windows_of_the_same_mode(self, guarded_dir): """Two windows applying the same mode: the inner exit must NOT restore. - This is what a refcount used to buy; the stack gives it for free, because - the inner pop re-applies the outer's (identical) mode. + The stack gives this for free: the inner pop re-applies the outer's + (identical) mode. """ original = _mode(guarded_dir) @@ -311,26 +304,24 @@ async def test_out_of_order_release_of_differing_modes(self, guarded_dir): assert _mode(guarded_dir) == original async def test_crash_handlers_install_from_the_event_loop_thread(self, guarded_dir, monkeypatch): - """The advertised crash-safety property: a killed run must not leave the - tree at mode 000. - - Asserted through the PUBLIC ``set_permissions`` entry point, not by - calling ``push`` directly. That distinction is the whole bug this test - exists for: installation used to happen inside ``push``, which only ever - runs on an ``asyncio.to_thread`` worker, where ``signal.signal`` raises - ``ValueError`` into a swallowing ``except`` — so SIGTERM had no restore - at all in production while a push-level test reported it installed. + """A killed run must not leave the tree at mode 000. + + Pins: opening a window through the PUBLIC ``set_permissions`` entry point + registers the atexit restore and installs SIGINT/SIGTERM handlers once. + Hazard: do not assert through ``push`` directly — it runs on an + ``asyncio.to_thread`` worker, where ``signal.signal`` raises ``ValueError``. + + Rationale: .claude/notes/permissions.md § Locking and crash safety """ registered: list[object] = [] installed_signals: list[int] = [] monkeypatch.setattr("coder_eval.fs_permissions.atexit.register", registered.append) def _fake_signal(signum, _handler): - # Reproduce the property that made the original bug invisible: the - # real signal.signal raises off the main thread. A permissive stub - # records an install that CPython would have refused, which is - # exactly how a push()-time install passed its own test while doing - # nothing in production. + # The real signal.signal raises off the main thread. A permissive + # stub would record an install that CPython refuses, so a + # push()-time install would pass this test while doing nothing in + # production. if threading.current_thread() is not threading.main_thread(): raise ValueError("signal only works in main thread of the main interpreter") installed_signals.append(signum) @@ -353,9 +344,8 @@ def _fake_signal(signum, _handler): async def test_install_failure_is_not_latched(self, guarded_dir, monkeypatch): """A failed install must be retried, not recorded as done. - ``_install_crash_handlers`` used to swallow the failure internally and - latch ``_handlers_installed = True`` regardless, so the one retry that - could have succeeded (from the main thread) never happened. + Pins: after a refused install, the next ``ensure_crash_handlers`` call + attempts both signals again, so a retry from the main thread can succeed. """ attempts: list[int] = [] @@ -366,13 +356,11 @@ def _refuse(signum, _handler): registry = _PermissionStack() # SCOPE the patches, do not lift them by hand. `signal.signal` is patched - # process-wide, and the async teardown restores SIGINT by calling - # `signal.signal(SIGINT, default_int_handler)` — which hits `_refuse` and raises - # `ValueError` out of teardown, failing the test for something it does not test. - # An explicit `undo()` after the calls fixed the happy path only: if - # `ensure_crash_handlers` itself raised — the very regression this test exists to - # catch — the undo would be skipped and the teardown ValueError would MASK the - # real failure. A context manager restores on every exit path. + # process-wide, and the async teardown restores SIGINT through it, which hits + # `_refuse` and raises `ValueError` out of teardown. A manual `undo()` is skipped + # if `ensure_crash_handlers` itself raises (the regression this test catches), + # and the teardown ValueError then MASKS the real failure. A context manager + # restores on every exit path. with monkeypatch.context() as mp: mp.setattr("coder_eval.fs_permissions.atexit.register", lambda _fn: None) mp.setattr("coder_eval.fs_permissions.signal.signal", _refuse) @@ -432,10 +420,10 @@ async def test_signal_handler_restores_then_chains(self, guarded_dir, monkeypatc assert killed == [signal.SIGTERM] async def test_sig_ign_previous_is_not_re_raised(self, guarded_dir, monkeypatch): - """SIG_IGN is neither callable nor SIG_DFL — the one disposition the - original chain fell through entirely. Restoring and returning is correct; - killing the process would override a deliberate `signal.signal(SIGINT, - SIG_IGN)` by the embedding application.""" + """A SIG_IGN previous disposition restores and returns without a re-raise. + + SIG_IGN is neither callable nor SIG_DFL. Killing the process would override + a deliberate `signal.signal(SIGINT, SIG_IGN)` by the embedding application.""" monkeypatch.setattr("coder_eval.fs_permissions.atexit.register", lambda _fn: None) captured: dict[int, Any] = {} killed: list[int] = [] @@ -454,9 +442,10 @@ async def test_sig_ign_previous_is_not_re_raised(self, guarded_dir, monkeypatch) assert killed == [] async def test_failed_restore_keeps_the_entry_for_the_crash_path(self, guarded_dir, monkeypatch): - """``pop`` used to ``del`` the entry BEFORE the restoring chmod, so a - failed chmod stripped ``restore_all`` of the only record of the original - mode — turning a recoverable failure into a permanently-000 tree.""" + """A failed restoring chmod in ``pop`` keeps the registry entry. + + Pins: ``restore_all`` still holds the original mode and recovers the path, + instead of leaving a permanently-000 tree.""" registry = _PermissionStack() original = _mode(guarded_dir) assert registry.push(guarded_dir, RESTRICTED_MODE) is True diff --git a/tests/test_regrade.py b/tests/test_regrade.py index d1435dee6..d92a96f2b 100644 --- a/tests/test_regrade.py +++ b/tests/test_regrade.py @@ -327,22 +327,13 @@ def test_a_sandbox_path_outside_the_run_dir_is_refused(tmp_path: Path) -> None: default_workspace(run_dir, _result(sandbox_path=str(outside))) -# -------------------------------------------------------------------------- -# Grading a `driver: docker` row — inside a container of its own image -# -# Such a task's criteria address the IMAGE's paths and toolchain, so grading -# them on the host answers a different question. Demonstrated on -# `tasks/byod_smoke_test.yaml`, whose criterion is `test -f /opt/byod_marker` -# (baked into the BYOD image): the identical row scores SUCCESS 1.000 graded in -# a container and FAILURE 0.000 graded on the host. That is not a flaky -# difference — it is the host answering "is the marker on THIS machine", which -# nobody asked. -# -# So the docker row is now DISPATCHED to a grading container rather than -# refused. `--allow-host-grading` keeps its old meaning: grade here anyway (no -# docker available, or criteria known to be host-portable), and wear the -# `graded_on_host` stamp. -# -------------------------------------------------------------------------- +# Grading a `driver: docker` row — inside a container of its own image. Such a task's +# criteria address the IMAGE's paths and toolchain, so grading them on the host answers +# a different question: on `tasks/byod_smoke_test.yaml` the identical row scores SUCCESS +# 1.000 in a container and FAILURE 0.000 on the host. The row is therefore DISPATCHED to +# a grading container rather than refused, and `--allow-host-grading` keeps its old +# meaning: grade here anyway, and wear the `graded_on_host` stamp. +# Rationale: .claude/notes/isolation.md § Grading a docker row inside a container class _RunnerDouble: @@ -875,13 +866,11 @@ def test_with_no_exemption_the_baseline_command_is_scanned(self, monkeypatch: py class TestEvaluateDispatchesADockerRow: - """The reordering in `evaluate_command` is the fix; nothing pinned it. + """A docker row reaches the container dispatch from `evaluate`. - `delegates_to_regrade` exists solely because `grading_sandbox_config` -- - whose job is to REFUSE `driver: docker` -- was being called BEFORE the branch - that no longer needs it, so no docker row could ever reach the container - dispatch. Revert the hoist and every docker detached grade becomes a hard - refusal again, with the suite still green. + Hazard: `grading_sandbox_config` REFUSES `driver: docker`, so `evaluate_command` + must take the in-place `regrade_in_place` branch BEFORE building a sandbox + config. Build it first and every docker detached grade becomes a hard refusal. """ @staticmethod @@ -953,9 +942,8 @@ class TestContainerFailureKeepsItsEvidence: `_grade_in_container` runs the whole dispatch inside a `TemporaryDirectory`, and every DockerRunner diagnostic — the container's merged stdout+stderr, the captured build log, the in-container FATAL guards — is written into it. - Folding out only on SUCCESS destroyed precisely the evidence, while - DockerRunError's own text says `See {log_path} for container output`, naming - a path that no longer existed by the time it was printed. + Pins: the log is folded out of it on failure as well as on success, and the + error names the rescued path, which still exists when it is printed. """ async def test_the_container_log_survives_a_failed_grade( diff --git a/tests/test_reports_junit.py b/tests/test_reports_junit.py index 3d47ce425..5d0f6bd0d 100644 --- a/tests/test_reports_junit.py +++ b/tests/test_reports_junit.py @@ -727,7 +727,7 @@ def test_parity_real_producer_output_through_writer(write_run_json: Callable[... (``run_record.eval_result_to_task_dict``, the batch.py path) so a producer-side rename of ``status`` / ``task_path`` / ``total_cost_usd`` / ``model_used`` / ``total_tokens`` / ``visible_turns`` / ``weighted_score`` - (RunSummary.task_results is an untyped ``list[dict[str, Any]]``) can no longer + (RunSummary.task_results is an untyped ``list[dict[str, Any]]``) cannot silently drop a property/classname or mis-bucket a row with zero failing assertions. """ diff --git a/tests/test_run_helpers.py b/tests/test_run_helpers.py index cc43f35a8..19fe8860b 100644 --- a/tests/test_run_helpers.py +++ b/tests/test_run_helpers.py @@ -30,8 +30,8 @@ def _names(paths): class TestRecursiveGlob: - # The published snippets all use `**`, and the lint rule that used to ban it - # was removed in favour of this promise. A switch to `glob.glob`, which is + # The published snippets all use `**`, and this test is the promise that it + # recurses. A switch to `glob.glob`, which is # non-recursive by default, would silently drop the top-level task. def test_double_star_matches_both_depths(self, tasks_tree): assert _names(expand_task_files([Path("tasks/**/*.yaml")])) == [ diff --git a/tests/test_run_limits_resolver.py b/tests/test_run_limits_resolver.py index 93d7828c7..59debf6d7 100644 --- a/tests/test_run_limits_resolver.py +++ b/tests/test_run_limits_resolver.py @@ -131,8 +131,9 @@ def test_variant_unset_does_not_clear_task_block(self): assert resolved.run_limits.max_usd == 5.0 def test_legacy_agent_max_turns_now_rejected(self): - """Legacy variant.agent.max_turns is no longer hoisted — it fails loudly via - the agent model's extra='forbid' (the hoist shim was removed).""" + """A variant `agent.max_turns` is rejected by the agent model's extra='forbid'. + + It is not hoisted into `run_limits`.""" default_exp = _default_exp() task = _make_task() exp = ExperimentDefinition( diff --git a/tests/test_run_metrics.py b/tests/test_run_metrics.py index 6866a87f4..d52643249 100644 --- a/tests/test_run_metrics.py +++ b/tests/test_run_metrics.py @@ -1,18 +1,14 @@ """Run-level derived metrics: one pass-rate denominator, and honest cost totals. -Two bugs are pinned here. - -**The denominator.** ``pass_rate`` used to be ``succeeded / (run - error)``, which -paid a bonus for erroring: the more a run fell over, the smaller its denominator -got, up to the degenerate case of a run rendering as a perfect score while passing -a handful of rows. Every surface now divides by ``tasks_graded`` — every -dispatched task except the ones that were never measured at all (``coder-eval +**The denominator.** Every surface divides ``pass_rate`` by ``tasks_graded`` — +every dispatched task except the ones never measured at all (``coder-eval execute`` leaves rows ``NOT_GRADED``, and those leave BOTH sides of the rate). +Hazard: a ``run - error`` denominator pays a bonus for erroring, since the more a +run falls over, the smaller it gets. -**The bill.** Cost was summed over whatever rows happened to carry one, so a run -whose model was missing from the rate card, or whose turns were killed before the -backend reported a cost, understated its spend silently. Unpriced spend is now -counted and the total is labelled a floor. +**The bill.** Spend on a model missing from the rate card, or on turns killed +before the backend reported a cost, is counted as unpriced, and the total is +labelled a floor. """ from __future__ import annotations diff --git a/tests/test_sandbox.py b/tests/test_sandbox.py index f1f2175f7..62b3bfc69 100644 --- a/tests/test_sandbox.py +++ b/tests/test_sandbox.py @@ -1094,9 +1094,8 @@ def test_check_parent_node_modules_contamination_reports_scoped_offender(tmp_pat def test_check_parent_node_modules_contamination_reports_unscoped_offender(tmp_path, caplog): """An ancestor node_modules/ holding any installed package is reported. - Locks in the generic check: previously the helper only caught - ``@uipath/*`` and would have missed unscoped packages, leaving - coder_eval consumers in other ecosystems uncovered. + Pins: the check is generic, not limited to ``@uipath/*``, so an unscoped + package is reported and consumers in other ecosystems are covered. """ parent = tmp_path / "fake-home" contam = parent / "node_modules" / "lodash" @@ -1320,16 +1319,13 @@ def test_absent_when_the_task_declares_no_reference(self, tmp_path): def test_default_venv_can_import_system_site_packages(): """The sandbox venv must not shadow the interpreter it is layered over. - `SandboxConfig.python` defaults to a `PythonEnvConfig()` instance, so every - task gets a venv. Built ISOLATED, that venv split the toolchain inside a task - image that provisions packages globally: `python` resolved to the venv and - could not import them, while `pip` -- which `uv venv` never places in the venv - -- fell through to the image's global pip and reported them present. Measured - in a task image: `import langchain` raised ModuleNotFoundError while - `pip list` showed `langchain 1.3.14`. + Pins: the default `SandboxConfig` (whose `python` is a `PythonEnvConfig()` + instance, so every task gets a venv) writes `include-system-site-packages = + true` into `pyvenv.cfg`. Asserted through `pyvenv.cfg` rather than a live + import so the test holds on a host whose base interpreter has nothing + installed. - Asserted through `pyvenv.cfg` rather than a live import so the test is - hermetic: it holds on a host whose base interpreter has nothing installed. + Rationale: .claude/notes/isolation.md § Why the venv gets system site packages """ config = SandboxConfig(driver="tempdir") assert config.python is not None, "default is an instance, not None -- the case this guards" diff --git a/tests/test_sandbox_record_cli.py b/tests/test_sandbox_record_cli.py index c466b1f52..f778dcf67 100644 --- a/tests/test_sandbox_record_cli.py +++ b/tests/test_sandbox_record_cli.py @@ -500,11 +500,10 @@ def test_rendered_shim_does_not_execute_anything(self, spec): @pytest.mark.parametrize("module", SIDECAR_MODULES) def test_the_sidecar_does_not_execute_anything_either(self, module): - """Restores coverage the sidecar refactor silently dropped. + """The sidecar modules reach no subprocess either. - While `argv_match.py` was SPLICED into the shim, - `test_rendered_shim_does_not_execute_anything` scanned its body too. As a - separate file it is no longer in that scan, and CE057 cannot stand in: + `test_rendered_shim_does_not_execute_anything` scans only the rendered + shim, not the sidecar files, and CE057 cannot stand in: `os` is on its STDLIB_ALLOWED (the matcher genuinely needs it), so `os.system(...)` in the sidecar would pass lint, typecheck, and ship into every sandbox. "It stubs a tool; it does not proxy one" is a documented @@ -864,13 +863,12 @@ def test_the_sidecar_import_survives_pythonsafepath(self): sandbox.cleanup(preserve=False) def test_an_unevaluable_response_rule_is_rejected_at_load(self, monkeypatch): - """The authoring fault that `rule_error` used to escalate for, caught where - the agent cannot participate. + """An unevaluable response rule is rejected at load, where the agent cannot participate. - `cli_called` can only score a `rule_error` 0.0 -- the log is agent-writable, - so a fault there cannot be attributed to the task author. Attribution has to - happen before a sandbox exists, so `RecordedCli` runs the real matcher over - every rule at load time. + Pins: `RecordedCli` runs the real matcher over every rule at load time, so an + authoring fault is a validation error before a sandbox exists. + + Rationale: .claude/notes/contracts.md § The five refuse-to-score paths are uniform at a gating 0.0 """ from coder_eval.models import cli_match diff --git a/tests/test_sandbox_templates.py b/tests/test_sandbox_templates.py index 6c737292e..a337a81bc 100644 --- a/tests/test_sandbox_templates.py +++ b/tests/test_sandbox_templates.py @@ -180,14 +180,11 @@ def test_template_ignores_venv(self, tmp_path): # Verify main.py copied assert (sandbox_path / "main.py").exists() - # The .venv present here is the sandbox's own (setup creates one); - # assert the TEMPLATE's copy did not land, via a marker file the real - # venv can never contain. Unconditional on purpose: guarding this - # behind `if venv_bin.exists()` made the whole check vacuous the - # moment provisioning changed, which is exactly how it went - # unnoticed. The marker is also layout-independent -- asserting on - # `.venv/bin/python` would only work on POSIX, since a real venv puts - # its interpreter in `Scripts/python.exe` on Windows. + # The .venv here is the sandbox's own (setup creates one); assert the + # TEMPLATE's copy did not land, via a marker file the real venv never + # contains. Unconditional on purpose: an `if venv_bin.exists()` guard made + # the check vacuous, unnoticed, when provisioning changed. The marker is + # also layout-independent: `.venv/bin/python` exists only on POSIX. assert (sandbox_path / ".venv").exists() assert not (sandbox_path / ".venv" / "from_template.marker").exists() finally: diff --git a/tests/test_sandbox_venv_live.py b/tests/test_sandbox_venv_live.py index fd80ca8e0..23cad1df9 100644 --- a/tests/test_sandbox_venv_live.py +++ b/tests/test_sandbox_venv_live.py @@ -2,22 +2,12 @@ Gated: needs a real docker daemon and the `coder-eval-agent` base image. -This is the one thing the unit tests cannot prove. `tests/test_sandbox.py` asserts -the venv is created with system site packages by reading `pyvenv.cfg`, which is a -property of the flag, not of the outcome. The outcome only exists inside an image -that provisions packages GLOBALLY — the shape every task image has (the framework -image installs with `uv pip install --system`; skillsbench task images do -`RUN pip install ...`). There, an isolated sandbox venv split the toolchain: -`python` resolved to the venv and could not import the image's packages, while -`pip` fell through to the image's global pip and reported them present. - -Measured against this test's own scenario: - - main (isolated venv) python -c "import pydantic" -> exit 1 - with --system-site-packages python -c "import pydantic" -> exit 0 - -`pydantic` is a coder_eval runtime dependency, so the base image already has it -installed globally — no build and no network are needed to reproduce the shape. +Pins the outcome inside an image that installs packages globally: a criterion's +`python` imports the image's `pydantic`, and its `sys.prefix` is still the sandbox +venv. `pydantic` is a coder_eval runtime dependency, so the base image has it +globally with no build and no network. + +Rationale: .claude/notes/isolation.md § Why the venv gets system site packages """ from __future__ import annotations diff --git a/tests/test_seed_from_prior_result.py b/tests/test_seed_from_prior_result.py index 8edceecb6..9ea722d48 100644 --- a/tests/test_seed_from_prior_result.py +++ b/tests/test_seed_from_prior_result.py @@ -229,11 +229,14 @@ def test_grader_environment_is_kept_beside_the_run_s_not_over_it(tmp_path: Path) def test_the_evaluate_only_path_selects_the_same_gate_as_the_agent_path(tmp_path: Path) -> None: - """C1: gate selection is FIRED-ONLY, and a detached grade reaches the verdict - through the evaluate-only branch. That branch used to call - ``all_criteria_passed`` unconditionally, so re-grading an early-stopped run - applied the full-run strict-AND gate to a truncated trajectory and could flip - SUCCESS into FAILURE. Both paths must go through ``_select_gate``.""" + """C1: the evaluate-only and agent branches select the gate the same way. + + Pins: both call ``_select_gate`` and neither calls ``all_criteria_passed`` + inline, so re-grading an early-stopped run gates on the armed subset instead + of applying the full-run strict-AND gate to a truncated trajectory. + + Rationale: .claude/notes/orchestration.md § Gate selection is fired-only + """ import inspect source = inspect.getsource(Orchestrator._evaluation_loop) @@ -294,17 +297,13 @@ def test_grading_cannot_overturn_an_execution_fact() -> None: def test_max_turns_exhausted_is_not_an_execution_fact() -> None: """The one status that reads like an execution fact and is not one. - It is SUBORDINATE to the verdict: `run` returns SUCCESS for a max-turns - trajectory whose criteria pass and only falls through to this status when - they do not — which is why `_terminal_status` puts the `grade=False` arm - above it. The table said True while that method's docstring argued the - opposite, so a prior max-turns row re-graded through `evaluate` was written - back as MAX_TURNS_EXHAUSTED *holding weighted_score 1.000* and exited 1 — a - combination `run` can never produce for the same trajectory. - - Its own test, not a line in the loop above, because the two statements - ("grading may not launder a crash into a pass" and "grading decides this - one") are different contracts that happened to share a fixture. + Pins: `FinalStatus.MAX_TURNS_EXHAUSTED.is_execution_fact` is False, so a + detached grade decides it, and the fact survives on the + `EvaluationResult.max_turns_exhausted` field. Keep it out of the loop above: + "grading may not launder a crash into a pass" and "grading decides this one" + are different contracts. + + Rationale: .claude/notes/orchestration.md § The terminal-status chain """ assert not FinalStatus.MAX_TURNS_EXHAUSTED.is_execution_fact # The fact is not lost; it just lives somewhere a verdict cannot contradict. diff --git a/tests/test_streaming_events.py b/tests/test_streaming_events.py index 478fed533..d0b34a171 100644 --- a/tests/test_streaming_events.py +++ b/tests/test_streaming_events.py @@ -30,11 +30,7 @@ def _telemetry(tool_name: str = "Bash", tool_id: str = "tool_123", **params: obj def test_agent_start_event_creation(): - """AgentStartEvent stores iteration/prompt info and is a StreamEvent. - - (Migrated from the old TurnStartEvent, which used to mean the - orchestrator/agent-start boundary — that role is now AgentStartEvent.) - """ + """AgentStartEvent stores iteration/prompt info and is a StreamEvent.""" event = AgentStartEvent( task_id="test-task", iteration=1, diff --git a/tests/test_sub_agent_runner.py b/tests/test_sub_agent_runner.py index 4828323ea..a313726a6 100644 --- a/tests/test_sub_agent_runner.py +++ b/tests/test_sub_agent_runner.py @@ -617,10 +617,9 @@ async def test_runner_reference_dir_with_nested_underscore_reference_preserved( ) -> None: """A nested ``_reference/`` inside the user's reference dir is NOT stripped. - Regression for finding #9: the pre-fix code reused the sandbox-side - ``ignore_patterns`` (which includes ``_reference``) for the reference-side - copytree, so a customer who happened to have a nested ``_reference/`` subdir - in their reference bundle would silently lose it. + Pins: the reference-side copytree does not reuse the sandbox-side + ``ignore_patterns`` (which include ``_reference``), so a nested + ``_reference/`` subdir in a customer's reference bundle survives. """ (tmp_path / "Main.xaml").write_text("") diff --git a/tests/test_suite_rollup.py b/tests/test_suite_rollup.py index 9151b3b1c..719b5e675 100644 --- a/tests/test_suite_rollup.py +++ b/tests/test_suite_rollup.py @@ -518,10 +518,11 @@ def test_task_json_relpath_uses_replicate_index(self, tmp_path: Path) -> None: class TestStackedSameTypeAggregation: - """A task can stack multiple criteria of the SAME type (e.g. activation's - per-skill skill_triggered) and each gets its OWN across-row aggregate, sliced - by position and keyed by description — not one type-pooled number repeated per - instance (the pre-fix behavior). This is what makes per-skill recall real.""" + """Stacked criteria of the SAME type each get their OWN across-row aggregate. + + A task can stack them (e.g. activation's per-skill skill_triggered). Each + aggregate is sliced by position and keyed by description, not one type-pooled + number repeated per instance. This is what makes per-skill recall real.""" @staticmethod def _row(row_id: str, scores: tuple[float, float]) -> TaskResult: diff --git a/tests/test_tags.py b/tests/test_tags.py index e49f921cd..3faadfcad 100644 --- a/tests/test_tags.py +++ b/tests/test_tags.py @@ -276,19 +276,14 @@ def _entry(self, task: TaskDefinition): def test_dispatch_is_proved_against_a_real_shim_log(self): """The authoritative detector, checked against a log the SHIM actually wrote. - `cli_called` matches argv only, so it passes whether a rule answered or the - entry fallback did. `captured.txt` is transcribable: this YAML is serialised - to /work/input and mounted at /work/task_dir, both readable. And a codegen - regression that renders `RULES = []` raises nothing, so neither `rule_error` - nor `sidecar_error` is booked. Only the `"rule": N` key catches that. - - So this test does not compare the YAML against itself. It generates the - task's own record_cli entry, RUNS each stubbed command, and requires the - criteria's regexes to match the resulting real log lines -- because the - needle's exact spelling (`"rule": 0`, with the space) belongs to - `json.dumps`'s default separators in `invocation_log.record`, not to this - test. Switching the shim to compact separators would otherwise leave this - green while the blocking CI probe failed. + Pins: the task's own record_cli entry is generated, each stubbed command is + RUN, and every must-match `file_matches_regex` criterion on the log matches + the real log lines the shim wrote, including the `"rule": N` key. Hazard: + the needle's spelling (`"rule": 0`, with the space) belongs to `json.dumps`'s + default separators in `invocation_log.record`, so never compare the YAML + against itself. + + Rationale: .claude/notes/contracts.md § Why the record_cli probe reads a real shim log """ task = self._task() entry = self._entry(task) diff --git a/tests/test_task_definition_unknown_fields.py b/tests/test_task_definition_unknown_fields.py index 1d955208e..7fb034a19 100644 --- a/tests/test_task_definition_unknown_fields.py +++ b/tests/test_task_definition_unknown_fields.py @@ -2,8 +2,8 @@ ``TaskDefinition`` deliberately does NOT declare ``extra='forbid'`` (see the class docstring) because the downstream skills task repo carries a long tail -of stale top-level fields (``max_iterations``, ``llm_reviewer``, ``skip``, …) -that used to be silently dropped. Instead, ``_warn_on_unknown_fields`` logs a +of stale top-level fields (``max_iterations``, ``llm_reviewer``, …). +Instead of rejecting or silently dropping them, ``_warn_on_unknown_fields`` logs a ``DeprecationWarning`` per unknown key so authors see the typo at load time without blocking the run. """ diff --git a/tests/test_teardown_interrupt.py b/tests/test_teardown_interrupt.py index ff5920198..b4ba849d7 100644 --- a/tests/test_teardown_interrupt.py +++ b/tests/test_teardown_interrupt.py @@ -3,12 +3,12 @@ The task-timeout watchdog cancels the asyncio task via ``loop.call_soon_threadsafe(task.cancel)``, delivering a ``CancelledError`` at the next await. When that await is inside ``_run_post_run_commands`` (post-run -commands do awaited I/O), the cancellation used to land in ``run()``'s -``finally`` block and abort it wholesale — skipping ``_cleanup()`` (tempdir -leaked; workspace never preserved) AND ``_finalize_result()`` (task.json lost, -so the task silently vanished from the run). These tests pin that a post-run -interrupt — cancellation or a plain exception — still runs the full teardown -(cleanup + finalize) and then re-raises the original exception unchanged. +commands do awaited I/O), the cancellation lands in ``run()``'s ``finally`` +block. These tests pin that a post-run interrupt — cancellation or a plain +exception — still runs the full teardown (``_cleanup()`` + ``_finalize_result()``) +and then re-raises the original exception unchanged. + +Rationale: .claude/notes/orchestration.md § Teardown must be interrupt-proof """ import asyncio diff --git a/tests/test_telemetry.py b/tests/test_telemetry.py index 5cefa9d8c..1c380b417 100644 --- a/tests/test_telemetry.py +++ b/tests/test_telemetry.py @@ -105,9 +105,9 @@ def test_connection_string_resolves_from_each_documented_env_alias(monkeypatch, # The three documented env vars are the entire activation contract. Prove each # actually populates telemetry_connection_string through pydantic-settings' # real env resolution (fresh Settings, no .env), so dropping or renaming an - # AliasChoices entry can no longer silently disable telemetry with no test - # failure. (Other tests monkeypatch the already-resolved field, which can't - # catch a broken alias.) + # AliasChoices entry fails a test instead of silently disabling telemetry. + # (Other tests monkeypatch the already-resolved field, which can't catch a + # broken alias.) from coder_eval.config import Settings for v in ("TELEMETRY_CONNECTION_STRING", "APPLICATIONINSIGHTS_CONNECTION_STRING", "UIPATH_AI_CONNECTION_STRING"): diff --git a/tests/test_timeout_exceptions.py b/tests/test_timeout_exceptions.py index 428becd14..1d2d26532 100644 --- a/tests/test_timeout_exceptions.py +++ b/tests/test_timeout_exceptions.py @@ -89,7 +89,7 @@ def test_base_fields(self): class TestTurnTimeoutErrorNoPartialField: - """TurnTimeoutError no longer carries partial_turn_record.""" + """TurnTimeoutError has no partial_turn_record attribute.""" def test_constructs_without_partial_kwarg(self): err = TurnTimeoutError(30.0, iteration=1) diff --git a/tests/test_timing_close_window.py b/tests/test_timing_close_window.py index e69c11ba8..1539a7e5a 100644 --- a/tests/test_timing_close_window.py +++ b/tests/test_timing_close_window.py @@ -36,9 +36,8 @@ def _load_decompose_run(): class TestCloseWindow: """The RAW window: where it opens, where it ends, and the clamp. - The tool subtraction these cases used to cover moved to - `timing.py::subtract_tool_time`, where it happens once for all - five harnesses instead of five times in five reducers — see + The tool subtraction lives in `timing.py::subtract_tool_time`, once for + all five harnesses — see `tests/test_event_collector.py::TestSubtractToolTime`, which carries the union, grouping, clamping and non-mutation cases. What is left here is the part that is genuinely per-reducer: the mark. @@ -144,7 +143,7 @@ def test_one_wording_for_every_call_site(self): pin only whichever one it happened to call. What varies between sites is deliberate and only that: the field name, and which side is aware. Everything after that clause is the advice, and it must be identical or - the sites are no longer sharing a helper. + the sites do not share a helper. """ advice = set() for call in ( @@ -269,10 +268,10 @@ def test_a_fresh_clock_anchors_on_its_own_pair(self): Note what this deliberately does NOT assert: that two clocks report different times. They should AGREE, and closely, because both derive from the same monotonic source — re-anchoring exists to correct drift - against real wall time, not to introduce an offset. An earlier version - of this test asserted `second.now() != first.now()`; that passed only - on sub-microsecond skew between the two constructors' reads, so it was - flaky under load and asserted the opposite of the design. + against real wall time, not to introduce an offset. Asserting + `second.now() != first.now()` would pass only on sub-microsecond skew + between the two constructors' reads: flaky under load, and the opposite + of the design. """ import time as _time @@ -302,8 +301,8 @@ def _command(tool_id: str, started, completed) -> dict: def _turn(*, commands: list[dict], messages: list[dict] | None = None, duration_seconds: float = 3.0) -> dict: """A `task.json` turn dict that validates as a `TurnRecord`. - The two sensors no longer parse stamps out of a raw dict; they validate and - call the typed selector, so a fixture below the model's required fields + The two sensors validate and call the typed selector rather than parse + stamps out of a raw dict, so a fixture below the model's required fields would fail in validation rather than on the thing the test is about. """ return { @@ -320,23 +319,17 @@ class TestTheThreeToolUnionsAgree: """Three readers answer "how long did this turn's tools run". They must agree. * `timing.main_thread_tool_spans` + `union_ms` — the TYPED selector the - collector subtracts from its generation windows and measures the head and - tail against, and which now writes `TurnRecord.tool_union_ms`. + collector subtracts from its generation windows, measures the head and + tail against, and writes `TurnRecord.tool_union_ms` from. * `tests/_fixtures/golden_streams/_scrub.py::_tool_union_ms` — the golden corpus's identity check, which validates the dump and calls that selector. * `scripts/timing/decompose_run.py::_tool_ms` — the LIVE two-sided residual - gate, which `.github/workflows/pr-checks.yml` runs against a real run, and - which does the same. - - The three used to be three COPIES of the selection rule, and they agreed by - luck once at a real cost: the collector filtered its GENERATIONS to the main - thread and then passed EVERY command as a tool span. A child nests inside - the parent Agent call, whose interval the union already covers, so nothing - failed — but Codex's recovered child tools carry the CHILD's clock, so the - nesting is not guaranteed. Now there is ONE selector and two callers of it, - and what remains worth pinning is that neither sensor has grown a second - path back, and that each still computes its OWN union rather than reading - the producer's stored answer. + gate, which does the same. + + Pins: all three exclude a sub-agent's own tools, and each sensor computes + its OWN union rather than reading the producer's stored answer. + + Rationale: .claude/notes/timing.md § main_thread_tool_spans """ @staticmethod @@ -580,11 +573,11 @@ def test_a_union_disagreement_exits_one_even_with_no_gate_asked_for(self, tmp_pa assert main([self._record(tmp_path, stored=999.0), "--min-turn-ms", "0"]) == 1 def test_a_union_disagreement_exits_one_even_when_every_turn_is_too_short_to_gate(self, tmp_path): - """The arm-order regression, asserted directly. + """A union breach outranks the no-gateable-turns arm. The turn is excluded from the share columns and the gate, so the - no-gateable-turns arm fires — and used to return before the breach was - ever consulted. + no-gateable-turns arm fires — and it must not return before the breach + is consulted. """ main = _load_decompose_run().main assert main([self._record(tmp_path, stored=999.0), "--min-turn-ms", "999999999"]) == 1 diff --git a/tests/test_timing_identity_contract.py b/tests/test_timing_identity_contract.py index 382157324..1a479acc6 100644 --- a/tests/test_timing_identity_contract.py +++ b/tests/test_timing_identity_contract.py @@ -2,44 +2,17 @@ head + Σ generation + UNION(tool) + tail == the turn's own span -This is the committed MAGNITUDE sensor, and it exists because nothing else in -the suite is one: - -* the golden corpus masks ``generation_duration_ms``, both window bounds, both - ``execution_*_at`` stamps and both head/tail fields to a placeholder - (``_scrub.py::SCRUB_KEYS``), so a snapshot records that a window was measured - and never what it measured — a timing value can move by seconds with every - golden test still green; -* ``_scrub.py::assert_timing_captured``'s own identity check is ONE-SIDED - (``overshoot <= ...``), so an UNDERCOUNT — a bucket claiming less time than - it should, which is the defect class this whole area keeps producing — passes - it silently. It cannot be made two-sided either: the replays run in ~0.3 ms of - synthetic wall clock, where a relative bound is vacuous; -* ``scripts/timing/decompose_run.py --max-residual-pct`` IS two-sided, but needs - live ``task.json`` files. - -Magnitudes are only real where a scripted clock makes them real, so each case -drives the harness's own REDUCER with a clock it moves by hand, then feeds the -messages and commands it produced through a real ``EventCollector`` — the same -seam production uses to compute the head and the tail. Every number asserted is -therefore one the harness computed, against a span the test declared. - -Three clock-injection styles are needed, and all three already exist in the -per-harness suites (this module reuses their idiom rather than inventing a -fourth): - -* an injected ``TurnClock`` — pi, antigravity and claude-code take ``clock=`` - / build one through a patched ``TurnClock`` factory; -* a ``datetime`` SUBCLASS monkeypatched onto the module — opencode, which also - calls ``datetime.fromtimestamp`` through the same global (see - ``tests/test_opencode_agent.py``'s ``_SteppedClock`` for why a stub breaks); -* ``time.monotonic`` patched ON TOP of an injected clock — claude-code, whose - ``turn_start_time``, turn deadline and measured tool durations still read - ``time.monotonic()``, so scripting only the clock leaves the reducer - straddling a real clock and a scripted one. - -Codex is the fifth and takes its stamps from SDK epoch milliseconds rather than -from any host clock, so its case scripts those stamps directly. +Pins: the identity closes ms-exact on every built-in harness. Each case drives +the harness's own REDUCER with a hand-moved clock and reduces its output through +a real ``EventCollector``, so every asserted number is computed by the harness. + +HAZARD: a case must script every clock its reducer reads — an injected +``TurnClock`` (pi, antigravity, claude-code), a ``datetime`` subclass on the +module (opencode), ``time.monotonic`` on top of the injected clock +(claude-code), or SDK epoch-ms stamps (codex). A real clock left in makes the +case pass by accident. + +Rationale: .claude/notes/timing.md § Why the ms-exact identity contract exists """ from __future__ import annotations @@ -189,7 +162,7 @@ class _InjectedClock: Injected rather than monkeypatched: pi and antigravity derive every wall stamp from their per-turn clock, so patching the module's ``datetime`` - would no longer reach them and the case would quietly measure the real + would not reach them and the case would quietly measure the real clock and pass by accident. """ @@ -433,33 +406,15 @@ def _codex_turn() -> Turn: def _claude_turn(monkeypatch: pytest.MonkeyPatch) -> Turn: """A tool call between two emissions, with a real head and a real tail. - The clock is INJECTED, like pi's and antigravity's: every wall stamp this - reducer records now derives from the turn's ``TurnClock``, and a derived - stamp escapes a monkeypatched module ``datetime`` entirely — the case would - quietly measure the real clock and pass by accident. ``time.monotonic`` is - still patched off the same counter, because ``turn_start_time``, the - deadline and the tool call's own measured duration read it; leaving it real - leaves the reducer straddling a scripted clock and a live one, and the tool - span (a monotonic duration subtracted back off a clock reading) would be - nonsense. - - The first `message_start` re-seeds the window, so the CLI spawn and the - query build before it are head rather than msg0's generation. That a LATER - one must not re-seed is asserted directly in - `tests/test_agent_telemetry.py`; here it shows up as the windows still - tiling. - - Its windows TILE across the tool result, and this case only proved that by - accident until the reducer was fixed. The mark used to be reset when the - result arrived, so the interval between the emission that ISSUED the call - and the result landed in no bucket. Here that interval IS the tool's - execution exactly — the case scripts the result at the instant the tool - ends — so the tool bucket happened to claim the same milliseconds and the - identity closed anyway. On a real turn the two differ: a 21.5 ms `Write` - can be followed by a 2.5 s round trip, and 21% of the turn goes missing. - `test_a_slow_tool_result_round_trip_is_not_lost` is the case that - discriminates; this one deliberately keeps the coincident shape so the two - read as a pair. + Pins: the first `message_start` re-seeds the window, so the CLI spawn before + it is head. The result lands the instant the tool ends, so this cannot tell + a tiled window from one reset at the result; see + `test_a_slow_tool_result_round_trip_is_not_lost`. + + HAZARD: the clock is INJECTED and `time.monotonic` is patched off the same + counter; leave either real and the reducer straddles two clocks. + + Rationale: .claude/notes/timing.md § Why a coincident tool result cannot catch an un-tiled window """ from coder_eval.agents import claude_code_agent as claude_module from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeTurnState @@ -524,21 +479,14 @@ def _monotonic() -> float: def _claude_slow_result_turn(monkeypatch: pytest.MonkeyPatch) -> Turn: - """A FAST tool followed by a SLOW result round trip — the shape that hid a defect. - - ``_claude_turn`` above scripts the tool result at the instant the tool - finishes, so the un-tiled interval and the tool's own span were the same - milliseconds and the identity closed even while the mark was being reset. - Every live probe had the same blind spot from the other direction: three - concurrent ``sleep 3`` calls make the tool union so large that the round - trip rounds away (measured: 0.05% residual). - - Here the tool runs for 20 ms and its result takes 2000 ms to come back, - which is `tasks/dataset_example.yaml` — the task CI actually runs, where a - 21.5 ms ``Write`` met a 2511.7 ms round trip and 21% of the turn was - accounted to nothing. The identity closing here is the whole point: the - window after the result must tile from the previous emission, not open - when the result lands. + """A FAST tool followed by a SLOW result round trip. + + Pins: the window after a tool result tiles from the previous emission and + does not open when the result lands. The tool runs for 20 ms and its result + takes 2000 ms to come back, the shape traced off + `tasks/dataset_example.yaml`; the identity closing is the assertion. + + Rationale: .claude/notes/timing.md § Why a coincident tool result cannot catch an un-tiled window """ from coder_eval.agents import claude_code_agent as claude_module from coder_eval.agents.claude_code_agent import ClaudeCodeAgent, _ClaudeTurnState @@ -587,17 +535,13 @@ def _monotonic() -> float: message_id="m1", ) ) - # The tool itself is 20 ms. What follows is the shape a live turn actually - # has, traced off `tasks/dataset_example.yaml`: the SDK delivers TWO user - # messages, the second ~2 s after the first. The old code reset the mark on - # each, so the next window opened at the LAST one and that 2 s vanished. - # - # One user message is not enough to catch it, and that is exactly why this - # shipped: claude-code reconstructs `execution_started_at` by subtracting - # the measured duration from the resolve instant, so with a single message - # the discarded interval and the tool's own span are the SAME milliseconds - # — `subtract_tool_time` removes them either way and the identity closes - # with or without the bug. The second message is what separates them. + # The tool itself is 20 ms. What follows is the shape a live turn actually has, + # traced off `tasks/dataset_example.yaml`: the SDK delivers TWO user messages, the + # second ~2 s after the first. The old code reset the mark on each, so the next + # window opened at the LAST one and that 2 s vanished. One user message is not + # enough to catch it — with a single message the discarded interval and the tool's + # own span are the SAME milliseconds, so the identity closes either way. + # Rationale: .claude/notes/timing.md § subtract_tool_time clock.at_ms = 1000 state.on_user_message(UserMessage("c1", False, "written")) clock.at_ms = 3000 @@ -673,15 +617,14 @@ def test_every_built_in_harness_has_a_case(): def test_the_sensor_sees_a_window_that_stops_tiling(): """The gating mutation check, as a committed test rather than an attestation. - A window seeded from its own turn start instead of from the previous - flush's close is the defect pi shipped with, and the whole point of this - module is that the SUITE notices it rather than a reviewer reproducing it - by hand. The golden corpus cannot: it masks every value involved. + Pins: a pi window seeded from its own turn start, not from the previous + flush's close, fails ``assert_identity_closes`` and loses exactly the 600 ms + scripted between one ``turn_end`` and the next ``turn_start``. + + Assert the MAGNITUDE as well as the failure: "it raised" alone would also + pass if the mutation broke the case in some unrelated way. - Asserted on the MAGNITUDE as well as on the failure, because "it raised" - would also pass if the mutation broke the case in some unrelated way. The - 600 ms is the scripted gap between one ``turn_end`` and the next - ``turn_start`` — real model time, which untiling books to nothing. + Rationale: .claude/notes/timing.md § Why the ms-exact identity contract exists """ healthy = _pi_turn() mutated = _pi_turn(untile=True) diff --git a/tests/test_ungraded_reporting.py b/tests/test_ungraded_reporting.py index 93450ca9c..c232e558d 100644 --- a/tests/test_ungraded_reporting.py +++ b/tests/test_ungraded_reporting.py @@ -276,8 +276,10 @@ def test_summary_points_at_the_grading_form_that_keeps_the_trajectory(tmp_path: def test_summary_still_reports_a_ratio_for_an_empty_run(tmp_path: Path) -> None: - """Both counters are falsy for tasks_run == 0; independent `if`s would print - no Results line at all, where it previously printed 0/0.""" + """An empty run still prints a `0/0 succeeded` Results line. + + Both counters are falsy for tasks_run == 0, so independent `if`s would print + no Results line at all.""" empty = _summary(tasks_run=0, tasks_succeeded=0, tasks_not_graded=0) assert "0/0 succeeded" in _summary_output(empty, tmp_path) diff --git a/tests/test_verify_published_workflow.py b/tests/test_verify_published_workflow.py index d3f25b4cf..6b7dd7d35 100644 --- a/tests/test_verify_published_workflow.py +++ b/tests/test_verify_published_workflow.py @@ -1,25 +1,19 @@ """``verify-published-action.yml`` couples to things nothing else asserts. -The workflow cannot be exercised before merge — ``workflow_run`` and ``schedule`` only -fire from the default branch — so every coupling it makes to another file is a place -where a rename passes ``make verify`` green and the gate silently rots in production. -Four such couplings, each with an executable binding here: - -1. **``workflow_run: workflows: ["Release"]``** matches ``release.yml``'s ``name:`` by - display string. GitHub does not error on an unmatched name; the trigger simply never - fires, degrading the gate to schedule-only with no signal. -2. **The Marketplace slug** is derived by a shell pipeline, a *second* slugger next to - the tested ``tests/lint/action_docs.py::marketplace_slug`` that CE026 uses for the doc - links. They agree today only because ``action.yml``'s ``name:`` is ``coder_eval`` — the - one input for which both are the identity function. -3. **The ``# <-- kept in sync`` pin anchor** now has three readers with three different - whitespace tolerances (``release.yml``'s sed, this workflow's sed, and - ``tests/test_action_version_pin.py``). A reformat can leave one reporting "parity OK" - on a pin another silently refused to bump. -4. **The inline consumer task YAML** is a whole ``TaskDefinition`` document that no test - validates, while CE029 already validates that exact shape in Markdown. Any field - rename (or an ``extra="forbid"`` violation) would surface only as an opaque failure in - the paid nightly. +``workflow_run`` and ``schedule`` fire only from the default branch, so this module +pins each of the workflow's cross-file couplings before merge: + +1. ``workflow_run: workflows: ["Release"]`` names ``release.yml``'s ``name:``. +2. The workflow's shell slug pipeline agrees with + ``tests/lint/action_docs.py::marketplace_slug`` (CE026). +3. ``release.yml``'s sed, this workflow's sed and ``tests/test_action_version_pin.py`` + read the same ``# <-- kept in sync`` pin from ``action.yml``. +4. The inline consumer task YAML loads and declares run limits. + +It also runs ``release.yml``'s promote guards and checks that the mechanics gate +classifies every ``FinalStatus``. + +Rationale: .claude/notes/reporting.md § Couplings of verify-published-action.yml """ from __future__ import annotations @@ -396,7 +390,6 @@ def test_monotonicity_guard_refuses_when_no_release_tag_exists(tmp_path: Path): # -------------------------------------------------------------------------------------- # 6. the mechanics gate reasons over FinalStatus values as bare strings -# # The gate's whole job is deciding which run.json `status` values mean "the published # action is broken". It spells them as string literals inside a heredoc, so a renamed # member, or a NEW member nobody classified here, is invisible to every other gate —