Conversation
Adds `tests/lint/prose_budget.py`: a stdlib-only measurement of the essay-shaped prose in `src/coder_eval` — docstrings over 150 words (Typer commands exempt) plus comment runs of 3+ lines — gated against a module baseline of 79,754 words. It also resolves every `Rationale: <path> § <heading>` pointer, and under `--assert-code-unchanged <ref>` proves a commit moved prose only, by comparing the docstring-stripped AST and the multiset of functional directive comments. The gate is wired in both seams: `make docs-budget` / `make verify`, and the `quality-gate` job in pr-checks.yml — CI restates every step and never invokes `make`, so the Makefile line alone would gate nothing on a PR. Relocates the 12 sections of `.claude/architecture-notes.md` verbatim into `.claude/notes/<subsystem>.md` and deletes it, so rationale has exactly one home and the tree has exactly one index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2
Cuts the four densest files in the tree from 6,874 essay words to 533,
leaving the caller-facing contract and the couplings a future editor
would break, and moving the design rationale to
`.claude/notes/{timing,permissions,persistence}.md`.
Kept in the source: the `timing.ts` parity claim and its shared fixture,
`TurnClock`'s not-for-deadlines hazard, the raw-total ordering hazard and
its division guard, the stacked-window contract, and every CE reference
whose text stayed. Deleted: history git already holds, and `TurnClock`'s
per-harness roster, which `docs/agents/HARNESS_PARITY.md` owns.
Also corrects a stale relocated claim: the notes said the task directory
was not shielded, but `Orchestrator._communicate_with_retry` chmods it
alongside the reference.
No executable statement changed — proved per file by
`prose_budget --assert-code-unchanged`, which compares the
docstring-stripped AST and the functional directive comments.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2
Cuts the five harness adapters and their eight helpers from 17,227 essay words to 4,456, and gives `.claude/notes/agents.md` the shared story the adapters kept telling five times over: the turn lifecycle, first-window seeding, per-harness generation marks, token accounting and cost resolution, why a clean exit can still be a crash, and how each CLI is reaped. Every per-harness parity claim either stays in the adapter or is already a row in docs/agents/HARNESS_PARITY.md, which the taxonomy makes the SSOT for cross-harness facts. Both SDK#24168 FIXMEs stay FIXMEs. No executable statement changed — proved per file by `prose_budget --assert-code-unchanged`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JcdMjPKFc2wdg4J6Ezg4E2
Cuts orchestrator.py and orchestration/ from 17,724 essay words to 3,980, extending the four sections Phase 1 relocated into orchestration.md rather than opening rival headings for the same topics: the terminal-status chain and the four grading sites under execute-vs-run, the fired-only gate and the ceiling/floor bounds under early stop. New sections cover what the orchestrator alone owns — recording the task as authored, the three separately-resolved routes, interrupt-proof teardown, restoring a PATH from an untrusted run directory, the dialog loop, embedded commands and experiment resolution. regrade's container rationale went to isolation.md, where detached grading already lives. `prose_budget` now resolves a pointer against `##` or `###`, so appending to an existing section does not force the pointer up to its parent. No executable statement changed — proved by `prose_budget --assert-code-unchanged`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13,289 words of essay-shaped prose across 11 files down to 2,960, with no code change (AST + directive multiset verified against the phase-start SHA). docker_runner.py 4,993 -> ~1,100 and sandbox.py 3,259 -> ~590 are the bulk. The repo's largest comment block, 34 lines on --cap-drop, is now the four COUNTERPART lines that state the coupling plus a pointer; the capability argument it carried is rationale and moved. isolation.md gains Capability drops and the anti-cheat window, What crosses into the container, Trusting what the container sends back, and The sandbox the criteria run in. Detached grading from the CLI joins the existing detached-grading section rather than rivalling it, and the in-container driver rewrite stays in orchestration.md, which already owned it. The Phase-1 relocated bullet covered nine of these topics already, so the single-home rule needed excision as well as writing: seven spans moved out of it, each left as a cross-reference to the section that now owns them. The --cap-drop/FOWNER known gap stays in docs/DOCKER_ISOLATION.md; the notes link it and say so. All eight --help outputs are byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…/notes 14,406 words across 34 files down to about 3,300, with no code change and no generated surface touched. Two constraints made this phase different from the others, and both were verified by parsing rather than by trusting a downstream check: the first non-blank line of every criterion class docstring, which CE033 copies into the plugin reference, is byte-identical; and no Field(description=...) string under models/ changed, which is what CE030 ripples into the task guide. The LiveVerdict determinism and monotonicity properties stay at the definition site in criteria/base.py, because they are the contract an author has to satisfy. What moved is the derivation: why CE036 replays trajectories, and what CE025 cannot see. contracts.md gains the live_verdict contract, the checker base class, route resolution, judge context and untrusted text, sub-agent judging, and recording a CLI invocation. persistence.md gains judge persistence; agents.md the sdk_options pass-through; orchestration.md the rates, the early-stop guardrail placement, and the armed gate. Docstrings that restated a Field description, or the task guide's own YAML examples, were deleted rather than moved — the guide is the source of truth. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…de/notes 10,234 words across 30 files down to about 2,900, finishing the plan. The tree goes from 79,754 essay-shaped words to 18,163, and from 126 docstrings over 150 words to one. That one is agent.py::communicate, the plugin SPI contract every third-party agent author reads: the Args/Returns/Raises, the rule that a mid-turn failure sets pending_turn before raising and the caller rolls the counter back, and the streaming event protocol. It is the exemption the plan predicted, and the only one taken, so the 150-word bar held for the whole tree. reporting.md gains the Agent ABC contract, telemetry emission, cost joining, report rollups and the HTML twin, Harbor export, and the ATIF trajectory bridge. Two hazards the plan named had to be ADDED rather than kept: neither the pricing.py/pricing.ts mirror nor the "evalboard's static twin" parity claim was ever stated in its own source file — both lived only in CLAUDE.md. Editing one side of either pair without the other is exactly what they exist to prevent, so they now say so where the editor will be. errors/categories.py is untouched: its per-member retryability notes are the contract errors/categorization.py dispatches on, and they are one-line comments the plan puts out of scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check_pointers proves a pointer RESOLVES. It cannot see the defect that actually shipped: a block replacement anchored on the wrong line leaves the tail of the replaced prose stranded after the pointer, where the $-anchored regex stops looking. The file parses, the pointer resolves, and the comment carries a severed half-sentence. Three phases of the prose-reduction work shipped that shape to review before this check existed; a fourth ran clean because the check was running from the first edit. It also reports an orphaned docstring terminator, which is worse than it looks: the file stops parsing, and the measurement silently reports it as zero words rather than failing. A docstring may still follow its pointer with Args:/Returns:/Raises: — that is the house shape, and the one case this must not flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Self-contained (inline CSS/JS, no external fonts or images)" is caller-facing — it is why the file can be opened offline and uploaded as a CI artifact — and the 7/7 rewrite dropped it while adding the static-twin claim. Both belong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fifth was promoted (b8b1dcf). These four are deferred with the reason, so the next person does not rediscover them: the general severed-fragment shape needs an allowlist to be usable, the single-home rule needs a similarity measure, a pointer landing unhelpfully is probably not mechanizable at all, and the generated-surface guard needs a commit-scoped diff the lint harness cannot see. The single-home one is the highest-value unbuilt guard in the notes design — it bit every phase of this run, once against a file the phase never opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit of the finished refactor found nine places where compression either dropped a contract or, worse, replaced it with a plausible wrong reason. The bad one is `_resolve_backend_route`. Its docstring said the function raises "so it must survive -O" — true of a different function, transplanted here — while the actual contract was gone from the source AND from the notes: an explicit backend override that cannot be honored must fail loudly, not degrade to a backend the task author never asked for. A reader told the raise is only about -O can reasonably "improve" it into a fallback, which is exactly the regression the deleted sentence forbade. The rest: - write_text_atomic claimed mode 0o644 as if guaranteed. It is a ceiling; the umask narrows it, and 0o077 yields 0600 — which breaks the docker driver, the failure the directive exists to prevent. - A NOTE in sandbox.py said three methods do not validate path traversal. Two of them go through resolve_files (and so through _within_sandbox and _reject_escaped); only list_files still does not. Someone reasoning about criterion containment from that comment reached the wrong conclusion. - enforces_permission_windows kept the clause making the window SAFE and lost the one making it REAL: without the container's DAC cap drops a mode-000 directory is still readable by root. - capture_to's copytree flags lost the reason one of them is load-bearing. - adopt no longer said criteria can still write to the adopted tree, under a docstring headlined "materializing nothing". - The docker log constants lost why there are two of them, and had no pointer. - armed_criteria_passed said "weighted score", which names a different quantity in the same class; the binarisation that makes gate_threshold=1.0 an exact equivalence was gone. - judge_context said "these tokens" with no list after it. - _build_run_command_env counted eight env facets and sets nine. The baseline rises 18,163 -> 18,549. That is the ratchet working as intended: restoring real contract is a legitimate reason to raise it, and the reason belongs here rather than in a silent constant bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review B found one shape repeated six times: compression kept a categorical claim and dropped the words that made it true. Each is the only fact its comment now carries, sitting above the code someone would edit. - claude_code_agent: the task-notification guard is checked SECOND, not first. What matters is that it precedes _is_sdk_result_message. - opencode_agent: "the condition is the TELEMETRY, not the event vocabulary" — but one of the two arms IS the vocabulary, and a later comment says so. The dropped word was "alone". Deleting the arm as redundant re-opens scoring SUCCESS 1.0 on zero turns. - batch: the predicate is NOT_GRADED **or** executed. "Executed is a required half" reads as `and`, which would leave zero-iteration execute rows ungraded forever. - reference_comparison: "every failure below raises" — three branches below return a gating 0.0 on purpose, one of them carrying a CE039 noqa. - docker_runner: the stdout limit does not mirror _POST_RUN_STREAM_LIMIT; it is 256x larger, deliberately. Unifying them downward reinstates the mid-stream teardown that lost whole paid tasks. - cli_called: "harness fault, not agent behaviour" contradicted the block 50 lines down explaining that all five paths are agent-reachable. Also restores the judge ignore floor's MECHANISM: it is a copy-time control, the same list passed as copytree's `ignore`, not only an SDK setting. Without that, dropping .claude / .mcp.json from it looks redundant and lets an agent-planted settings file into the judge's own working directory. _resolve_backend_route's contract now has a home in the notes, and the docstring points at it rather than at the LiteLLM section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--tmpfs` is never emitted. `_build_argv` does not construct one, `test_reference_inside_task_dir_needs_no_tmpfs_mask` asserts `not _tmpfs(argv)`, and docker_runner.py's own `_reference_mount_args` docstring opens "No tmpfs mask any more". Three of its neighbours said the opposite, two of them lines this refactor wrote. The claim predates the branch, so this is not a regression — but the refactor read every one of those lines and compressed them instead of questioning them, and it authored two new canonical homes for the falsehood. Re-ratifying it is worse than never having looked. What is actually true: the task dir is a shielded read-write COPY, so a reference embedded in it is covered by the same mode-000 window rather than hidden under a layered filesystem. `orchestration/evaluation.py`'s hard-fail was justified by the mask and by an EROFS on a `:ro` bind — both false now, and the real reason (the fallback resolves to a tree the window was not opened over) is the one that survives. Also restructures `permissions.md`'s Phase-1 blob. Its single 4,450-char line duplicated two of its own sibling sections, restated the FOWNER/CHOWN rationale that isolation.md explicitly delegates to docs/ saying "they are not restated here", and carried the tmpfs claim. All four inbound source pointers still resolve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ESSAY_BASELINE_WORDS was the weakest part of the gate. It was a single tree-wide number, hand-edited eleven times in this work alone, that a reviewer had to take on trust; it let growth in one file hide behind shrinkage in another; and it said nothing at all about a file that did not exist yet. It is gone. Two rules replace it, and neither has a number anyone maintains: own-line comments per file <= MAX(20, 0.15 * file length) no docstring over 150 words of PROSE The comment budget is proportional, so deleting code takes its budget with it and a new file is governed from its first commit. Own-line only: a trailing `# noqa` is a directive and a per-member annotation on an enum is the contract a dispatcher reads — counting either would push against documenting them. The floor is what protects a constants module at one comment per constant, which is where the tree's natural maximum sits. The essay rule now counts prose, not structure: an Args/Returns/Raises block is interface documentation, and counting it pushed exactly the docstrings that document their contract best over the line. Exemptions are categorical rather than numeric — an @AbstractMethod docstring IS the contract implementers read, so the plugin SPI is covered by kind, and a new abstract method is covered automatically. The old "at most two, by fiat" allowance is unnecessary: the tree now has ZERO essays. Getting there took 44 comment lines out of 7 files. Most came out by reflowing two lines into one or cutting a section divider; the one real compression was a 22-line block in early_stop.py restating the floor bound that orchestration.md already owns, and the distractor-exclusion rule it uniquely held moved there rather than being dropped. One reflow silently merged `# pyright: reportImportCycles=false` into the prose line above it, which would have stopped pyright honouring it. --assert-code-unchanged caught it. That is the second time the directive multiset has earned its place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uipreliga
requested review from
CarlesUIPath,
akshaylive,
bai-uipath and
tmatup
as code owners
September 15, 2026 05:46
|
Claude finished @uipreliga's task in 1m 18s —— View job
|
…nd clarifications Co-Authored-By: Claude <noreply@anthropic.com>
Open
5 tasks
bai-uipath
approved these changes
Sep 15, 2026
bai-uipath
left a comment
bai-uipath
left a comment
Collaborator
There was a problem hiding this comment.
Fix what you agree with, otherwise lgtm. The prose-only invariant holds: docstring-stripped ASTs are identical across all 99 changed Python files, directive comments are unchanged, Typer help text is byte-identical, and the judge's verdict schema is untouched. The pointer convention is real rather than decorative, with 363 pointers in src resolving to 83% of the note headings and none dangling.
Worth stating before the cuts: src prose drops from 77,954 words to 17,752, but the notes go from 9,356 to 41,391, so the new home is 4.4x the old one. The cuts below take roughly a quarter off that.
Cuts
- The CE lint-rule catalogue in the notes README. Around 2,165 words duplicating each rule's own module docstring, disclaimed as non-authoritative in the two sentences directly above it, and targeted by no pointer. Fix: delete the catalogue, keep the sentences that name where the authority lives.
- The per-file abstract lines. Seven single-line summaries, the largest 19,000 characters, restating the sections immediately beneath them: about 6,900 words, 17% of the notes. Fix: delete them, since the contents list and the headings already navigate.
- "Not wired up yet" in the permissions note. 769 words, nearly half that file, designing code that does not exist. Fix: open an issue and delete the section.
- The CLAUDE.md commit. It asserts that an adapter rejects an unsupported run-limit field at load time; the opencode and pi adapters both log a warning and continue, and the parity doc still records those divergences as table rows. Fix: revert that sentence. The communication-style and delete-before-you-guard edits in the same commit aren't about prose relocation either.
Worth fixing
- The gate penalizes usage examples. Args/Returns/Raises blocks are excluded from the word count but Example: blocks are not, so a call example counts against the bar. Two were deleted here, including the keyword contract for the retry helper. Fix: add Example:/Examples: to the excluded sections and restore them.
Minor: the Typer exemption set has no stale-membership test, unlike the other allowlists in this repo; and the abstract-method exemption is carrying one 315-word docstring worth trimming on its own merits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts 79,754 words of essay-shaped prose in
src/coder_evalto 17,752, and 126 docstrings over 150 words to zero, by moving design rationale into.claude/notes/and leaving a resolvable pointer at each source site.No code changed.
python -m tests.lint.prose_budget --assert-code-unchanged 9af3db3dcompares a docstring-strippedast.dumpplus a per-file multiset of functional directive comments (# noqa,# type: ignore,# pyright:,# nosec,# pragma:,# fmt:) and reports nothing across all 14 commits. All 116 directives intact.What moved, and what stayed
Every passage was classified: CONTRACT and HAZARD stay in source (compressed); RATIONALE moves to
.claude/notes/<subsystem>.md; HISTORY is deleted (git holds it); a CROSS-MODULE CLAIM is deleted and linked to its owner..claude/architecture-notes.mdis replaced by nine subsystem files under one index.The gate
make docs-budget, wired into bothmake verifyandpr-checks.yml'squality-gate(CI restates every step as.venv/bin/...and never callsmake, so both seams were needed). Four checks:Rationale: <path> § <heading>pointer resolves$-anchored regex cannot see itMAX(20, 0.15 × file length)The last two replaced a hand-maintained tree-wide word count. Nothing in the gate is now a number anyone maintains: delete code and the comment budget shrinks with it, and a new file is governed from its first commit. Exemptions are categorical — an
Args:/Returns:/Raises:block is structure not prose, and an@abstractmethoddocstring IS the interface contract implementers read.Reviewing this
The diff is large but uniform. The highest-value thing to check is whether any compressed docstring now asserts something false — that produced every real defect found here, and the worst case was a contract not dropped but replaced with a plausible wrong reason that invited the regression it originally forbade. Those are fixed in
0fe3e0a7,fd72e9aaandf7b5a193; the commit messages name each one.Three known items, all recorded rather than hidden:
isolation.mdstill carries a 19KB single-line bullet from Phase 1. A reviewer wanted it deleted as duplicated; measurement says later phases paraphrased rather than copied it, so deleting would lose detail. Left as a judgement call.tests/(105,417 lines) is unmeasured — the plan's own open question..claude/harness-candidates.md; the strongest is a single-home detector for the notes tree.Verification
5,733 tests / 2 skipped ·
make lint635 · pyright 0 · coverage 92.60% ·make evalboard-verifygreen ·make plugin-referenceandmake docs-indexesproduce no diff · all 8 public--helpoutputs byte-identical to before the work.🤖 Generated with Claude Code