feat(lint): gate tests/ on the prose rules, and cap the comment run as well as the file total

[uipreliga]

Applies the prose rules this repo enforces on src/coder_eval to tests/, turns the gate on for it, and — with tests/lint/ finally in scope — fixes the one gate rule that turned out to be measuring the wrong thing. Follows #177, which did the same for src/, and is rebased onto #176.

Four things happen here.

1. The gate now scans tests/ as well as src/

tests/lint/prose_budget.py scans a _ROOTS tuple instead of one hard-coded root, reports repo-relative paths, and fails when a root does not exist — in the scan and in --assert-code-unchanged, so a renamed root can never pass on zero files. _python_files is the single traversal; main prints collect_failures(). Phase 8 flips the default to (src/coder_eval, tests).

That put the essay police on their own beat for the first time. 67 docstrings over 150 prose words and 5 files over their comment budget, cleared:

Phase	Scope	Result
3	30 rule docstrings in tests/lint/rules/	8,347 → 3,328 prose words
4	11 doc-surface lint tools + 4 TestCExxx classes	4,976 → 1,928 prose words
5	_scrub.py, _bracket_clock.py, opencode_fixtures.py	5 essays cleared
6	14 plain test modules	15 essays + 3 comment budgets
7	all of tests/	~120 HISTORY lines, 16 long test docstrings

Across the branch, tests/lint/ goes from 18,265 to 11,668 docstring prose words and all of tests/ from 105,305 to 95,722.

Each docstring keeps its contract and hazards; the defect story moves behind a resolving Rationale: <path> § <heading> pointer. Rule rationale now lives in the new .claude/notes/lint-rules.md (one section per CE id, split verbatim out of .claude/notes/README.md); test rationale goes to the subsystem note it belongs to.

Dangling citations fixed: harbor/packager.py and tests/test_harbor_packager.py cited a module docstring that no longer held the claim; tests/test_harbor_agent.py cited one that never did.

Prose-only, proved

Every prose phase passes --assert-code-unchanged, which compares the docstring-stripped AST and the directive comments against the base.

Gate holes the final review found, and closed here

Three ways the tool could pass without checking what it claims (each fixed test-first):

• assert_code_unchanged compared directive comments one way, so adding a # noqa passed a "prose-only" proof.
• main() ran the ordinary gate for any argv, so a mistyped --assert-code-unchanged exited 0 having proved nothing.
• The file list came from git diff --name-only, which never names an untracked file, so a new .py file full of code passed.

Two further gaps needed design work and are deferred in .claude/harness-candidates.md: narrative after an Args: block that follows a pointer, and pointer targets outside .claude/notes/.

2. Rebased onto #176, which brought five new essays

396c22cc (the reports split) landed three new modules and reworded two more, all over the 150-word bar once this branch's gate sees them:

tests/lint/rules/_layers.py                       432w -> 137w
tests/lint/rules/ce066_no_report_imports_in_core   313w -> 121w
tests/lint/pricing_mirror.py                       311w -> 143w
tests/test_custom_lint.py::TestRuffExternal...     246w ->  93w
tests/lint/rules/no_cli_imports_in_core.py         235w -> 117w

New notes sections carry what moved: CE004, CE065, CE066, _layers, TestRuffExternalCoversEveryRule. Three merge conflicts resolved: ce053's docstring (kept the slimmed side), harness-candidates.md (kept both appends) and the Makefile (kept pricing-mirror and the new docs-budget help text).

3. CE023 deleted — it guarded a package that no longer exists

src/coder_eval/proxy/ is gone; CHANGELOG.md:2487 records deleting the coder_eval.proxy.pricing shim. The rule banning imports of it stayed behind, so it could not fire, and its docstring still described the shim as present.

Removed from all five places a rule lives: the module, its runner.py import and ALL_RULES entry, its test class, and its id in [tool.ruff.lint] external. The id stays retired, as runner.py already requires for CE062 — both are now recorded in .claude/notes/lint-rules.md § The CE id space, which replaces the 14-line comment that used to carry it.

46 → 45 registered rules.

4. The comment budget was inverted, so the rule is now the comment RUN

With tests/lint/ in scope I measured what MAX(20, 0.15 × lines) on a file's total comment lines was actually doing. It was backwards:

file	shape	old rule
isolation/docker_runner.py	229 comments, 62 runs of 1–2 lines, longest run 6	229/229 — blocked
tests/test_regrade.py	one 16-line paragraph	29% of budget — fine
src/coder_eval/criteria/base.py	one 13-line paragraph	31% of budget — fine

It taxed short annotations pinned to the line they explain, and waved through the paragraphs that .claude/notes/ exists to hold. Four files had settled at exactly 100.0% of budget (docker_runner.py 229/229, criteria/agent_judge.py 65/65, evaluation/sub_agent.py 34/34, config.py 33/33) — nobody lands on 229/229 by accident; the budget had become a target.

So a third rule is added, and it is the primary one:

No own-line comment run may exceed 8 lines.

A run reads through one blank line, so splitting a paragraph does not duck it, and through none at two, which is the separation PEP 8 already puts between a banner and its section. It caps a shape, with no per-file allowance: carry as many one-line notes as earn their place.

Measured before choosing 8: 78.5% of runs in the tree are 1–2 lines, 2.5% are 6 or more, and the longest was 16. The cap is below the current maximum, as a cap rule must be. 24 runs across 19 files were shortened or moved to .claude/notes/ behind a pointer — 6 in src/, 18 in tests/.

The file total is kept as the backstop underneath, unchanged at MAX(20, 0.15 × lines): a file can pass the run cap with every block short and still be mostly commentary. test_it_catches_what_the_run_cap_cannot pins that seam — thirty three-line blocks pass the run cap and fail the total — so neither rule can later be deleted as redundant without a red test.

make docs-budget now enforces three rules on both roots: comment run ≤ 8 lines, file total ≤ MAX(20, 0.15 × lines), docstring ≤ 150 prose words.

Worth a human's eye

• The four files still at 100% of their total budget are frozen: a comment added to one has to be paid for by deleting another, or by raising the ratio. Recorded in .claude/notes/README.md with the measurements, so the next person to hit it sees the trade rather than bumping the number blind.
• src/ is touched, in 6 files — all of them one over-long comment run each, moved to notes behind a pointer. No behaviour change.
• .claude/notes/README.md — docs: move design rationale out of src/ into .claude/notes, gate it, and slim CLAUDE.md #177 had rewritten that section to say no prose summary is kept there; this branch points it at lint-rules.md. The rule file owns invariant/scope/blind spots, lint-rules.md owns the motivating defect, and the rule file wins on disagreement.
• Prose moved rather than shrank. tests/ loses ~12,700 docstring words and .claude/notes/ gains ~12,400. .claude/ has no prose budget, so the repo total is roughly flat; that is the design (rationale has one home) but it is worth stating plainly rather than claiming a reduction.

Verification

make verify: 5,816 passed, 2 skipped, 92.63% coverage, 671 custom lint rules — ruff format, ruff check, pyright, lint suite, prose gate, full test run. The gate reports 0 essays, 0 over-long runs and 0 files over budget with both roots in scope.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 31s —— View job

---

Code Review In Progress

Todo List:

• Read code review guidelines from .github/code_review.md
• Read project conventions from CLAUDE.md
• Get full PR diff with git diff origin/main...HEAD
• Review key changed files:
  • tests/lint/prose_budget.py (the gate implementation)
  • .claude/notes/lint-rules.md (new rule rationale file)
  • Sample rule files to verify prose cleanup
• Perform cross-file consistency checks
• Check "what's missing" analysis
• Verify prose-only proof claim
• Write comprehensive review feedback

Starting review now...