From 0e9b39bca09f6e95dabf44380309779174ea8d9d Mon Sep 17 00:00:00 2001 From: RasputinKaiser <178525839+RasputinKaiser@users.noreply.github.com> Date: Mon, 20 Jul 2026 23:57:49 -0400 Subject: [PATCH] =?UTF-8?q?feat(score):=20recalibrate=20audit=20scorer=20?= =?UTF-8?q?=E2=80=94=20de-saturate=20&=20discriminate=20(U1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the bottom-compressed, saturated auto-scorer with a graduated, N/A-redistributing engine so scores discriminate across the quality range. - N/A redistribution: each dimension carries `applicable` + `ceiling`; a dimension that doesn't apply (no hooks, no skills) is DROPPED from both the numerator and denominator of the effective total. No more free 10 for a no-hooks plugin. - score_skill_quality: flat 0 -> graduated per-skill proxies (imperative body ~8, actionable specifics ~5, failure handling ~5; ~7 judgment residue). A good plugin now earns ~16; a poor one ~0. - score_context: budget-utilization gradient (full at <=50% of budget, linear decay to 0 at >=100%) for bodies (600w) and descriptions (400c) — a 599w body scores far below a 300w one. - score_trigger: NOT-clause credit now requires a redirect to an alternative ("not for X, use Y"), not a bare "not" token. - score_manifest: coarse 4/2/0 steps softened into finer gradients. - score_hooks: graduated when present, N/A (redistributed) when absent. - Effective total renormalized to /100 with a `grade` field (bands: 92+ Exceptional / 82 Strong / 68 Solid / 50 Needs work / <50 Poor). - Schema freeze respected: total.{auto,max}, dimensions[k].{auto,max, needs_judgment} preserved; only additive keys. --min-baseline now skips pre-recalibration baselines (scale change) instead of false-failing. - Selftest: 4 band fixtures (poor/fair/solid/excellent) + de-saturation invariants (hooks-absent dropped, skill_quality>0, lean>near-budget context, 92+ achievable). deterministic-scoring.md rewritten to match exactly. Co-Authored-By: Claude Opus 4.8 --- scripts/score.py | 636 +++++++++++++----- .../references/deterministic-scoring.md | 91 ++- 2 files changed, 529 insertions(+), 198 deletions(-) diff --git a/scripts/score.py b/scripts/score.py index d3709d9..5ee2cb3 100644 --- a/scripts/score.py +++ b/scripts/score.py @@ -5,15 +5,31 @@ python3 scripts/score.py # human table python3 scripts/score.py --json # structured object python3 scripts/score.py --md # markdown - python3 scripts/score.py --min N # exit 1 if total auto < N + python3 scripts/score.py --min N # exit 1 if effective total < N python3 scripts/score.py --min-baseline PATH # exit 1 on regression python3 scripts/score.py selftest # deterministic fixtures Scores the OBJECTIVE parts of the 100-pt rubric in skills/plugin-audit/references/scoring-rubric.md. For each dimension it emits -{auto, max, needs_judgment}: `auto` = points a machine can verify, `max` = the -dimension's rubric ceiling, `needs_judgment` = the sub-points only a human/LLM -can score. Judgment-heavy dimensions (skill quality) report auto≈0. +{auto, max, ceiling, applicable, needs_judgment}: `auto` = machine-verifiable +points earned, `max` = the dimension's nominal rubric ceiling (unchanged), +`ceiling` = the machine-achievable slice of that max (auto can never exceed it), +`applicable` = whether the dimension applies to this plugin at all. + +Calibration (2026 recalibration — de-saturate + discriminate): + * No free points. Every dimension awards graduated credit; validity earns the + MIDDLE of a sub-point, not its max. + * N/A -> redistribute. A dimension that genuinely does not apply (no hooks, no + skills) is DROPPED — excluded from both numerator and denominator — so its + weight spreads proportionally across the applicable dimensions. No dimension + ever hands out a free maximum for being absent. + * The effective total renormalizes the earned auto against the achievable auto + ceiling of the *applicable* dimensions, scaled to 100, so totals stay + comparable across plugins with different applicable sets. A perfect plugin + can reach 100; a mature-but-improvable one lands in the 70s-80s. + * Grade bands (on the effective total): + 92-100 Exceptional | 82-91 Strong | 68-81 Solid | + 50-67 Needs work | <50 Poor Self-contained: reuses the parsing patterns from scripts/validate.py but does not import it (a sibling may be editing it). @@ -28,10 +44,16 @@ SEMVER = re.compile(r"^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.\-]+)*$") # Quoted trigger phrases in a description: 'like this', "like this", `like this`. QUOTED = re.compile(r"'([^']{3,60})'|\"([^\"]{3,60})\"|`([^`]{3,60})`") -# NOT-clause heuristics — negative scope signals in a description. -NOT_CLAUSE = re.compile( - r"\bnot for\b|\bnot when\b|\bdon't use\b|\bdo not use\b|\bnot to\b|" - r"\brather than\b|\binstead of\b|\bnot\b.*\buse\b|\(not\b", +# NOT-clause with a genuine redirect to an alternative/sibling — merely containing +# the token "not" earns nothing; the description must point somewhere else. +NOT_ALT = re.compile( + r"\binstead of\b|" # "instead of X" + r"\brather than\b|" # "rather than X" + r"\buse\s+[`'\"]?[\w-]+[`'\"]?\s+instead\b|" # "use Y instead" + r"\bnot for\b[^.]{0,80}?\buse\b|" # "not for X ... use Y" + r"\bnot\b[^.]{0,80}?[\(;,—-]\s*use\b|" # "not X, use Y" / "not X (use Y)" + r"\bdon'?t use\b[^.]{0,60}?\buse\b|" # "don't use ... use Y" + r"\bdo not use\b[^.]{0,60}?\buse\b", re.I, ) WHEN_SIGNAL = re.compile(r"\buse when\b|\buse for\b|\bwhen asked\b|\btrigger\b|\bwhen a\b|\bwhen the\b", re.I) @@ -43,6 +65,61 @@ "plugins", "skill", "skills", "claude", "code", "codex", "harness", } +# Imperative verbs an agent-instruction body tends to open lines with. +IMPERATIVE_VERBS = { + "run", "read", "write", "create", "add", "use", "check", "verify", "edit", + "open", "load", "call", "score", "compute", "parse", "return", "emit", + "report", "list", "find", "set", "print", "build", "install", "fix", "avoid", + "ensure", "do", "make", "keep", "note", "pass", "save", "delete", "remove", + "update", "generate", "apply", "follow", "start", "stop", "copy", "move", + "rename", "replace", "insert", "append", "split", "join", "merge", "fetch", + "send", "click", "type", "select", "choose", "pick", "review", "audit", + "scan", "test", "validate", "confirm", "ask", "reject", "accept", "skip", + "include", "exclude", "define", "declare", "assert", "raise", "catch", + "handle", "render", "draw", "show", "hide", "enable", "disable", "count", + "sum", "map", "filter", "sort", "group", "trim", "strip", "format", + "normalize", "extract", "collect", "gather", "record", "log", "track", + "measure", "rank", "weight", "prefer", "treat", "name", "label", "mark", + "flag", "gate", "guard", "wrap", "identify", "determine", "decide", + "compare", "match", "search", "grep", "cd", "echo", "cat", "give", "grade", + "reason", "walk", "iterate", "loop", "collapse", "expand", "inspect", +} + +# Failure-handling signals in a body. +FAILURE_KEYWORDS = [ + "if missing", "if it fails", "if fails", "fallback", "falls back", "fall back", + "error", "ambiguous", "absent", "does not exist", "doesn't exist", "cannot", + "can't", "unavailable", "invalid", "retry", "when missing", "if absent", + "if unavailable", "if the file", "if no ", "not found", "fails", "failure", + "timeout", "times out", "crash", "exit 1", "unreadable", "when in doubt", + "if unclear", "edge case", "missing", "otherwise", "when none", "if empty", +] + +# Machine-achievable ceilings per dimension (sum of auto sub-point weights). +CEILINGS = { + "manifest_integrity": 15.0, + "skill_quality": 18.0, + "trigger_precision": 13.0, + "context_economy": 14.0, + "hooks_health": 5.0, + "distribution": 8.0, +} + +GRADE_BANDS = [ + (92.0, "Exceptional"), + (82.0, "Strong"), + (68.0, "Solid"), + (50.0, "Needs work"), + (0.0, "Poor"), +] + + +def grade_for(effective): + for floor, label in GRADE_BANDS: + if effective >= floor: + return label + return "Poor" + # ---------- parsing helpers (self-contained) ---------- @@ -136,6 +213,15 @@ def clamp(v, lo, hi): return max(lo, min(hi, v)) +def budget_fraction(used, budget): + """Utilization gradient: full credit at <=50% of budget, linear decay to 0 at + >=100%. A body/description that hugs the budget scores far below a lean one.""" + if budget <= 0: + return 0.0 + u = used / budget + return clamp(2.0 * (1.0 - u), 0.0, 1.0) + + # ---------- collision detection (inline, stdlib) ---------- def collision_count(skills): @@ -159,6 +245,56 @@ def collision_count(skills): return count, notes +# ---------- skill-quality proxies ---------- + +def _first_word(line): + s = re.sub(r"^[#>*\-\d.\)\(`\s]+", "", line.strip()) + m = re.match(r"[A-Za-z']+", s) + return m.group(0).lower() if m else "" + + +def skill_quality_components(body): + """Return (imperative<=8, actionable<=5, failure<=5) machine proxies for one + skill body. Averaged across skills upstream.""" + lines = [l for l in body.splitlines() if l.strip()] + if not lines: + return (0.0, 0.0, 0.0) + + # imperative body (8): verb-first density + step markers + verb-first heading. + firsts = [_first_word(l) for l in lines] + verb_lines = sum(1 for w in firsts if w in IMPERATIVE_VERBS) + density = verb_lines / len(lines) + imp = 5.0 * clamp(density / 0.35, 0.0, 1.0) + has_steps = bool(re.search(r"(?m)^\s*\d+[.)]\s", body)) or bool(re.search(r"\bstep\s*\d", body, re.I)) + imp += 2.0 if has_steps else 0.0 + heading_verb = any(_first_word(l) in IMPERATIVE_VERBS + for l in lines if l.lstrip().startswith("#")) + imp += 1.0 if heading_verb else 0.0 + imp = clamp(imp, 0.0, 8.0) + + # actionable specifics (5): code fences + inline code + file paths. + spec = 0.0 + fences = body.count("```") + spec += 2.0 if fences >= 2 else (1.0 if fences == 1 else 0.0) + inline = len(re.findall(r"`[^`\n]+`", body)) + spec += 1.5 * clamp(inline / 3.0, 0.0, 1.0) + # A real path reference — a known extension, a ./ ~/ ../ prefix, or a slash + # inside inline code. Deliberately NOT a bare `word/word` (excludes prose like + # "and/or", "read/write", "pass/fail"). + pathlike = bool(re.search(r"[\w./-]+\.(py|md|json|sh|js|ts|txt|ya?ml|toml)\b", body)) \ + or bool(re.search(r"(?:\.\.?/|~/)[\w./-]+", body)) \ + or bool(re.search(r"`[^`\n]*/[^`\n]*`", body)) + spec += 1.5 if pathlike else 0.0 + spec = clamp(spec, 0.0, 5.0) + + # failure handling (5): distinct failure-signal keywords. + low = body.lower() + fh_hits = sum(1 for kw in FAILURE_KEYWORDS if kw in low) + fail = 5.0 * clamp(fh_hits / 3.0, 0.0, 1.0) + + return (imp, spec, fail) + + # ---------- dimension scorers ---------- def score_manifest(root): @@ -166,29 +302,29 @@ def score_manifest(root): judgment, auto = [], 0.0 claude, codex = m.get("claude"), m.get("codex") present = {k: v for k, v in m.items() if v is not None} - # m keys == manifest files that exist on disk (value None means it failed to parse). files_present = len(m) corrupt = any(v is None for v in m.values()) - # 4pt — each present manifest valid JSON, kebab name, semver version. + # 4pt — validity, graduated: kebab name (2), semver version (1.5), description (0.5). if present and not corrupt: names = {v.get("name") for v in present.values()} - vers_ok = all(SEMVER.match(v.get("version", "") or "") for v in present.values()) name_ok = all(n and KEBAB.match(n) for n in names) - auto += 4 if (name_ok and vers_ok) else (2 if (name_ok or vers_ok) else 0) - judgment.append("description accuracy (4pt sub-point) is judgment") + vers_ok = all(SEMVER.match(v.get("version", "") or "") for v in present.values()) + desc_ok = all((v.get("description") or "").strip() for v in present.values()) + auto += 2.0 * name_ok + 1.5 * vers_ok + 0.5 * desc_ok + judgment.append("description accuracy (part of the 4pt validity sub-point) is judgment") - # 4pt — cross-harness parity: both manifests, identical name, agreeing version. + # 4pt — cross-harness parity: identical name (2) + agreeing version (2). if files_present == 2 and claude and codex: same_name = claude.get("name") == codex.get("name") same_ver = base_version(claude.get("version", "")) == base_version(codex.get("version", "")) - auto += 4 if (same_name and same_ver) else (2 if (same_name or same_ver) else 0) + auto += 2.0 * same_name + 2.0 * same_ver elif files_present == 1 and len(present) == 1: - auto += 4 # single-harness plugin earns full parity credit for its one manifest + auto += 4.0 # single-harness plugin earns full parity credit for its one manifest else: judgment.append("cross-harness parity unverifiable (a manifest missing or failed to parse)") - # 3pt — component pointers ./-prefixed and inside root. + # 3pt — component pointers ./-prefixed, in-root, resolvable (proportional). ptr_keys = ("skills", "hooks", "mcpServers", "apps") ptr_vals = [] for v in present.values(): @@ -196,52 +332,123 @@ def score_manifest(root): if k in v and isinstance(v[k], str): ptr_vals.append(v[k]) if ptr_vals: - good = all(p.startswith("./") and not p.startswith("../") - and (root / p.lstrip("./")).exists() for p in ptr_vals) - auto += 3 if good else 0 + good = sum(1 for p in ptr_vals + if p.startswith("./") and not p.startswith("../") + and (root / p.lstrip("./")).exists()) + auto += 3.0 * good / len(ptr_vals) else: - auto += 3 # no string pointers to get wrong - # 2pt — layout: manifest dir holds only plugin.json / marketplace.json. - layout_ok = True - for mdir in (".claude-plugin", ".codex-plugin"): - d = root / mdir - if d.is_dir(): + auto += 3.0 # no string pointers to get wrong + + # 2pt — layout: each manifest dir holds only plugin.json / marketplace.json (proportional). + mdirs = [root / d for d in (".claude-plugin", ".codex-plugin") if (root / d).is_dir()] + if mdirs: + clean = 0 + for d in mdirs: stray = [f.name for f in d.iterdir() if f.name not in ("plugin.json", "marketplace.json")] if stray: - layout_ok = False - auto += 2 if layout_ok else 0 + judgment.append(f"{d.name}/ holds stray files: {stray}") + else: + clean += 1 + auto += 2.0 * clean / len(mdirs) + else: + auto += 2.0 # 2pt — publisher metadata: author present (mechanical core). has_author = any(v.get("author") for v in present.values()) - auto += 2 if has_author else 0 + auto += 2.0 if has_author else 0.0 judgment.append("publisher-metadata appropriateness to distribution level (2pt) is judgment") - return {"auto": round(auto, 1), "max": 15, "needs_judgment": judgment} + return {"auto": round(auto, 1), "max": 15, "ceiling": CEILINGS["manifest_integrity"], + "applicable": True, "needs_judgment": judgment} -def score_context(root): + +def score_skill_quality(root): skills = load_skills(root) + if not skills: + return {"auto": 0.0, "max": 25, "ceiling": 0.0, "applicable": False, + "needs_judgment": ["no skills — dimension N/A, weight redistributed"]} + imp = spec = fail = 0.0 + weak = [] + for s in skills: + i, sp, fh = skill_quality_components(s["body"]) + imp += i + spec += sp + fail += fh + if (i + sp + fh) < 6.0: + weak.append(s["name"]) + n = len(skills) + auto = imp / n + spec / n + fail / n # each component averaged, ceiling 8+5+5=18 + judgment = [ + "one-job-per-skill / no two skills same job — judgment", + "imperative-body proxy scored (~8): verb-first density, step markers, verb headings", + "actionable-specifics proxy scored (~5): file paths, backticked commands, code fences", + "failure-handling proxy scored (~5): missing/fails/fallback/error/ambiguous/absent signals", + "remaining ~7pt (genuine one-job separation & instruction depth) is judgment residue", + ] + if weak: + judgment.append(f"skills scoring low on machine proxies (read the body): {weak}") + return {"auto": round(auto, 1), "max": 25, "ceiling": CEILINGS["skill_quality"], + "applicable": True, "needs_judgment": judgment} + + +def score_trigger(root): + skills = load_skills(root) + if not skills: + return {"auto": 0.0, "max": 20, "ceiling": 0.0, "applicable": False, + "needs_judgment": ["no skills — dimension N/A, weight redistributed"]} judgment, auto = [], 0.0 + n = len(skills) + + # 3pt (of 7) — description within budget AND carries a when/trigger signal or quoted phrase. + good_when = sum(1 for s in skills + if s["description"] and len(s["description"]) <= 400 + and (WHEN_SIGNAL.search(s["description"]) or quoted_phrases(s["description"]))) + auto += 3.0 * good_when / n + judgment.append("what+when completeness & trigger-phrase quality (remaining 4pt) is judgment") + + # 5pt — negative scope: NOT-clause that redirects to an alternative/sibling. + has_alt = sum(1 for s in skills if NOT_ALT.search(s["description"])) + auto += 5.0 * has_alt / n + if has_alt < n: + judgment.append(f"{n - has_alt}/{n} skills lack a redirecting NOT-clause (\"not for X, use Y\")") + + # 5pt — no trigger collisions (light sibling collision count). + count, notes = collision_count(skills) + auto += clamp(5.0 - count, 0.0, 5.0) + if count: + judgment.extend(notes) + + judgment.append("risky-skill guarding (allow_implicit_invocation / negative scope, 3pt) is judgment") + return {"auto": round(auto, 1), "max": 20, "ceiling": CEILINGS["trigger_precision"], + "applicable": True, "needs_judgment": judgment} + + +def score_context(root): + skills = load_skills(root) if not skills: - return {"auto": 0.0, "max": 20, "needs_judgment": ["no skills to score"]} + return {"auto": 0.0, "max": 20, "ceiling": 0.0, "applicable": False, + "needs_judgment": ["no skills — dimension N/A, weight redistributed"]} + judgment, auto = [], 0.0 + n = len(skills) - # 8pt — bodies within 600-word budget. - within = sum(1 for s in skills if len(s["body"].split()) <= 600) - over = [s["name"] for s in skills if len(s["body"].split()) > 600] - auto += 8 * within / len(skills) + # 8pt — body utilization gradient (600-word budget; lean scores far above near-budget). + body_frac = sum(budget_fraction(len(s["body"].split()), 600) for s in skills) / n + auto += 8.0 * body_frac + over = [f"{s['name']}={len(s['body'].split())}w" for s in skills if len(s["body"].split()) > 600] if over: - judgment.append(f"bodies over 600 words: {over}") + judgment.append(f"bodies over the 600-word budget: {over}") judgment.append("progressive disclosure (detail pushed to references/) is judgment") # 6pt — no duplicated content across skills — judgment. judgment.append("no duplicated content across skills (6pt) is judgment") - # 4pt — descriptions within 400-char budget. - d_within = sum(1 for s in skills if len(s["description"]) <= 400) - d_over = [s["name"] for s in skills if len(s["description"]) > 400] - auto += 4 * d_within / len(skills) + # 4pt — description utilization gradient (400-char budget). + desc_frac = sum(budget_fraction(len(s["description"]), 400) for s in skills) / n + auto += 4.0 * desc_frac + d_over = [f"{s['name']}={len(s['description'])}c" for s in skills if len(s["description"]) > 400] if d_over: - judgment.append(f"descriptions over 400 chars: {d_over}") + judgment.append(f"descriptions over the 400-char budget: {d_over}") # 2pt — no dead weight: empty files / empty reference dirs. dead = [] @@ -252,55 +459,25 @@ def score_context(root): refs = s["dir"] / "references" if refs.is_dir() and not any(refs.iterdir()): dead.append(str(refs.relative_to(root)) + "/ (empty)") - auto += 2 if not dead else 0 + auto += 2.0 if not dead else 0.0 if dead: judgment.append(f"dead/empty files: {dead}") - return {"auto": round(auto, 1), "max": 20, "needs_judgment": judgment} - - -def score_trigger(root): - skills = load_skills(root) - judgment, auto = [], 0.0 - if not skills: - return {"auto": 0.0, "max": 20, "needs_judgment": ["no skills to score"]} - n = len(skills) - - # 7pt — states what AND when, concrete trigger phrases. Mechanical proxy (3): - # description within budget AND carries a when/trigger signal. - good_when = sum(1 for s in skills - if s["description"] and len(s["description"]) <= 400 - and (WHEN_SIGNAL.search(s["description"]) or quoted_phrases(s["description"]))) - auto += 3 * good_when / n - judgment.append("what+when completeness & trigger-phrase quality (remaining 4pt) is judgment") - - # 5pt — negative scope: NOT-clause presence heuristic. - has_not = sum(1 for s in skills if NOT_CLAUSE.search(s["description"])) - auto += 5 * has_not / n - if has_not < n: - judgment.append(f"{n - has_not}/{n} skills lack a detectable NOT-clause") - - # 5pt — no trigger collisions (light sibling collision count). - count, notes = collision_count(skills) - auto += clamp(5 - count, 0, 5) - if count: - judgment.extend(notes) - - # 3pt — risky/niche skills guarded — judgment. - judgment.append("risky-skill guarding (allow_implicit_invocation / negative scope, 3pt) is judgment") - return {"auto": round(auto, 1), "max": 20, "needs_judgment": judgment} + return {"auto": round(auto, 1), "max": 20, "ceiling": CEILINGS["context_economy"], + "applicable": True, "needs_judgment": judgment} def score_hooks(root): judgment = [] hj = root / "hooks" / "hooks.json" if not hj.is_file(): - return {"auto": 10.0, "max": 10, - "needs_judgment": ["no hooks present — full credit awarded"]} + return {"auto": 0.0, "max": 10, "ceiling": 0.0, "applicable": False, + "needs_judgment": ["no hooks present — dimension N/A, weight redistributed"]} auto = 0.0 try: data = load_json(hj) except Exception as e: - return {"auto": 0.0, "max": 10, "needs_judgment": [f"hooks.json invalid JSON: {e}"]} + return {"auto": 0.0, "max": 10, "ceiling": CEILINGS["hooks_health"], "applicable": True, + "needs_judgment": [f"hooks.json invalid JSON: {e}"]} # 3pt — valid shape: event -> matcher group -> handlers. shape_ok = isinstance(data, dict) and bool(data) if shape_ok: @@ -312,38 +489,39 @@ def score_hooks(root): if not (isinstance(g, dict) and isinstance(g.get("hooks"), list)): shape_ok = False break - auto += 3 if shape_ok else 0 + auto += 3.0 if shape_ok else 0.0 + if not shape_ok: + judgment.append("hooks.json shape is not event -> matcher group -> handlers") # 2pt — paths use ${CLAUDE_PLUGIN_ROOT} or ${PLUGIN_ROOT}. strings = list(walk_strings(data)) cmds = [s for s in strings if "/" in s and (".py" in s or ".sh" in s or ".js" in s)] paths_ok = all(("${CLAUDE_PLUGIN_ROOT}" in s or "${PLUGIN_ROOT}" in s) for s in cmds) if cmds else True - auto += 2 if paths_ok else 0 + auto += 2.0 if paths_ok else 0.0 if not paths_ok: judgment.append("some hook paths do not use ${CLAUDE_PLUGIN_ROOT}/${PLUGIN_ROOT}") judgment.append("per-event contract correctness & harness-limit respect (5pt) is judgment") - return {"auto": round(auto, 1), "max": 10, "needs_judgment": judgment} + return {"auto": round(auto, 1), "max": 10, "ceiling": CEILINGS["hooks_health"], + "applicable": True, "needs_judgment": judgment} def score_distribution(root): judgment, auto = [], 0.0 - # 3pt — README covers what / skills / install. + # 3pt — README covers what / skills / install (graduated by signals present). readme = None for cand in ("README.md", "readme.md", "Readme.md"): if (root / cand).is_file(): readme = (root / cand).read_text(encoding="utf-8").lower() break if readme: - hits = sum(kw in readme for kw in ("install", "skill", "## what")) - # "what it does" section + skills list + install steps sig = ("install" in readme) + ("skill" in readme) + bool(re.search(r"^#", readme, re.M)) - auto += 3 * clamp(sig, 0, 3) / 3 + auto += 3.0 * clamp(sig, 0, 3) / 3 judgment.append("README completeness (does it truly cover all sections) is judgment") else: judgment.append("no README found") # 3pt — version discipline: changelog or ledger exists. has_log = any((root / f).is_file() for f in ("CHANGELOG.md", "LEDGER.md", "CHANGES.md")) - auto += 3 if has_log else 0 + auto += 3.0 if has_log else 0.0 if not has_log: judgment.append("no CHANGELOG/LEDGER found") judgment.append("version-bumped-with-changes discipline is judgment") @@ -367,21 +545,12 @@ def score_distribution(root): listed = name in {e.get("name") for e in entries if isinstance(e, dict)} except Exception: pass - auto += 2 if listed else 0 + auto += 2.0 if listed else 0.0 if not listed: judgment.append("Claude Code marketplace.json does not list this plugin") judgment.append("Codex marketplace registration (user-global) unverifiable from plugin root (2pt)") - return {"auto": round(auto, 1), "max": 10, "needs_judgment": judgment} - - -def score_skill_quality(root): - skills = load_skills(root) - return {"auto": 0.0, "max": 25, "needs_judgment": [ - f"one-job-per-skill / no two skills same job ({len(skills)} skills) — judgment", - "bodies are imperative agent instructions (not docs/marketing) — judgment", - "steps actionable: file paths, commands, exact formats — judgment", - "failure handling: missing file / failed command / ambiguous input — judgment", - ]} + return {"auto": round(auto, 1), "max": 10, "ceiling": CEILINGS["distribution"], + "applicable": True, "needs_judgment": judgment} DIMENSIONS = [ @@ -399,15 +568,26 @@ def score(root): dims = {} for key, fn in DIMENSIONS: try: - dims[key] = fn(root) + d = fn(root) except Exception as e: - dims[key] = {"auto": 0.0, "max": 0, "needs_judgment": [f"scorer crashed: {e}"]} - total_auto = round(sum(d["auto"] for d in dims.values()), 1) - total_max = sum(d["max"] for d in dims.values()) + d = {"auto": 0.0, "max": 0, "ceiling": 0.0, "applicable": False, + "needs_judgment": [f"scorer crashed: {e}"]} + d.setdefault("applicable", True) + d.setdefault("ceiling", d.get("max", 0)) + dims[key] = d + + applicable = {k: v for k, v in dims.items() + if v.get("applicable", True) and v.get("ceiling", 0) > 0} + sum_ceiling = sum(v["ceiling"] for v in applicable.values()) + sum_auto = sum(v["auto"] for v in applicable.values()) + effective = round(100.0 * sum_auto / sum_ceiling, 1) if sum_ceiling else 0.0 return { "target": str(root), "dimensions": dims, - "total": {"auto": total_auto, "max": total_max}, + "grade": grade_for(effective), + # total.auto = effective score out of 100 (N/A dims dropped & redistributed). + # total.effective mirrors it; total.max stays 100 for cross-plugin comparability. + "total": {"auto": effective, "max": 100, "effective": effective}, } @@ -415,13 +595,14 @@ def score(root): def render_table(result): lines = [f"target: {result['target']}", ""] - lines.append(f"{'dimension':<20} {'auto':>6} {'max':>5}") - lines.append("-" * 34) + lines.append(f"{'dimension':<20} {'auto':>6} {'max':>5} applicable") + lines.append("-" * 46) for key, d in result["dimensions"].items(): - lines.append(f"{key:<20} {d['auto']:>6} {d['max']:>5}") + app = "yes" if d.get("applicable", True) and d.get("ceiling", 0) > 0 else "N/A (dropped)" + lines.append(f"{key:<20} {d['auto']:>6} {d['max']:>5} {app}") t = result["total"] - lines.append("-" * 34) - lines.append(f"{'TOTAL (auto)':<20} {t['auto']:>6} {t['max']:>5}") + lines.append("-" * 46) + lines.append(f"{'EFFECTIVE / 100':<20} {t['auto']:>6} {t['max']:>5} grade: {result['grade']}") lines.append("") lines.append("needs judgment (score on top of the deterministic floor):") for key, d in result["dimensions"].items(): @@ -432,11 +613,13 @@ def render_table(result): def render_md(result): lines = [f"# Deterministic score — `{result['target']}`", "", - "| Dimension | Auto | Max |", "|---|---:|---:|"] + f"**Effective: {result['total']['auto']}/100 — {result['grade']}**", "", + "| Dimension | Auto | Max | Applicable |", "|---|---:|---:|:--|"] for key, d in result["dimensions"].items(): - lines.append(f"| {key} | {d['auto']} | {d['max']} |") + app = "yes" if d.get("applicable", True) and d.get("ceiling", 0) > 0 else "N/A — dropped" + lines.append(f"| {key} | {d['auto']} | {d['max']} | {app} |") t = result["total"] - lines.append(f"| **Total (auto)** | **{t['auto']}** | **{t['max']}** |") + lines.append(f"| **Effective / 100** | **{t['auto']}** | **{t['max']}** | {result['grade']} |") lines.append("\n## Needs judgment") for key, d in result["dimensions"].items(): for note in d["needs_judgment"]: @@ -452,81 +635,173 @@ def _write(base, rel, content): p.write_text(content, encoding="utf-8") -def _make_good_plugin(base): - _write(base, ".claude-plugin/plugin.json", json.dumps({ - "name": "good-plugin", "version": "1.0.0", - "description": "A good plugin.", "author": {"name": "RasputinKaiser"}, - "skills": "./skills/"})) - _write(base, ".codex-plugin/plugin.json", json.dumps({ - "name": "good-plugin", "version": "1.0.0+codex.1", - "description": "A good plugin.", "author": {"name": "RasputinKaiser"}, - "skills": "./skills/"})) - _write(base, ".claude-plugin/marketplace.json", json.dumps({ - "name": "good-plugin", "plugins": [{"name": "good-plugin", "source": "."}]})) - _write(base, "README.md", "# good-plugin\n## What it does\n## Skills\n## Install\nsteps here\n") - _write(base, "CHANGELOG.md", "# Changelog\n- 1.0.0\n") - _write(base, "skills/alpha/SKILL.md", - "---\nname: alpha\ndescription: Convert widgets to gadgets. " - "Use when asked to transmute a widget. Not for gadget deletion.\n---\n" - "Do the alpha job. Step 1. Step 2.\n") - _write(base, "skills/beta/SKILL.md", - "---\nname: beta\ndescription: Render invoices to PDF. " - "Use when the user wants a printable bill. Not for spreadsheets.\n---\n" - "Do the beta job. Step 1. Step 2.\n") - - -def _make_broken_plugin(base): - # version drift, name mismatch, over-budget description, empty file, no NOT-clauses, - # colliding descriptions, no marketplace listing, no changelog. +_RICH_BODY = ( + "## Read the manifest\n" + "1. Read the manifest at `.claude-plugin/plugin.json` and parse it.\n" + "2. Run `python3 scripts/score.py .` to compute the floor.\n" + "3. Verify the `--json` output matches the expected format.\n" + "4. Report the effective total and grade.\n\n" + "If the manifest is missing, emit an error and stop. If the command fails or the\n" + "input is ambiguous, fall back to a manual pass and note it. When in doubt, skip.\n" +) + + +def _lean_body(): + return _RICH_BODY # ~55 words, well under budget + + +def _near_budget_body(): + # ~590 words of imperative-ish prose, just under the 600-word body budget. + sentence = "Run the next step and verify the file exists before you continue. " + return _RICH_BODY + "\n" + (sentence * 75) + + +def _make_plugin(base, *, dual=True, skills=None, hooks=False, readme=True, + changelog=True, marketplace=True, author=True, name="demo-plugin", + codex_name=None, codex_ver="1.0.0+codex.1", ver="1.0.0"): + man = {"name": name, "version": ver, "description": "A demo plugin.", "skills": "./skills/"} + if author: + man["author"] = {"name": "RasputinKaiser"} + _write(base, ".claude-plugin/plugin.json", json.dumps(man)) + if dual: + cman = dict(man) + cman["name"] = codex_name or name + cman["version"] = codex_ver + _write(base, ".codex-plugin/plugin.json", json.dumps(cman)) + if marketplace: + _write(base, ".claude-plugin/marketplace.json", json.dumps({ + "name": name, "plugins": [{"name": name, "source": ".", "category": "dev"}]})) + if readme: + _write(base, "README.md", + "# demo\n## What it does\nthings\n## Skills\nlist\n## Install\nsteps here\n") + if changelog: + _write(base, "CHANGELOG.md", "# Changelog\n- 1.0.0 initial\n") + if hooks: + _write(base, "hooks/hooks.json", json.dumps({ + "PreToolUse": [{"matcher": "Bash", + "hooks": [{"type": "command", + "command": "python3 ${CLAUDE_PLUGIN_ROOT}/hooks/guard.py"}]}]})) + _write(base, "hooks/guard.py", "print('ok')\n") + for rel, desc, body in (skills or []): + _write(base, rel, f"---\nname: {rel.split('/')[1]}\ndescription: {desc}\n---\n{body}") + + +def _excellent(base): + skills = [ + ("skills/alpha/SKILL.md", + "Convert widgets to gadgets. Use when the user asks to transmute a widget " + "into a gadget. Not for gadget deletion (use the beta skill instead).", + _lean_body()), + ("skills/beta/SKILL.md", + "Render invoices to a printable PDF. Use when the user wants a printed bill " + "or receipt. Not for spreadsheet export, rather than a ledger dump.", + _lean_body()), + ] + _make_plugin(base, dual=True, skills=skills, hooks=True, name="excellent-plugin") + + +def _solid(base): + # mature-but-improvable: rich lean bodies, but one skill lacks a redirecting + # NOT-clause, no changelog, no hooks. Should land Solid/Strong (68-91). + skills = [ + ("skills/alpha/SKILL.md", + "Convert widgets to gadgets. Use when the user asks to transmute a widget. " + "Not for gadget deletion (use the beta skill instead).", + _lean_body()), + ("skills/beta/SKILL.md", + "Render invoices to a printable PDF. Use when the user wants a printed bill.", + _lean_body()), + ] + _make_plugin(base, dual=True, skills=skills, hooks=False, changelog=False, + name="solid-plugin") + + +def _fair(base): + # thin bodies, near-budget on one, weak triggers, no marketplace/changelog. + skills = [ + ("skills/alpha/SKILL.md", + "Manage the widget gadget thing for the user in various situations.", + "Do the alpha job.\nStep one then step two.\n"), + ("skills/beta/SKILL.md", + "Handle beta stuff when needed by the operator somehow.", + _near_budget_body()), + ] + _make_plugin(base, dual=True, skills=skills, hooks=False, readme=True, + changelog=False, marketplace=False, name="fair-plugin") + + +def _poor(base): + # name mismatch, version drift, over-budget colliding descriptions, empty file, + # no NOT-clauses, no marketplace listing, no changelog, thin bodies. _write(base, ".claude-plugin/plugin.json", json.dumps({ - "name": "broken-plugin", "version": "1.0.0", "description": "x"})) + "name": "poor-plugin", "version": "1.0.0", "description": "x"})) _write(base, ".codex-plugin/plugin.json", json.dumps({ - "name": "broke-plugin", "version": "2.0.0", "description": "x"})) + "name": "por-plugin", "version": "2.0.0", "description": ""})) _write(base, ".claude-plugin/marketplace.json", json.dumps({"plugins": []})) long_desc = "manage the widget gadget thing " * 20 # > 400 chars, no when/not signal - _write(base, "skills/alpha/SKILL.md", - f"---\nname: alpha\ndescription: {long_desc}\n---\nbody\n") - _write(base, "skills/beta/SKILL.md", - f"---\nname: beta\ndescription: {long_desc}\n---\nbody\n") + _write(base, "skills/alpha/SKILL.md", f"---\nname: alpha\ndescription: {long_desc}\n---\nbody\n") + _write(base, "skills/beta/SKILL.md", f"---\nname: beta\ndescription: {long_desc}\n---\nbody\n") _write(base, "skills/alpha/references/empty.md", "") def selftest(): failures = [] with tempfile.TemporaryDirectory() as td: - good = Path(td) / "good" - _make_good_plugin(good) - r = score(good) - dims = r["dimensions"] + base = Path(td) + + exc = base / "excellent"; _excellent(exc); re_ = score(exc) + sol = base / "solid"; _solid(sol); rs = score(sol) + fai = base / "fair"; _fair(fai); rf = score(fai) + poo = base / "poor"; _poor(poo); rp = score(poo) + + ed, sd, fd, pd_ = (r["dimensions"] for r in (re_, rs, rf, rp)) + et, st, ft, pt = (r["total"]["auto"] for r in (re_, rs, rf, rp)) + checks = [ - ("good manifest_integrity == 15", dims["manifest_integrity"]["auto"] == 15), - ("good hooks == 10 (no hooks)", dims["hooks_health"]["auto"] == 10), - ("good context_economy == 14 (auto ceiling)", dims["context_economy"]["auto"] == 14), - ("good trigger no collisions (>=13)", dims["trigger_precision"]["auto"] >= 13), - ("good distribution == 8", dims["distribution"]["auto"] == 8), - ("good skill_quality auto == 0", dims["skill_quality"]["auto"] == 0), - ("good total auto >= 55", r["total"]["auto"] >= 55), - ("total max == 100", r["total"]["max"] == 100), + # --- band membership of effective totals --- + (f"excellent in Exceptional (>=92), got {et}", et >= 92), + (f"solid in Solid/Strong band [68,91], got {st}", 68 <= st <= 91), + (f"fair in Needs-work band [50,67], got {ft}", 50 <= ft <= 67), + (f"poor in Poor band (<50), got {pt}", pt < 50), + ("monotonic ordering poor 0 for a good plugin", + ed["skill_quality"]["auto"] > 0 and sd["skill_quality"]["auto"] > 0), + ("(b2) skill_quality auto reaches ~16 for excellent (rich imperative body)", + ed["skill_quality"]["auto"] >= 16), + ("(d) demanding-but-not-impossible: excellent is 92+ yet NOT a trivial 100", + 92 <= et < 100), + # --- schema freeze --- + ("total.max == 100", re_["total"]["max"] == 100), + ("total.effective mirrors total.auto", re_["total"]["effective"] == et), + ("grade computed from the band table", re_["grade"] == "Exceptional"), + ("dimension keys unchanged", + list(ed.keys()) == [k for k, _ in DIMENSIONS]), + ("nominal maxes preserved 15/25/20/20/10/10", + [ed[k]["max"] for k, _ in DIMENSIONS] == [15, 25, 20, 20, 10, 10]), ] for label, ok in checks: if not ok: - failures.append(f"{label} (got dims={ { k:v['auto'] for k,v in dims.items()} })") - - broken = Path(td) / "broken" - _make_broken_plugin(broken) - rb = score(broken) - db = rb["dimensions"] - bchecks = [ - ("broken manifest < 15", db["manifest_integrity"]["auto"] < 15), - ("broken parity penalized (manifest <= 9)", db["manifest_integrity"]["auto"] <= 9), - ("broken context < 20", db["context_economy"]["auto"] < 20), - ("broken trigger < good trigger", db["trigger_precision"]["auto"] < dims["trigger_precision"]["auto"]), - ("broken distribution < 8", db["distribution"]["auto"] < 8), - ("broken total < good total", rb["total"]["auto"] < r["total"]["auto"]), - ] - for label, ok in bchecks: - if not ok: - failures.append(f"{label} (got dims={ { k:v['auto'] for k,v in db.items()} })") + failures.append(label) + + # --- (c) context strictly higher for lean vs near-budget body --- + lean = base / "ctx_lean" + _make_plugin(lean, dual=False, name="lean-plugin", + skills=[("skills/a/SKILL.md", "Do a. Use when a is needed. Not for b (use c).", + _lean_body())]) + near = base / "ctx_near" + _make_plugin(near, dual=False, name="near-plugin", + skills=[("skills/a/SKILL.md", "Do a. Use when a is needed. Not for b (use c).", + _near_budget_body())]) + lc = score(lean)["dimensions"]["context_economy"]["auto"] + nc = score(near)["dimensions"]["context_economy"]["auto"] + if not (lc > nc): + failures.append(f"(c) lean context {lc} not > near-budget context {nc}") if failures: print("SELFTEST FAILED:") @@ -583,18 +858,25 @@ def main(argv): total_auto = result["total"]["auto"] rc = 0 if min_n is not None and total_auto < min_n: - print(f"\nGATE FAIL: total auto {total_auto} < --min {min_n}", file=sys.stderr) + print(f"\nGATE FAIL: effective total {total_auto} < --min {min_n}", file=sys.stderr) rc = 1 if baseline is not None: try: prev = load_json(baseline) - prev_auto = prev.get("total", {}).get("auto", prev.get("auto")) \ + prev_total = prev.get("total", {}) if isinstance(prev, dict) else {} + prev_auto = prev_total.get("auto", prev.get("auto")) \ if isinstance(prev, dict) else None - if prev_auto is None: + # `total.auto` switched to the renormalized effective scale; a baseline + # captured before that (no `effective` key) is not comparable. + if isinstance(prev_total, dict) and prev_auto is not None \ + and "effective" not in prev_total: + print(f"\nwarning: baseline {baseline} predates effective-scoring; " + "scales differ, skipping regression gate", file=sys.stderr) + elif prev_auto is None: print(f"\nwarning: baseline {baseline} has no total.auto; skipping regression gate", file=sys.stderr) elif total_auto < prev_auto: - print(f"\nGATE FAIL: total auto {total_auto} < baseline {prev_auto} ({baseline})", + print(f"\nGATE FAIL: effective total {total_auto} < baseline {prev_auto} ({baseline})", file=sys.stderr) rc = 1 else: diff --git a/skills/plugin-audit/references/deterministic-scoring.md b/skills/plugin-audit/references/deterministic-scoring.md index 6ecb0c4..55ef42e 100644 --- a/skills/plugin-audit/references/deterministic-scoring.md +++ b/skills/plugin-audit/references/deterministic-scoring.md @@ -1,6 +1,6 @@ # Deterministic scoring — the machine floor -`score.py` computes the objective slice of the 100-point rubric so the agent only judges what a script cannot. It reads the same manifests, skill frontmatter, and budgets the human rubric uses, and returns a per-dimension floor plus a list of what still needs judgment. +`score.py` computes the objective slice of the 100-point rubric so the agent only judges what a script cannot. It reads the same manifests, skill frontmatter, and budgets the human rubric uses, and returns a per-dimension floor, a redistributed effective total, and a list of what still needs judgment. Run it against a plugin root (the dir holding `.claude-plugin/plugin.json` and/or `.codex-plugin/plugin.json`): @@ -8,49 +8,98 @@ Run it against a plugin root (the dir holding `.claude-plugin/plugin.json` and/o python3 scripts/score.py --json ``` -Output shape, one entry per rubric dimension: +Output shape, one entry per rubric dimension plus a renormalized total and a grade: ``` { - "manifest-integrity": {"auto": 11, "max": 15, "needs_judgment": ["description accuracy"]}, - "context-economy": {"auto": 18, "max": 20, "needs_judgment": ["duplication across skills"]}, - ... - "_total": {"auto": 71, "max": 100} + "dimensions": { + "manifest_integrity": {"auto": 13.5, "max": 15, "ceiling": 15.0, "applicable": true, "needs_judgment": [...]}, + "hooks_health": {"auto": 0.0, "max": 10, "ceiling": 0.0, "applicable": false, "needs_judgment": ["no hooks present — dimension N/A, weight redistributed"]}, + ... + }, + "grade": "Solid", + "total": {"auto": 78.2, "max": 100, "effective": 78.2} } ``` -`auto` is the machine FLOOR: points the script is confident about. `max` is the dimension cap (unchanged from `scoring-rubric.md`). `needs_judgment` names the sub-criteria the script cannot decide — the agent scores those, on top of `auto`, up to `max`. The final dimension score is `auto` + (judged points), never below `auto`. +Per dimension: -## What is auto-scored vs judged, per dimension +- `auto` — machine-verifiable points earned, scored against the dimension's nominal `max`. +- `max` — the dimension's nominal rubric ceiling (unchanged: 15 / 25 / 20 / 20 / 10 / 10). +- `ceiling` — the machine-achievable slice of `max` (`auto` never exceeds it; the remainder is judgment residue the agent scores on top). +- `applicable` — whether the dimension applies to this plugin at all. A dropped dimension (`false`) is excluded from the total. +- `needs_judgment` — the sub-criteria the script cannot decide. -| Dimension (max) | `score.py` auto-scores | Agent judges (`needs_judgment`) | +`total.auto` (== `total.effective`) is the **effective score out of 100**: the earned auto renormalized against the achievable auto ceiling of the *applicable* dimensions. `total.max` stays `100` so totals stay comparable across plugins. The final dimension score is `auto` + (judged points), never below `auto`. + +## Calibration principles + +1. **Graduated credit, no coarse pass/fail.** Every sub-point awards a gradient. Validity earns the MIDDLE of a sub-point, not its max. +2. **No free points.** A dimension never hands out a free maximum for being absent. +3. **N/A → redistribute.** A dimension that genuinely does not apply (no hooks; no skills) is DROPPED — excluded from both the numerator and the denominator of the effective total — so its weight spreads proportionally across the applicable dimensions. Renormalized to 100. +4. **Reward leanness.** Budget dimensions reward being well UNDER budget via a gradient (see Context economy). +5. **Skill quality gets machine signal.** Per-skill proxies for imperative bodies, actionable specifics, and failure handling replace the old flat `0`. + +## Effective total & redistribution + +Let `A` = the set of applicable dimensions (`applicable == true` and `ceiling > 0`). + +``` +effective = 100 * (sum of auto over A) / (sum of ceiling over A) +``` + +A dimension that is N/A contributes to neither sum, so the remaining dimensions absorb its weight proportionally — no free max, no dead weight. A plugin with no hooks is scored purely on the dimensions that apply to it; a plugin with no skills drops `skill_quality`, `trigger_precision`, and `context_economy` the same way. + +## Grade bands (on the effective total) + +| Band | Range | +|---|---| +| Exceptional | 92–100 | +| Strong | 82–91 | +| Solid | 68–81 | +| Needs work | 50–67 | +| Poor | < 50 | + +A mature-but-improvable plugin lands in the Solid/Strong range (~78–84); 92+ is rare but achievable when every applicable dimension is strong. + +## Auto formulas, per dimension + +| Dimension (max / auto ceiling) | `score.py` auto-scores | Agent judges (`needs_judgment`) | |---|---|---| -| Manifest integrity (15) | JSON parses; kebab-case `name`; semver `version`; cross-harness `name`/`version` parity; component pointers `./`-prefixed and in-root; manifest-dir layout | `description` accuracy; whether publisher metadata suits the distribution level | -| Skill quality (25) | — (all judgment) | one-job-per-skill; imperative bodies; actionable steps; failure handling | -| Trigger precision (20) | Description char budget; NOT-clause presence heuristic; collision count from the trigger graph (`G_t`) | Whether triggers match real user phrasing; genuine vs apparent collisions; guard adequacy for risky skills | -| Context economy (20) | Description chars vs budget; body words vs budget; dead/empty files and unused dirs | Cross-skill duplication; progressive-disclosure quality (aided by `tokens.py`) | -| Hooks health (10) | `hooks.json` shape; `${CLAUDE_PLUGIN_ROOT}`/`${PLUGIN_ROOT}` paths; referenced scripts exist and are executable | Per-event contract correctness; per-harness capability limits; runtime health (from `errscan.py`) | -| Distribution readiness (10) | README sections present; changelog/ledger exists; a marketplace entry per targeted harness | Interface/presentation quality (`presentation.md`); asset craft; copy quality | +| Manifest integrity (15 / 15) | validity graduated — kebab `name` (2) + semver `version` (1.5) + `description` present (0.5); cross-harness parity — identical `name` (2) + agreeing `version` (2), single-harness earns full 4; component pointers `./`-prefixed & resolvable (proportional, 3); manifest-dir layout clean (proportional, 2); `author` present (2) | `description` accuracy; publisher-metadata appropriateness | +| Skill quality (25 / 18) | per-skill proxies averaged — imperative body (≤8: verb-first line density vs a 35% target, numbered/step markers, verb-first heading); actionable specifics (≤5: code fences, inline `` `code` ``, file paths); failure handling (≤5: `missing`/`fails`/`fallback`/`error`/`ambiguous`/`absent`/… signals, 3+ distinct → full) | one-job-per-skill; genuine instruction depth (~7pt residue) | +| Trigger precision (20 / 13) | description ≤400 chars AND carries a when/trigger signal or quoted phrase (proportional, 3); redirecting NOT-clause — `"instead of"` / `"rather than"` / `"not for X … use Y"` / `"use Y instead"`, a bare `not` earns nothing (proportional, 5); no collisions — `clamp(5 − collision_count, 0, 5)` | what+when completeness & trigger-phrase quality; risky-skill guarding | +| Context economy (20 / 14) | body utilization gradient (≤600-word budget: full at ≤50%, linear decay to 0 at ≥100%, averaged, 8); description utilization gradient (≤400-char budget, same shape, 4); no dead/empty files or empty ref dirs (2) | cross-skill duplication (6); progressive-disclosure quality | +| Hooks health (10 / 5) | **N/A → dropped when no `hooks/hooks.json`** (no free 10); when present: valid shape event→matcher→handlers (3), paths use `${CLAUDE_PLUGIN_ROOT}`/`${PLUGIN_ROOT}` (2) | per-event contract correctness; per-harness capability limits; runtime health | +| Distribution readiness (10 / 8) | README signals present — `install` + `skill` + a heading (proportional, 3); changelog/ledger exists (3); Claude `marketplace.json` lists the plugin (2) | README completeness; version-bump discipline; Codex registration (user-global, unverifiable, 2) | -Skill quality is fully judged — nothing mechanical stands in for reading the bodies. Every other dimension has a floor. +The **budget gradient** (`budget_fraction`) is the leanness reward: `clamp(2 * (1 − used/budget), 0, 1)` — full credit at ≤50% of budget, 0 at ≥100%. A 599/600-word body scores far below a 300-word one. ## Mapping to the rubric -- The `auto`/`max` pairs correspond one-to-one to the six dimensions in `scoring-rubric.md`; the sub-point weights there are unchanged. `score.py` never invents points — it only fills the objective fraction of each dimension. +- The `auto`/`max` pairs correspond one-to-one to the six dimensions in `scoring-rubric.md`; the nominal maxes are unchanged. `score.py` never invents points — it only fills the objective fraction of each dimension, capped at `ceiling`. - Feed `tokens.py` output into the Context-economy judgment and `errscan.py` output into the Hooks-health judgment (see `scoring-rubric.md` for the evidence notes). -- Report the split explicitly (see the Diagnostics block in `report-style.md`): floor total, judged delta, final, and which dimensions carried `needs_judgment` notes. +- Report the split explicitly (see the Diagnostics block in `report-style.md`): effective total, grade, which dimensions were dropped as N/A, and which carried `needs_judgment` notes. ## Gates and CI `score.py` can also enforce a floor instead of just reporting one: ``` -python3 scripts/score.py --min 55 # fail if the auto total drops below 55 -python3 scripts/score.py --min-baseline # fail if below the stored .plugin-improver/score-baseline.json +python3 scripts/score.py --min 55 # fail if the effective total drops below 55 +python3 scripts/score.py --min-baseline PATH # fail if below the stored baseline JSON ``` -CI runs the `--min-baseline` form so a change that mechanically regresses the plugin (a blown budget, a broken pointer, a lost manifest) fails the build before any human judgment is applied. When auditing, run the plain report form first; treat a red gate as an automatic 🔴 finding. +Both gates read `total.auto` (the effective /100 score). CI runs a floor form so a change that mechanically regresses the plugin (a blown budget, a broken pointer, a lost manifest) fails the build before any human judgment is applied. When auditing, run the plain report form first; treat a red gate as an automatic 🔴 finding. ## When the scorer is unavailable If `score.py` is missing or errors, score every dimension by hand from `scoring-rubric.md` and note `floor: manual` in the Diagnostics block so the report stays honest about provenance. The floor is an accelerator, not a dependency — the rubric is still the source of truth. + +## Selftest + +``` +python3 scripts/score.py selftest +``` + +Runs deterministic temp-dir fixtures spanning poor / fair / solid / excellent and asserts band membership of their effective totals, monotonic ordering, and the de-saturation invariants: (a) hooks is dropped (not auto-max) when absent and graduated when present; (b) `skill_quality` auto is > 0 for a good plugin; (c) context scores strictly higher for a lean body than a near-budget one; (d) a demanding-but-not-impossible fixture reaches 92+. Stdlib-only, no network, no fixtures on disk.