diff --git a/README.md b/README.md index 376355d..07217b8 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,10 @@ Reusable agent plugins and skills following AGENTS/skills conventions. ## Plugins -| Plugin | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [vault](./vault/) | Zettelkasten-style vault workflows for Obsidian notes | -| [development](./development/) | Agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness and marketing, session log audits, decision records, and self-hosted runners | +| Plugin | Description | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [vault](./vault/) | Zettelkasten-style vault workflows for Obsidian notes | +| [development](./development/) | Agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness and marketing, session log audits, bug verification, decision records, and self-hosted runners | ## Layout diff --git a/development/.codex-plugin/plugin.json b/development/.codex-plugin/plugin.json index e64705d..ed429e8 100644 --- a/development/.codex-plugin/plugin.json +++ b/development/.codex-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "development", - "version": "0.0.7", - "description": "Developer workflow skills for agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness, OSS marketing, session log audits, decision records, and GitHub Actions self-hosted runners.", + "version": "0.0.8", + "description": "Developer workflow skills for agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness, OSS marketing, session log audits, bug verification, decision records, and GitHub Actions self-hosted runners.", "author": { "name": "Mark Phelps" }, @@ -41,13 +41,17 @@ "architecture", "whiteboard", "design-rationale", - "trade-offs" + "trade-offs", + "verify-bug", + "bug-verification", + "false-positives", + "triage" ], "skills": "./skills/", "interface": { "displayName": "Development", - "shortDescription": "Agent context, PR fixup, CLI design, landing copy, architecture mapping and whiteboard defense, OSS readiness, marketing, session log audit, decision records, and runner workflows", - "longDescription": "Development workflow skills for auditing agent context files, fixing existing GitHub PRs, designing humane command-line interfaces, writing product landing-page copy backed by shipped code, mapping codebase architecture and design rationale, preparing GitHub repos for open source release, sharpening OSS README positioning, mining local agent session logs for papercuts in your own tools, writing and maintaining decision and plan records (ADRs), and running GitHub Actions self-hosted runners on a Linux host.", + "shortDescription": "Agent context, PR fixup, CLI design, landing copy, architecture mapping and whiteboard defense, OSS readiness, marketing, session log audit, bug verification, decision records, and runner workflows", + "longDescription": "Development workflow skills for auditing agent context files, fixing existing GitHub PRs, designing humane command-line interfaces, writing product landing-page copy backed by shipped code, mapping codebase architecture and design rationale, preparing GitHub repos for open source release, sharpening OSS README positioning, mining local agent session logs for papercuts in your own tools, verifying whether claimed bugs are real, writing and maintaining decision and plan records (ADRs), and running GitHub Actions self-hosted runners on a Linux host.", "developerName": "Mark Phelps", "category": "Development", "capabilities": ["Read", "Write"], @@ -71,7 +75,9 @@ "Add a second repository runner to this machine and verify it is listening.", "Write a decision record for switching our job queue to Postgres.", "Supersede record 012 with a new plan and update the registry.", - "Set up a docs/records directory with a registry README for this repo." + "Set up a docs/records directory with a registry README for this repo.", + "Is this bug from the code review actually real?", + "Verify the suspected bugs in docs/map/ and write the verdicts back." ] } } diff --git a/development/AGENTS.md b/development/AGENTS.md index d4ac1a1..5daf938 100644 --- a/development/AGENTS.md +++ b/development/AGENTS.md @@ -37,6 +37,8 @@ agent context. release, emphasizing first-run and first-contribution developer experience - `session-log-audit`: mine local agent session logs (Claude Code, Pi, Codex) for papercuts in a project the user builds with agents, then rank the fixes +- `verify-bug`: decide whether claimed bugs are real through an isolated + Prover/Skeptic/Referee hearing, with repro tests in throwaway worktrees ## Workflow Policy @@ -54,6 +56,9 @@ agent context. - For whiteboard maps, never invent design rationale. Tag rationale claims with evidence, keep defense records local under `.map/`, and never modify source code while mapping or defending. +- For bug verification, never touch the working tree (repros run only in a + `git worktree` under `.verify/`), never fix code or open issues, and post + verdicts to GitHub only after the user confirms each post. - For self-hosted runners, resolve the runner account instead of assuming a user or home directory, keep one namespaced instance per target, treat tokens as secrets that never reach a file that persists, and confirm no job is diff --git a/development/README.md b/development/README.md index 1a8a01a..e3a8c6b 100644 --- a/development/README.md +++ b/development/README.md @@ -3,8 +3,8 @@ Developer workflow skills for repo-facing agent context, GitHub PR follow-up, command-line interface design, codebase-grounded landing copy, architecture mapping and whiteboard defense, open source release preparation, audits of your -own agent session logs, decision records, and GitHub Actions self-hosted -runners. +own agent session logs, bug-claim verification, decision records, and GitHub +Actions self-hosted runners. ## Skill Set @@ -17,6 +17,7 @@ runners. | `github-pr-fixup` | Addresses unresolved GitHub PR review comments and failing CI on the existing source branch. | When a user gives you an existing PR URL and wants review feedback or CI failures fixed without a new PR. | | `github-self-hosted-runner` | Installs, registers, verifies, or removes namespaced GitHub Actions self-hosted runners on a Linux host. | When a user gives a repository or organization URL and wants a persistent runner managed by systemd. | | `whiteboard` | Maps architecture and design rationale, then runs whiteboard-defense quizzes. | When the user explicitly invokes `/whiteboard` to map a codebase, explore a region, refresh a map, or defend their understanding. | +| `verify-bug` | Rules on whether claimed bugs are real via an isolated Prover/Skeptic/Referee hearing. | When the user asks whether a bug is real, wants findings or review comments verified, or points at suspected bugs from a map. | | `oss-marketing` | Sharpens README and public-doc positioning for first-time visitors. | When a repo needs launch copy, clearer positioning, or a README that explains what the project is. | | `oss-repo-readiness` | Audits and prepares a repo for open source release, focused on developer experience. | When making a repo public, writing CONTRIBUTING or issue templates, or running a pre-launch checklist. | | `session-log-audit` | Mines local agent session logs for papercuts and produces a ranked fix list. | When the user wants to know what is annoying about their own tool, or why they work around it. | @@ -40,6 +41,11 @@ runners. - Whiteboard is explicit-invocation only, never modifies source code, and keeps personal defense records under `.map/` while committed maps live under `docs/map/`. +- Whiteboard logs `suspected-bug` leads without investigating them; `verify-bug` + rules on them. Neither skill depends on the other. +- Bug verification never modifies the working tree. Repro tests run only in a + throwaway `git worktree` under `.verify/`, and GitHub replies are posted only + after the user confirms each one. - Codebase landing copy is for a product's own landing page and must trace claims to shipped code; use OSS marketing for open-source README positioning. - Session log audits need a project the user built with agents and used diff --git a/development/skills/verify-bug/SKILL.md b/development/skills/verify-bug/SKILL.md new file mode 100644 index 0000000..d6c1f01 --- /dev/null +++ b/development/skills/verify-bug/SKILL.md @@ -0,0 +1,173 @@ +--- +name: verify-bug +description: + Decide whether claimed bugs are real by running an adversarial hearing on each + claim — a Prover argues it's reachable, a Skeptic hunts for what prevents it, + and an independent Referee rules confirmed, refuted, or unclear — with an + isolated repro test when that's cheap. Use when the user asks whether a bug is + real, wants findings or review comments verified, asks if a reported issue + reproduces, or points at suspected bugs from a codebase map. Accepts pasted + claims, code-review comments, GitHub issues, or `suspected-bug` entries in + `docs/map/`. Does not hunt for new bugs, and never fixes code; don't use it + for ordinary debugging of a known failure or for general code review. +--- + +# Verify bug + +Turn claimed bugs into verdicts you can act on. A claim is a lead, not a fact: +code that looks wrong is often protected by validation in another file, a +framework guarantee, a lock, a type, or configuration. And real bugs described +vaguely get dismissed. This skill gives every claim a fair hearing with an +advocate on each side, then records the verdict with its evidence. + +## Hard rules + +- **Never modify the user's working tree.** Reads happen anywhere; repro tests + happen only in a throwaway `git worktree` (see "Reproduction"). +- **Never fix bugs, open issues, or post comments on your own.** Posting a + verdict back to GitHub requires the user's explicit confirmation for that + specific post. +- **No verdict without receipts.** Every argument must point at code + (`path:line` or `path:Symbol`). Arguments without receipts are discounted. + +## Workflow + +1. **Collect claims** from the source the user gave (see "Sources"). +2. **Normalize** each into a claim record (below). +3. **Dedupe:** merge claims that point at the same code path with the same + trigger. Agents often report one root cause several ways; keep the clearest + scenario and list the merged sources. +4. **Rank** by severity _if true_: data loss or corruption, security, silent + wrong results, crashes or outages, degraded behavior, cosmetic. Don't try to + judge likelihood here; that's what the hearing is for. +5. **Cap:** verify the top 10 by default. If more remain, say how many and let + the user pick or raise the cap. Unverified claims keep an `unverified` + verdict; nothing is silently dropped. +6. **Hear** each claim (see "The hearing"). +7. **Record** verdicts (see "Output") and summarize in chat. + +### Claim record + +```markdown +- id: +- source: +- claim: +- scenario: +- receipt: +- severity-if-true: +- verdict: unverified +``` + +If a claim has no concrete scenario ("this might be racy"), ask the Prover to +construct one as its first job. If it can't, the verdict is +`unclear: no concrete trigger`. + +## Sources + +| Source | How to read it | Where the verdict goes | +| ------------------------------ | ------------------------------------------------------ | ----------------------------------------- | +| `docs/map/*.md` friction lists | `suspected-bug` entries in "Open questions & friction" | In place, replacing `verdict: unverified` | +| GitHub issue or PR comment | `gh issue view`, `gh pr view --comments`, or the API | Report file; draft reply on request | +| Pasted list or review output | The conversation | Report file | + +For map entries, keep the entry's wording and replace only the verdict slot. +Preserve any existing verdict unless the code at its receipt has changed since +it was recorded. + +## The hearing + +Run three roles per claim. When the harness supports subagents, run each role in +its own isolated context, because an agent that has read the claim's framing, or +another role's reasoning, anchors on it. Pass only what each role needs. Role +briefs, including exact output formats, are in `references/roles.md`; read it +before the first hearing. + +1. **Prover** — gets the claim record and repo access. Builds the strongest case + that the bug is reachable: the entry point, the path through the code, the + state at the failure point, and the observable effect. May attempt a repro + (see "Reproduction"). +2. **Skeptic** — gets the claim record and repo access, _not_ the Prover's + output. Hunts for anything that prevents the bug: upstream validation, + framework or language guarantees, locks and ordering, types, config defaults, + dead code paths, or the claim simply misreading the code. +3. **Referee** — gets the claim record, both structured outputs, and repo + access. Checks each receipt, weighs the arguments, and rules. + +Run hearings for different claims in parallel when possible; the three roles for +one claim run Prover and Skeptic in parallel, then Referee. + +**Without subagents**, run the roles sequentially yourself: write the Prover +case, then deliberately set it aside and write the Skeptic case from the claim +alone, then referee. Tell the Referee step to discount any argument without a +receipt, since sequential roles bleed into each other. + +### Verdicts + +| Verdict | Meaning | +| --------------------------- | -------------------------------------------------------------------- | +| `confirmed (reproduced)` | A failing test in an isolated worktree demonstrates it | +| `confirmed (static)` | A complete trigger path with receipts, and no valid prevention found | +| `refuted: ` | Something concrete prevents it, with a receipt | +| `unclear: ` | Neither side made its case; say what would settle it | + +Prefer `unclear` over a forced call. A wrong `refuted` buries a real bug, and a +wrong `confirmed` wastes someone's afternoon; `unclear` with a precise "what's +missing" is useful. + +## Reproduction + +A failing test is the strongest evidence and the start of a fix, so the Prover +should attempt one when it's cheap: the repo has a working test command, the +trigger can be expressed as a unit or integration test, and no external services +are needed. + +1. Create an isolated worktree on a scratch branch inside the excluded + `.verify/` directory: + `git worktree add -b verify-bug/ .verify/worktrees/ HEAD` +2. Write the smallest test that should fail if the bug is real, and run only + that test with the repo's own test command, from inside the worktree. +3. Save the test as a patch. New files are untracked, so stage first: + `git -C .verify/worktrees/ add -A && git -C .verify/worktrees/ diff --cached > .verify/patches/.patch` +4. Remove the worktree and branch: + `git worktree remove --force .verify/worktrees/ && git branch -D verify-bug/` + +A test that fails for an unrelated reason (compile error, missing fixture) +proves nothing; fix the test or fall back to a static argument. + +**Ask the user once before** installing dependencies, starting services or +containers, making network calls, or running anything other than the repo's own +test command. Concurrency bugs, external-service bugs, and UI bugs are usually +not cheap to reproduce; don't force it. + +## Output + +Every run writes a local report, plus in-place verdicts for map sources. + +- **Report:** `.verify/-.md`, with patches in + `.verify/patches/`. On first run, before creating any worktree, add `.verify/` + to `.git/info/exclude` so nothing lands in the repo's tracked files. The + report lists every claim (including capped, unverified ones) with its verdict, + the Prover's trigger path, the Skeptic's prevention argument, the Referee's + reasoning, and any patch link. Use the template in `references/roles.md`. +- **Map sources:** replace `verdict: unverified` with the verdict and a one-line + reason, plus a link to the report. For example: + + ```text + verdict: refuted: Sync holds mu across the check (sync/engine.go:88); see .verify/2026-09-24-sync.md + ``` + + The report path is local, so keep the one-line reason self-sufficient. + +- **GitHub sources:** after the summary, offer to draft a reply for each + verified issue or comment. Show the draft; post only after the user confirms + that post. Keep replies factual: verdict, trigger path or prevention, and the + repro test inline if one exists. + +### Chat summary + +Keep it short: counts by verdict, then one line per confirmed bug (claim, +severity, reproduced or static, patch link), then refuted and unclear claims in +a compact list. Mention how many claims were capped. When confirmed bugs exist, +say they're ready to fix and that the repro patch is the first half of the fix. +Don't restate the report. diff --git a/development/skills/verify-bug/references/roles.md b/development/skills/verify-bug/references/roles.md new file mode 100644 index 0000000..068c7c4 --- /dev/null +++ b/development/skills/verify-bug/references/roles.md @@ -0,0 +1,122 @@ +# Role briefs + +Give each role only its brief, the claim record, and repo access. Isolation is +the point: a role that has seen another role's reasoning anchors on it. + +## Contents + +- Prover +- Skeptic +- Referee +- Report template + +## Prover + +> You are arguing that this bug is real. Build the strongest honest case. +> +> Find the entry point where the triggering input or event arrives, and trace +> the path through the code to the failure point, with a receipt at each hop. +> Describe the state at the failure point and the observable effect for a user +> or caller. If the claim has no concrete scenario, construct one; if you can't, +> say so. +> +> If a repro is cheap (working test command, no external services), write the +> smallest failing test in the isolated worktree you've been given and run only +> that test. Report the exact command and output. +> +> Don't invent receipts. If a hop in the path is uncertain, mark it uncertain +> rather than smoothing it over; a weak link stated plainly is more useful than +> a confident wrong one. + +Output: + +```markdown +- trigger: +- path: + 1. + 2. ... +- failure: +- uncertain hops: +- repro: +- patch: +``` + +## Skeptic + +> You are arguing that this bug is not real. Find what prevents it, honestly. +> +> Check upstream validation and middleware, framework and language guarantees, +> locking and ordering, type constraints, config defaults, whether the code path +> is reachable at all, and whether the claim misreads the code. Every prevention +> needs a receipt. If you look hard and find nothing that prevents the bug, say +> so; a Skeptic who concedes a real bug is doing the job. + +Output: + +```markdown +- preventions: + - +- misreadings: +- gaps: +- concede: +``` + +## Referee + +> You rule on whether this bug is real. You have the claim, the Prover's case, +> and the Skeptic's case. Neither is trustworthy by default. +> +> Open every receipt and check it says what's claimed. Discount any argument +> without a receipt. A single valid prevention that holds under the claim's +> scenario refutes it. A complete path with no valid prevention confirms it. A +> passing Prover repro counts against the claim; a repro that fails for the +> stated reason confirms it. If neither side made its case, rule unclear and say +> exactly what would settle it. + +Output: + +```markdown +- verdict: +- reason: +- checked receipts: +- would settle it: +``` + +## Report template + +```markdown +# Verify bug: + +- Date: +- Repo HEAD: +- Source: +- Claims: → · Verified: · Capped: + +## Summary + +| ID | Claim | Severity if true | Verdict | +| --- | ----- | ---------------- | ------- | + +## : + +- Source: (merged: ) +- Scenario: +- Verdict: — + +### Prover + + + +### Skeptic + + + +### Referee + + + +## Capped (unverified) + +- : () +``` diff --git a/development/skills/whiteboard/SKILL.md b/development/skills/whiteboard/SKILL.md index 4db85f0..df9c2b5 100644 --- a/development/skills/whiteboard/SKILL.md +++ b/development/skills/whiteboard/SKILL.md @@ -194,15 +194,27 @@ Each entry is typed: | `confusing` | Misleading names, surprising indirection, comment contradicts code | zoom | | `inconsistent` | Two patterns doing the same job; docs disagree with code | scout | | `cleanup` | Dead code, abandoned dependency, ancient TODO | any | +| `suspected-bug` | A concrete scenario where the code looks wrong or unhandled | zoom, defense | | `defense-exposed` | The user missed a question because the code misled them | defense | The signal here is simple: if an agent with full read access struggled to understand something, a new contributor will too. Keep entries to one line plus a receipt. Don't fix anything. -When `unexplained` items accumulate, tell the user they are good candidates for -decision records (ADRs) documenting the rationale. Use that plain wording and -nothing more; this skill has no dependency on any records tooling. +A `suspected-bug` entry must name the triggering scenario, not just a feeling: +"two concurrent `Sync` calls both pass the `running` check before either sets it +(`sync/engine.go:Run`)", not "possible race in sync." Don't investigate further +or try to reproduce it; a suspected bug is a lead, and the map treats it as +unverified. Leave the verdict slot empty (see `references/templates.md`) so a +later verification pass can fill it in. + +Handoffs use plain wording and nothing more; this skill depends on no other +tooling: + +- When `unexplained` items accumulate, tell the user they are good candidates + for decision records (ADRs) documenting the rationale. +- When `suspected-bug` items exist, tell the user these suspected bugs could be + verified before anyone acts on them. ### Scout notes diff --git a/development/skills/whiteboard/references/defense.md b/development/skills/whiteboard/references/defense.md index a4957a9..27a6225 100644 --- a/development/skills/whiteboard/references/defense.md +++ b/development/skills/whiteboard/references/defense.md @@ -107,6 +107,12 @@ it, what would the alternative have cost. These are optional extras in a session, after core questions, and are often the most valuable part for repos the user owns. +Suspected bugs with a `refuted` verdict make excellent `adversarial` questions, +because the code looks wrong and isn't: "Two `Sync` calls can overlap here. Why +isn't that a race?" The verdict's evidence is the ground truth, so these can be +core questions. Entries still `unverified` or `unclear` are framed like inferred +decisions: ask the user whether it's a real bug and why, and don't grade. + ## Write-backs When the user gives a rationale for an inferred or unexplained decision that diff --git a/development/skills/whiteboard/references/templates.md b/development/skills/whiteboard/references/templates.md index a025b4a..59374df 100644 --- a/development/skills/whiteboard/references/templates.md +++ b/development/skills/whiteboard/references/templates.md @@ -107,9 +107,16 @@ short paragraph. State machines for anything with a lifecycle.> - **:** (``) +If a scenario looks unhandled, also log it as a `suspected-bug` below. + ## Open questions & friction - `` (``) +- `suspected-bug` (``) — verdict: unverified + +A verification pass may later replace `unverified` with `confirmed: `, +`refuted: `, or `unclear: `. Preserve +whatever verdict is there when refreshing. ## Zoom candidates diff --git a/skills/verify-bug/SKILL.md b/skills/verify-bug/SKILL.md new file mode 100644 index 0000000..d6c1f01 --- /dev/null +++ b/skills/verify-bug/SKILL.md @@ -0,0 +1,173 @@ +--- +name: verify-bug +description: + Decide whether claimed bugs are real by running an adversarial hearing on each + claim — a Prover argues it's reachable, a Skeptic hunts for what prevents it, + and an independent Referee rules confirmed, refuted, or unclear — with an + isolated repro test when that's cheap. Use when the user asks whether a bug is + real, wants findings or review comments verified, asks if a reported issue + reproduces, or points at suspected bugs from a codebase map. Accepts pasted + claims, code-review comments, GitHub issues, or `suspected-bug` entries in + `docs/map/`. Does not hunt for new bugs, and never fixes code; don't use it + for ordinary debugging of a known failure or for general code review. +--- + +# Verify bug + +Turn claimed bugs into verdicts you can act on. A claim is a lead, not a fact: +code that looks wrong is often protected by validation in another file, a +framework guarantee, a lock, a type, or configuration. And real bugs described +vaguely get dismissed. This skill gives every claim a fair hearing with an +advocate on each side, then records the verdict with its evidence. + +## Hard rules + +- **Never modify the user's working tree.** Reads happen anywhere; repro tests + happen only in a throwaway `git worktree` (see "Reproduction"). +- **Never fix bugs, open issues, or post comments on your own.** Posting a + verdict back to GitHub requires the user's explicit confirmation for that + specific post. +- **No verdict without receipts.** Every argument must point at code + (`path:line` or `path:Symbol`). Arguments without receipts are discounted. + +## Workflow + +1. **Collect claims** from the source the user gave (see "Sources"). +2. **Normalize** each into a claim record (below). +3. **Dedupe:** merge claims that point at the same code path with the same + trigger. Agents often report one root cause several ways; keep the clearest + scenario and list the merged sources. +4. **Rank** by severity _if true_: data loss or corruption, security, silent + wrong results, crashes or outages, degraded behavior, cosmetic. Don't try to + judge likelihood here; that's what the hearing is for. +5. **Cap:** verify the top 10 by default. If more remain, say how many and let + the user pick or raise the cap. Unverified claims keep an `unverified` + verdict; nothing is silently dropped. +6. **Hear** each claim (see "The hearing"). +7. **Record** verdicts (see "Output") and summarize in chat. + +### Claim record + +```markdown +- id: +- source: +- claim: +- scenario: +- receipt: +- severity-if-true: +- verdict: unverified +``` + +If a claim has no concrete scenario ("this might be racy"), ask the Prover to +construct one as its first job. If it can't, the verdict is +`unclear: no concrete trigger`. + +## Sources + +| Source | How to read it | Where the verdict goes | +| ------------------------------ | ------------------------------------------------------ | ----------------------------------------- | +| `docs/map/*.md` friction lists | `suspected-bug` entries in "Open questions & friction" | In place, replacing `verdict: unverified` | +| GitHub issue or PR comment | `gh issue view`, `gh pr view --comments`, or the API | Report file; draft reply on request | +| Pasted list or review output | The conversation | Report file | + +For map entries, keep the entry's wording and replace only the verdict slot. +Preserve any existing verdict unless the code at its receipt has changed since +it was recorded. + +## The hearing + +Run three roles per claim. When the harness supports subagents, run each role in +its own isolated context, because an agent that has read the claim's framing, or +another role's reasoning, anchors on it. Pass only what each role needs. Role +briefs, including exact output formats, are in `references/roles.md`; read it +before the first hearing. + +1. **Prover** — gets the claim record and repo access. Builds the strongest case + that the bug is reachable: the entry point, the path through the code, the + state at the failure point, and the observable effect. May attempt a repro + (see "Reproduction"). +2. **Skeptic** — gets the claim record and repo access, _not_ the Prover's + output. Hunts for anything that prevents the bug: upstream validation, + framework or language guarantees, locks and ordering, types, config defaults, + dead code paths, or the claim simply misreading the code. +3. **Referee** — gets the claim record, both structured outputs, and repo + access. Checks each receipt, weighs the arguments, and rules. + +Run hearings for different claims in parallel when possible; the three roles for +one claim run Prover and Skeptic in parallel, then Referee. + +**Without subagents**, run the roles sequentially yourself: write the Prover +case, then deliberately set it aside and write the Skeptic case from the claim +alone, then referee. Tell the Referee step to discount any argument without a +receipt, since sequential roles bleed into each other. + +### Verdicts + +| Verdict | Meaning | +| --------------------------- | -------------------------------------------------------------------- | +| `confirmed (reproduced)` | A failing test in an isolated worktree demonstrates it | +| `confirmed (static)` | A complete trigger path with receipts, and no valid prevention found | +| `refuted: ` | Something concrete prevents it, with a receipt | +| `unclear: ` | Neither side made its case; say what would settle it | + +Prefer `unclear` over a forced call. A wrong `refuted` buries a real bug, and a +wrong `confirmed` wastes someone's afternoon; `unclear` with a precise "what's +missing" is useful. + +## Reproduction + +A failing test is the strongest evidence and the start of a fix, so the Prover +should attempt one when it's cheap: the repo has a working test command, the +trigger can be expressed as a unit or integration test, and no external services +are needed. + +1. Create an isolated worktree on a scratch branch inside the excluded + `.verify/` directory: + `git worktree add -b verify-bug/ .verify/worktrees/ HEAD` +2. Write the smallest test that should fail if the bug is real, and run only + that test with the repo's own test command, from inside the worktree. +3. Save the test as a patch. New files are untracked, so stage first: + `git -C .verify/worktrees/ add -A && git -C .verify/worktrees/ diff --cached > .verify/patches/.patch` +4. Remove the worktree and branch: + `git worktree remove --force .verify/worktrees/ && git branch -D verify-bug/` + +A test that fails for an unrelated reason (compile error, missing fixture) +proves nothing; fix the test or fall back to a static argument. + +**Ask the user once before** installing dependencies, starting services or +containers, making network calls, or running anything other than the repo's own +test command. Concurrency bugs, external-service bugs, and UI bugs are usually +not cheap to reproduce; don't force it. + +## Output + +Every run writes a local report, plus in-place verdicts for map sources. + +- **Report:** `.verify/-.md`, with patches in + `.verify/patches/`. On first run, before creating any worktree, add `.verify/` + to `.git/info/exclude` so nothing lands in the repo's tracked files. The + report lists every claim (including capped, unverified ones) with its verdict, + the Prover's trigger path, the Skeptic's prevention argument, the Referee's + reasoning, and any patch link. Use the template in `references/roles.md`. +- **Map sources:** replace `verdict: unverified` with the verdict and a one-line + reason, plus a link to the report. For example: + + ```text + verdict: refuted: Sync holds mu across the check (sync/engine.go:88); see .verify/2026-09-24-sync.md + ``` + + The report path is local, so keep the one-line reason self-sufficient. + +- **GitHub sources:** after the summary, offer to draft a reply for each + verified issue or comment. Show the draft; post only after the user confirms + that post. Keep replies factual: verdict, trigger path or prevention, and the + repro test inline if one exists. + +### Chat summary + +Keep it short: counts by verdict, then one line per confirmed bug (claim, +severity, reproduced or static, patch link), then refuted and unclear claims in +a compact list. Mention how many claims were capped. When confirmed bugs exist, +say they're ready to fix and that the repro patch is the first half of the fix. +Don't restate the report. diff --git a/skills/verify-bug/references/roles.md b/skills/verify-bug/references/roles.md new file mode 100644 index 0000000..068c7c4 --- /dev/null +++ b/skills/verify-bug/references/roles.md @@ -0,0 +1,122 @@ +# Role briefs + +Give each role only its brief, the claim record, and repo access. Isolation is +the point: a role that has seen another role's reasoning anchors on it. + +## Contents + +- Prover +- Skeptic +- Referee +- Report template + +## Prover + +> You are arguing that this bug is real. Build the strongest honest case. +> +> Find the entry point where the triggering input or event arrives, and trace +> the path through the code to the failure point, with a receipt at each hop. +> Describe the state at the failure point and the observable effect for a user +> or caller. If the claim has no concrete scenario, construct one; if you can't, +> say so. +> +> If a repro is cheap (working test command, no external services), write the +> smallest failing test in the isolated worktree you've been given and run only +> that test. Report the exact command and output. +> +> Don't invent receipts. If a hop in the path is uncertain, mark it uncertain +> rather than smoothing it over; a weak link stated plainly is more useful than +> a confident wrong one. + +Output: + +```markdown +- trigger: +- path: + 1. + 2. ... +- failure: +- uncertain hops: +- repro: +- patch: +``` + +## Skeptic + +> You are arguing that this bug is not real. Find what prevents it, honestly. +> +> Check upstream validation and middleware, framework and language guarantees, +> locking and ordering, type constraints, config defaults, whether the code path +> is reachable at all, and whether the claim misreads the code. Every prevention +> needs a receipt. If you look hard and find nothing that prevents the bug, say +> so; a Skeptic who concedes a real bug is doing the job. + +Output: + +```markdown +- preventions: + - +- misreadings: +- gaps: +- concede: +``` + +## Referee + +> You rule on whether this bug is real. You have the claim, the Prover's case, +> and the Skeptic's case. Neither is trustworthy by default. +> +> Open every receipt and check it says what's claimed. Discount any argument +> without a receipt. A single valid prevention that holds under the claim's +> scenario refutes it. A complete path with no valid prevention confirms it. A +> passing Prover repro counts against the claim; a repro that fails for the +> stated reason confirms it. If neither side made its case, rule unclear and say +> exactly what would settle it. + +Output: + +```markdown +- verdict: +- reason: +- checked receipts: +- would settle it: +``` + +## Report template + +```markdown +# Verify bug: + +- Date: +- Repo HEAD: +- Source: +- Claims: → · Verified: · Capped: + +## Summary + +| ID | Claim | Severity if true | Verdict | +| --- | ----- | ---------------- | ------- | + +## : + +- Source: (merged: ) +- Scenario: +- Verdict: — + +### Prover + + + +### Skeptic + + + +### Referee + + + +## Capped (unverified) + +- : () +``` diff --git a/skills/whiteboard/SKILL.md b/skills/whiteboard/SKILL.md index 4db85f0..df9c2b5 100644 --- a/skills/whiteboard/SKILL.md +++ b/skills/whiteboard/SKILL.md @@ -194,15 +194,27 @@ Each entry is typed: | `confusing` | Misleading names, surprising indirection, comment contradicts code | zoom | | `inconsistent` | Two patterns doing the same job; docs disagree with code | scout | | `cleanup` | Dead code, abandoned dependency, ancient TODO | any | +| `suspected-bug` | A concrete scenario where the code looks wrong or unhandled | zoom, defense | | `defense-exposed` | The user missed a question because the code misled them | defense | The signal here is simple: if an agent with full read access struggled to understand something, a new contributor will too. Keep entries to one line plus a receipt. Don't fix anything. -When `unexplained` items accumulate, tell the user they are good candidates for -decision records (ADRs) documenting the rationale. Use that plain wording and -nothing more; this skill has no dependency on any records tooling. +A `suspected-bug` entry must name the triggering scenario, not just a feeling: +"two concurrent `Sync` calls both pass the `running` check before either sets it +(`sync/engine.go:Run`)", not "possible race in sync." Don't investigate further +or try to reproduce it; a suspected bug is a lead, and the map treats it as +unverified. Leave the verdict slot empty (see `references/templates.md`) so a +later verification pass can fill it in. + +Handoffs use plain wording and nothing more; this skill depends on no other +tooling: + +- When `unexplained` items accumulate, tell the user they are good candidates + for decision records (ADRs) documenting the rationale. +- When `suspected-bug` items exist, tell the user these suspected bugs could be + verified before anyone acts on them. ### Scout notes diff --git a/skills/whiteboard/references/defense.md b/skills/whiteboard/references/defense.md index a4957a9..27a6225 100644 --- a/skills/whiteboard/references/defense.md +++ b/skills/whiteboard/references/defense.md @@ -107,6 +107,12 @@ it, what would the alternative have cost. These are optional extras in a session, after core questions, and are often the most valuable part for repos the user owns. +Suspected bugs with a `refuted` verdict make excellent `adversarial` questions, +because the code looks wrong and isn't: "Two `Sync` calls can overlap here. Why +isn't that a race?" The verdict's evidence is the ground truth, so these can be +core questions. Entries still `unverified` or `unclear` are framed like inferred +decisions: ask the user whether it's a real bug and why, and don't grade. + ## Write-backs When the user gives a rationale for an inferred or unexplained decision that diff --git a/skills/whiteboard/references/templates.md b/skills/whiteboard/references/templates.md index a025b4a..59374df 100644 --- a/skills/whiteboard/references/templates.md +++ b/skills/whiteboard/references/templates.md @@ -107,9 +107,16 @@ short paragraph. State machines for anything with a lifecycle.> - **:** (``) +If a scenario looks unhandled, also log it as a `suspected-bug` below. + ## Open questions & friction - `` (``) +- `suspected-bug` (``) — verdict: unverified + +A verification pass may later replace `unverified` with `confirmed: `, +`refuted: `, or `unclear: `. Preserve +whatever verdict is there when refreshing. ## Zoom candidates