Skip to content

feat: add verify-bug skill and suspected-bug friction to whiteboard #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markphelps merged 1 commit into from
Sep 24, 2026
Merged

feat: add verify-bug skill and suspected-bug friction to whiteboard #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ Reusable agent plugins and skills following AGENTS/skills conventions.

## Plugins

| Plugin | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [vault](./vault/) | Zettelkasten-style vault workflows for Obsidian notes |
| [development](./development/) | Agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness and marketing, session log audits, decision records, and self-hosted runners |
| Plugin | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [vault](./vault/) | Zettelkasten-style vault workflows for Obsidian notes |
| [development](./development/) | Agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness and marketing, session log audits, bug verification, decision records, and self-hosted runners |

## Layout

Expand Down
18 changes: 12 additions & 6 deletions development/.codex-plugin/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "development",
"version": "0.0.7",
"description": "Developer workflow skills for agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness, OSS marketing, session log audits, decision records, and GitHub Actions self-hosted runners.",
"version": "0.0.8",
"description": "Developer workflow skills for agent context, PR fixups, CLI design, codebase-grounded landing copy, architecture mapping and whiteboard defense, OSS readiness, OSS marketing, session log audits, bug verification, decision records, and GitHub Actions self-hosted runners.",
"author": {
"name": "Mark Phelps"
},
Expand Down Expand Up @@ -41,13 +41,17 @@
"architecture",
"whiteboard",
"design-rationale",
"trade-offs"
"trade-offs",
"verify-bug",
"bug-verification",
"false-positives",
"triage"
],
"skills": "./skills/",
"interface": {
"displayName": "Development",
"shortDescription": "Agent context, PR fixup, CLI design, landing copy, architecture mapping and whiteboard defense, OSS readiness, marketing, session log audit, decision records, and runner workflows",
"longDescription": "Development workflow skills for auditing agent context files, fixing existing GitHub PRs, designing humane command-line interfaces, writing product landing-page copy backed by shipped code, mapping codebase architecture and design rationale, preparing GitHub repos for open source release, sharpening OSS README positioning, mining local agent session logs for papercuts in your own tools, writing and maintaining decision and plan records (ADRs), and running GitHub Actions self-hosted runners on a Linux host.",
"shortDescription": "Agent context, PR fixup, CLI design, landing copy, architecture mapping and whiteboard defense, OSS readiness, marketing, session log audit, bug verification, decision records, and runner workflows",
"longDescription": "Development workflow skills for auditing agent context files, fixing existing GitHub PRs, designing humane command-line interfaces, writing product landing-page copy backed by shipped code, mapping codebase architecture and design rationale, preparing GitHub repos for open source release, sharpening OSS README positioning, mining local agent session logs for papercuts in your own tools, verifying whether claimed bugs are real, writing and maintaining decision and plan records (ADRs), and running GitHub Actions self-hosted runners on a Linux host.",
"developerName": "Mark Phelps",
"category": "Development",
"capabilities": ["Read", "Write"],
Expand All @@ -71,7 +75,9 @@
"Add a second repository runner to this machine and verify it is listening.",
"Write a decision record for switching our job queue to Postgres.",
"Supersede record 012 with a new plan and update the registry.",
"Set up a docs/records directory with a registry README for this repo."
"Set up a docs/records directory with a registry README for this repo.",
"Is this bug from the code review actually real?",
"Verify the suspected bugs in docs/map/ and write the verdicts back."
]
}
}
5 changes: 5 additions & 0 deletions development/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ agent context.
release, emphasizing first-run and first-contribution developer experience
- `session-log-audit`: mine local agent session logs (Claude Code, Pi, Codex)
for papercuts in a project the user builds with agents, then rank the fixes
- `verify-bug`: decide whether claimed bugs are real through an isolated
Prover/Skeptic/Referee hearing, with repro tests in throwaway worktrees

## Workflow Policy

Expand All @@ -54,6 +56,9 @@ agent context.
- For whiteboard maps, never invent design rationale. Tag rationale claims with
evidence, keep defense records local under `.map/`, and never modify source
code while mapping or defending.
- For bug verification, never touch the working tree (repros run only in a
`git worktree` under `.verify/`), never fix code or open issues, and post
verdicts to GitHub only after the user confirms each post.
- For self-hosted runners, resolve the runner account instead of assuming a
user or home directory, keep one namespaced instance per target, treat tokens
as secrets that never reach a file that persists, and confirm no job is
Expand Down
10 changes: 8 additions & 2 deletions development/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Developer workflow skills for repo-facing agent context, GitHub PR follow-up,
command-line interface design, codebase-grounded landing copy, architecture
mapping and whiteboard defense, open source release preparation, audits of your
own agent session logs, decision records, and GitHub Actions self-hosted
runners.
own agent session logs, bug-claim verification, decision records, and GitHub
Actions self-hosted runners.

## Skill Set

Expand All @@ -17,6 +17,7 @@ runners.
| `github-pr-fixup` | Addresses unresolved GitHub PR review comments and failing CI on the existing source branch. | When a user gives you an existing PR URL and wants review feedback or CI failures fixed without a new PR. |
| `github-self-hosted-runner` | Installs, registers, verifies, or removes namespaced GitHub Actions self-hosted runners on a Linux host. | When a user gives a repository or organization URL and wants a persistent runner managed by systemd. |
| `whiteboard` | Maps architecture and design rationale, then runs whiteboard-defense quizzes. | When the user explicitly invokes `/whiteboard` to map a codebase, explore a region, refresh a map, or defend their understanding. |
| `verify-bug` | Rules on whether claimed bugs are real via an isolated Prover/Skeptic/Referee hearing. | When the user asks whether a bug is real, wants findings or review comments verified, or points at suspected bugs from a map. |
| `oss-marketing` | Sharpens README and public-doc positioning for first-time visitors. | When a repo needs launch copy, clearer positioning, or a README that explains what the project is. |
| `oss-repo-readiness` | Audits and prepares a repo for open source release, focused on developer experience. | When making a repo public, writing CONTRIBUTING or issue templates, or running a pre-launch checklist. |
| `session-log-audit` | Mines local agent session logs for papercuts and produces a ranked fix list. | When the user wants to know what is annoying about their own tool, or why they work around it. |
Expand All @@ -40,6 +41,11 @@ runners.
- Whiteboard is explicit-invocation only, never modifies source code, and keeps
personal defense records under `.map/` while committed maps live under
`docs/map/`.
- Whiteboard logs `suspected-bug` leads without investigating them; `verify-bug`
rules on them. Neither skill depends on the other.
- Bug verification never modifies the working tree. Repro tests run only in a
throwaway `git worktree` under `.verify/`, and GitHub replies are posted only
after the user confirms each one.
- Codebase landing copy is for a product's own landing page and must trace
claims to shipped code; use OSS marketing for open-source README positioning.
- Session log audits need a project the user built with agents and used
Expand Down
173 changes: 173 additions & 0 deletions development/skills/verify-bug/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
---
name: verify-bug
description:
Decide whether claimed bugs are real by running an adversarial hearing on each
claim — a Prover argues it's reachable, a Skeptic hunts for what prevents it,
and an independent Referee rules confirmed, refuted, or unclear — with an
isolated repro test when that's cheap. Use when the user asks whether a bug is
real, wants findings or review comments verified, asks if a reported issue
reproduces, or points at suspected bugs from a codebase map. Accepts pasted
claims, code-review comments, GitHub issues, or `suspected-bug` entries in
`docs/map/`. Does not hunt for new bugs, and never fixes code; don't use it
for ordinary debugging of a known failure or for general code review.
---

# Verify bug

Turn claimed bugs into verdicts you can act on. A claim is a lead, not a fact:
code that looks wrong is often protected by validation in another file, a
framework guarantee, a lock, a type, or configuration. And real bugs described
vaguely get dismissed. This skill gives every claim a fair hearing with an
advocate on each side, then records the verdict with its evidence.

## Hard rules

- **Never modify the user's working tree.** Reads happen anywhere; repro tests
happen only in a throwaway `git worktree` (see "Reproduction").
- **Never fix bugs, open issues, or post comments on your own.** Posting a
verdict back to GitHub requires the user's explicit confirmation for that
specific post.
- **No verdict without receipts.** Every argument must point at code
(`path:line` or `path:Symbol`). Arguments without receipts are discounted.

## Workflow

1. **Collect claims** from the source the user gave (see "Sources").
2. **Normalize** each into a claim record (below).
3. **Dedupe:** merge claims that point at the same code path with the same
trigger. Agents often report one root cause several ways; keep the clearest
scenario and list the merged sources.
4. **Rank** by severity _if true_: data loss or corruption, security, silent
wrong results, crashes or outages, degraded behavior, cosmetic. Don't try to
judge likelihood here; that's what the hearing is for.
5. **Cap:** verify the top 10 by default. If more remain, say how many and let
the user pick or raise the cap. Unverified claims keep an `unverified`
verdict; nothing is silently dropped.
6. **Hear** each claim (see "The hearing").
7. **Record** verdicts (see "Output") and summarize in chat.

### Claim record

```markdown
- id: <short slug>
- source: <map path | issue URL | review comment URL | pasted>
- claim: <one-line statement of what goes wrong>
- scenario: <the concrete trigger: input, interleaving, config, or sequence>
- receipt: <path:line or path:Symbol where it supposedly goes wrong>
- severity-if-true: <data | security | silent-wrong | crash | degraded |
cosmetic>
- verdict: unverified
```

If a claim has no concrete scenario ("this might be racy"), ask the Prover to
construct one as its first job. If it can't, the verdict is
`unclear: no concrete trigger`.

## Sources

| Source | How to read it | Where the verdict goes |
| ------------------------------ | ------------------------------------------------------ | ----------------------------------------- |
| `docs/map/*.md` friction lists | `suspected-bug` entries in "Open questions & friction" | In place, replacing `verdict: unverified` |
| GitHub issue or PR comment | `gh issue view`, `gh pr view --comments`, or the API | Report file; draft reply on request |
| Pasted list or review output | The conversation | Report file |

For map entries, keep the entry's wording and replace only the verdict slot.
Preserve any existing verdict unless the code at its receipt has changed since
it was recorded.

## The hearing

Run three roles per claim. When the harness supports subagents, run each role in
its own isolated context, because an agent that has read the claim's framing, or
another role's reasoning, anchors on it. Pass only what each role needs. Role
briefs, including exact output formats, are in `references/roles.md`; read it
before the first hearing.

1. **Prover** — gets the claim record and repo access. Builds the strongest case
that the bug is reachable: the entry point, the path through the code, the
state at the failure point, and the observable effect. May attempt a repro
(see "Reproduction").
2. **Skeptic** — gets the claim record and repo access, _not_ the Prover's
output. Hunts for anything that prevents the bug: upstream validation,
framework or language guarantees, locks and ordering, types, config defaults,
dead code paths, or the claim simply misreading the code.
3. **Referee** — gets the claim record, both structured outputs, and repo
access. Checks each receipt, weighs the arguments, and rules.

Run hearings for different claims in parallel when possible; the three roles for
one claim run Prover and Skeptic in parallel, then Referee.

**Without subagents**, run the roles sequentially yourself: write the Prover
case, then deliberately set it aside and write the Skeptic case from the claim
alone, then referee. Tell the Referee step to discount any argument without a
receipt, since sequential roles bleed into each other.

### Verdicts

| Verdict | Meaning |
| --------------------------- | -------------------------------------------------------------------- |
| `confirmed (reproduced)` | A failing test in an isolated worktree demonstrates it |
| `confirmed (static)` | A complete trigger path with receipts, and no valid prevention found |
| `refuted: <why>` | Something concrete prevents it, with a receipt |
| `unclear: <what's missing>` | Neither side made its case; say what would settle it |

Prefer `unclear` over a forced call. A wrong `refuted` buries a real bug, and a
wrong `confirmed` wastes someone's afternoon; `unclear` with a precise "what's
missing" is useful.

## Reproduction

A failing test is the strongest evidence and the start of a fix, so the Prover
should attempt one when it's cheap: the repo has a working test command, the
trigger can be expressed as a unit or integration test, and no external services
are needed.

1. Create an isolated worktree on a scratch branch inside the excluded
`.verify/` directory:
`git worktree add -b verify-bug/<id> .verify/worktrees/<id> HEAD`
2. Write the smallest test that should fail if the bug is real, and run only
that test with the repo's own test command, from inside the worktree.
3. Save the test as a patch. New files are untracked, so stage first:
`git -C .verify/worktrees/<id> add -A && git -C .verify/worktrees/<id> diff --cached > .verify/patches/<id>.patch`
4. Remove the worktree and branch:
`git worktree remove --force .verify/worktrees/<id> && git branch -D verify-bug/<id>`

A test that fails for an unrelated reason (compile error, missing fixture)
proves nothing; fix the test or fall back to a static argument.

**Ask the user once before** installing dependencies, starting services or
containers, making network calls, or running anything other than the repo's own
test command. Concurrency bugs, external-service bugs, and UI bugs are usually
not cheap to reproduce; don't force it.

## Output

Every run writes a local report, plus in-place verdicts for map sources.

- **Report:** `.verify/<YYYY-MM-DD>-<slug>.md`, with patches in
`.verify/patches/`. On first run, before creating any worktree, add `.verify/`
to `.git/info/exclude` so nothing lands in the repo's tracked files. The
report lists every claim (including capped, unverified ones) with its verdict,
the Prover's trigger path, the Skeptic's prevention argument, the Referee's
reasoning, and any patch link. Use the template in `references/roles.md`.
- **Map sources:** replace `verdict: unverified` with the verdict and a one-line
reason, plus a link to the report. For example:

```text
verdict: refuted: Sync holds mu across the check (sync/engine.go:88); see .verify/2026-09-24-sync.md
```

The report path is local, so keep the one-line reason self-sufficient.

- **GitHub sources:** after the summary, offer to draft a reply for each
verified issue or comment. Show the draft; post only after the user confirms
that post. Keep replies factual: verdict, trigger path or prevention, and the
repro test inline if one exists.

### Chat summary

Keep it short: counts by verdict, then one line per confirmed bug (claim,
severity, reproduced or static, patch link), then refuted and unclear claims in
a compact list. Mention how many claims were capped. When confirmed bugs exist,
say they're ready to fix and that the repro patch is the first half of the fix.
Don't restate the report.
Loading
Loading