Add the cheat-proof tests skill

README.md

@@ -10,6 +10,7 @@ collections that power ML inference and the Smart Window, and growing to host ot
 | Tool | Location | What it is |
 |---|---|---|
 | **docs** | [`./docs/`](./docs/) | A live dashboard visualizing the Remote Settings collections behind Firefox's on-device ML inference and the Smart Window. Served by GitHub Pages from `/docs`. |
+| **skills** | [`./skills/`](./skills/) | Claude Code skills |
 
 ## Task runner
 

Taskfile.yml

@@ -4,6 +4,9 @@ includes:
   docs:
     taskfile: ./docs/Taskfile.yml
     dir: ./docs
+  skills:
+    taskfile: ./skills/Taskfile.yml
+    dir: ./skills
 
 tasks:
   default:

skills/README.md

@@ -0,0 +1,24 @@
+# Skills
+
+[Claude Code skills](https://docs.claude.com/en/docs/claude-code/skills) for working in the Firefox
+tree. Each skill is one directory with a `SKILL.md` at its root.
+
+| Skill | What it is |
+|---|---|
+| [`write-cheatproof-test`](./write-cheatproof-test/) | Writing end-to-end integration tests in the Firefox tree. One rule: if the test passes, it must prove the code works when it leaves the harness.  |
+
+## Install
+
+```sh
+task skills:install     # symlink every skill into ~/.claude/skills
+task skills:uninstall
+```
+
+It symlinks rather than copies, so `git pull` updates the skills in place, and it stops short of
+clobbering a real directory in `~/.claude/skills`. Or skip the task runner and copy a skill
+directory wherever your agent reads skills from.
+
+## Adding a skill
+
+Drop a new directory here with a `SKILL.md` inside. `task skills:install` picks it up with no
+further wiring — nothing outside this folder needs to change beyond a row in the table above.

skills/Taskfile.yml

@@ -0,0 +1,28 @@
+version: '3'
+
+vars:
+  DEST: $HOME/.claude/skills
+
+tasks:
+  install:
+    desc: Symlink every skill here into ~/.claude/skills so Claude Code loads it
+    cmds:
+      - |
+        mkdir -p "{{.DEST}}"
+        for dir in */SKILL.md; do
+          skill=$(dirname "$dir")
+          if [ -e "{{.DEST}}/$skill" ] && [ ! -L "{{.DEST}}/$skill" ]; then
+            echo "{{.DEST}}/$skill is a real directory; move it aside first" >&2
+            exit 1
+          fi
+          ln -sfn "$PWD/$skill" "{{.DEST}}/$skill"
+        done
+
+  uninstall:
+    desc: Remove the skill symlinks from ~/.claude/skills
+    cmds:
+      - |
+        for dir in */SKILL.md; do
+          skill=$(dirname "$dir")
+          if [ -L "{{.DEST}}/$skill" ]; then rm -f "{{.DEST}}/$skill"; fi
+        done

skills/write-cheatproof-test/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: write-cheatproof-test
+description: Write a cheat-proof end-to-end test in the Firefox tree: real browser, real page, real UI, with only the expensive external boundary mocked. Use when writing, adding or fixing a test for Smart Window / AI Window, PageExtractor or toolkit/components/ml code, when reviewing whether a test proves anything, or when reproducing a bug as a test. Trigger on "write a test for X", "add test coverage", "reproduce this bug as a test", "is this test cheat-proof", "does this test prove anything".
+---
+
+# Cheat-proof tests
+
+One rule: **if the test passes, it must prove the code works when it leaves the harness.** A
+test that passes because someone mocked it into passing convinces a domain expert that broken
+code works.
+
+You are writing an **end-to-end integration test**: a real browser window, a real page served
+over a real HTTP server, the component reached through the same wiring production uses, and one
+mocked external service. Leave the collaborators real.
+
+**Mirror an exemplar, do not invent an approach.** The exemplars below encode where the
+acceptable mock boundary sits, so read one for that judgment instead of reconstructing it here.
+
+## The method
+
+Work through these in order, and do not skip step 1 or step 5.
+
+1. **Pick the exemplar** from the index below and `Read` it in full, plus its `head.js`.
+2. **State the unit of work** in one sentence: the component whose behavior you are proving.
+   Treat everything around it as environment.
+3. **Reuse the vocabulary** below. When no helper exists for a seam you need, build the helper
+   by hand rather than reaching into internals from the test.
+4. **Structure it as arrange / act / assert**, with declarative input.
+5. **Prove it can fail** with the mutation loop below. Skip it and you have no evidence.
+6. **Cite the exemplar** in your report and in the test's doc comment.
+
+## Exemplar index
+
+Three areas, each with its own `head.js` and fixtures. Paths are relative to the area root.
+Read the file in full, plus every `head.js` the adjacent `browser.toml` loads.
+`references/exemplars.md` annotates five of these and inventories the fixtures.
+
+### `browser/components/aiwindow`, tests in `ui/test/browser/`
+
+| Testing this | Read and mirror |
+| --- | --- |
+| A user flow through the Smart Window UI, with a controlled LLM | `browser_security_chat.js` |
+| Security properties (private data / untrusted content) | `browser_security_chat.js`, the base example whose own doc comment asks you to cite it |
+| A tool call being made or blocked | `browser_security_run_search.js` |
+| A Lit custom element in isolation | `browser_aiwindow_website_chip.js` + its `test_website_chip_page.html` |
+
+### `toolkit/components/pageextractor`, tests in `tests/browser/`
+
+| Testing this | Read and mirror |
+| --- | --- |
+| A component against a real web page | `browser_dom_extractor.js`, whose later tasks also show a table-driven option matrix |
+| A different page source or scope | `browser_dom_extractor_pdf.js`, `browser_dom_extractor_reader_mode.js`, `browser_viewport_extractor.js`, `browser_page_metadata.js` |
+| Extraction with no user-visible tab | `browser_headless_extractor.js`, `browser_anonymous_headless_extractor.js` |
+| Extraction driven by a tool call | `browser_dom_extractor_search_tool.js` |
+
+### `toolkit/components/ml`, tests in `tests/browser/` and `tests/xpcshell/`
+
+| Testing this | Read and mirror |
+| --- | --- |
+| A real engine end to end: init, RS config enrichment, parallel runs, wasm download failure | `tests/browser/browser_ml_engine_lifetime.js` |
+| Cancelling an engine mid-run | `tests/browser/browser_ml_engine_e2e.js` |
+| The OpenAI chat protocol at the network layer | `tests/browser/browser_ml_openai.js` |
+| Glean metrics emitted by an inference run | `tests/browser/browser_ml_telemetry.js` |
+| `PipelineOptions` merge and modelHub revision semantics | `tests/browser/browser_ml_engine_process.js` |
+| The mock LLM engine API itself | `tests/browser/browser_ml_mock_llm_engine.js` |
+| Windowless policy and security logic | `tests/xpcshell/test_security_orchestrator.js` and siblings. xpcshell is correct here |
+
+Some neighbors in these directories fail the bar, so read before copying.
+`browser_ml_privatebrowsing.js` asserts `deepEqual(models, [])` against a fake hub holding no
+models, which passes whether or not private browsing changes anything (kill question 6).
+`browser_ml_engine_security.js` checks that the security layer "is correctly integrated" without
+making it reach a decision (kill question 3).
+
+Outside these three areas: locate the nearest `browser.toml`, read the `head.js` next to it, and
+pick the existing test that runs closest to real. When nothing nearby is cheat-proof, say so
+instead of copying a bad neighbor.
+
+## Vocabulary
+
+Reach for these names. Each marks the boundary between acceptable environment and application
+internals. `references/exemplars.md` carries the full parameter lists.
+
+**A real page in a real tab**, from `MLTestUtils`
+(`resource://testing-common/MLTestUtils.sys.mjs`):
+
+```js
+const { html } = MLTestUtils.serveHTMLInTab({ browser: gBrowser });
+const { tab, url, getPageExtractor, cleanup } = await html`
+  <h1>News Article</h1>
+  <p>This is a news article about technology.</p>
+`;
+```
+
+`serveSharedHTMLInTab` when you need many tabs from one server, or a URL with no tab behind it.
+`serveStalledPage` and `serveRedirect` for load-timeout and redirect behavior.
+
+**A controlled language model**, from `AIWindowTestUtils`
+(`resource://testing-common/AIWindowTestUtils.sys.mjs`):
+
+```js
+const mockEngineManager = new MockEngineManager();   // before any Smart Window opens
+await mockEngineManager.respondTo({ purpose: "chat", response: "This page has no title." });
+```
+
+`captureRequest({ purpose })` returns `{ request, respond }` and lets you assert on what real
+code sent before you decide the reply. The assertion runs against real inputs rather than
+test-fed values, which makes it the strongest cheat-proof move on offer. `MockSearchManager` has
+the same shape for `ExaSearchProvider._fetch`.
+
+**The real UI**, from `browser/components/aiwindow/ui/test/browser/head.js`:
+`openAIWindowWithSidebar` · `typeInSmartbar` · `submitSmartbar` · `clickNewChatButton` ·
+`getSidebarChatMessages` · `checkForElementInChatMessage` · `spawnBounded` /
+`waitForMutationBounded` for waits that fail fast instead of hanging until the harness aborts.
+
+Drive the UI the way a user does: click the button, type in the field. Following Testing
+Library's guiding principles, prefer accessible, user-visible handles over structural selectors.
+You get behavior correctness and cheat-proof accessibility out of the same assertion.
+
+**A real engine over faked Remote Settings**, from `head.js` beside
+`toolkit/components/ml/tests/browser/`:
+
+```js
+const { remoteClients, cleanup } = await setup({ prefs: [["browser.ml.enable", true]] });
+```
+
+`setup()` mocks Remote Settings, pushes the standard `browser.ml.*` prefs and resets FOG. Its
+`cleanup()` waits on `EngineProcess.areAllEnginesTerminated()`, so a leaked engine fails the
+test rather than the next one. Release model downloads with
+`remoteClients[name].resolvePendingDownloads(n)`, and prove the process started with
+`checkForRemoteType("inference")`.
+
+## Shape: arrange, act, assert
+
+Each `add_task` runs three phases, in that order, once each. Comment the boundary wherever it is
+not obvious.
+
+```js
+add_task(async function test_dom_extractor_default_options() {
+  // Arrange: a real page in a real tab, served over a real HttpServer.
+  const { html } = await MLTestUtils.serveHTMLInTab({ browser: gBrowser });
+  const { getPageExtractor, cleanup } = await html`
+    <article>
+      <h1>Hello World</h1>
+      <p>This is a paragraph</p>
+    </article>
+  `;
+
+  // Act: exercise the real actor, reached the way production reaches it.
+  const { text } = await getPageExtractor().getText();
+
+  // Assert: expected value written as data.
+  is(
+    text,
+    ["Hello World", "This is a paragraph"].join("\n"),
+    "Text can be extracted from the page."
+  );
+
+  return cleanup();
+});
+```
+
+For a Smart Window flow: arrange is `new MockEngineManager()` then `openAIWindowWithSidebar()`
+then `serveHTMLInTab`; act is `typeInSmartbar` / `submitSmartbar` then `respondTo` or
+`captureRequest`; assert reads the rendered DOM and the conversation state.
+
+Three consequences worth knowing:
+
+- **One act per task**, and nothing in arrange asserts about the subject. A task that interleaves
+  acts and assertions cannot tell a reader which step regressed.
+- **`respondTo` belongs to act**, because it unblocks real code that is already in flight. Only
+  `new MockEngineManager()` sits in arrange, constructed before the window opens.
+  **`captureRequest` straddles act and assert by design**: assert on the real request mid-flow,
+  release it, and note in a comment that you did.
+- **Capture pre-act state during arrange** when the assertion is about a *change*.
+  `browser_security_chat.js` grabs the conversation before submit so the initial-state
+  assertions run before `getRealTimeInfo` mutates it. That ordering is behavior.
+
+## Hard rules
+
+- **Never stub the code under test.** Mock the external service and leave your own component
+  real. When the stub is what makes the assertion pass, the test proves nothing.
+- **In `browser/components/aiwindow/ui`, avoid `stubEngineNetworkBoundaries`,
+  `startMockOpenAI` and `withServer`.** That `head.js` marks them `@deprecated` in favor of
+  `MockEngineManager` (Bug 2045844). The JSDoc grants one exception, a test whose subject is the
+  OpenAI chat network protocol, and taking it costs you a sentence in the doc comment. These
+  deprecations are area-scoped: `toolkit/components/ml` has its own undeprecated
+  `startMockOpenAI`, so use the local copy there. See `references/exemplars.md`.
+- **Never write an xpcshell test for AI Window / Smart Window UI behavior**, which needs a real
+  window, sidebar and content process. In `toolkit/components/ml`, xpcshell is right for
+  windowless policy and security logic and wrong once an engine or a page is involved.
+- **No `waitForCondition` on a bare timer** to paper over a race. Wait on the condition itself,
+  bounded.
+- **Register new support files** in the adjacent `browser.toml` `support-files` list, or the test
+  passes locally and fails in CI.
+- **Assertion messages are sentences** stating the behavior you proved rather than the mechanism:
+  `"The conversation gets marked as private as the tab info is added to it."`
+- Clean up: `cleanupMocks()`, `BrowserTestUtils.closeWindow(win)`, and every `cleanup()` the
+  fixtures handed you.
+
+## Prove it can fail
+
+A test you have only ever seen pass is not evidence. Run the loop:
+
+```
+1. ./mach test --headless <test path>          -> must be GREEN
+2. break the code under test (one real edit)
+3. ./mach test --headless <test path>          -> must be RED
+4. revert the break
+5. ./mach test --headless <test path>          -> must be GREEN again
+```
+
+Step 2 edits the source file, not the test: flip the boolean the assertion reads, drop the
+sanitizer call, return early from the handler. Pick a break resembling a plausible regression.
+
+**A test that stays green at step 3 is worthless. Rewrite it or delete it.** Do not debug around
+it, and do not report it as done. Never leave the break in the tree: revert it, then confirm
+step 5 before reporting.
+
+Editing an existing JS/HTML test file needs no build step. Run `./mach build faster` when a
+`.sys.mjs` was **added**, by you or by a pull, because unpackaged files fail to load at runtime
+with a misleading `Failed to load moz-src:///…` and a stale test result. When a fixture behaves
+impossibly, suspect a stale objdir before suspecting your test.
+
+Redirect slow test output to a file under `artifacts/` and read that rather than piping through
+`tail` or `grep`.
+
+## Reproducing a bug as a test
+
+Given STR / ER / AR, mirror the STR line for line: same order, same user actions, same page. A
+reader should be able to diff the bug report against the test. Write the test first, watch it
+fail for the reported reason, then fix the code.
+
+## Report
+
+State the unit of work, the exemplar you mirrored, the one thing you mocked and why, and the
+mutation that proved the test can fail. When you bend a hard rule, name which one and why.