diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 639fbd7..4444dc3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -67,6 +67,25 @@ jobs: exit 1 fi + # story 055.W4.2 (D17 + D20(4)) — capability analysis. Deliberately ONE additive step: the + # analyzer's own unit tests (test/capability-analyzer.test.mjs) already run in the + # "Unit tests" step above via the test/*.test.mjs glob, and the publish-time gates (AC1 + # blocking, AC3 blocking, AC6 warn-and-display) run inside publisher/publish.mjs and are + # covered by test/publish-cli.test.mjs against the real CLI. What is left for CI to prove is + # that the analyzer BINARY actually runs end-to-end over a real artifact, which is what a + # unit test cannot show. + - name: Capability analysis runs over a real artifact (D20(4) — derived, never self-declared) + run: | + set -e + tmp="$(mktemp -d)" + mkdir -p "$tmp/pkg/skills/demo" + printf 'MIT\n' > "$tmp/pkg/LICENSE" + printf -- '---\nname: demo\ndescription: demo\nallowed-tools: Read, Bash\n---\n\nRun `bash scripts/run-thing.sh` first.\n' \ + > "$tmp/pkg/skills/demo/SKILL.md" + tar -czf "$tmp/demo.tar.gz" -C "$tmp/pkg" . + node scripts/analyze-capabilities.mjs --artifact "$tmp/demo.tar.gz" --require-allowed-tools + echo "OK — analyzer ran end-to-end over a real artifact" + - name: No obvious secret shapes committed (base guard — NOT the D20(1) blocking scanner) run: | set -e diff --git a/README.md b/README.md index 237803a..6f2052e 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,24 @@ catalog side of those decisions; the Cockpit-side consumer lives in the product | `docs/SCHEMA.md` | Field-by-field explanation of the index entry schema | — | | `docs/INVARIANTS.md` | The four no-going-back invariants (D24 a/b/c + D21's `AC8`), how each is verified, and the explicitly-named design boundaries | — | +## Capability analysis + mandatory `allowed-tools` (`055.W4.2`) + +Every publishable skill MUST declare `allowed-tools` (kebab-case; comma/space string or YAML list). +Publishing without it fails, unconditionally. Capabilities shown to the user are **DERIVED** by +AIOX-side static analysis from the artifact's bytes — the publisher has no field in which to +declare them, and a manifest that tries is refused. + +This delivers **visibility, not containment**: nothing here sandboxes a plugin. v1 **warns and +displays**; the blocking path exists in code, off by configuration, with the documented trigger +"when opening to externals". + +```bash +node scripts/analyze-capabilities.mjs --artifact --require-allowed-tools +node scripts/analyze-capabilities.mjs --dir --json +``` + +Full design reasoning, the two signals, and what the analysis **cannot** see: `docs/CAPABILITIES.md`. + ## Testing ```bash diff --git a/docs/CAPABILITIES.md b/docs/CAPABILITIES.md new file mode 100644 index 0000000..e0e4dbf --- /dev/null +++ b/docs/CAPABILITIES.md @@ -0,0 +1,168 @@ +# Derived capabilities + mandatory `allowed-tools` + +Story `055.W4.2` · ADR-COCKPIT-ENTERPRISE-PREMIUM-PACK **D17** + **D20(4)**. + +> **Read this first: this delivers VISIBILITY, not CONTAINMENT.** +> Nothing in this document sandboxes a plugin. Real containment needs an execution host, and the +> ADR rejects a third-party WASM host **with evidence** — *"o Zed provou que mesmo com wasmtime +> completo a extensão não desenha UI"*. What ships here lets a user **see** what a plugin is +> capable of before installing it. If you read any of the below as "the plugin is now contained", +> you have read it wrong. + +--- + +## 1. The threat model is not the obvious one + +The intuitive frontier — *declarative markdown is safe, scripts are dangerous* — does not match the +real risk. In an app where **an agent executes what it reads**, markdown is not the safe class: it +is the class with the **widest reach and the least visibility**. + +Measured in the product repo's SOT (`.aiox-core/skills/`), 2026-08-10: + +| Measurement | Result | +|---|---| +| Skills declaring `allowed-tools` before this story | **0 of 35** | +| Skills that ship their own `scripts/` (`OWNS_SCRIPTS`) | **6 of 35** | +| Skills whose body instructs executing a script (`INSTRUCTS_EXECUTION`) | **9 of 35** | +| Is an MCP server an inspectable artifact? | **No — a `{command, args}` pointer** (`mcp.rs:68`) | + +D17's prohibition on third-party `scripts/` remains valid, but it is verifiable **over the folder, +not over the behaviour**. That is why the analyzer emits two signals and never one number. + +## 2. Two independent signals, never collapsed + +``` +OWNS_SCRIPTS the skill ships its own scripts/ directory → it HOSTS code +INSTRUCTS_EXECUTION the body tells the agent to EXECUTE a script → it RUNS code +``` + +They are **orthogonal**, and both matter: *who executes third-party script matters as much as who +hosts one.* A skill can host scripts it never runs; a skill can run scripts it does not host. +`INSTRUCTS_EXECUTION` is further classified by **whose** script: + +| Form | Meaning | Real example measured in the SOT | +|---|---|---| +| `own` | the skill's own `scripts/` | `close` → `node /scripts/write-ack.mjs` | +| `other-skill` | **another skill's** script | `validate` → `full-cycle/scripts/full-cycle-guard.mjs` | +| `repo` | the repository's top-level `scripts/` | `update-cockpit` → `bash scripts/run-aiox.sh` | +| `ambient` | a **bare name**, resolved from PATH/cwd at runtime | `criar-sot` → ``rode `registry-governance-check.js` `` | + +### Why the count is not an acceptance criterion + +**A measured number rots.** Three different counts circulated for `INSTRUCTS_EXECUTION` while this +story was being written (7 → 8 → 9) and *each was correct for its moment*: `validate` gained a +script reference after the ADR was written. + +Worse, the naive regex everyone was quoting — +`grep -rlE 'scripts/[a-z]|\.mjs|\.sh' .aiox-core/skills/*/SKILL.md` — returns **9**, and the +correct answer is **also 9**, but they are **different sets**: + +- it **wrongly includes** `self-heal` (a false positive — see §3), and +- it **wrongly excludes** `criar-sot` (a false negative: `.js`, no `scripts/` prefix, so the + pattern never sees it). + +Two opposite errors that happened to cancel out in the total. **Had the acceptance criterion fixed +a number, a wrong set would have passed it.** So the criterion requires the two signals, the +negative fixture, and coverage of every execution form — never a count. + +## 3. `self-heal` — the mandatory negative fixture + +`self-heal` has **no `scripts/` directory at all**, and its single script-shaped match is prose +citing another tool **by analogy**: + +> **Adapters mirror the `sync.mjs` ADAPTERS pattern** (add a CLI = add an adapter; the loop is +> untouched). + +The discriminant is **execution instruction vs. mention by pattern analogy**. The analyzer requires +either an interpreter in front of the token (`node …`, `bash …`, `powershell -File …`, `./…`) or an +imperative run-verb close in front of it (`run`, and PT-BR `rode`/`execute` — this corpus is +bilingual), and it rejects negated forms (*"never calls `wave-launch.js`"*). + +**If the analyzer marks `self-heal`, it has reproduced inside itself the exact calibration error +D17 exists to correct.** `test/capability-analyzer.test.mjs` asserts it is in **neither** signal. + +## 4. `allowed-tools` is mandatory (AC1) — and it is *not* a sandbox + +Every publishable skill MUST declare `allowed-tools` in its frontmatter. Publishing without it +**fails**, unconditionally — no flag, no env var, no branch disables it +(`lib/capability-analyzer.mjs::checkAllowedToolsDeclared`, wired into `publisher/publish.mjs`). + +The canonical contract, verified against Claude Code's own documentation: + +- Spelling is **kebab-case** `allowed-tools`. `allowed_tools`/`allowedTools` are silently ignored + by the runtime, so this repo **refuses** them rather than accepting a declaration that would + never take effect. +- The value may be a **comma-separated** string, a **space-separated** string, or a **YAML list**. + All three parse here. +- `allowed-tools: *` is refused — a wildcard passes a presence check while declaring nothing. + +> **It GRANTS, it does not RESTRICT.** `allowed-tools` pre-approves tools for the turn that invokes +> the skill; a tool *absent* from the list remains callable and merely prompts. So the declaration +> is a **floor on stated intent, never a ceiling on reach** — which is precisely why the displayed +> capabilities are derived from the body instead of read off this field. + +## 5. Capabilities are DERIVED, never self-declared (AC3) + +A capability declared by an untrusted party is worth zero. In Zed it is imposed by the `wasmtime` +host; over a stdio subprocess it is a label the author chose. + +So **the publisher has no field in which to assert its own capabilities.** This is enforced as a +refusal, not as "we ignore that field" — a field that is ignored today gets read by accident +tomorrow. A manifest carrying any of `capabilities`, `permissions`, `grants`, `sandbox` or +`trust_level` **fails the publish** (`checkNoSelfDeclaredCapabilities`). + +The derived block is written onto the entry by `publisher/publish.mjs`, computed from the +artifact's **actual bytes**, and carries `self_declared: false` as **data** — so a reader of the raw +index can tell at a glance that it was computed, not accepted. `lib/entry-schema.mjs` re-checks that +the flag is literally `false`. + +The declaration from §4 has exactly one role: a **second opinion to diff against**. When a skill's +body plainly does more than it declares, that surfaces as a `discrepancy` — it never shrinks the +displayed capability set. + +## 6. What this analysis CANNOT see (AC5) + +These travel **with** the capabilities, always — onto the entry, and into the rendered catalog. The +coupling is deliberate: **a capability list displayed without its blind spots lies by omission.** + +- **An MCP server is a runtime-resolved pointer, not an artifact.** The manifest supplies + `{command, args}` (product repo `mcp.rs:68`, typically `npx `) against a registry AIOX + does not control. This analysis covers the **pointer**; it has never opened the **target**. The + `npx` target is never inspected, downloaded or executed. +- A signature over the index covers the pointer, not the pointed-at package. **Provenance is not + behaviour.** +- This is **static analysis of natural-language prose**. An execution instruction phrased in a form + the probes do not match will be missed. **Absence of a signal is not proof of absence of the + behaviour.** +- An `ambient` instruction resolves from PATH/cwd **on the user's machine**; which file actually + runs is not knowable here. +- `allowed-tools` grants rather than restricts (§4). +- **Only skills are analyzed.** Other executable content (binaries, hooks, postinstall) is outside + this pass. + +## 7. v1 warns and displays; the blocking path is off (AC6) + +Per D17 this is an explicit decision, not laxity. The blocking path **exists in code** and is +exercised by tests — `BLOCK_ON_CAPABILITY_FINDINGS` / `capabilityFindingsAreBlocking(override)` and +the `--strict` flag on the CLI. + +**Documented trigger to turn it on: when the catalog opens to EXTERNAL publishers.** + +## 8. Running it + +```bash +# over a plugin artifact (the publish-time / CI input) +node scripts/analyze-capabilities.mjs --artifact [--json] [--require-allowed-tools] + +# over a directory of skills (e.g. a SOT skills tree) — same analyzer, different input adapter, +# so the numbers a maintainer reproduces here are the numbers the publish path computes +node scripts/analyze-capabilities.mjs --dir [--json] +``` + +At publish time all of this is automatic: AC1 blocks, AC3 blocks, the derived block is attached to +the entry, and the report is printed for the operator. + +--- + +*Story 055.W4.2 — AIOX Cockpit plugin system. See `docs/INVARIANTS.md` for D24's no-going-back +invariants and `docs/SCHEMA.md` for the entry shape.* diff --git a/docs/SCHEMA.md b/docs/SCHEMA.md index 15161e7..22d4322 100644 --- a/docs/SCHEMA.md +++ b/docs/SCHEMA.md @@ -29,6 +29,7 @@ the production index was still empty, which is the only moment it costs nothing | `publisher.subject` | yes | The entitlement subject that published this entry (D22). Never a GitHub handle — publishing is itself an entitlement (D16). | | `published_at` | yes | ISO-8601 timestamp. | | `license.spdx_or_path` | yes | SPDX id or in-package path to the license found at the package root, checked at publish (D24(c)). | +| `capabilities` | no | **DERIVED** capability analysis (D17 + D20(4), story `055.W4.2`). Written EXCLUSIVELY by `publisher/publish.mjs` from `lib/capability-analyzer.mjs`, computed from the artifact's own bytes. Carries `self_declared: false` as data, the per-skill **two signals** (`owns_scripts` / `instructs_execution`), the derived `union`, and a **non-empty `limits`** array. The publisher has **no manifest field** that feeds this — a manifest carrying `capabilities`/`permissions`/`grants`/`sandbox`/`trust_level` is refused outright. See `docs/CAPABILITIES.md`. | | `overlay.shadows` | no | Present only when the plugin declares shadowing one or more base skills (D23). `{ "": "" }`. This is **REUSE**, not a new mechanism — it mirrors the identical `overlay.shadows` block already defined and enforced in the product repo's `.aiox-core/sync/OVERLAY-MANIFEST.md` (story `055.W2.2`). The catalog only *renders* it (AC9). | ## What is deliberately absent from this version of the schema @@ -38,6 +39,10 @@ the production index was still empty, which is the only moment it costs nothing now, before the mechanism exists, would make the schema imply behavior the repo doesn't have. - A signature field over the entry or the index. Signing the index (D20(3)) is story `055.W4.3`, with its own key material in a vault kept separate from the entitlement signing key. +- Any field in which a PUBLISHER could state its own capabilities. Its absence is the point, not an + omission (D17/AC3, story `055.W4.2`): a capability asserted by the party being assessed is worth + zero, so the only capability data in an entry is the DERIVED `capabilities` block above. This is + enforced as a refusal at publish time, never as a silently-ignored field. ## Versioning this schema diff --git a/lib/capability-analyzer.mjs b/lib/capability-analyzer.mjs new file mode 100644 index 0000000..a6560ef --- /dev/null +++ b/lib/capability-analyzer.mjs @@ -0,0 +1,499 @@ +// lib/capability-analyzer.mjs — story 055.W4.2, D17 + D20(4). +// +// WHAT THIS IS: a STATIC analyzer that DERIVES what a plugin's skills are capable of doing, by +// reading the skills' own bodies. It is the AIOX side of the wire, run by AIOX-operated tooling +// (publish-time + CI), never by the publisher. +// +// WHAT THIS IS NOT — and this framing is load-bearing, not a disclaimer: this delivers +// VISIBILITY, NOT CONTAINMENT. Nothing here sandboxes anything. Real containment needs an +// execution host, and the ADR rejects a third-party WASM host with evidence ("o Zed provou que +// mesmo com wasmtime completo a extensão não desenha UI"). Per D17, v1 WARNS AND DISPLAYS; the +// blocking path exists below (see `BLOCK_ON_CAPABILITY_FINDINGS`) but is OFF by configuration, +// with its documented trigger: "when opening to externals". +// +// WHY DERIVED AND NOT DECLARED (D17): a capability declared by an untrusted party is worth zero. +// In Zed it is imposed by the `wasmtime` host; over a stdio subprocess it is a label. So the +// publisher has NO manifest field in which to assert its own capabilities — see +// `checkNoSelfDeclaredCapabilities` below, which refuses the publish outright if one appears. +// +// THE THREAT MODEL IS NOT THE OBVIOUS ONE (finding C2, measured). In an app where an agent +// EXECUTES WHAT IT READS, declarative markdown is not the safe class — it is the class with the +// widest reach and the least visibility. The prohibition on third-party `scripts/` is verifiable +// over the FOLDER, not over the BEHAVIOUR, so this analyzer emits TWO INDEPENDENT SIGNALS per +// skill and never collapses them into one number: +// +// OWNS_SCRIPTS — the skill ships its own `scripts/` directory (it HOSTS code). +// INSTRUCTS_EXECUTION — the skill's body tells the agent to EXECUTE a script: its own, +// ANOTHER skill's, or the repo's. +// +// They are orthogonal and both matter: who EXECUTES third-party script matters as much as who +// HOSTS one. A skill can host scripts it never runs, and can run scripts it does not host. + +import { execFileSync } from "node:child_process"; +import { readFileSync, readdirSync, existsSync, mkdtempSync, rmSync, statSync } from "node:fs"; +import { join, sep } from "node:path"; +import { tmpdir } from "node:os"; + +// ── frontmatter ──────────────────────────────────────────────────────────────────────────────── + +// `allowed-tools` is kebab-case and accepts a space- or comma-separated string OR a YAML list. +// All three shapes are valid Claude Code skill frontmatter, so all three must parse here — a +// parser that only understood one shape would report "absent" for a perfectly valid declaration +// and turn AC1 into a false accusation. +export function parseFrontmatter(text) { + const m = text.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/); + if (!m) return { present: false, fields: {}, body: text }; + + const fields = {}; + const lines = m[1].split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const kv = lines[i].match(/^([A-Za-z0-9_-]+)\s*:\s*(.*)$/); + if (!kv) continue; + const [, key, rawValue] = kv; + let value = rawValue.trim(); + if (value === "" || value === "|" || value === ">") { + // block scalar or YAML list — consume the indented continuation + const items = []; + let block = []; + for (let j = i + 1; j < lines.length; j++) { + const li = lines[j].match(/^\s*-\s+(.*)$/); + if (li) { items.push(li[1].trim()); i = j; continue; } + if (/^\s+\S/.test(lines[j])) { block.push(lines[j].trim()); i = j; continue; } + break; + } + fields[key] = items.length ? items : block.join(" "); + continue; + } + fields[key] = value.replace(/^["']|["']$/g, ""); + } + return { present: true, fields, body: text.slice(m[0].length) }; +} + +export function parseAllowedTools(value) { + if (value == null) return null; + if (Array.isArray(value)) return value.map((v) => String(v).trim()).filter(Boolean); + const s = String(value).trim(); + if (!s) return []; + return s.split(/[,\s]+/).map((t) => t.trim()).filter(Boolean); +} + +// ── AC1 — `allowed-tools` is MANDATORY on every publishable skill ────────────────────────────── + +// A skill with no `allowed-tools` is a skill whose author never had to state, even once, what it +// reaches for. Refusing is cheap and the declaration is the only thing the analyzer can COMPARE +// its derived findings against (see `discrepancies` below) — without it there is no second +// opinion, only the analyzer's own. +export function checkAllowedToolsDeclared(skills) { + const errors = []; + for (const skill of skills) { + const { present, fields } = parseFrontmatter(skill.content); + if (!present) { + errors.push(`skill "${skill.name}" (${skill.path}): no YAML frontmatter at all — refusing: \`allowed-tools\` is REQUIRED on every publishable skill (D17/AC1)`); + continue; + } + const raw = fields["allowed-tools"] ?? fields["allowed_tools"] ?? fields["allowedTools"]; + if (raw === undefined) { + errors.push(`skill "${skill.name}" (${skill.path}): frontmatter declares no \`allowed-tools\` — refusing (D17/AC1). Declare the tools this skill actually uses, e.g. \`allowed-tools: Read, Grep, Bash\`. The field is kebab-case and accepts a comma- or space-separated string, or a YAML list.`); + continue; + } + if (fields["allowed_tools"] !== undefined || fields["allowedTools"] !== undefined) { + errors.push(`skill "${skill.name}" (${skill.path}): the field is spelled \`allowed-tools\` (kebab-case). \`allowed_tools\`/\`allowedTools\` are silently ignored by the runtime, so accepting them here would let a skill ship with a declaration that never takes effect.`); + continue; + } + const tools = parseAllowedTools(raw); + if (!tools || tools.length === 0) { + errors.push(`skill "${skill.name}" (${skill.path}): \`allowed-tools\` is present but empty — refusing: an empty declaration is indistinguishable from no declaration.`); + continue; + } + if (tools.includes("*")) { + errors.push(`skill "${skill.name}" (${skill.path}): \`allowed-tools: *\` is refused — a wildcard passes the presence check while declaring nothing. Enumerate the tools the skill actually uses.`); + } + } + return errors; +} + +// ── AC3 — the publisher has NO field in which to declare its own capabilities ────────────────── + +// The literal proof AC3 asks for: not "we ignore the field", but "the field is refused". A +// manifest that tries to assert capabilities fails the publish, so no declared value can ever +// reach the user-facing display by any path. +const SELF_DECLARED_CAPABILITY_FIELDS = ["capabilities", "permissions", "grants", "sandbox", "trust_level"]; + +export function checkNoSelfDeclaredCapabilities(manifest) { + const errors = []; + for (const field of SELF_DECLARED_CAPABILITY_FIELDS) { + if (manifest && Object.prototype.hasOwnProperty.call(manifest, field)) { + errors.push( + `manifest declares \`${field}\` — refusing (D17/AC3): capabilities are DERIVED by static analysis on the AIOX side, NEVER self-declared by the publisher. A capability asserted by the party being assessed is worth zero (in Zed it is imposed by the wasmtime host; over a stdio subprocess it is just a label), and displaying it would let an author choose the label the user calibrates trust against. Remove the field; the catalog computes and displays the derived analysis instead.`, + ); + } + } + return errors; +} + +// ── the two signals ──────────────────────────────────────────────────────────────────────────── + +// A script-file token. The basename must start LOWERCASE, which is what keeps framework names out +// of the corpus: "Node.js", "Next.js", "Vue.js" are capitalized and therefore never match, while +// "run-aiox.sh", "write-ack.mjs" and "registry-governance-check.js" do. Both path separators are +// accepted because the Windows branch of a real instruction uses backslashes +// (`-File scripts\run-aiox.ps1`). +// The leading lookbehind is what keeps FRAMEWORK NAMES out of the corpus. Without it, "Next.js" +// matches from its `e` ("ext.js") and "Node.js" from its `o`. Requiring a non-word character +// immediately before the match, plus a lowercase first character, rejects both while still +// accepting `run-aiox.sh`, `write-ack.mjs` and `registry-governance-check.js`. +const SCRIPT_TOKEN = /(?/scripts/write-ack.mjs`) and the placeholder's `<`/`>` cannot be part of the +// token. What the gap MAY contain is tightly constrained — flags and path-ish characters only, +// never an English word — so "the shell script `foo.sh`" still does not qualify. +const INTERPRETER = /\b(?:node|bash|sh|zsh|python3?|npx|deno|ruby|perl|pwsh|powershell)\b/gi; + +// True when `before` ends in a genuine command line for one of the interpreters above. +// +// Written as code rather than one regex because the gap between the interpreter and the script is +// where the real cases live and they are not uniform: `node /`, `bash `, and +// `powershell -NoProfile -ExecutionPolicy Bypass -File ` all occur verbatim in the corpus. The +// last of those is why a naive "flags only" pattern fails — `Bypass` is a bare word that is a +// FLAG'S VALUE, not prose. +// +// The rule that keeps prose out: every chunk between the interpreter and the script must be a +// flag, a path-ish token, or the value of a preceding flag. That admits `-ExecutionPolicy Bypass` +// while rejecting "each node in `graph.js`" — where the gap is the bare English word "in". +function hasInterpreterBefore(before) { + INTERPRETER.lastIndex = 0; + let end = -1; + let m; + while ((m = INTERPRETER.exec(before)) !== null) end = m.index + m[0].length; + if (end < 0) return false; + + const gap = before.slice(end); + if (gap.length > 120) return false; // an interpreter mentioned far away is not this command + if (gap.length > 0 && !/^\s/.test(gap)) return false; // must be whitespace-separated from its argv + + const chunks = gap.trim().split(/\s+/).filter(Boolean); + for (let i = 0; i < chunks.length; i++) { + const c = chunks[i]; + if (!/^[-\w:.$<>{}()\\/,=[\]"']+$/.test(c)) return false; // backticks, prose punctuation, etc. + const isFlag = /^-/.test(c); + const isPathish = /[/\\<>.$]/.test(c); + const prevIsFlag = i > 0 && /^-/.test(chunks[i - 1]); + if (!isFlag && !isPathish && !prevIsFlag) return false; // a bare English word ends it + } + return true; +} + +// An imperative run-verb close in front of the token. PT-BR is first-class here: this corpus is +// bilingual and `rode` is exactly as much an execution instruction as `run`. Backticks ARE allowed +// in the gap (a code span is the normal way to write a script name); sentence terminators are not. +const RUN_VERB_BEFORE = /\b(?:run|runs|rode|rodar|rodando|execute|executar|executes|exec|invoke|invokes|invoking|call|called|calling|calls)\b([^.!?]{0,40})$/i; + +// ...unless it is negated. "never calls it at all" / "não roda" must not count as an instruction +// to execute. Only the span BETWEEN the verb and the token is examined. +const NEGATION_BETWEEN = /\b(?:never|not|no|n[aã]o|nunca|without|sem)\b/i; + +function isExecutionContext(before) { + if (hasInterpreterBefore(before)) return "interpreter"; + if (/\.[/\\]$/.test(before)) return "interpreter"; // ./script.sh + const verb = before.match(RUN_VERB_BEFORE); + if (verb && !NEGATION_BETWEEN.test(verb[1])) return "run-verb"; + return null; +} + +// Classify WHO owns the script an instruction points at. All four forms are real and all of them +// matter to D17's threat model, which is exactly why the signal is not "owns a script": +// own — the skill's own scripts/ dir +// other-skill — ANOTHER skill's script (a real, measured pattern: /validate runs full-cycle's guard) +// repo — the repository's top-level scripts/ +// ambient — a BARE script name with no path, resolved from PATH/cwd at runtime. Which file +// actually runs is decided on the user's machine; see ANALYZER_LIMITS. +function classifyTarget(target, skillName, ownedScripts = []) { + const norm = target.replace(/\\/g, "/"); + + // An explicit `skills//` segment is the most specific evidence available — check it first, + // before any placeholder heuristic, so `/.aiox-core/skills/full-cycle/scripts/x.mjs` + // is correctly `other-skill` and never collapses to `repo`. + const m = norm.match(/(?:^|\/)skills\/([^/]+)\//); + if (m) return m[1] === skillName ? "own" : "other-skill"; + + // A `<...>` placeholder that MENTIONS a skill (``, ``, + // ``) denotes the skill's own directory. `` does not, and must + // fall through — otherwise every placeholder path would be misread as self-owned. + const ph = norm.match(/^<([^>]*)>\//); + if (ph && /skill/i.test(ph[1])) return "own"; + + const rest = norm.replace(/^<[^>]*>\//, ""); + + // A bare relative `scripts/` is genuinely ambiguous in prose — it can mean the repo's + // top-level scripts/ or the skill's own, written skill-relative. Resolve it against the + // package's REAL file listing rather than guessing: if this skill actually ships a script of + // that name, the reference is to its own. Evidence beats convention. + const basename = rest.split("/").pop(); + if (basename && ownedScripts.some((f) => f.replace(/\\/g, "/").endsWith(`/${basename}`))) return "own"; + + if (rest.includes("/")) return "repo"; + return "ambient"; +} + +export function detectExecutionInstructions(body, skillName, ownedScripts = []) { + const hits = []; + const lines = body.split(/\r?\n/); + lines.forEach((line, idx) => { + SCRIPT_TOKEN.lastIndex = 0; + let m; + while ((m = SCRIPT_TOKEN.exec(line)) !== null) { + const before = line.slice(0, m.index); + const how = isExecutionContext(before); + if (!how) continue; + // Re-attach a `<...>/` placeholder the token regex could not absorb (`<` and `>` are not + // path characters). Without this the owner of `node /scripts/write-ack.mjs` + // reads as the repo's top-level scripts/ instead of the skill's own — a wrong ANSWER from + // a right DETECTION, which is the more dangerous of the two failures. + const phMatch = before.match(/<[^>]*>\/$/); + const target = (phMatch ? phMatch[0] : "") + m[0]; + hits.push({ + line: idx + 1, + text: line.trim().slice(0, 220), + target, + owner: classifyTarget(target, skillName, ownedScripts), + matched_by: how, + }); + } + }); + return hits; +} + +// A skill OWNS a script when the file lives under that skill's own directory. Matched on the +// path segment `/scripts/` so it works for every layout this repo actually sees: +// `skills//scripts/x` (SOT dir), `/skills//scripts/x` (npm-pack style +// tarball) and a bare `/scripts/x`. The repo's OWN top-level `scripts/` is deliberately +// NOT owned by anybody — a skill that runs it is `repo`, which is a different signal. +export function detectOwnedScripts(files, skillName) { + const needle = `/${skillName}/scripts/`; + return files + .map((f) => f.replace(/\\/g, "/")) + .filter((f) => `/${f}`.includes(needle)) + .sort(); +} + +// ── derived capabilities ─────────────────────────────────────────────────────────────────────── + +// Derived from the BODY, never from `allowed-tools`. The declaration is recorded separately and +// only ever used as a second opinion to diff against (see `discrepancies`). +const CAPABILITY_PROBES = [ + { id: "filesystem:read", re: /\b(?:Read|Grep|Glob)\b|\bread the\b|\bleia\b/ }, + { id: "filesystem:write", re: /\b(?:Write|Edit|NotebookEdit)\b|\bwrite (?:the|a)\b|\bescrev[ae]\b/ }, + { id: "shell:execute", re: /\bBash\b|```(?:bash|sh|shell|console)\b|\bgit \w+|\bcargo \w+|\bgh \w+/ }, + { id: "network:fetch", re: /\b(?:WebFetch|WebSearch)\b|\bcurl\b|\bnpx\b|https?:\/\// }, + { id: "agent:spawn", re: /\bAgent\(|\bsubagent_type\b|\bTeamCreate\b|\bAgent tool\b/ }, + { id: "user:prompt", re: /\bAskUserQuestion\b/ }, + { id: "mcp:server", re: /\bmcp(?:Servers)?\b|\bMCP\b/ }, +]; + +export function deriveCapabilities(body, signals) { + const caps = new Set(); + for (const probe of CAPABILITY_PROBES) if (probe.re.test(body)) caps.add(probe.id); + if (signals.owns_scripts.value) caps.add("script:host"); + if (signals.instructs_execution.value) { + caps.add("script:execute"); + caps.add("shell:execute"); // executing a script IS shell reach, whether or not "Bash" is written + } + return [...caps].sort(); +} + +// ── AC5 — what this analyzer CANNOT see ──────────────────────────────────────────────────────── + +// Without this, the capability display lies by omission — the C2 finding in its most expensive +// form. Every report carries these limits; they are not an appendix, they are part of the result. +export const ANALYZER_LIMITS = Object.freeze([ + "An MCP server is a RUNTIME-RESOLVED POINTER, not an inspectable artifact: the manifest supplies `{command, args}` (see the product repo's mcp.rs:68, typically `npx `), resolved against a registry AIOX does not control. This analysis covers the POINTER; it has never opened the TARGET. The `npx` target is NEVER inspected, downloaded or executed here, so nothing below constrains what that package does once it runs.", + "A signature over the index covers the pointer, not the pointed-at package. Provenance is not behaviour.", + "This is STATIC analysis of skill prose. It cannot see what an agent will actually decide to do at runtime, and a skill body is natural language — an execution instruction phrased in a form these probes do not match will be missed. Absence of a signal is NOT proof of absence of the behaviour.", + "An execution instruction classified `ambient` (a bare script name with no path) resolves from PATH/cwd at runtime. Which file actually runs is decided on the user's machine and is not knowable here.", + "`allowed-tools` GRANTS pre-approval for a turn; it does NOT restrict. A tool absent from the declaration remains callable (it merely prompts), so the declaration is a floor on intent, never a ceiling on reach.", + "Only skills are analyzed. Any other executable content in the package (binaries, hooks, postinstall) is outside this pass.", +]); + +// Blocking is DELIBERATELY off (D17: v1 warns and displays). The path exists so that turning it +// on is a configuration change with a test behind it, not a re-design. +// TRIGGER TO ENABLE: when the catalog opens to EXTERNAL publishers. Until then every finding is +// surfaced to the user and to the operator, and none of them refuses a publish. +export const BLOCK_ON_CAPABILITY_FINDINGS = false; + +export function capabilityFindingsAreBlocking(override = undefined) { + return override === undefined ? BLOCK_ON_CAPABILITY_FINDINGS : Boolean(override); +} + +// ── the analysis ─────────────────────────────────────────────────────────────────────────────── + +export function analyzeSkill(skill, allFiles) { + const { fields, body } = parseFrontmatter(skill.content); + const declared = parseAllowedTools( + fields["allowed-tools"] ?? fields["allowed_tools"] ?? fields["allowedTools"], + ); + + const ownedScripts = detectOwnedScripts(allFiles, skill.name); + const execHits = detectExecutionInstructions(body, skill.name, ownedScripts); + + const signals = { + // Signal 1 — HOSTS code. + owns_scripts: { value: ownedScripts.length > 0, evidence: ownedScripts }, + // Signal 2 — INSTRUCTS execution (own / other-skill / repo / ambient). + instructs_execution: { + value: execHits.length > 0, + forms: [...new Set(execHits.map((h) => h.owner))].sort(), + evidence: execHits, + }, + }; + + const derived = deriveCapabilities(body, signals); + + // The declaration's ONLY role: a second opinion to diff against. It never feeds the display. + const discrepancies = []; + if (declared) { + if (signals.instructs_execution.value && !declared.includes("Bash")) { + discrepancies.push( + `instructs executing a script (${signals.instructs_execution.evidence[0].target}) but \`allowed-tools\` does not list Bash — the declaration under-states the skill's reach`, + ); + } + for (const [cap, tool] of [["filesystem:write", "Write"], ["network:fetch", "WebFetch"]]) { + if (derived.includes(cap) && declared.length && !declared.some((t) => t.startsWith(tool))) { + discrepancies.push(`derived \`${cap}\` is not reflected in the declared \`allowed-tools\``); + } + } + } + + return { + skill: skill.name, + path: skill.path, + declared_allowed_tools: declared, + signals, + derived_capabilities: derived, + discrepancies, + }; +} + +export function analyzeSkills(skills, allFiles) { + const perSkill = skills.map((s) => analyzeSkill(s, allFiles)).sort((a, b) => a.skill.localeCompare(b.skill)); + return { + analyzer_version: "1.0.0", + derived_by: "aiox-plugins/lib/capability-analyzer.mjs (static analysis, AIOX-operated)", + self_declared: false, + skills: perSkill, + totals: { + skills: perSkill.length, + owns_scripts: perSkill.filter((s) => s.signals.owns_scripts.value).length, + instructs_execution: perSkill.filter((s) => s.signals.instructs_execution.value).length, + }, + union_capabilities: [...new Set(perSkill.flatMap((s) => s.derived_capabilities))].sort(), + limits: [...ANALYZER_LIMITS], + enforcement: BLOCK_ON_CAPABILITY_FINDINGS ? "blocking" : "warn-and-display", + }; +} + +// The compact projection stored on a catalog entry and rendered to the user (D20(4): +// "capabilities derivadas e VISÍVEIS"). `self_declared: false` is written as DATA, not prose, so +// a reader of the raw index can tell at a glance that this block was computed, not accepted. +// `limits` travels WITH the capabilities on purpose — a capability list that can be displayed +// without its blind spots is a capability list that will eventually be displayed without them. +export function toEntryCapabilities(report) { + return { + analyzer_version: report.analyzer_version, + derived_by: report.derived_by, + self_declared: false, + enforcement: report.enforcement, + union: report.union_capabilities, + skills: report.skills.map((s) => ({ + skill: s.skill, + capabilities: s.derived_capabilities, + owns_scripts: s.signals.owns_scripts.value, + instructs_execution: s.signals.instructs_execution.value + ? s.signals.instructs_execution.forms + : [], + })), + limits: report.limits, + }; +} + +// ── collectors: a directory of skills, or an artifact tarball ────────────────────────────────── + +function walk(dir, base = dir, out = []) { + for (const e of readdirSync(dir, { withFileTypes: true })) { + const full = join(dir, e.name); + if (e.isDirectory()) walk(full, base, out); + else out.push(full.slice(base.length + 1).split(sep).join("/")); + } + return out; +} + +// A directory laid out as //SKILL.md (a SOT skills dir), or /skills//SKILL.md. +export function collectSkillsFromDir(root) { + if (!existsSync(root) || !statSync(root).isDirectory()) throw new Error(`not a directory: ${root}`); + const base = existsSync(join(root, "skills")) ? join(root, "skills") : root; + const files = walk(base).map((f) => `skills/${f}`); + const skills = readdirSync(base, { withFileTypes: true }) + .filter((d) => d.isDirectory() && existsSync(join(base, d.name, "SKILL.md"))) + .map((d) => ({ + name: d.name, + path: `skills/${d.name}/SKILL.md`, + content: readFileSync(join(base, d.name, "SKILL.md"), "utf8"), + })) + .sort((a, b) => a.name.localeCompare(b.name)); + return { skills, files }; +} + +// Extracts the artifact to a temp dir and reads the real bytes — same posture as +// lib/license-check.mjs (shell out to the system `tar`, no new npm dependency). +export function collectSkillsFromTarball(tarPath) { + const dir = mkdtempSync(join(tmpdir(), "aiox-plugins-capability-")); + try { + execFileSync("tar", ["-xzf", tarPath, "-C", dir]); + const all = walk(dir); + const skillMds = all.filter((f) => /(?:^|\/)SKILL\.md$/.test(f)); + const skills = skillMds.map((rel) => { + const parts = rel.split("/"); + return { + name: parts.length >= 2 ? parts[parts.length - 2] : "(root)", + path: rel, + content: readFileSync(join(dir, rel), "utf8"), + }; + }).sort((a, b) => a.name.localeCompare(b.name)); + return { skills, files: all }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} + +// ── human-readable rendering (AC6 — the catalog entry must SHOW this) ────────────────────────── + +export function renderCapabilityReport(report) { + const out = []; + out.push(`Derived capabilities — ${report.totals.skills} skill(s) analyzed`); + out.push(`(DERIVED by static analysis on the AIOX side; never self-declared by the publisher — D17)`); + out.push(""); + out.push(`Signals: OWNS_SCRIPTS ${report.totals.owns_scripts}/${report.totals.skills} · INSTRUCTS_EXECUTION ${report.totals.instructs_execution}/${report.totals.skills}`); + out.push(`Union of capabilities: ${report.union_capabilities.join(", ") || "(none detected)"}`); + out.push(""); + for (const s of report.skills) { + const flags = [ + s.signals.owns_scripts.value ? "OWNS_SCRIPTS" : null, + s.signals.instructs_execution.value ? `INSTRUCTS_EXECUTION(${s.signals.instructs_execution.forms.join("/")})` : null, + ].filter(Boolean); + out.push(` ${s.skill}`); + out.push(` capabilities: ${s.derived_capabilities.join(", ") || "(none detected)"}`); + out.push(` signals: ${flags.join(" · ") || "none"}`); + out.push(` declared: ${s.declared_allowed_tools ? s.declared_allowed_tools.join(", ") : "(NONE — refused at publish, AC1)"}`); + for (const h of s.signals.instructs_execution.evidence) { + out.push(` ↳ line ${h.line} [${h.owner}] ${h.target}`); + } + for (const d of s.discrepancies) out.push(` ! ${d}`); + } + out.push(""); + out.push(`Enforcement: ${report.enforcement.toUpperCase()} (D17 — v1 warns and displays; blocking path exists, off by configuration, trigger: "when opening to externals")`); + out.push(""); + out.push("WHAT THIS ANALYSIS CANNOT SEE:"); + for (const l of report.limits) out.push(` - ${l}`); + return out.join("\n"); +} diff --git a/lib/entry-schema.mjs b/lib/entry-schema.mjs index 37d64a9..4119d03 100644 --- a/lib/entry-schema.mjs +++ b/lib/entry-schema.mjs @@ -82,6 +82,28 @@ export function validateEntryShape(entry) { errors.push(`${tag}: publisher.subject required (D22 — entitlement subject, never a GitHub handle)`); if (!entry.published_at) errors.push(`${tag}: published_at required`); if (!entry.license?.spdx_or_path) errors.push(`${tag}: license.spdx_or_path required (D24(c))`); + // story 055.W4.2 (D17/D20(4)) — the derived-capability block. Two things are checked, and the + // second is the one that matters: `self_declared` must be LITERALLY false. The block is written + // by the AIOX-side analyzer at publish time; an entry arriving with `self_declared: true` (or + // with the flag missing) is either hand-edited or produced by something that is not the + // analyzer, and in both cases the value cannot be shown to a user as derived provenance. + if (entry.capabilities !== undefined) { + const c = entry.capabilities; + if (typeof c !== "object" || c === null || Array.isArray(c)) { + errors.push(`${tag}: capabilities must be an object produced by the capability analyzer (D20(4))`); + } else { + if (c.self_declared !== false) + errors.push( + `${tag}: capabilities.self_declared must be literally false (D17/AC3) — capabilities are DERIVED by AIOX-side static analysis, never asserted by the publisher; an entry claiming otherwise cannot be displayed as derived provenance`, + ); + if (!Array.isArray(c.union)) + errors.push(`${tag}: capabilities.union must be an array (may be empty) — the derived capability set`); + if (!Array.isArray(c.limits) || c.limits.length === 0) + errors.push( + `${tag}: capabilities.limits must be a non-empty array (D17/AC5) — an analysis displayed without its blind spots lies by omission; the MCP-pointer limitation in particular must travel with the result`, + ); + } + } if (entry.overlay?.shadows && typeof entry.overlay.shadows === "object") { for (const [skill, reason] of Object.entries(entry.overlay.shadows)) { if (!reason || typeof reason !== "string" || !reason.trim()) { diff --git a/publisher/publish.mjs b/publisher/publish.mjs index 34256df..b80b094 100644 --- a/publisher/publish.mjs +++ b/publisher/publish.mjs @@ -67,6 +67,15 @@ import { } from "../lib/entry-schema.mjs"; import { checkLicenseInPackageRoot } from "../lib/license-check.mjs"; import { loadLedger, saveLedger, recordPublish } from "../lib/ledger.mjs"; +import { + analyzeSkills, + collectSkillsFromTarball, + checkAllowedToolsDeclared, + checkNoSelfDeclaredCapabilities, + toEntryCapabilities, + renderCapabilityReport, + capabilityFindingsAreBlocking, +} from "../lib/capability-analyzer.mjs"; function usageAndExit(msg) { if (msg) console.error(`error: ${msg}\n`); @@ -125,6 +134,29 @@ function main() { ? args["emit-tiers"].split(",").map((t) => t.trim()).filter(Boolean) : manifest.tiers; + // ── story 055.W4.2 (D17 + D20(4)) — allowed-tools + DERIVED capabilities ──────────────────── + // + // Order matters. AC3 first: if the manifest tries to assert its own capabilities, refuse before + // doing any analysis at all — there is no "we'll just ignore that field" path, because a field + // that is ignored today is a field that gets read by accident tomorrow. + const selfDeclaredErrs = checkNoSelfDeclaredCapabilities(manifest); + if (selfDeclaredErrs.length) { + console.error("REFUSED — manifest self-declares capabilities (D17/AC3):"); + for (const e of selfDeclaredErrs) console.error(` - ${e}`); + process.exit(1); + } + + // Read the skills out of the artifact's ACTUAL BYTES — the same posture as check (c): a + // manifest field is an assertion, the tarball is the thing the user will run. + const { skills, files } = collectSkillsFromTarball(args.artifact); + + // AC1 — BLOCKING and unconditional. No flag, no env var, no fixture path disables it. + const allowedToolsErrs = checkAllowedToolsDeclared(skills); + + // AC3/AC6 — capabilities DERIVED here, on the AIOX side, from the body. Nothing the publisher + // wrote about itself contributes to this value. + const capabilityReport = analyzeSkills(skills, files); + const entry = { schema_version: ENTRY_SCHEMA_VERSION, plugin_id: manifest.plugin_id, @@ -138,6 +170,7 @@ function main() { publisher: { subject: args.subject }, published_at: new Date().toISOString(), license: { spdx_or_path: manifest.license }, + capabilities: toEntryCapabilities(capabilityReport), ...(manifest.overlay ? { overlay: manifest.overlay } : {}), }; @@ -151,13 +184,24 @@ function main() { const idImmutabilityErrs = checkIdImmutabilityAgainstLedger(ledger, entry); const burnedErrs = checkNameNotBurned(ledger, entry); - const allErrs = [...shapeErrs, ...bindingErrs, ...hostErrs, ...tierErrs, ...licenseErrs, ...idImmutabilityErrs, ...burnedErrs]; + const allErrs = [...shapeErrs, ...bindingErrs, ...hostErrs, ...tierErrs, ...licenseErrs, ...idImmutabilityErrs, ...burnedErrs, ...allowedToolsErrs]; if (allErrs.length) { console.error("REFUSED — entry fails validation:"); for (const e of allErrs) console.error(` - ${e}`); process.exit(1); } + // AC6 — v1 WARNS AND DISPLAYS. The derived analysis is printed for the operator and stored on + // the entry for the user; it does NOT refuse the publish. This is D17's explicit decision, not + // laxity: real containment needs an execution host, and the ADR rejects a third-party WASM host + // with evidence. The blocking path below exists and is exercised by tests; its documented + // trigger is "when opening to externals". + console.error(renderCapabilityReport(capabilityReport)); + if (capabilityFindingsAreBlocking() && capabilityReport.union_capabilities.length) { + console.error("REFUSED — capability findings are blocking (the catalog is open to external publishers)"); + process.exit(1); + } + if (!existsSync(args.target)) usageAndExit(`--target ${args.target} does not exist`); const index = JSON.parse(readFileSync(args.target, "utf8")); index.entries ??= []; diff --git a/schema/index-entry.schema.json b/schema/index-entry.schema.json index 70c2d2f..9f7b71e 100644 --- a/schema/index-entry.schema.json +++ b/schema/index-entry.schema.json @@ -103,6 +103,56 @@ } } }, + "capabilities": { + "type": "object", + "additionalProperties": false, + "required": ["analyzer_version", "derived_by", "self_declared", "union", "limits"], + "description": "DERIVED capability analysis (ADR D17 + D20(4), story 055.W4.2). Written EXCLUSIVELY by publisher/publish.mjs from lib/capability-analyzer.mjs, computed from the artifact's own bytes. THE PUBLISHER HAS NO MANIFEST FIELD THAT FEEDS THIS — a manifest carrying `capabilities`/`permissions`/`grants`/`sandbox`/`trust_level` is refused outright at publish time (lib/capability-analyzer.mjs::checkNoSelfDeclaredCapabilities), because a capability asserted by the party being assessed is worth zero. This block delivers VISIBILITY, NOT CONTAINMENT: nothing here sandboxes the plugin.", + "properties": { + "analyzer_version": { "type": "string" }, + "derived_by": { "type": "string", "description": "The analyzer that produced this block — provenance of the derivation itself." }, + "self_declared": { + "type": "boolean", + "const": false, + "description": "ALWAYS literally false, and re-checked in code (lib/entry-schema.mjs::validateEntryShape). Encoded as data rather than prose so a reader of the raw index can tell the block was computed, not accepted." + }, + "enforcement": { + "type": "string", + "enum": ["warn-and-display", "blocking"], + "description": "D17: v1 is `warn-and-display`. The blocking path exists in code, off by configuration, with the documented trigger 'when opening to externals'." + }, + "union": { + "type": "array", + "items": { "type": "string" }, + "description": "Union of derived capabilities across every skill in the package (may be empty)." + }, + "skills": { + "type": "array", + "description": "Per-skill breakdown carrying the TWO INDEPENDENT SIGNALS. They are never collapsed into one number: hosting a script and being instructed to execute one are orthogonal, and both matter to the threat model.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["skill", "capabilities", "owns_scripts", "instructs_execution"], + "properties": { + "skill": { "type": "string" }, + "capabilities": { "type": "array", "items": { "type": "string" } }, + "owns_scripts": { "type": "boolean", "description": "SIGNAL 1 — the skill ships its own scripts/ directory (it HOSTS code)." }, + "instructs_execution": { + "type": "array", + "items": { "type": "string", "enum": ["own", "other-skill", "repo", "ambient"] }, + "description": "SIGNAL 2 — the body instructs EXECUTING a script, and whose: its own, another skill's, the repo's, or `ambient` (a bare name resolved from PATH/cwd at runtime, so which file runs is decided on the user's machine). Empty array = not detected." + } + } + } + }, + "limits": { + "type": "array", + "minItems": 1, + "items": { "type": "string" }, + "description": "REQUIRED and non-empty (AC5). What the analysis CANNOT see — above all that an MCP server is a runtime-resolved {command, args} pointer (product repo mcp.rs:68) whose `npx` target is never inspected. Travels WITH the capabilities so the two can never be displayed apart; a capability list shown without its blind spots lies by omission." + } + } + }, "overlay": { "type": "object", "additionalProperties": false, diff --git a/scripts/analyze-capabilities.mjs b/scripts/analyze-capabilities.mjs new file mode 100644 index 0000000..e574365 --- /dev/null +++ b/scripts/analyze-capabilities.mjs @@ -0,0 +1,73 @@ +#!/usr/bin/env node +// scripts/analyze-capabilities.mjs — story 055.W4.2, D17 + D20(4). +// +// Runs the static capability analyzer (lib/capability-analyzer.mjs) over either a plugin ARTIFACT +// (.tar.gz — the publish-time / CI input) or a DIRECTORY of skills (the shape a SOT skills tree +// has). Same analyzer either way: the input adapter differs, the derivation does not, so the +// numbers a maintainer reproduces over a directory are the numbers the publish path computes over +// a tarball. +// +// v1 is WARN-AND-DISPLAY (D17): this exits 0 even when it finds capabilities. `--strict` opts into +// the blocking path that exists for "when opening to externals". AC1 (`allowed-tools` mandatory) +// is a SEPARATE, always-blocking check at publish time — see publisher/publish.mjs. +// +// usage: +// node scripts/analyze-capabilities.mjs --artifact [--json] [--strict] +// node scripts/analyze-capabilities.mjs --dir [--json] [--strict] + +import { + analyzeSkills, + collectSkillsFromDir, + collectSkillsFromTarball, + renderCapabilityReport, + checkAllowedToolsDeclared, + capabilityFindingsAreBlocking, +} from "../lib/capability-analyzer.mjs"; + +function usageAndExit(msg) { + if (msg) console.error(`error: ${msg}\n`); + console.error("usage: node scripts/analyze-capabilities.mjs (--artifact | --dir ) [--json] [--strict] [--require-allowed-tools]"); + process.exit(1); +} + +const argv = process.argv.slice(2); +const args = { json: argv.includes("--json"), strict: argv.includes("--strict"), requireAllowedTools: argv.includes("--require-allowed-tools") }; +for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--artifact") args.artifact = argv[++i]; + if (argv[i] === "--dir") args.dir = argv[++i]; +} +if (!args.artifact && !args.dir) usageAndExit("one of --artifact / --dir is required"); +if (args.artifact && args.dir) usageAndExit("--artifact and --dir are mutually exclusive"); + +let collected; +try { + collected = args.artifact ? collectSkillsFromTarball(args.artifact) : collectSkillsFromDir(args.dir); +} catch (e) { + console.error(`error: could not read the input: ${e.message}`); + process.exit(2); +} + +if (collected.skills.length === 0) { + console.error("error: no SKILL.md found in the input — nothing to analyze (this is a defect in the input, not a clean result)"); + process.exit(2); +} + +const report = analyzeSkills(collected.skills, collected.files); + +console.log(args.json ? JSON.stringify(report, null, 2) : renderCapabilityReport(report)); + +// AC1 as an explicit, opt-in gate for the directory/CI path (at publish time it is unconditional). +if (args.requireAllowedTools) { + const errors = checkAllowedToolsDeclared(collected.skills); + if (errors.length) { + console.error("\nREFUSED — `allowed-tools` is mandatory (D17/AC1):"); + for (const e of errors) console.error(` - ${e}`); + process.exit(1); + } + console.error(`\nOK — all ${collected.skills.length} skill(s) declare \`allowed-tools\``); +} + +if (capabilityFindingsAreBlocking(args.strict || undefined) && report.union_capabilities.length) { + console.error("\nREFUSED — capability findings are blocking under --strict (the D17 path reserved for opening to externals)"); + process.exit(1); +} diff --git a/scripts/render-catalog.mjs b/scripts/render-catalog.mjs index 9e25b58..a2edba7 100644 --- a/scripts/render-catalog.mjs +++ b/scripts/render-catalog.mjs @@ -76,6 +76,39 @@ export function renderCatalog(index, label) { lines.push(`- Published by: \`${escapeMd(e.publisher?.subject)}\` at ${escapeMd(e.published_at)}`); lines.push(`- License: ${escapeMd(e.license?.spdx_or_path)}`); + // story 055.W4.2 (D20(4)) — DERIVED capabilities, shown on the entry the user reads BEFORE + // installing. Rendered from `e.capabilities`, which publisher/publish.mjs computes with the + // static analyzer from the artifact's own bytes; the publisher has no field that feeds this. + // + // The `limits` block is rendered UNCONDITIONALLY alongside the capability list, and that + // coupling is the point (AC5): an MCP server is a runtime-resolved `{command, args}` pointer + // and the `npx` target is never inspected. A capability display that omits what it cannot see + // lies by omission — so the two cannot be rendered apart. + const caps = e.capabilities; + if (caps) { + lines.push(""); + lines.push(`### What this plugin can do — \`${escapeMd(e.plugin_id)}\``); + lines.push(""); + lines.push( + `_DERIVED by AIOX-side static analysis (\`${escapeMd(caps.derived_by)}\`), **never self-declared by the publisher** (D17). Mode: ${escapeMd(caps.enforcement)}._`, + ); + lines.push(""); + lines.push(`- Capabilities: ${(caps.union ?? []).length ? caps.union.map((c) => `\`${escapeMd(c)}\``).join(", ") : "_(none detected)_"}`); + for (const s of caps.skills ?? []) { + const flags = [ + s.owns_scripts ? "ships its own `scripts/`" : null, + (s.instructs_execution ?? []).length ? `instructs script execution (${s.instructs_execution.map(escapeMd).join(", ")})` : null, + ].filter(Boolean); + lines.push( + ` - \`${escapeMd(s.skill)}\` — ${(s.capabilities ?? []).map((c) => `\`${escapeMd(c)}\``).join(", ") || "_none detected_"}${flags.length ? ` · **${flags.join(" · ")}**` : ""}`, + ); + } + lines.push(""); + lines.push("> **What this analysis CANNOT see** — read this before calibrating trust:"); + lines.push(">"); + for (const l of caps.limits ?? []) lines.push(`> - ${escapeMd(l)}`); + } + const shadows = e.overlay?.shadows; if (shadows && Object.keys(shadows).length > 0) { lines.push(""); diff --git a/test/capability-analyzer.test.mjs b/test/capability-analyzer.test.mjs new file mode 100644 index 0000000..d33d7bb --- /dev/null +++ b/test/capability-analyzer.test.mjs @@ -0,0 +1,281 @@ +// test/capability-analyzer.test.mjs — story 055.W4.2 (D17 + D20(4)). +// +// The fixtures below are not invented: each is the REAL shape measured in the product repo's SOT +// (`.aiox-core/skills/`) on 2026-08-10, reduced to the line that carries the signal. That matters +// because the analyzer's whole reason to exist is calibration — a fixture set built from +// imagination would calibrate it against imagination. +// +// The negative fixture (`self-heal`) is MANDATORY and is the most important test in this file. +// A naive regex (`scripts/[a-z]|\.mjs|\.sh`) marks it, and it is a FALSE POSITIVE: the skill has +// no scripts/ dir at all and its only match is prose citing sync.mjs BY ANALOGY. If the analyzer +// marks it, the analyzer has reproduced inside itself the exact calibration error D17 exists to +// correct. + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { + analyzeSkills, + analyzeSkill, + detectExecutionInstructions, + detectOwnedScripts, + parseFrontmatter, + parseAllowedTools, + checkAllowedToolsDeclared, + checkNoSelfDeclaredCapabilities, + toEntryCapabilities, + ANALYZER_LIMITS, + BLOCK_ON_CAPABILITY_FINDINGS, + capabilityFindingsAreBlocking, +} from "../lib/capability-analyzer.mjs"; + +const fm = (tools) => `---\nname: x\ndescription: y\nallowed-tools: ${tools}\n---\n`; +const skill = (name, body, tools = "Read, Bash") => ({ + name, + path: `skills/${name}/SKILL.md`, + content: fm(tools) + body, +}); + +// ── frontmatter / allowed-tools parsing ──────────────────────────────────────────────────────── + +test("parseAllowedTools accepts all three canonical shapes", () => { + assert.deepEqual(parseAllowedTools("Read, Grep, Bash"), ["Read", "Grep", "Bash"]); + assert.deepEqual(parseAllowedTools("Read Grep Bash"), ["Read", "Grep", "Bash"]); + assert.deepEqual(parseAllowedTools(["Read", "Grep"]), ["Read", "Grep"]); + assert.equal(parseAllowedTools(undefined), null); +}); + +test("parseFrontmatter reads a YAML-list allowed-tools", () => { + const { fields } = parseFrontmatter("---\nname: x\nallowed-tools:\n - Read\n - Bash\n---\nbody\n"); + assert.deepEqual(fields["allowed-tools"], ["Read", "Bash"]); +}); + +// ── AC1 — allowed-tools mandatory ────────────────────────────────────────────────────────────── + +test("AC1: a skill WITHOUT allowed-tools is refused", () => { + const s = { name: "no-decl", path: "skills/no-decl/SKILL.md", content: "---\nname: no-decl\ndescription: d\n---\nbody" }; + const errs = checkAllowedToolsDeclared([s]); + assert.equal(errs.length, 1); + assert.match(errs[0], /declares no `allowed-tools`/); +}); + +test("AC1: a skill WITH allowed-tools passes", () => { + assert.deepEqual(checkAllowedToolsDeclared([skill("ok", "body")]), []); +}); + +test("AC1: `allowed-tools: *` is refused — a wildcard passes presence while declaring nothing", () => { + const errs = checkAllowedToolsDeclared([skill("wild", "body", "*")]); + assert.equal(errs.length, 1); + assert.match(errs[0], /wildcard/); +}); + +test("AC1: an EMPTY allowed-tools is refused", () => { + const s = { name: "empty", path: "p", content: "---\nname: empty\nallowed-tools:\n---\nbody" }; + assert.match(checkAllowedToolsDeclared([s])[0], /empty/); +}); + +test("AC1: snake_case/camelCase spellings are refused, not silently accepted", () => { + const s = { name: "snake", path: "p", content: "---\nname: snake\nallowed_tools: Read\n---\nbody" }; + const errs = checkAllowedToolsDeclared([s]); + assert.equal(errs.length, 1); + assert.match(errs[0], /kebab-case/); +}); + +test("AC1: no frontmatter at all is refused", () => { + const errs = checkAllowedToolsDeclared([{ name: "raw", path: "p", content: "# just a heading\n" }]); + assert.match(errs[0], /no YAML frontmatter/); +}); + +// ── AC3 — no self-declared capabilities ──────────────────────────────────────────────────────── + +test("AC3: a manifest declaring `capabilities` is REFUSED (not ignored)", () => { + const errs = checkNoSelfDeclaredCapabilities({ plugin_id: "p", capabilities: ["filesystem:write"] }); + assert.equal(errs.length, 1); + assert.match(errs[0], /DERIVED by static analysis/); +}); + +test("AC3: every self-assertion field is refused, not just `capabilities`", () => { + for (const field of ["permissions", "grants", "sandbox", "trust_level"]) { + const errs = checkNoSelfDeclaredCapabilities({ plugin_id: "p", [field]: "anything" }); + assert.equal(errs.length, 1, `${field} must be refused`); + } +}); + +test("AC3: a clean manifest passes", () => { + assert.deepEqual(checkNoSelfDeclaredCapabilities({ plugin_id: "p", tiers: ["base"] }), []); +}); + +test("AC3: the derived entry block marks itself self_declared:false", () => { + const report = analyzeSkills([skill("a", "Read the file.")], ["skills/a/SKILL.md"]); + assert.equal(toEntryCapabilities(report).self_declared, false); +}); + +// ── AC4 — the two independent signals ────────────────────────────────────────────────────────── + +test("AC4 signals are INDEPENDENT: a skill can own scripts without instructing execution", () => { + const s = skill("hoarder", "This skill ships helpers but the body never runs them."); + const r = analyzeSkill(s, ["skills/hoarder/SKILL.md", "skills/hoarder/scripts/unused.mjs"]); + assert.equal(r.signals.owns_scripts.value, true); + assert.equal(r.signals.instructs_execution.value, false); +}); + +test("AC4 signals are INDEPENDENT: a skill can instruct execution without owning scripts", () => { + const s = skill("borrower", "Run `bash scripts/run-aiox.sh` from the repo root."); + const r = analyzeSkill(s, ["skills/borrower/SKILL.md"]); + assert.equal(r.signals.owns_scripts.value, false); + assert.equal(r.signals.instructs_execution.value, true); +}); + +// --- positive fixture: OWN script (real shape — `close`) --- +test("AC4 positive/own: `node /scripts/write-ack.mjs` is detected and owned by the skill", () => { + const s = skill("close", "node /scripts/write-ack.mjs --story-id --ack close"); + const r = analyzeSkill(s, ["skills/close/SKILL.md", "skills/close/scripts/write-ack.mjs"]); + assert.equal(r.signals.instructs_execution.value, true); + assert.deepEqual(r.signals.instructs_execution.forms, ["own"]); +}); + +// --- positive fixture: ANOTHER skill's script (real shape — `validate` runs full-cycle's guard) --- +test("AC4 positive/other-skill: a script under skills//scripts is classified other-skill", () => { + const s = skill("validate", "node /.aiox-core/skills/full-cycle/scripts/full-cycle-guard.mjs inspect "); + const r = analyzeSkill(s, ["skills/validate/SKILL.md"]); + assert.deepEqual(r.signals.instructs_execution.forms, ["other-skill"]); +}); + +// --- positive fixture: REPO script (real shape — `update-cockpit`) --- +test("AC4 positive/repo: `bash scripts/run-aiox.sh` is classified repo", () => { + const s = skill("update-cockpit", " - macOS/Linux: `bash scripts/run-aiox.sh`"); + const r = analyzeSkill(s, ["skills/update-cockpit/SKILL.md"]); + assert.deepEqual(r.signals.instructs_execution.forms, ["repo"]); +}); + +// --- positive fixture: AMBIENT (real shape — `criar-sot`, PT-BR, missed by the naive grep) --- +test("AC4 positive/ambient: a bare script name behind a PT-BR run verb is detected", () => { + const s = skill("criar-sot", "- Em repos com governança de registry, rode `registry-governance-check.js --mode advisory` se aplicável."); + const r = analyzeSkill(s, ["skills/criar-sot/SKILL.md"]); + assert.equal(r.signals.instructs_execution.value, true); + assert.deepEqual(r.signals.instructs_execution.forms, ["ambient"]); +}); + +test("AC4: the Windows branch of a real instruction (backslashes, -File) is detected", () => { + const s = skill("update-cockpit", " - Windows: `powershell -NoProfile -ExecutionPolicy Bypass -File scripts\\run-aiox.ps1`"); + const r = analyzeSkill(s, ["skills/update-cockpit/SKILL.md"]); + assert.equal(r.signals.instructs_execution.value, true); +}); + +// --- THE MANDATORY NEGATIVE FIXTURE --- +test("AC4 NEGATIVE (mandatory): `self-heal` is in NEITHER signal — prose analogy is not execution", () => { + // Verbatim from .aiox-core/skills/self-heal/SKILL.md:32 — its ONLY script-shaped match, and the + // reason the naive regex returns 9 instead of the correct set. + const s = skill( + "self-heal", + "**Adapters mirror the `sync.mjs` ADAPTERS pattern** (add a CLI = add an adapter; the loop is untouched).", + ); + const r = analyzeSkill(s, ["skills/self-heal/SKILL.md"]); // no scripts/ dir — matches disk + assert.equal(r.signals.owns_scripts.value, false, "self-heal has no scripts/ dir on disk"); + assert.equal( + r.signals.instructs_execution.value, + false, + "marking self-heal reproduces inside the analyzer the calibration error D17 exists to correct", + ); +}); + +test("AC4-bis NEGATIVE (generic): a skill that mentions no script at all is in neither signal", () => { + const r = analyzeSkill(skill("pure", "Read the story and report a verdict. No commands."), ["skills/pure/SKILL.md"]); + assert.equal(r.signals.owns_scripts.value, false); + assert.equal(r.signals.instructs_execution.value, false); +}); + +test("AC4 NEGATIVE: a NEGATED reference ('never calls wave-launch.js') is not an instruction", () => { + const s = skill("wave-execute", "the legacy `wave-launch.js` env-contract mirrored below never calls it at all"); + assert.equal(analyzeSkill(s, ["skills/wave-execute/SKILL.md"]).signals.instructs_execution.value, false); +}); + +test("AC4 NEGATIVE: framework names (Node.js / Next.js) are never mistaken for scripts", () => { + const s = skill("tech", "Run the build. Frameworks: React, Next.js, Vue. Serviço Node.js documentado."); + const hits = detectExecutionInstructions(s.content, "tech"); + assert.deepEqual(hits, [], `expected no hits, got ${JSON.stringify(hits)}`); +}); + +test("AC4 NEGATIVE: a test file cited as evidence is not an execution instruction", () => { + const s = skill("close", "Evidence: `scripts/write-ack.test.mjs` (4 cases — canonical path, field override)."); + assert.equal(analyzeSkill(s, ["skills/close/SKILL.md"]).signals.instructs_execution.value, false); +}); + +test("detectOwnedScripts matches a wrapped tarball layout, not only a bare SOT dir", () => { + const files = ["pkg-1.0.0/skills/close/SKILL.md", "pkg-1.0.0/skills/close/scripts/write-ack.mjs"]; + assert.deepEqual(detectOwnedScripts(files, "close"), ["pkg-1.0.0/skills/close/scripts/write-ack.mjs"]); + assert.deepEqual(detectOwnedScripts(files, "other"), []); +}); + +test("the repo's own top-level scripts/ is owned by NOBODY", () => { + assert.deepEqual(detectOwnedScripts(["scripts/run-aiox.sh", "skills/x/SKILL.md"], "x"), []); +}); + +// ── AC5 — the analyzer states what it cannot see ─────────────────────────────────────────────── + +test("AC5: every report carries the limits, and the MCP pointer limitation is among them", () => { + const report = analyzeSkills([skill("a", "body")], ["skills/a/SKILL.md"]); + assert.ok(report.limits.length >= 1); + const joined = report.limits.join("\n"); + assert.match(joined, /MCP/); + assert.match(joined, /\{command, args\}/); + assert.match(joined, /npx/i, "the npx target must be named as not inspected"); + assert.match(joined, /never inspected/i); +}); + +test("AC5: the limits survive into the entry block that reaches the catalog", () => { + const report = analyzeSkills([skill("a", "body")], ["skills/a/SKILL.md"]); + const entryCaps = toEntryCapabilities(report); + assert.ok(entryCaps.limits.length >= 1); + assert.match(entryCaps.limits.join("\n"), /MCP/); +}); + +test("AC5: ANALYZER_LIMITS records that allowed-tools GRANTS rather than RESTRICTS", () => { + assert.match(ANALYZER_LIMITS.join("\n"), /does NOT restrict/); +}); + +// ── AC6 — warn and display, blocking path present but off ────────────────────────────────────── + +test("AC6: v1 is warn-and-display — the blocking flag is off by configuration", () => { + assert.equal(BLOCK_ON_CAPABILITY_FINDINGS, false); + assert.equal(capabilityFindingsAreBlocking(), false); +}); + +test("AC6: the blocking path EXISTS and can be turned on (it is code, not a promise)", () => { + assert.equal(capabilityFindingsAreBlocking(true), true); +}); + +test("AC6: the report announces its own enforcement mode", () => { + const report = analyzeSkills([skill("a", "body")], ["skills/a/SKILL.md"]); + assert.equal(report.enforcement, "warn-and-display"); +}); + +// ── derived, not declared ────────────────────────────────────────────────────────────────────── + +test("capabilities are derived from the BODY, not from allowed-tools", () => { + // Declares only Read, but the body plainly instructs writing and shelling out. The derivation + // must follow the body — otherwise a publisher could shrink its displayed reach by under-declaring. + const s = skill("liar", "Write the report to disk, then run `bash scripts/deploy.sh`.", "Read"); + const r = analyzeSkill(s, ["skills/liar/SKILL.md"]); + assert.ok(r.derived_capabilities.includes("filesystem:write")); + assert.ok(r.derived_capabilities.includes("shell:execute")); + assert.ok(r.derived_capabilities.includes("script:execute")); +}); + +test("an under-declared allowed-tools surfaces as a discrepancy, not a silent pass", () => { + const s = skill("under", "Run `bash scripts/deploy.sh` now.", "Read"); + const r = analyzeSkill(s, ["skills/under/SKILL.md"]); + assert.ok(r.discrepancies.some((d) => /does not list Bash/.test(d))); +}); + +test("totals never collapse the two signals into one number", () => { + const report = analyzeSkills( + [ + skill("owns-only", "ships helpers, runs nothing"), + skill("exec-only", "Run `bash scripts/x.sh`"), + ], + ["skills/owns-only/SKILL.md", "skills/owns-only/scripts/h.mjs", "skills/exec-only/SKILL.md"], + ); + assert.equal(report.totals.owns_scripts, 1); + assert.equal(report.totals.instructs_execution, 1); + assert.equal(report.totals.skills, 2); +}); diff --git a/test/helpers/tarball.mjs b/test/helpers/tarball.mjs index c2d4398..c410941 100644 --- a/test/helpers/tarball.mjs +++ b/test/helpers/tarball.mjs @@ -25,18 +25,41 @@ export function buildTarball(files) { } } +// story 055.W4.2: `allowed-tools` became MANDATORY on every publishable skill (D17/AC1), so every +// fixture skill below now carries it. This is deliberately applied to the NEGATIVE fixtures too: +// a license-negative fixture that ALSO fails the allowed-tools gate would still make its test pass +// (the assertion greps for the license message), but for the wrong reason — and a test that can +// pass for the wrong reason stops being evidence the moment the right reason disappears. Each +// fixture is therefore invalid in EXACTLY ONE way. +export const FIXTURE_SKILL = `--- +name: fixture-skill +description: A minimal valid fixture skill. +allowed-tools: Read +--- + +# fixture skill +`; + +// A valid fixture skill carrying a distinguishing marker in its body. The D24(a)/F9 lineage tests +// depend on artifacts having DIFFERENT BYTES from one another (that is the property under test), +// so they cannot all share one constant — the marker is what keeps the digests distinct while the +// frontmatter keeps them publishable. +export function fixtureSkill(marker) { + return `---\nname: fixture-skill\ndescription: A minimal valid fixture skill.\nallowed-tools: Read\n---\n\n# ${marker}\n`; +} + // A minimal, valid plugin artifact: LICENSE at the true top level. export function buildValidArtifact() { return buildTarball({ "LICENSE": "MIT License\n\nCopyright (c) AIOX\n", - "SKILL.md": "# fixture skill\n", + "SKILL.md": FIXTURE_SKILL, }); } // A plugin artifact with NO license anywhere — the negative fixture for check (c). export function buildArtifactWithoutLicense() { return buildTarball({ - "SKILL.md": "# fixture skill, no license anywhere in the package\n", + "SKILL.md": FIXTURE_SKILL, "src/index.js": "// no license\n", }); } @@ -46,6 +69,33 @@ export function buildArtifactWithoutLicense() { export function buildArtifactWithBuriedLicense() { return buildTarball({ "src/nested/deep/LICENSE": "MIT License (buried, not at package root)\n", - "SKILL.md": "# fixture skill\n", + "SKILL.md": FIXTURE_SKILL, + }); +} + +// story 055.W4.2 — the AC1 negative fixture at CLI level: a licensed, otherwise-perfect artifact +// whose skill declares NO `allowed-tools`. Everything else about it is valid, so a publish that +// still succeeds proves the gate is decorative. +export function buildArtifactWithoutAllowedTools() { + return buildTarball({ + "LICENSE": "MIT License\n\nCopyright (c) AIOX\n", + "SKILL.md": "---\nname: fixture-skill\ndescription: declares no allowed-tools.\n---\n\n# fixture skill\n", + }); +} + +// story 055.W4.2 — a skill that ships its own scripts/ AND instructs executing it. Used to prove +// the two derived signals reach the published entry and the rendered catalog. +export function buildArtifactWithExecutingSkill() { + return buildTarball({ + "LICENSE": "MIT License\n\nCopyright (c) AIOX\n", + "skills/runner/SKILL.md": `--- +name: runner +description: Runs its own helper. +allowed-tools: Read, Bash +--- + +Run \`node /scripts/helper.mjs --go\` before anything else. +`, + "skills/runner/scripts/helper.mjs": "// helper\n", }); } diff --git a/test/publish-cli.test.mjs b/test/publish-cli.test.mjs index 5314606..3a84e41 100644 --- a/test/publish-cli.test.mjs +++ b/test/publish-cli.test.mjs @@ -18,7 +18,7 @@ import { createHash } from "node:crypto"; import { tmpdir } from "node:os"; import { join, dirname } from "node:path"; import { fileURLToPath } from "node:url"; -import { buildTarball, buildValidArtifact, buildArtifactWithoutLicense, buildArtifactWithBuriedLicense } from "./helpers/tarball.mjs"; +import { buildTarball, buildValidArtifact, buildArtifactWithoutLicense, buildArtifactWithBuriedLicense, buildArtifactWithoutAllowedTools, buildArtifactWithExecutingSkill, fixtureSkill } from "./helpers/tarball.mjs"; const here = dirname(fileURLToPath(import.meta.url)); const publishScript = join(here, "..", "publisher", "publish.mjs"); @@ -329,7 +329,7 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( const ledger = writeEmptyLedger(dir); // v1 under the original name — legitimate, lands - const artifactV1 = buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v1.0.0\n" }); + const artifactV1 = buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v1.0.0") }); assert.equal( tryPublish({ dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE, version: "1.0.0", artifact: artifactV1 }), null, @@ -337,7 +337,7 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( ); // v1.1.0 under a NEW name, REBUILT — different bytes, so digest lineage is blind to it - const artifactV2 = buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v1.1.0 — rebuilt, different content\n" }); + const artifactV2 = buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v1.1.0 — rebuilt, different content") }); const digestV1 = createHash("sha256").update(readFileSync(artifactV1)).digest("hex"); const digestV2 = createHash("sha256").update(readFileSync(artifactV2)).digest("hex"); assert.notEqual(digestV1, digestV2, "the fixture is only meaningful if the bytes genuinely differ"); @@ -367,11 +367,11 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( const target = writeEmptyIndex(dir); const ledger = writeEmptyLedger(dir); assert.equal( - tryPublish({ dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# a\n" }) }), + tryPublish({ dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("a") }) }), null, ); assert.equal( - tryPublish({ dir, target, ledger, plugin_id: "sinkra-os", lineage_id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb", version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# b\n" }) }), + tryPublish({ dir, target, ledger, plugin_id: "sinkra-os", lineage_id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb", version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("b") }) }), null, "a distinct plugin with a distinct lineage must not be caught by the rename check", ); @@ -384,8 +384,8 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( const target = writeEmptyIndex(dir); const ledger = writeEmptyLedger(dir); const args = { dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE }; - assert.equal(tryPublish({ ...args, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v1\n" }) }), null); - assert.equal(tryPublish({ ...args, version: "1.1.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v2\n" }) }), null); + assert.equal(tryPublish({ ...args, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v1") }) }), null); + assert.equal(tryPublish({ ...args, version: "1.1.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v2") }) }), null); const ledgerData = JSON.parse(readFileSync(ledger, "utf8")); assert.equal(ledgerData.plugins["aiox-enterprise"].history.length, 2); assert.equal(ledgerData.plugins["aiox-enterprise"].lineage_id, LIN_ENTERPRISE, "the identity is carried, not rewritten"); @@ -397,7 +397,7 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( const target = writeEmptyIndex(dir); const ledger = writeEmptyLedger(dir); assert.equal( - tryPublish({ dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v1\n" }) }), + tryPublish({ dir, target, ledger, plugin_id: "aiox-enterprise", lineage_id: LIN_ENTERPRISE, version: "1.0.0", artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v1") }) }), null, ); const stderr = tryPublish({ @@ -405,7 +405,7 @@ describe("check (a) — id immutability (D24(a), AC1, AC5 negative fixtures)", ( plugin_id: "aiox-enterprise", lineage_id: "cccccccc-cccc-4ccc-8ccc-cccccccccccc", // step 1 of the evasion: free up the old lineage version: "1.1.0", - artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": "# v2\n" }), + artifact: buildTarball({ LICENSE: "MIT\n", "SKILL.md": fixtureSkill("v2") }), }); assert.ok(stderr !== null, "expected the relabelling to be REFUSED"); assert.match(stderr, /is on record with lineage_id/); @@ -532,3 +532,115 @@ describe("check (b) — burned name survives despublish (D24(b), AC2, VC-1, AC5 }); }); }); + +// ── story 055.W4.2 — allowed-tools (AC1) + DERIVED capabilities (AC3/AC5/AC6) ────────────────── +// +// Proven through the REAL CLI as a subprocess, never a mocked one — same posture as the 055.W3.3 +// invariant fixtures. A gate that has only ever been exercised as a pure function has not been +// shown to be wired into the thing that actually publishes. +describe("055.W4.2 — `allowed-tools` mandatory + capabilities DERIVED, never declared", () => { + test("AC1 — publishing a skill with NO `allowed-tools` is REFUSED (everything else about it is valid)", () => { + withTempDir((dir) => { + const target = writeEmptyIndex(dir); + const ledger = writeEmptyLedger(dir); + const manifest = writeManifest(dir); + const artifact = buildArtifactWithoutAllowedTools(); + assert.throws( + () => + publish({ + manifest, target, ledger, + subject: "sub-1", + "r2-key": `plugins/aiox-enterprise/1.0.0/x.tar.gz`, + artifact, + "mirror-url": `https://${GOOD_HOST}/plugins/aiox-enterprise/1.0.0/x.tar.gz`, + "no-push": true, + }), + (e) => { + const out = String(e.stderr ?? "") + String(e.stdout ?? ""); + assert.match(out, /allowed-tools` is REQUIRED|declares no `allowed-tools`/); + return true; + }, + ); + // the refusal must be REAL: nothing appended + assert.equal(JSON.parse(readFileSync(target, "utf8")).entries.length, 0); + }); + }); + + test("AC3 — a manifest that self-declares `capabilities` is REFUSED, not ignored", () => { + withTempDir((dir) => { + const target = writeEmptyIndex(dir); + const ledger = writeEmptyLedger(dir); + const manifest = writeManifest(dir, { capabilities: ["filesystem:read"] }); + const artifact = buildValidArtifact(); + assert.throws( + () => + publish({ + manifest, target, ledger, + subject: "sub-1", + "r2-key": `plugins/aiox-enterprise/1.0.0/x.tar.gz`, + artifact, + "mirror-url": `https://${GOOD_HOST}/plugins/aiox-enterprise/1.0.0/x.tar.gz`, + "no-push": true, + }), + (e) => { + const out = String(e.stderr ?? "") + String(e.stdout ?? ""); + assert.match(out, /self-declares capabilities|NEVER self-declared/); + return true; + }, + ); + assert.equal(JSON.parse(readFileSync(target, "utf8")).entries.length, 0); + }); + }); + + test("AC6 — a published entry CARRIES the derived capabilities, its two signals, and its limits", () => { + withTempDir((dir) => { + const target = writeEmptyIndex(dir); + const ledger = writeEmptyLedger(dir); + const manifest = writeManifest(dir); + const artifact = buildArtifactWithExecutingSkill(); + publish({ + manifest, target, ledger, + subject: "sub-1", + "r2-key": `plugins/aiox-enterprise/1.0.0/x.tar.gz`, + artifact, + "mirror-url": `https://${GOOD_HOST}/plugins/aiox-enterprise/1.0.0/x.tar.gz`, + "no-push": true, + }); + const entry = JSON.parse(readFileSync(target, "utf8")).entries[0]; + assert.ok(entry.capabilities, "the entry must carry a derived capability block"); + assert.equal(entry.capabilities.self_declared, false); + assert.equal(entry.capabilities.enforcement, "warn-and-display"); + + const runner = entry.capabilities.skills.find((s) => s.skill === "runner"); + assert.equal(runner.owns_scripts, true, "SIGNAL 1 — it ships its own scripts/"); + assert.deepEqual(runner.instructs_execution, ["own"], "SIGNAL 2 — and instructs running it"); + assert.ok(entry.capabilities.union.includes("script:execute")); + + // AC5 — the blind spots travel WITH the capabilities, all the way onto the entry. + assert.ok(entry.capabilities.limits.length >= 1); + assert.match(entry.capabilities.limits.join("\n"), /MCP/); + assert.match(entry.capabilities.limits.join("\n"), /npx/i); + }); + }); + + test("AC6 — capability findings WARN but do not block: the publish with capabilities SUCCEEDS", () => { + withTempDir((dir) => { + const target = writeEmptyIndex(dir); + const ledger = writeEmptyLedger(dir); + const manifest = writeManifest(dir); + const artifact = buildArtifactWithExecutingSkill(); + publish({ + manifest, target, ledger, + subject: "sub-1", + "r2-key": `plugins/aiox-enterprise/1.0.0/x.tar.gz`, + artifact, + "mirror-url": `https://${GOOD_HOST}/plugins/aiox-enterprise/1.0.0/x.tar.gz`, + "no-push": true, + }); + const entry = JSON.parse(readFileSync(target, "utf8")).entries[0]; + assert.ok(entry.capabilities.union.length > 0, "it DID find capabilities..."); + // ...and published anyway. That is D17's decision, made visible as a test. + assert.equal(JSON.parse(readFileSync(target, "utf8")).entries.length, 1); + }); + }); +});