Version-aware CLI detection: know *which* agent version we're guarding, not just *whether* one exists

[chhhee10]

Problem / Motivation

Umbrella issue. Core mechanism + a per-CLI contributor checklist at the bottom. Good first issues throughout — each CLI row can be claimed independently.

TL;DR

failproofai integrates with 12 agent CLIs by wiring hooks into each one. Today detection is a boolean: binaryExists("copilot") → which copilot → true/false. But every CLI's hook contract (payload shape, tool-input keys, deny-response format) drifts between versions — and when it drifts, enforcement breaks silently. failproofai is currently version-blind: it cannot tell it's talking to a version whose contract it has never verified.

This issue proposes making failproofai version-aware: detect each CLI's version, record it, warn when it's outside the range we've verified, and — where a contract genuinely changed — branch behavior per version. Detection cost is essentially free (we already spawn to detect presence).

Why this matters (two real, recent case studies)

1. GitHub Copilot CLI 1.0.41 → 1.0.71 — the hook payload's file-tool input keys changed (file_path → path, plus file_text/old_str/new_str), and .env reads silently sailed past block-env-files. Enforcement looked "installed and working" but was a no-op. We only caught it by manually capturing live payloads.
2. Factory droid 0.171 → 0.172 — suspected the same class of silent drift; our contract was verified against 0.171 only.

In both cases, version awareness would have turned a silent failure into a visible warning ("Copilot 1.0.72 detected; failproofai has verified 1.0.71 — file policies may not enforce; please report").

Current behavior

// src/hooks/integrations.ts
function binaryExists(name: string): boolean {
  const cmd = process.platform === "win32" ? `where ${name}` : `which ${name}`;
  try { execSync(cmd, { stdio: "pipe" }); return true; } catch { return false; }
}

// each integration:
detected() { return binaryExists("copilot"); }

Detection is binary. failproofai knows that Copilot exists, never which Copilot.

Proposed Solution

1. Three-state detection (do NOT collapse presence into version)

A version probe has three outcomes, not two. The critical one is the middle: a binary that exists but whose --version hangs, errors, or prints something unparseable must still count as present so failproofai keeps enforcing — never silently disable enforcement because a version string couldn't be read.

interface CliDetection {
  present: boolean;        // binary is on PATH (today's signal — preserved)
  version: string | null;  // parsed semver-ish string, or null if unknown/unreadable
  raw?: string;            // raw `--version` output, for diagnostics
}

// present:false, version:null            → not installed
// present:true,  version:"1.0.71"        → installed, known version
// present:true,  version:null            → installed, version UNKNOWN → still enforce, but flag it

detected() stays derivable as detect().present, so nothing downstream breaks.

2. Per-CLI version command + parser

Each CLI prints its version differently and to different streams. Every integration declares how to read and parse it:

interface Integration {
  // ...existing...
  versionCommand?: string;                      // e.g. "copilot --version"
  parseVersion?(raw: string): string | null;    // e.g. /Copilot CLI ([\d.]+)/ → "1.0.71"
  verifiedVersions?: string;                     // semver range we've verified, e.g. ">=1.0.71 <1.1.0"
}

Probe with a short timeout (a misbehaving --version must never hang failproofai) and cache the result for the process.

3. Where it runs (cost)

which is a cheap PATH lookup; <cli> --version spawns the real binary (Node startup, sometimes a prompt). So version probing runs only off the hot path — in the config wizard, audit, and a doctor/policies view — never per hook invocation. Result is cached.

What we do with the version (full scope)

Phased so it ships incrementally, but the end-state is full version-aware support:

• Phase 1 — Detect + record. Add detect() returning CliDetection; attach cli_version to hook + audit telemetry. Instantly gives us the real version distribution across users and an early-warning signal for drift. (Guaranteed-shippable core.)
• Phase 2 — Surface + warn. Show detected version + verified range in failproofai doctor / policies output. Warn when a CLI's version falls outside verifiedVersions ("newer than verified — enforcement may be affected, please open an issue"). This is the user-facing "failproofai has your back" moment.
• Phase 3 — Version-branched contracts. Where a contract genuinely changed between versions (e.g. Copilot's input keys), select the right canonicalization/response shape by detected version instead of accepting-both-shapes forever. Keep this narrow and well-tested — it's the most fragile part; accept-both-shapes remains the default and version-branching is only for cases where the shapes actually conflict.

API sketch (illustrative)

export function detectCli(name: string, integ: Integration): CliDetection {
  if (!binaryExists(name)) return { present: false, version: null };
  try {
    const raw = execSync(integ.versionCommand ?? `${name} --version`,
      { encoding: "utf8", stdio: "pipe", timeout: 3000 });
    return { present: true, version: integ.parseVersion?.(raw) ?? null, raw };
  } catch {
    return { present: true, version: null }; // present but unreadable → still enforce
  }
}

Acceptance criteria

• CliDetection type + detectCli() with a hard timeout and per-process caching.
• detected() preserved (derived from .present) — no downstream regressions.
• versionCommand / parseVersion / verifiedVersions wired for each of the 12 CLIs (see table).
• cli_version added to hook + audit telemetry (Phase 1).
• doctor/policies surfaces version + verified range, and warns on out-of-range (Phase 2).
• Unit tests: each parser against a captured real --version string; the three-state matrix (absent / known / present-but-unreadable); timeout + fail-safe (unreadable ⇒ present:true).
• A misbehaving/hanging --version never hangs or disables failproofai (timeout + fail-open to present:true).

Per-CLI checklist (claim a row!)

Each row = one self-contained good-first-issue: find the CLI's version command, write the parse regex, record the version(s) we've verified. ? = needs a contributor to verify on a real install.

CLI	Binary	Version command	Parse target (example output)	Verified version(s)
Claude Code	claude / claude-code	claude --version	?	?
Codex	codex	codex --version	?	?
Copilot	copilot	copilot --version	GitHub Copilot CLI 1.0.71. → 1.0.71	1.0.71 (broke at 1.0.41→1.0.71)
Cursor	cursor-agent / agent	cursor-agent --version ?	?	docs 2026-05-08
OpenCode	opencode	opencode --version	?	1.14.33 / 1.17.x
Pi	pi	pi --version ?	?	0.73.1, 0.80.3
Hermes	hermes	? (gateway — may differ)	?	?
OpenClaw	openclaw	openclaw --version ?	?	2026.7.1
Factory (droid)	droid	droid --version	0.171.0	0.171.0
Devin	devin	devin --version	v3000.1.27 → 3000.1.27	3000.1.27
Antigravity	agy	agy --version	1.1.2	1.1.2
Goose	goose	goose --version	1.43.0	1.43.0

Notes for contributors: some CLIs print to stderr, append a trailing ., prefix v, or (gateways like Hermes/OpenClaw) may not expose a conventional --version — capture the raw output verbatim in your PR so the parser is written against reality, not a guess. Please don't guess a version format you can't run.

Out of scope / risks

• Not calling --version on the hot hook path (latency).
• Version-branching (Phase 3) stays narrow: accept-both-shapes remains the default; branch only where versions genuinely conflict.
• Gateways (Hermes/OpenClaw) may need a different version source than a CLI flag — flagged in the table.

---

Alternatives Considered

No response

Additional Context

Context: this proposal came directly out of debugging the Copilot 1.0.71 file-policy regression — failproofai enforced nothing on a .env read yet reported healthy. Version awareness is how we make that class of drift loud instead of silent.