[failproofai-570] fix: validate translated docs pages at generation time and retry on failure

.github/workflows/translate-docs.yml

@@ -46,9 +46,13 @@ jobs:
     strategy:
       fail-fast: false
       # Cap matrix fan-out so the LiteLLM proxy at ANTHROPIC_BASE_URL
-      # never sees the full 14-way burst. With 4 jobs x MAX_CONCURRENT=2
-      # = 8 simultaneous requests, well under the proxy's empirical
-      # connection-drop knee point (see scripts/translate-docs/cli.ts).
+      # never sees the full 14-way burst. With max-parallel 4 jobs x
+      # MAX_CONCURRENT=4 (the cli.ts default; the env block below does not set
+      # TRANSLATE_MAX_CONCURRENT) that is 16 simultaneous requests. A
+      # generation-time validation retry (TRANSLATE_MAX_ATTEMPTS) re-translates
+      # inside the worker's existing slot, so it lengthens that slot's
+      # wall-clock rather than adding a concurrent request — peak concurrency
+      # stays 16 no matter how many attempts a page needs.
       max-parallel: 4
       matrix:
         lang: ${{ fromJson(needs.prepare.outputs.languages) }}

CHANGELOG.md

@@ -12,6 +12,7 @@
 - Redirect the 13 AgentEye pages the upstream syncs deleted (`/agenteye/deployment`, `/agenteye/kubernetes-deployment`, `/agenteye/troubleshooting`, …) to a live page instead of hard-404ing. (#556)
 
 ### Fixes
+- Validate every translated docs page at generation time and re-translate with the validation error fed back, instead of discovering the breakage in `consolidate` after all 14 language jobs have already spent their tokens. The daily `translate-docs` run kept failing (5 of the last 15 runs) on errors like `de/agenteye/cli-skill.mdx: There is a syntax error in your frontmatter on line 2` — the model re-emitting an escaped inner quote from a `description:` value as an unescaped `"`, which breaks the YAML. The repo's own `validate:mdx` net could not catch this class at all: `findMdxParseError` blanks the frontmatter before compiling, so a frontmatter YAML error was structurally invisible to it and only the last-step `mintlify validate` in `consolidate` ever saw it — one bad page in one language sank the whole batch and nothing published. `validate-mdx.ts` now exports `findFrontmatterError` (YAML parse via the already-present `yaml` dep, file-relative line numbers) and `findPageError` (frontmatter then body), and `validate:mdx` runs the latter, so the CI net is now a strict superset of `mintlify validate` across all 647 pages. At generation time each translator renders the exact bytes it will write (sanitizers + link-rewrite for MDX pages, disclaimer + RTL `<div>` wrapper for the README) and validates them through a shared `findTranslationError` — frontmatter YAML, frontmatter key-parity against the English source (catches a dropped or renamed block, which is still valid YAML and which `mintlify` tolerates), and MDX body — re-translating with the error appended to the prompt until it passes or `TRANSLATE_MAX_ATTEMPTS` (integer ≥ 1, default 3, distinct from the SDK-transport `TRANSLATE_MAX_RETRIES`) is reached. On exhaustion it throws *before* the write, so a broken page is never written to disk and never cached — the cache keys only the English source hash, so a cached-invalid page could otherwise never self-heal. System-prompt rule 3 now carries the same frontmatter-quoting guidance rule 2 has always given for JSX attributes, cutting the failure off at the source. (#570)
 - Stop shipping npm lifecycle scripts, removing the `npm warn` every install printed. npm 12 (`allowScripts`, released 2026-07-08 and now the `latest` dist-tag) blocks dependency install scripts by default, so `npm install failproofai` warned `1 package had install scripts blocked … failproofai@… (postinstall: node scripts/postinstall.mjs)` and silently skipped the script; npm 11.16+ prints the advisory `allow-scripts` variant and still runs it. `allowScripts` is purely consumer-side — a package cannot opt itself in (verified: a package declaring `allowScripts`/`trustedDependencies` for *itself* is ignored) — so the only fix is to ship no install scripts. The `postinstall` script's install telemetry (`first_install` / `version_changed` / `package_installed`, with identical event names and properties) now fires from the CLI's first non-hook invocation via `lib/install-check.ts`, reporting at most once per version and no-op'ing on the steady-state path; it is deliberately kept out of `bin/failproofai.mjs`'s `--hook` fast path, which runs on every tool call. This also *recovers* telemetry already being dropped for bun, pnpm, and Yarn ≥4.14 users, which have blocked install scripts for some time. The script's `server.js` check and shadowed-PATH diagnosis were redundant — `scripts/launch.ts` already performs both at launch, with a better error message. `package_installed` now measures install→activation rather than raw installs, and same-version reinstalls (`direction: "reinstall"`) are no longer reported, as the CLI has no signal to detect them. The install-time welcome message is dropped; the `config` wizard's first-run redirect already handles onboarding. (#560)
 - Remove `scripts/preuninstall.mjs`, which never ran. npm honours no uninstall lifecycle scripts (verified with a probe package: `postinstall` fired, `preuninstall` did not), so the hook cleanup it appeared to perform has never happened — uninstalling failproofai leaves `__failproofai_hook__` entries behind in settings files. Removing the dead code makes the gap visible; cleanup needs an explicit `failproofai policies --uninstall` before removing the package. (#560)
 - Stop this repo's dogfood hooks from silently no-op'ing when `bun` isn't on the hook's PATH. All 75 hook commands across the 8 project-level agent-CLI configs fired `bun bin/failproofai.mjs --hook <Event>` directly, so whenever the agent CLI was handed a PATH without bun — `npm i -g bun` installs into a single nvm version's bin dir, so `nvm use <other>` drops it; a macOS GUI launch inherits a launchd PATH built without `~/.zshrc` — every event died with exit 127 and the session ran with **zero** policy enforcement, saying nothing. The configs now call a dev-only `node scripts/dev-hook.mjs` launcher that locates bun across `PATH`, `$BUN_INSTALL/bin`, `~/.bun/bin`, the node execPath sibling, Homebrew/`/usr/local`, and every `~/.nvm/versions/node/*/bin`; installs it via npm if it is genuinely absent; builds `dist/index.js` when missing so `.failproofai/policies/*.mjs` can resolve `import ... from 'failproofai'`; then re-execs the real binary with `stdio: "inherit"` (byte-exact stdout — the deny contracts are JSON on stdout) and propagates the exit code verbatim (2 still means deny; signals map to 128+signum as `sh` did). A `command -v node` pre-check fronts each command so a missing node is a loud one-liner rather than silence — exit 2 on tool events, exit 1 on stop-class events, where exit 2 would mean "retry" and loop forever. The `.opencode` dev shim shares the same resolver and no longer reports a never-run hook as `exitCode: 0` (a silent allow). Production users on `npx -y failproofai` are unaffected. A new drift-guard test reads every committed dogfood config and asserts the launcher form, guard, exit-code split, and per-file command counts — these configs are generated by nothing and read by nothing, which is how #337's opencode shim drifted and silently no-op'd `block-read-outside-cwd` repo-wide. (#564)

__tests__/scripts/translate-docs/mdx-translator.test.ts

@@ -1,18 +1,63 @@
 // @vitest-environment node
-import { describe, it, expect, beforeEach, afterEach } from "vitest";
-import { mkdtempSync, mkdirSync, writeFileSync, existsSync, rmSync } from "node:fs";
+import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
+import {
+  mkdtempSync,
+  mkdirSync,
+  writeFileSync,
+  readFileSync,
+  existsSync,
+  rmSync,
+} from "node:fs";
 import { tmpdir } from "node:os";
-import { join } from "node:path";
+import { join, dirname } from "node:path";
+
+// Mock the Anthropic SDK so the validation-gate tests below drive translation
+// output deterministically. Harmless to the pure/real-tree tests in this file,
+// which never call the translator.
+const { streamMock } = vi.hoisted(() => ({ streamMock: vi.fn() }));
+vi.mock("@anthropic-ai/sdk", () => ({
+  default: class MockAnthropic {
+    messages = { stream: streamMock };
+  },
+}));
+
 import {
   rewriteInternalLinks,
   sanitizeJsxAttributes,
   stripStrayTrailingFence,
   convertHtmlComments,
   getEnglishMdxPages,
   pruneOrphanedTranslations,
+  translateMdxPage,
 } from "@/scripts/translate-docs/mdx-translator";
 import type { TranslationCache } from "@/scripts/translate-docs/types";
 
+/** Queue ONE `end_turn` translation response; call once per expected attempt. */
+function queueTranslation(text: string): void {
+  streamMock.mockReturnValueOnce({
+    finalMessage: async () => ({
+      stop_reason: "end_turn",
+      content: [{ type: "text", text }],
+      usage: { input_tokens: 10, output_tokens: 20 },
+    }),
+  });
+}
+
+/** Sticky response — every attempt returns the same text. */
+function stickyTranslation(text: string): void {
+  streamMock.mockReturnValue({
+    finalMessage: async () => ({
+      stop_reason: "end_turn",
+      content: [{ type: "text", text }],
+      usage: { input_tokens: 10, output_tokens: 20 },
+    }),
+  });
+}
+
+function emptyCache(): TranslationCache {
+  return { sourceHash: "", lastUpdated: "", translations: {} };
+}
+
 describe("getEnglishMdxPages", () => {
   it("includes AgentEye pages in automatic translation", () => {
     const pages = getEnglishMdxPages();
@@ -404,3 +449,106 @@ describe("convertHtmlComments", () => {
     expect(convertHtmlComments(input)).toBe(input);
   });
 });
+
+describe("translateMdxPage validation gate", () => {
+  const EN_SOURCE = `---\ntitle: "Skill"\ndescription: "A page"\n---\n\n# Body\n`;
+  const REL = "agenteye/cli-skill.mdx";
+  const VALID_DE = `---\ntitle: "Fähigkeit"\ndescription: "Eine Seite"\n---\n\n# Körper\n`;
+  // An unescaped inner quote in the description — the reported failure class.
+  const BROKEN_FM_DE = `---\ntitle: "Fähigkeit"\ndescription: "Fragen Sie "ist kaputt?""\n---\n\n# Körper\n`;
+
+  let docsDir: string;
+  let srcPath: string;
+  let outputPath: string;
+
+  beforeEach(() => {
+    streamMock.mockReset();
+    vi.spyOn(console, "warn").mockImplementation(() => {});
+    docsDir = mkdtempSync(join(tmpdir(), "mdx-gate-"));
+    srcPath = join(docsDir, REL);
+    mkdirSync(dirname(srcPath), { recursive: true });
+    writeFileSync(srcPath, EN_SOURCE);
+    outputPath = join(docsDir, "de", REL);
+  });
+
+  afterEach(() => {
+    rmSync(docsDir, { recursive: true, force: true });
+    vi.restoreAllMocks();
+  });
+
+  it("writes the translation when it validates on the first try", async () => {
+    queueTranslation(VALID_DE);
+    const cache = emptyCache();
+
+    const result = await translateMdxPage(srcPath, "de", { docsDir, cache });
+
+    expect(streamMock).toHaveBeenCalledTimes(1);
+    expect(existsSync(outputPath)).toBe(true);
+    expect(result.attempts).toBe(1);
+  });
+
+  it("writes the valid retry after a broken-frontmatter first attempt", async () => {
+    queueTranslation(BROKEN_FM_DE);
+    queueTranslation(VALID_DE);
+    const cache = emptyCache();
+
+    const result = await translateMdxPage(srcPath, "de", { docsDir, cache });
+
+    expect(streamMock).toHaveBeenCalledTimes(2);
+    expect(result.attempts).toBe(2);
+    expect(existsSync(outputPath)).toBe(true);
+    // The VALID translation is what got written, not the broken first attempt.
+    expect(readFileSync(outputPath, "utf-8")).toContain("Eine Seite");
+  });
+
+  it("throws and writes no file when every attempt fails validation", async () => {
+    stickyTranslation(BROKEN_FM_DE);
+    const cache = emptyCache();
+
+    await expect(
+      translateMdxPage(srcPath, "de", { docsDir, cache }),
+    ).rejects.toThrow(/still fails validation/);
+    expect(existsSync(outputPath)).toBe(false);
+  });
+
+  it("adds no cache entry when every attempt fails validation", async () => {
+    stickyTranslation(BROKEN_FM_DE);
+    const cache = emptyCache();
+
+    await expect(
+      translateMdxPage(srcPath, "de", { docsDir, cache }),
+    ).rejects.toThrow();
+    expect(cache.translations).not.toHaveProperty(`${REL}::de`);
+    expect(Object.keys(cache.translations)).toHaveLength(0);
+  });
+
+  it("validates the sanitized, link-rewritten bytes rather than the raw model output", async () => {
+    // Raw output has a stray doubled quote in a JSX attribute (invalid MDX);
+    // sanitizeJsxAttributes fixes it before validation, so it passes on the
+    // first attempt — proving validation runs on the RENDERED bytes. Only one
+    // response is queued, so a wrongly-triggered retry would fail the test.
+    const RAW_FIXABLE = `---\ntitle: "Fähigkeit"\ndescription: "Eine Seite"\n---\n\n<Card title="Foo"" />\n`;
+    queueTranslation(RAW_FIXABLE);
+    const cache = emptyCache();
+
+    const result = await translateMdxPage(srcPath, "de", { docsDir, cache });
+
+    expect(streamMock).toHaveBeenCalledTimes(1);
+    expect(result.attempts).toBe(1);
+    const written = readFileSync(outputPath, "utf-8");
+    expect(written).toContain('title="Foo"');
+    expect(written).not.toContain('title="Foo""');
+  });
+
+  it("performs no validation and no model call under dryRun", async () => {
+    const result = await translateMdxPage(srcPath, "de", {
+      docsDir,
+      cache: emptyCache(),
+      dryRun: true,
+    });
+
+    expect(streamMock).not.toHaveBeenCalled();
+    expect(existsSync(outputPath)).toBe(false);
+    expect(result.cached).toBe(false);
+  });
+});

__tests__/scripts/translate-docs/translator.test.ts

@@ -12,13 +12,33 @@ vi.mock("@anthropic-ai/sdk", () => ({
   },
 }));
 
-import { translateContent } from "@/scripts/translate-docs/translator";
+import {
+  translateContent,
+  translateValidated,
+} from "@/scripts/translate-docs/translator";
 
-/** Configure the next `.finalMessage()` resolution. */
+/** Configure the next `.finalMessage()` resolution (sticky — same value forever). */
 function mockFinalMessage(message: unknown): void {
   streamMock.mockReturnValue({ finalMessage: async () => message });
 }
 
+/** Queue ONE `end_turn` response with the given text; call once per attempt. */
+function queueFinalMessage(
+  text: string,
+  usage: { input_tokens: number; output_tokens: number } = {
+    input_tokens: 10,
+    output_tokens: 20,
+  },
+): void {
+  streamMock.mockReturnValueOnce({
+    finalMessage: async () => ({
+      stop_reason: "end_turn",
+      content: [{ type: "text", text }],
+      usage,
+    }),
+  });
+}
+
 describe("translateContent", () => {
   beforeEach(() => {
     streamMock.mockReset();
@@ -69,3 +89,157 @@ describe("translateContent", () => {
     expect(requestArgs.max_tokens).toBe(64000);
   });
 });
+
+describe("translateValidated", () => {
+  const base = {
+    source: "# Doc",
+    lang: "de",
+    langName: "German",
+    label: "doc [de]",
+    render: (raw: string) => raw,
+  };
+  /** Reject any rendered output that still contains the marker `BAD`. */
+  const rejectBad = async (bytes: string): Promise<string | null> =>
+    bytes.includes("BAD") ? "the output still contains BAD" : null;
+
+  beforeEach(() => {
+    streamMock.mockReset();
+    // Silence the per-attempt retry warnings so test output stays readable.
+    vi.spyOn(console, "warn").mockImplementation(() => {});
+  });
+
+  it("makes exactly one model call when the first translation is valid", async () => {
+    queueFinalMessage("clean body");
+
+    const result = await translateValidated({
+      ...base,
+      validate: async () => null,
+    });
+
+    expect(result.rendered).toBe("clean body");
+    expect(result.attempts).toBe(1);
+    expect(streamMock).toHaveBeenCalledTimes(1);
+  });
+
+  it("retries with the validation error and returns the corrected translation", async () => {
+    queueFinalMessage("BAD body");
+    queueFinalMessage("good body");
+
+    const result = await translateValidated({ ...base, validate: rejectBad });
+
+    expect(result.rendered).toBe("good body");
+    expect(result.attempts).toBe(2);
+    expect(streamMock).toHaveBeenCalledTimes(2);
+  });
+
+  it("appends the retry feedback to the user turn and leaves the system prompt untouched", async () => {
+    queueFinalMessage("BAD body");
+    queueFinalMessage("good body");
+
+    await translateValidated({
+      ...base,
+      validate: async (bytes) =>
+        bytes.includes("BAD") ? "the frontmatter is broken on line 2" : null,
+    });
+
+    const first = streamMock.mock.calls[0][0] as {
+      system: unknown;
+      messages: Array<{ role: string; content: string }>;
+    };
+    const second = streamMock.mock.calls[1][0] as {
+      system: unknown;
+      messages: Array<{ role: string; content: string }>;
+    };
+    // First attempt carries no repair note.
+    expect(first.messages[0].content).not.toContain("REJECTED");
+    // The retry feeds the validation error back in the same user turn...
+    expect(second.messages[0].content).toContain(
+      "the frontmatter is broken on line 2",
+    );
+    expect(second.messages[0].content).toContain("Retry 2 of 3");
+    // ...and never mutates the (cached) system prompt.
+    expect(second.system).toEqual(first.system);
+  });
+
+  it("retries an empty translation instead of returning it", async () => {
+    queueFinalMessage("   "); // whitespace-only counts as empty
+    queueFinalMessage("good body");
+
+    const result = await translateValidated({
+      ...base,
+      validate: async () => null,
+    });
+
+    expect(result.rendered).toBe("good body");
+    expect(result.attempts).toBe(2);
+  });
+
+  it("throws after TRANSLATE_MAX_ATTEMPTS invalid translations", async () => {
+    mockFinalMessage({
+      stop_reason: "end_turn",
+      content: [{ type: "text", text: "BAD body" }],
+      usage: { input_tokens: 10, output_tokens: 20 },
+    });
+
+    await expect(
+      translateValidated({ ...base, validate: rejectBad }),
+    ).rejects.toThrow(/still fails validation after 3 attempt/);
+  });
+
+  it("stops calling the model once the attempt cap is reached", async () => {
+    mockFinalMessage({
+      stop_reason: "end_turn",
+      content: [{ type: "text", text: "BAD" }],
+      usage: { input_tokens: 1, output_tokens: 1 },
+    });
+
+    await expect(
+      translateValidated({ ...base, validate: rejectBad }),
+    ).rejects.toThrow();
+    expect(streamMock).toHaveBeenCalledTimes(3);
+  });
+
+  it("sums input and output tokens across attempts", async () => {
+    queueFinalMessage("BAD body", { input_tokens: 100, output_tokens: 200 });
+    queueFinalMessage("good body", { input_tokens: 50, output_tokens: 60 });
+
+    const result = await translateValidated({ ...base, validate: rejectBad });
+
+    expect(result.inputTokens).toBe(150);
+    expect(result.outputTokens).toBe(260);
+    expect(result.attempts).toBe(2);
+  });
+
+  it("does not retry a response truncated at max_tokens", async () => {
+    // translateContent throws on max_tokens before returning; that is not a
+    // validity failure, so translateValidated must let it propagate uncaught.
+    mockFinalMessage({
+      stop_reason: "max_tokens",
+      content: [{ type: "text", text: "partial…" }],
+      usage: { input_tokens: 1, output_tokens: 64000 },
+    });
+
+    await expect(
+      translateValidated({
+        ...base,
+        lang: "he",
+        langName: "Hebrew",
+        validate: async () => null,
+      }),
+    ).rejects.toThrow(/truncated at max_tokens/);
+    expect(streamMock).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not retry when the request itself throws", async () => {
+    streamMock.mockReturnValue({
+      finalMessage: async () => {
+        throw new Error("Connection error.");
+      },
+    });
+
+    await expect(
+      translateValidated({ ...base, validate: async () => null }),
+    ).rejects.toThrow(/Connection error/);
+    expect(streamMock).toHaveBeenCalledTimes(1);
+  });
+});