Skip to content

perf(repo): verify-perf / memory (tsc --build cache, typecheck concurrency, heap, workspaceConcurrency) - #11

Open
timoteo7 wants to merge 1 commit into
mainfrom
fix/pre-json-into-side-pnpm-concurrency
Open

timoteo7 wants to merge 1 commit into
mainfrom
fix/pre-json-into-side-pnpm-concurrency

Conversation

@timoteo7

@timoteo7 timoteo7 commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Problem

Two root causes parked task runs for hours on the 2026-09-23 nightly fleet (12GB desktop):

  1. .changeset/pre.json corruption on merge/rebase — when a branch rebased across a chore(release) commit (e.g. v0.78.0-beta.6) and the conflict was resolved from the older side, the beta anchor broke and check-pre-json-anchor failed deterministically on that branch. Two review tasks sat terminal-failed 8–9h with verification blocked. Same regression class as fix(RUFU-075): safest scheduler queue/mission-store writes for minimal stores Runfusion/Fusion#3472.
  2. Unbounded recursive parallelism — pnpm -r defaults to 4 concurrent projects, and pnpm 10.33 silently ignores the ~/.npmrc workspace-concurrency key (it echoes the value; behavior stays at 4). Each task typecheck fanned out 4 tsc --noEmit workers (500MB–1.4GB each) and thrashed the machine: memory PSI full 94%, engine HTTP timeouts, runs stuck in swap for 70+ minutes.

Fix

  • .gitattributes: .changeset/pre.json merge=binary — built-in git merge driver that keeps the copy from the branch being merged INTO (the exact rule scripts/check-pre-json-anchor.mjs prescribes manually). No per-clone configuration needed — the common merge=ours recipe requires git config merge.ours.driver true in every clone and otherwise leaves full conflict markers. The check stays as detection; this prevents the corruption at conflict-resolution time.

  • pnpm-workspace.yaml: workspaceConcurrency: 2 — caps recursive package commands at 2 projects at a time (the live pnpm 10.x config channel). Scope is one pnpm -r invocation only. The manifest comment now states why the value stays a constant — pnpm 10 has no live env channel for it, and this is the only backstop for an ad-hoc pnpm -r no script wraps — and that it is a blunt guardrail overridable per command with --workspace-concurrency, not a project requirement. Callers this repo owns already pass that flag explicitly (scripts/test-changed.mjs derives it from FUSION_TEST_WORKSPACE_CONCURRENCY).

  • scripts/verify-fast.mjs: TYPECHECK_PARALLEL_LIMIT now derives from RAM instead of core count — this is the guard that actually covers the typecheck fan-out of root cause 2. verify:fast spawns one pnpm --filter <pkg> typecheck process per changed package, and those bypass workspaceConcurrency entirely, so the key above cannot bound them. The old min(4, cpuCount / 2) gave 4 here; tsc is memory-bound (1–1.4GB RSS each), so the ceiling is now 1 worker per 5GB of total RAM, capped at 4 — 2 on this 12GB box, and a 64GB machine is no longer slowed down by it. FUSION_VERIFY_FAST_TYPECHECK_CONCURRENCY overrides the derived default, alongside the existing FUSION_VERIFY_FAST_SERIAL. Measured 2026-09-23: the 4-wide batch sat in D state at 3–7% CPU per worker for 10+ min with zero progress, drove memory PSI full to 80%, and froze the desktop twice. STATIC_CHECK_PARALLEL_LIMIT is untouched (those steps are node-startup dominated, not memory bound).

  • scripts/dev-with-memory.mjs: the dev heap ceiling was FUSION_DEV_MEMORY_MB || "8192" — a constant sized after one workstation, handing every node child an 8GB heap ceiling on a 12GB box. It now derives from os.totalmem() (half of RAM, clamped 1–8GB → 6381MB here) with the env override kept. The wrapper also stopped prepending a second --max-old-space-size when NODE_OPTIONS already pins one (callers like the local Fusion launcher pin 2048, and precedence between repeated V8 flags is not worth relying on).

Tests

  • Merge driver red/green: without the attribute, a pre.json rebase conflict leaves <<<<<<< markers with the older side able to win (bug reproduced); with merge=binary the file resolves to the INTO side automatically.
  • Concurrency bench: 6-package workspace measuring real process overlap — peak 4 without the key, peak 2 with it.
  • Typecheck cap: node --check scripts/verify-fast.mjs clean; importing the module asserts the derived ceiling lands in 1–4 (2 on this 12GB box) and that buildTypecheckStep("@fusion/core", { hasTypecheck: true }).parallelLimit equals that constant rather than a literal, so a future edit cannot silently leave the step at 4. Override path checked too: FUSION_VERIFY_FAST_TYPECHECK_CONCURRENCY=3 yields 3, while a garbage or 0 value falls back to the derived default. Exits non-zero if any assertion breaks.
  • Dev heap ceiling: node --check scripts/dev-with-memory.mjs clean; formula check asserts the derived value lands in the 1–8GB clamp (6381MB on this 12GB box, no clamping in play here), and the NODE_OPTIONS dedup asserts three cases — an inherited --max-old-space-size is left untouched, an empty inheritance injects one, and other inherited flags (--trace-warnings) are preserved. Exits non-zero if any assertion breaks.
  • node scripts/check-fnxc-future-dates.mjs — 0 future-dated stamps (both FNXC notes use a date -u stamp).
  • Grep for TYPECHECK_PARALLEL_LIMIT across the tree: no test asserts the old value of 4.
  • pnpm install --frozen-lockfile --ignore-scripts --offline, pnpm check:workspace-package-graph, pnpm check:pre-json — all exit 0.

Changeset

None — repository build/merge configuration only; no published package behavior changes.

Implements FUSI-021 / FUSI-022.

@timoteo7 timoteo7 self-assigned this Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

ThreatCrush Security Scan

4586 finding(s)

HIGH/CRITICAL: 43 | MEDIUM: 4032 | LOW: 511

Severity Rule Location
HIGH secret-database-url .github/workflows/full-suite.yml:55
HIGH secret-generic-credential .github/workflows/full-suite.yml:56
HIGH secret-database-url .github/workflows/full-suite.yml:284
HIGH secret-generic-credential .github/workflows/full-suite.yml:285
HIGH secret-database-url .github/workflows/full-suite.yml:324
HIGH secret-generic-credential .github/workflows/full-suite.yml:325
HIGH secret-database-url .github/workflows/pr-checks.yml:221
HIGH secret-generic-credential .github/workflows/pr-checks.yml:222
HIGH secret-generic-credential .github/workflows/release.yml:522
HIGH secret-generic-credential .github/workflows/release.yml:524
HIGH secret-generic-credential .github/workflows/test-release.yml:445
HIGH secret-generic-credential .github/workflows/test-release.yml:447
HIGH secret-generic-credential docs/cli-reference.md:80
HIGH secret-generic-credential docs/signals-connectors.md:34
HIGH secret-generic-credential docs/signals-connectors.md:77
HIGH secret-generic-credential docs/signals-connectors.md:94
HIGH secret-generic-credential docs/signals-connectors.md:117
HIGH secret-generic-credential docs/signals-connectors.md:159
HIGH secret-generic-credential packages/cli/STANDALONE.md:71
HIGH secret-database-url packages/core/src/postgres/credential-redact.ts:12
HIGH secret-database-url packages/core/src/postgres/credential-redact.ts:30
HIGH secret-database-url packages/core/src/postgres/credential-redact.ts:31
HIGH secret-database-url packages/core/src/postgres/credential-redact.ts:102
HIGH secret-database-url packages/core/src/postgres/credential-redact.ts:103
HIGH secret-generic-credential packages/core/src/postgres/embedded-lifecycle.ts:843
HIGH secret-database-url packages/core/src/postgres/embedded-lifecycle.ts:1574
HIGH secret-database-url packages/core/src/postgres/pg-backup.ts:756
HIGH js-ssrf-outbound-request packages/dashboard/app/public/sw.js:651
HIGH js-ssrf-outbound-request packages/dashboard/app/public/sw.js:727
HIGH js-host-header-trust packages/dashboard/src/cli-session-ws.ts:81
HIGH js-host-header-trust packages/dashboard/src/cli-session-ws.ts:115
HIGH js-ssrf-outbound-request packages/dashboard/src/routes.ts:1858
HIGH js-host-header-trust packages/dashboard/src/server.ts:2689
HIGH js-host-header-trust packages/dashboard/src/server.ts:2714
HIGH js-host-header-trust packages/dashboard/src/server.ts:3025
HIGH js-host-header-trust packages/dashboard/src/server.ts:3193
HIGH secret-slack-webhook plugins/examples/fusion-plugin-notification/README.md:46
HIGH secret-database-url scripts/pg-test-server.mjs:200
HIGH secret-database-url scripts/pg-test-server.mjs:231
HIGH secret-database-url scripts/pg-test-server.mjs:241
HIGH secret-generic-credential scripts/sync-fusion-skill-tools.mjs:550
HIGH secret-generic-credential scripts/verify-windows-elevated-restricted.mjs:81
HIGH secret-generic-credential scripts/verify-windows-encoding-recovery.mjs:41
MEDIUM redos-nested-quantifier docs/agents.md:1710
MEDIUM insecure-temp-file packages/cli/src/__tests__/bin.test.ts:136
MEDIUM insecure-temp-file packages/cli/src/__tests__/dev-with-memory-lib.test.ts:33
MEDIUM insecure-temp-file packages/cli/src/__tests__/dev-with-memory-lib.test.ts:34
MEDIUM insecure-temp-file packages/cli/src/__tests__/dev-with-memory-lib.test.ts:35
MEDIUM insecure-temp-file packages/cli/src/__tests__/dev-with-memory-lib.test.ts:43
MEDIUM insecure-temp-file packages/cli/src/__tests__/dev-with-memory-lib.test.ts:48

…and 4536 more. Full results in the Security tab.

Snippets are redacted; ThreatCrush never prints matched credential material.

@timoteo7
timoteo7 force-pushed the fix/pre-json-into-side-pnpm-concurrency branch from ce712e5 to c0dbd01 Compare September 23, 2026 22:04
@timoteo7 timoteo7 changed the title fix(repo): keep pre.json on the INTO side and cap pnpm recursion at 2 fix(repo): keep pre.json on the INTO side and bound the verify:fast typecheck fan-out Sep 23, 2026
@timoteo7
timoteo7 force-pushed the fix/pre-json-into-side-pnpm-concurrency branch from 464c5f4 to b84f2ef Compare September 23, 2026 22:35
FNXC:TriagePlanningRecovery 2026-09-19-04:04:
The sweep announced "Recovering specified triage task <id>" on every poll for cards it never
recovered, and recorded no outcome when the recovery declined or when its own gates skipped the
candidate. The recorded engine log held 180 announcements and zero reasons, so an operator could
paused: false,
steps: [{ title: "Implement", status: "pending" }],
workflowStepResults: [
{ workflowStepId: "plan-review", workflowStepName: "Plan Review", phase: "pre-merge", status: "passed", verdict: "APPROVE" },
@timoteo7
timoteo7 force-pushed the fix/pre-json-into-side-pnpm-concurrency branch 3 times, most recently from 82d59cc to 4af29b0 Compare September 24, 2026 00:26
timoteo7 pushed a commit that referenced this pull request Sep 24, 2026
…ing-spin + graph identity + session watchdog)

FNXC:BoardReliability 2026-09-24-00:46:
Consolidates the board-reliability fixes (the engine theme, kept separate from verify-perf in #11):
1. SOURCE-OF-TRUTH stranded count (stranded-commits.ts): fork-point + "0 task-unique = 0 stranded"; a failed fork-point
   falls back to ZERO, never the raw base. listStrandedCommits() uses the single source (the 2nd call site).
2. Planning-spin: an unchanged-but-valid authoritative PROMPT.md is SUCCESS (was re-planning forever).
3. Graph-failure identity: a parse/compile failure names the failing stage, not node 'unknown'.
4. Session-idle watchdog: aborts a runaway session (budget 250 tools / idle 30min) with RUNAWAY_SESSION (call fn_task_done).
5. Manual retry clears ANY non-user pause. Plus the typecheck step stays READ-ONLY (tsc --build --noEmit).

Symptom Verification:
- Original symptom: "N stranded commits since <base>" treadmill + planning-spin + node 'unknown' + 4h runaway sessions.
- Exact reproduction: a task branch at the main tip vs merge-base; a valid unchanged PROMPT.md; a session that never calls fn_task_done.
- Assertion it is gone: countStrandedCommits()==0 for shared lineage; planning accepts a valid unchanged spec; the watchdog aborts runaway sessions; the typecheck emits nothing (read-only).
timoteo7 pushed a commit that referenced this pull request Sep 24, 2026
…ead-only)

FNXC:VerifyPerf 2026-09-24-01:00:
MOVED from #17 (board-reliability) to #11 (verify-perf theme, per review). The typecheck step keeps the incremental
tsbuildinfo cache (warm runs recheck only changed files) while staying READ-ONLY (--noEmit: no dist emit, no partial dist
on a killed typecheck).

Symptom Verification: warm verify typecheck rechecks only changed files; the step emits nothing (read-only).
@timoteo7 timoteo7 changed the title fix(repo): keep pre.json on the INTO side and bound the verify:fast typecheck fan-out perf(repo): verify-perf / memory (tsc --build cache, typecheck concurrency, heap, workspaceConcurrency) Sep 24, 2026
timoteo7 pushed a commit that referenced this pull request Sep 24, 2026
@timoteo7
timoteo7 force-pushed the fix/pre-json-into-side-pnpm-concurrency branch from 2150a29 to 0dfb730 Compare September 25, 2026 04:15
@timoteo7

Copy link
Copy Markdown
Owner Author

Review notes for the audited SHA 0dfb730e1e2cc8b3e246335ec0476ffcf2c5e328:

The TypeScript build-cache change is worth keeping: a focused compiler check confirmed that --build --noEmit reuses the cache and still detects a newly introduced type error. The reported performance gain was not measured across the full workspace, so please avoid extrapolating the synthetic benchmark to all packages.

One description needs correction: Git's built-in merge=binary keeps the current-side content as its tentative result but leaves the path conflicted; it does not automatically finish the merge. The change can still prevent conflict-marker corruption, but the PR's “resolves automatically” wording overstates it. Also consider whether a global workspaceConcurrency: 2 is the right default for larger machines; the PR itself describes it as a coarse small-host guardrail. Prefer the existing per-command overrides where possible, and keep the measured cache improvement separate from these policy choices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants