Cleanup planning and bounded safety scans regress on large workspaces

[chubes4]

Problem

The documented DB-backed cleanup path and all reviewed fallback scans are unusable on a 446-worktree workspace. The retention planner fails before returning a run ID, while bounded safety scans that previously completed quickly now run silently beyond three minutes.

This regresses the behavior addressed by #736, #631, and #384. In particular, #736 recorded bounded-cleanup-eligible-apply --limit=100 completing in roughly 5.5 seconds; the same bounded family now times out with only 15 candidates.

Environment

• Workspace: /Users/chubes/Developer
• Inventory: 53 primaries, 446 worktrees
• Cleanup-eligible inventory rows: 15
• Free disk: 58.8 GiB
• No inventory-known dirty or unpushed blockers
• Two stale filesystem locks were safely pruned before retrying; subsequent lock status reported zero stale locks

Reproduction

Planner fails without a run ID

studio wp datamachine-code workspace cleanup plan \
  --mode=retention \
  --include-artifacts \
  --limit=100 \
  --top=20 \
  --until-budget=2m \
  --format=json

Observed:

Error: Failed to create cleanup run.

No run ID, reason code, child error, or recovery command is returned.

Bounded cleanup-eligible review hangs

studio wp datamachine-code workspace worktree bounded-cleanup-eligible-apply \
  --dry-run --limit=15 --format=json

Observed: no output before a 180-second shell timeout.

Active/no-signal merged review hangs

studio wp datamachine-code workspace worktree active-no-signal-merged-apply \
  --dry-run --limit=100 --offset=0 --format=json

Observed: no output before a 180-second shell timeout.

Full cleanup review hangs

studio wp datamachine-code workspace worktree cleanup --dry-run --format=json

Observed: no output before a 180-second shell timeout.

Cheap workspace hygiene and bounded size inventory still return, proving the top-level inventory itself is readable. No destructive cleanup was applied.

Safety impact

The inventory includes open-PR worktrees marked cleanup_eligible, including Blocks Engine PR #659 and Static Site Importer PR #714. Operators therefore cannot safely substitute an inventory-only apply: fresh Git/PR revalidation is required, but every path providing it hangs. This pushes operators toward manual deletion or retaining hundreds of stale worktrees.

Expected

• cleanup plan returns a persisted run ID or a structured creation failure with the owning job/database error.
• Every bounded command honors --limit and returns partial evidence within a normal operator timeout.
• Long GitHub/Git checks emit progress and a continuation cursor.
• Per-repository auth/API failures degrade to explicit skipped rows instead of stalling the whole page.
• Open PR worktrees remain protected even if lifecycle metadata says cleanup_eligible.

Acceptance criteria

• Reproduce with 400+ registered worktrees and 15 cleanup-eligible rows.
• bounded-cleanup-eligible-apply --dry-run --limit=15 returns within 30 seconds or returns a budgeted partial page.
• active-no-signal-merged-apply --dry-run --limit=100 emits progress/continuation rather than remaining silent.
• Planner creation errors include a stable reason code, underlying exception/job evidence, and retry command.
• Tests cover unavailable GitHub App token exchange, slow GitHub responses, and mixed open/merged PR candidates without N-times credential retry amplification.
• Existing dirty, unpushed, primary, containment, and open-PR protections remain fail-closed.

Related

• Cleanup scans should be bounded by default on large workspaces #736: cleanup scans bounded by default
• workspace cleanup plan is still too slow on large workspaces #631: retention planning latency
• Budget full worktree cleanup dry-runs #384: budget full cleanup dry-runs
• Keep direct metadata reconciliation apply bounded during revalidation #351: bounded metadata apply revalidation

AI assistance

• AI assistance: Yes
• Model: OpenAI GPT-5.6 Sol
• Tool: OpenCode
• Used for: command reproduction, duplicate search, safety analysis, and issue drafting

[chubes4]

Additional scale/performance evidence on 2026-07-28:

• Filesystem hygiene reports 515 managed worktree directories.
• Git worktree enumeration returns 702 rows because it also discovers 193 external worktrees; 174 are Homeboy controller-scratch attempts.
• Exact DMC worktree get on installed v0.53.6 took 5.36s in a successful sample and exceeded Homeboy’s 10s provider deadline twice.
• Target repo homeboy alone has 236 registered worktree rows.
• DMC Targeted worktree get and finalize commands hang on a clean managed worktree #964 / PR Bound targeted worktree lifecycle commands #965 fixes targeted lookup but is only present in draft v0.54.1; this site remains on v0.53.6.
• Inventory has 760 DB rows and 59 missing paths.

Large-workspace tests should include external Git worktrees and high-cardinality single-repository registries, not only top-level managed directories. Bounded commands need to avoid paying full repo registry/status costs for an exact handle.

[chubes4]

Additional large-workspace evidence from 2026-07-30: cleanup output and probing remained expensive enough that operator workflows relied on inventory-only previews whose outcomes diverged sharply from apply. A preview reported 85 lifecycle candidates while fresh apply removed 12. The bounded path should support a budgeted safety-probe mode that returns partial would_remove evidence and continuation rather than forcing a choice between fast/inexact and slow/exact review.

[chubes4]

Fresh production-scale reproduction on 2026-07-30 confirms the cleanup latency problem remains severe and can produce destructive partial progress without any final evidence.

Starting state:

• 475 worktrees
• 35.4 GiB free / 96.18% disk used
• 94 inventory cleanup candidates

Command:

studio wp datamachine-code workspace worktree cleanup --format=json

Observed:

• No output before the outer shell's 600-second timeout.
• The command nevertheless removed worktrees before termination.
• A follow-up hygiene scan showed worktrees dropped from 475 to 305 and free disk increased to 37.7 GiB.
• There was no returned summary, continuation cursor, durable cleanup run ID, or exact removed/skipped set.
• One DB lock remained active after the owner process exited and only became safely pruneable after expiry.

The same run also showed that homeboy cleanup waits beyond its displayed 30-second DMC provider-preview budget and remains hung until a 600-second outer timeout. Homeboy regression tracker: Extra-Chill/homeboy#10963.

This strengthens the requirement that apply itself be paginated and durable. A command that mutates 170 worktrees and then times out without output is not merely slow; it leaves the operator unable to prove what happened or resume from a known boundary.

Additional acceptance criteria:

• Apply returns a durable run ID before the first removal.
• Each bounded page persists removed/skipped rows incrementally.
• Outer timeout or client disconnect does not erase completion evidence.
• Continuation resumes after the last persisted row rather than rescanning from zero.
• Per-repository progress is emitted before potentially slow GitHub/provider calls.

AI assistance: OpenAI GPT-5.6 Sol via OpenCode was used for live reproduction and evidence drafting.