Fix deadlocked ProjectData archive sweep: derive the selection ceiling from the write budget

[simple-agent-manager]

Summary

The production ProjectData archive sweep reclaimed nothing for four days while reporting succeeded on all 226 hourly runs, and the root Durable Object climbed from 94% to 96.7% of its 10 GB ceiling (~5 days of headroom).

Two independently configured ceilings had to agree, and drifted:

• selectCandidates filtered ss.message_count <= PROJECT_DATA_ARCHIVE_SWEEP_MESSAGE_BUDGET and ordered message_count DESC ... LIMIT sweepProjects * sweepSessions (deployed 1 x 1).
• reserveArchiveWrites refuses any estimate above PROJECT_DATA_ARCHIVE_DAILY_WRITE_BUDGET — an affordability ceiling of floor((100000 - 1000) / 32) = 3093 write units, versus a selection ceiling of 5000 messages.

On 2026-09-08 a GitHub production Environment override lowered the write budget to 100000 to bound cost. Nobody lowered the selection ceiling with it. From then on every tick picked the same 4994-message session (5f4ac04c-153f-4add-b96b-629bffaf0b2c, ended 2026-04-23), estimated ~160,808 writes, was refused before reserveArchiveWrites touched D1 — so the UTC budget window also froze at 2026-09-08T00:00:00Z — and continued out of a one-element list. No journal row, no location change, no error, no skip reason.

What changed

1. The selection ceiling is now derived from the allowance (archiveAffordableWriteUnits / archiveAffordableMessageCeiling in apps/api/src/project-data-archive/write-budget.ts), used as the message_count <= ? bind. Two hand-set numbers that must agree will drift again; one derivation cannot. An explicit session-scoped operator canary still bypasses it — reserveArchiveWrites retains the final say on cost.
2. A refusal descends instead of ending the tick. selectCandidates over-reads PROJECT_DATA_ARCHIVE_SWEEP_FALLTHROUGH_DEPTH (4) spare candidates, and both per-tick bounds — session slots and the cumulative message budget — moved into the journaling loop so a refused candidate, which opens no migrating fence and moves no rows, consumes neither.
3. The two refusals are now distinguishable (.claude/rules/72): exceeds_allowance (waiting cannot help) vs window_exhausted (normal end-of-day backpressure, which happens on ~23 of 24 ticks once the pool is spent).
4. The failure is visible. PROJECT_DATA_ARCHIVE_BUDGET_STALL_ALERT_SWEEPS (3) consecutive ticks that migrate nothing and see only exceeds_allowance flip the cadence row to partial with an actionable last_error. Migration 0156 adds consecutive_budget_stalls; the increment, threshold comparison and status escalation all happen inside one atomic UPDATE ... RETURNING.
5. wrangler.toml: DAILY_WRITE_BUDGET 250000 → 100000 (matching the deployed production override, so staging and self-hosts derive the same ceiling instead of only production carrying it), SWEEP_MESSAGE_BUDGET 5000 → 2000, SWEEP_SESSIONS 1 → 2.

IMPORTANT — this fix does not reverse the storage curve, and is not meant to

Within the 100000/day allowance the sweep can spend at most floor((100000 - 1000) / 32) = 3093 write units/day ≈ ~1500–2000 messages/day. Measured reclaim from the 2026-09-05→09-08 drain was ~500–600 bytes/message (849 MB recovered across ~1.77 M messages), so the restored sweep reclaims on the order of 1–1.5 MB/day against ~66 MB/day of growth.

The deadlock is real and fixing it is necessary regardless. But this PR cannot stop the object filling. Raising PROJECT_DATA_ARCHIVE_DAILY_WRITE_BUDGET is deliberately out of scope — Raphaël rejected the projected ~$100/mo DO SQLite bill on 2026-09-08 — and no additional reclaimer is enabled (EVENT_LOG / GROUPED_FTS / TOOL_PAYLOAD cleanup all remain false, verified unchanged in the diff). That is a spend decision and it needs one. Options, cheapest first: lower PROJECT_DATA_ARCHIVE_WRITE_ESTIMATE_FACTOR (32 is a conservative scheduling estimate, not a measurement — if real amplification is lower, the budget is under-spending real capacity); raise the daily budget; or enable one of the other reclaimers.

Control-loop load review (.claude/rules/47)

Candidate volume per tick: min(remaining, sweepProjects * sweepSessions) + sweepFallthroughDepth = 6 rows (was 1). Worst case — every candidate refused, which is the steady state during the incident this fixes — is roughly 33 D1 + DO round-trips: ~11 fixed (cadence claim/finish, crash-gap, two selects), ≤6 estimate DO RPCs, ~1–2 ensureProjectId RPCs (now memoised per tick rather than per candidate), ≤6 D1 reservation upserts, ≤6 D1 journal writes for the ≤2 fenced sessions. The common case (first candidate affordable) is ~15. This runs hourly, not per-request. Pre-PR worst case was ~15; the first draft measured ~50 and was brought down by memoising owner stubs, lowering the fall-through depth 8 → 4, and lowering SWEEP_SESSIONS 4 → 2.

selectCandidates has no covering index for (status, ended_at, message_count DESC). Measured rather than assumed: production session_summaries holds 5,526 rows (5,289 terminal), so an hourly scan-and-sort is negligible and an index would tax every session-summary write. Not added.

Validation

• pnpm lint — clean (apps/api, and pnpm check:fast 13/13 tasks)
• pnpm typecheck — clean
• pnpm test — see below
• Additional validation: pnpm quality:migration-safety (183 FK relationships, 0 violations), pnpm quality:do-migration-safety, pnpm quality:wrangler-bindings
• Candidate-selection change states expected volume and worst-case per-candidate cost (above)

Discrimination proof (.claude/rules/62)

Seven fixes, reverted one at a time; each reddened exactly one test, then restored:

Revert	Test that went red
derived ceiling → sweepMessageBudget	migrates an affordable session when the largest eligible one exceeds the whole allowance
fall-through over-fetch removed	falls through to a smaller candidate when the top candidate passes the ceiling but the budget refuses it
refusal categories collapsed to budgetDeferred	does not count an exhausted daily pool as a stall
stall status escalation removed	stops reporting succeeded after consecutive sweeps that can afford nothing, and resets once one lands
pendingSlots not passed	journals at most sweepSessions candidates even when the over-fetch returns more affordable ones
message budget committed on admission	does not charge the cumulative message budget for a candidate the write budget refused
scoped path uses the derived ceiling	lets an operator-named session bypass the derived ceiling, but not a project-wide canary

All tests enter through the real runProjectDataArchiveSharding against real DO SQLite and real D1, with a mixed candidate set. Fixtures measure the DO's own cost model to size the allowance; they never tell the sweep which candidate to take. A fixture seeded only with affordable candidates would have passed throughout the outage.

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — runs 34684491860 and 34687207699, both success
• Live app verified via Playwright — authenticated in-browser against https://api.sammy.party/api/auth/token-login (200), then navigated https://app.sammy.party
• Existing workflows confirmed working — dashboard, projects and settings all render real data; 0 responses >= 500, 0 console errors
• New feature/fix verified on staging — see evidence below
• Infrastructure verification completed — N/A: no infra changes. This PR touches no packages/cloud-init/, packages/vm-agent/, DNS, TLS or VM provisioning path; it provisioned no VMs and consumed no Hetzner capacity.
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes

Staging Verification Evidence

Staging could not exercise this on its shipped config: it runs PROJECT_DATA_ARCHIVE_COMPACT_ENABLED=false, which makes the whole write-budget path inert. Four temporary staging Environment variables were set to make the path reachable and to force the affordability boundary onto staging's live candidate set. None of them overrides a value this PR changes — SWEEP_MESSAGE_BUDGET and DAILY_WRITE_BUDGET were deliberately left un-overridden so they deploy from wrangler.toml and stay verifiable per .claude/rules/70. All four are removed after verification:

PROJECT_DATA_ARCHIVE_COMPACT_ENABLED=true
PROJECT_DATA_ARCHIVE_WRITE_ESTIMATE_FACTOR=900      # forces the boundary onto live data
PROJECT_DATA_ARCHIVE_GLOBAL_SWEEP_INTERVAL_MS=300000
PROJECT_DATA_ARCHIVE_SESSION_GRACE_MS=86400000      # staging had 0 eligible sessions at 7d

Deployed values, read from the Cloudflare script-settings API for sam-api-staging (not the diff):

PROJECT_DATA_ARCHIVE_DAILY_WRITE_BUDGET          = 100000   <- this PR, no override
PROJECT_DATA_ARCHIVE_SWEEP_MESSAGE_BUDGET        = 2000     <- this PR, no override
PROJECT_DATA_ARCHIVE_SWEEP_SESSIONS              = 2        <- this PR, no override
PROJECT_DATA_ARCHIVE_SWEEP_UNIT_OVERHEAD_PERCENT = 100      <- new var, shipped default
PROJECT_DATA_ARCHIVE_SWEEP_FALLTHROUGH_DEPTH     = 4        <- new var, shipped default
PROJECT_DATA_ARCHIVE_BUDGET_STALL_ALERT_SWEEPS   = 3        <- new var, shipped default

The three new vars arriving at their defaults proves the sync-wrangler-config.ts + deploy-reusable.yml wiring end to end. The deploy log's Environment override: lines listed exactly the temporary vars and nothing else.

Migration 0156 applied to real staging D1 — consecutive_budget_stalls reads 0 on the live cadence row, which carries run_count in the 240s and a lease history.

The first sweep after deploy migrated nothing — investigated before assuming, and it was correct. Staging had zero eligible candidates: all 22 of its in-range terminal sessions are frozen precopy_refused inside the 7-day retry window. Replaying the real selector predicate including PRECOPY_REFUSAL_EXCLUSION_SQL returned 0 rows at the derived ceiling of 55 and 0 rows at 2000 — the pre-fix code would have selected nothing either. The new stall detector correctly did not fire: an empty candidate set is not a budget stall, which is precisely the distinction .claude/rules/72 asks for. Lowering SESSION_GRACE_MS to 1 day produced 8 real candidates.

Real reclamation, tick run_count=248 @ 2026-09-12 10:10:54Z:

• Selected the two largest affordable candidates under the derived ceiling: a 31-message and a 15-message session
• Both state=published, attempt_count=1, error_code=null
• Both project_data_session_locations.location_state moved root -> archive_shard — the data actually moved
• Exactly 2 sessions journaled against SWEEP_SESSIONS=2, while the over-fetch read 6 candidates. This is live confirmation of the pendingSlots fix; without it the tick would have journaled all 6.

project_data_archive_write_budget came unstuck — the production symptom, resolved:

window_started_at   2026-09-08 00:00:00  ->  2026-09-12 00:00:00
reserved_writes     8832                 ->  87700  = 4*1000 + 900*93 units over 4 attempts

Next tick run_count=249 @ 10:20:53Z: two 2-message sessions refused by the pre-existing pre-copy eligibility invariant (frozen/precopy_refused) and returned to root in the same tick. Not a budget refusal, no failure, no poison — the existing unwind path still behaves.

Cadence throughout: last_status=succeeded, consecutive_budget_stalls=0, last_error=null.

Post-Mortem

What broke

The production ProjectData archive sweep stopped reclaiming storage on 2026-09-08 and reported success for four days while the root Durable Object filled toward its 10 GB ceiling.

Root cause

A GitHub production Environment override lowered PROJECT_DATA_ARCHIVE_DAILY_WRITE_BUDGET to 100000 on 2026-09-08T14:05Z (bounding DO SQLite cost) without lowering PROJECT_DATA_ARCHIVE_SWEEP_MESSAGE_BUDGET, which ships from wrangler.toml at 5000 and has no override. Largest-first selection with LIMIT 1 then deterministically picked a session whose estimate exceeded the entire daily allowance, every hour, forever.

Class of bug

Two independently configured ceilings that must agree, with no derivation linking them — combined with a refusal path that neither descends nor reports. The refusal was structurally permanent (exceeds_allowance) but was counted in the same bucket as a transient one (window_exhausted), so the only signal was an absence.

Why it wasn't caught

reserveArchiveWrites returned a bare boolean, so no caller could distinguish "never affordable" from "pool spent". The sweep's success status was computed from failed/poisoned counts, which are both zero in a stall. And the existing tests seeded only affordable candidates — they would have passed unchanged throughout the outage.

Process fix included in this PR

• .claude/skills/env-reference/SKILL.md corrected (it stated the pre-fix deployed values and is what CLAUDE.md routes every env question to) and the three new vars documented.
• Root .env.example corrected — it carried live, uncommented DAILY_WRITE_BUDGET=250000 / SWEEP_SESSIONS=1 values that the quickstarts tell users to cp.
• A /changelog entry so the two superseded historical entries are no longer the newest word on these values.
• scripts/quality/deploy-reusable-workflow.test.ts now machine-checks that the new tunables reach both deploy sync blocks and the top-level [vars] — without the latter, listEnvironmentVarOverrides cannot log an override, which is the .claude/rules/70 failure mode verbatim.
• tasks/backlog/2026-09-12-split-project-data-archive-sharding-module.md for the rule-18 file-size violation (3,663 lines, pre-existing; splitting during an urgent production fix would have made the diff unreviewable).

Post-mortem file

tasks/active/2026-09-12-deadlocked-projectdata-archive-sweep-budget-mismatch.md

Specialist Review Evidence

• All local reviewers completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — N/A, all eight completed

Reviewer	Status	Outcome
constitution-validator	PASS	No CRITICAL/HIGH. 2 LOW fixed: import ARCHIVE_WRITE_FIXED_RESERVATION in tests instead of a bare 1000; document the estimateCap floor
task-completion-validator	ADDRESSED	WARN on unchecked task-file boxes and pending live evidence. Checklist/criteria checked off; throughput gap reported above
env-validator	ADDRESSED	1 CRITICAL (env-reference skill stated pre-fix values, omitted the 3 new vars) + 1 MEDIUM (root .env.example live stale values) — both fixed
cloudflare-specialist	ADDRESSED	Stall counter no longer read-then-write (a failed read silently restarted a streak); round-trips reduced; index finding measured and declined with row counts
doc-sync-validator	ADDRESSED	/changelog entry added; configuration.md double-floor formula and compact-only scoping corrected; env-reference fixed
architecture-reviewer	ADDRESSED	Found pendingSlots computed but never passed; fixed and made REQUIRED (.claude/rules/73). Spread inlined. File split deferred to a backlog task
performance-reviewer	ADDRESSED	Worst-case round-trips cut from ~50 to ~33 (stub memoisation, depth 8→4, sessions 4→2); estimate scan cap bounded so it cannot grow with the allowance
test-engineer	ADDRESSED	Independently reproduced the pendingSlots bug live (6 migrated against sweepSessions=2). Its two proven coverage gaps now have discriminating tests

The pendingSlots bug is worth calling out: it was introduced by the first cut of this change, found independently by two reviewers and by tracing consumers, and was invisible to all four original tests because none had more affordable candidates than slots. The parameter is now required rather than optional-with-a-silent-fallback.

CodeRabbit Review Evidence

• coderabbit-review label applied after local review, staging verification and CI gates passed — applied 2026-09-12 ~10:25Z, which dispatched coderabbit-bot-review.yml (runs 34688337501, 34688387290, both success) and posted the review command through the repository's human-scoped PAT.
• All CodeRabbit findings implemented or explicitly reviewed and closed/resolved — no findings were produced
• Incremental CodeRabbit review completed after final pushed fixes — blocked by rate limiting
• Latest CodeRabbit review has no unresolved feedback — cannot be established

CodeRabbit Notes

CodeRabbit is rate limited and has produced zero reviews and zero inline comments on this PR.

• 2026-09-12T09:16:17Z — CodeRabbit commented: "Review limit reached. Next included review available in 58 minutes... You've used all free OSS reviews for now." It enumerated all 21 changed files and the commit range, so it saw the PR, but emitted no findings.
• 2026-09-12T10:25:06Z — the coderabbit-review label dispatched the bot workflow, which posted @coderabbitai review via the human PAT.
• 2026-09-12T10:25:12Z — CodeRabbit replied: "Action not completed. Review rate limited."

State at the time of writing: gh pr view 2069 --json reviews returns 0; pulls/2069/comments returns 0. The PR itself is MERGEABLE / CLEAN with all 18 CI checks green and the SonarCloud Quality Gate passed (1 new issue, 0 security hotspots).

Per .claude/rules/25-review-merge-gate.md this is a hold, not a pass: CodeRabbit is unavailable, so its unresolved-feedback state cannot be inspected and the agent must not self-merge. The coderabbit-review label is deliberately left on so CodeRabbit performs an incremental review when its window resets, and the command has not been re-posted (the standing project rule is to trigger once and monitor rather than retry into a rate limit).

Context for the human decision: eight local specialist reviewers completed and their findings are addressed (table above), three of which found real defects. Raphaël has previously waived CodeRabbit for PRs #1984, #2030 and #2035 when it was unavailable or over its file limit. Production is at 96.84% of its Durable Object storage ceiling with roughly four days of headroom.

Exceptions

• Scope: apps/api/src/scheduled/project-data-archive-sharding.ts remains 3,663 lines, far over the .claude/rules/18 mandatory 800-line split threshold.
• Rationale: the violation is pre-existing (~3,372 lines before this change). Splitting it into seven modules during an urgent production fix, with the object at 96.7% of its ceiling, would have made a data-integrity-relevant diff unreviewable. The rule's own guidance is to split in a separate commit, which is what the backlog task specifies.
• Expiration: tasks/backlog/2026-09-12-split-project-data-archive-sharding-module.md, to be done before the next substantial change to the archive sweep.

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

N/A: no external API involved. All evidence came from first-party production state — the Cloudflare Worker script-settings API (deployed [vars] for sam-api-prod), the GitHub Environment variables API (override listing for production and staging), and D1 sam-prod (project_data_archive_write_budget, project_data_archive_migrations, project_data_archive_global_sweep_cadence, project_data_storage_telemetry_history, session_summaries).

Codebase Impact Analysis

• apps/api/src/scheduled/project-data-archive-sharding.ts — candidate selection, the journaling loop, cadence finish
• apps/api/src/project-data-archive/write-budget.ts — affordability derivation, discriminated reservation outcome
• apps/api/src/project-data-archive/contract.ts, apps/api/src/env.ts, apps/api/src/db/schema.ts
• apps/api/src/db/migrations/0156_archive_sweep_consecutive_budget_stalls.sql
• apps/api/wrangler.toml, scripts/deploy/sync-wrangler-config.ts, .github/workflows/deploy-reusable.yml (both sync blocks)
• Docs: apps/api/.env.example, root .env.example, apps/www/src/content/docs/docs/reference/configuration.md, .claude/skills/env-reference/SKILL.md, .claude/skills/changelog/SKILL.md

Consumers of the changed reserveArchiveWrites signature were enumerated: one production caller (processArchiveMigrationBatch.reserve) and the direct assertions in tests/workers/project-data-compact-archive.test.ts, all updated. ArchiveWriteReservation.maxMessages is consumed by the DO-side re-check in durable-objects/project-data/archive-sharding.ts:1101 — verified the coordinator and the DO still pass the same cap, so the re-check semantics are unchanged.

Documentation & Specs

apps/www/src/content/docs/docs/reference/configuration.md (table rows for the three new vars, the changed shipped values, and prose covering the derivation, the fall-through, the refusal split and the stall status). No specs/ file describes the archive sweep cadence or these vars.

Constitution & Risk Check

Principle XI (No Hardcoded Values): every new tunable ships as a DEFAULT_* + MAX_* pair consumed through envInt, declared in Env, shipped in [vars], forwarded through both deploy sync blocks, and documented in four places — verified by constitution-validator (PASS) and a new machine-checked test.

Principal risk is the migration. 0156 is ALTER TABLE ADD COLUMN with a constant default on a single-row table — metadata-only in SQLite, no rewrite, no backfill needed, and .claude/rules/31's DROP-TABLE prohibition is not approached. pnpm quality:migration-safety and quality:do-migration-safety both pass. The drizzle definition in schema.ts was updated to match; the drift between the two is what failed 20 unit tests and was caught by the full suite rather than the targeted runs.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: e721aff3-cdb2-4c5d-a708-7edfe3d7e742

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.4% Duplication on New Code

See analysis details on SonarQube Cloud

[raphaeltm]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[simple-agent-manager]

Merge held at the CodeRabbit gate

Everything else is green, but I am not self-merging this.

State: MERGEABLE / CLEAN, all 18 CI checks pass, SonarCloud Quality Gate passed, staging deployed and the fix verified end-to-end on live data. Eight local specialist reviewers completed; three found real defects and all are fixed.

Blocker: CodeRabbit is rate limited on the free OSS tier and has produced zero reviews and zero inline comments.

• 09:16:17Z — "Review limit reached. Next included review available in 58 minutes... You've used all free OSS reviews for now." It enumerated all 21 changed files and the commit range, so it saw the PR, but emitted no findings.
• 10:25:12Z — after the coderabbit-review label dispatched the bot workflow: "Action not completed. Review rate limited."
• Polled until 11:27Z: still reviews=0, inline comments=0. The window did not reset.

I have not re-posted the review command — the standing project rule is to trigger once and monitor rather than retry into a rate limit. The coderabbit-review label stays on so an incremental review runs if the window clears.

Per .claude/rules/25-review-merge-gate.md, CodeRabbit being unavailable means its unresolved-feedback state cannot be inspected, so this is a hold rather than a pass. needs-human-review applied.

@raphaeltm — you have waived CodeRabbit before for #1984, #2030 and #2035 when it was unavailable. If you are comfortable on the strength of the local reviewers, say so and I will merge and drive the production deploy plus the reclamation proof.

Separately and more importantly: this fix ends the deadlock but cannot reverse the storage curve. Production is at 96.86% and still climbing (9,686,147,072 bytes as of 11:14Z). At the deployed budget the restored sweep reclaims ~0.89 MB/day against ~73.9 MB/day of growth — 83x short, leaving roughly 4 days of headroom. Break-even needs ~8.2M writes/day; even assuming the write-estimate factor is 8x too conservative it still needs 10x the current budget. That is a spend/architecture decision I have deliberately not made — details in the open human-input request.