Stop an outage from spending a job's last attempt

[ralyodio]

The incident

Between 2026-08-26 and 08-28 every model provider ran out of budget at once — OpenAI credits at zero, Anthropic on the org spend cap (reported as a 400, not a 429). Site extraction is a model call, so every crawl failed:

Day	done	failed
Aug 25	261	0
Aug 26	8	1,985
Aug 27	6	1,745
Aug 28	5	1,123

Each failure was charged to the job as if the payload were at fault. 4,888 jobs burned through max_attempts and became failed, which nothing ever brings back.

When the chain recovered, the queue did not — because the outage had also consumed the work that regenerates work. Every refresh_research card had been approved, so workspacesWithInternalBacklog found nothing and queued no crawls. The pipeline sat at a fixed point it could not leave, and the daily digest reported zero crawled, zero found, zero sent — all of it true.

The fix

failJob takes an optional check for "this failure was not this job's fault". When it matches, the job returns to pending with its attempt refunded and a flat ten-minute hold, and can never reach failed.

Refunding is safe here in a way it deliberately is not in reclaimStalled: there the payload is the suspect, and the evidence is that the job killed its container. Here every other job in the queue is failing identically, which is exactly what makes the payload innocent.

Two supporting changes:

• drainQueue stops the tick on the first deferral rather than asking a provider that has already said no another 24 times.
• The server wires in isBudgetExhausted and says plainly that it is holding. The previous version of this outage was silent apart from a 400 in jobs.last_error that read like a bug in the crawler.

The predicate is injected rather than imported, so queue.ts keeps knowing nothing about what a job actually does.

Tests

Three new cases in queue.test.ts, all verified to fail without the fix:

• an outage does not spend the last attempt → without the fix the outcome is "dead" (the exact production behaviour)
• a genuine fault still dies, outage check or not → guards against the refund being too broad
• one outage holds the rest of the tick → without the break the handler is called 4 times instead of 1

bun test: 1,445 pass, 0 fail across 96 files. bun run typecheck clean, prettier clean.

Note on the live backlog

The 828 distinct domains killed by this incident were already requeued by hand in prod on 2026-08-29 (ids in ~/backups/outreachgraph-requeued-jobids-2026-08-29.txt), and the pipeline is generating its own work again. This PR is the fix that stops it recurring — no data migration needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_017o2oNWB7bwNxNsaRB7oTTY