feat(runs): retry transient provider errors with backoff

[yzxcj797]

Fixes the pi-retry and cron-adjacent backoff sections of #602 (sections 1 and the retry-cadence half of 2). Failover (3) and /healthz (4) are larger design decisions, deliberately not included.

Section 1: every pi provider error was terminal

piAssistantError fires on any assistant message with stopReason === "error", and both piLastAssistantTextOrThrow and piTurnError wrapped the formatted message in NonRetryableTurnError — so rate_limit_error and authentication_error took the same path, and the runs worker (retry: !(err instanceof NonRetryableTurnError)) parked the run after one attempt. Your deployment table says it exactly: every pi failure got 1 attempt; the only run that ever reached 3 was the claude-harness one, whose harness already treats everything but wall-clock timeout as retryable.

Fix: classification by provider error type. rate_limit_error, overloaded_error, timeout_error, api_error, server_error (plus the less-common siblings: internal_server_error, service_unavailable, temporarily_unavailable, model_service_overloaded) become plain Errors the worker retries; authentication_error / permission_error / invalid_request_error and non-JSON errors stay NonRetryableTurnError — a second attempt would fail identically. The formatted message is unchanged in both classes (Model provider API error (rate_limit_error): … still greps the same); only the class carries the decision.

The retry cadence: retries were immediate

Requeue sets status='pending' and the worker polls at 50ms — so even with retries, maxAttempts=3 hits a rate-limited provider back-to-back, "feeding the thing that caused it" (your section 2 observation, applied to the run path).

Fix: an ERROR-driven requeue now carries a not-before — retryBackoffMs(errorAttempts) = 5s × 2ⁿ capped at 60s — stored in the pending row's lease_expires_at and honored by both claim paths (claim and claimById, memory + Postgres; nothing else sets lease_expires_at on pending rows today, and the reaper only touches running rows, so the overload is backwards-compatible). Lease-expiry requeues — your suspected-crash path — keep their immediate retry and their own poison-pill budget, unchanged.

Tests (pi-retryable-errors.test.ts)

• rate_limit_error → not NonRetryableTurnError (the exact failure from your runs table), message still matches /rate_limit_error/.
• overload / api_error / server_error / timeout → retryable; authentication / permission / invalid_request → non-retryable; plain-text errors keep the historical class.
• retryBackoffMs: 5s → 10s → 20s … capped 60s.
• Memory-store behavior: an error-requeued run is not claimable inside the window (both claim and claimById); a fresh run claims immediately.

Existing suites updated for the intentional cadence change: run-store.test.ts / worker-reaper.test.ts pass retryBackoffMs: () => 0 where they model immediate claim-after-fail (the knob the memory factory now exposes); the lease-expiry reaper tests needed nothing. Full set: 87/87, tsc --noEmit clean.

What's deliberately not here

• Failover (section 3): resolveRuntimeChoice is a pure config read; making it provider-health-aware needs a health signal and a policy ("use B when A fails" needs somewhere to express B). Your note that the refusal-fallback machinery is proven and merely unwired is the right observation — wiring it to availability failures is a design PR of its own.
• Cron tick backoff (section 2's batch shape): one failing cron aborting its tick's batch deserves its own fix (isolation + per-cron backoff), separate from the run-level backoff landed here.
• /healthz (section 4): surfacing model reachability in a health endpoint is an ops contract decision (what does ok mean to your uptime monitor).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}