perf(search): restore index use for bank-scoped ANN and unit-date lookups

[kkroo]

The incident

2026-09-03 08:30-10:35Z, omar vault seat-harvest container. The Anthropic fleet went dark:
80 successes against 547 client errors per 10 min. An operator fixed it by hand and parked
the loop (mode = off in the switch file).

During a fleet-wide five-hour cap wave (every measured bench alias at 5h=100 with weekly
headroom intact) seat_float.decide() found no measured target, so its fallback promoted
UNMEASURED rows (no_usage_evidence / carry_too_old) — and the default
unmeasured_donor_scope = UNMEASURED_ANY_EXHAUSTED let a merely 5h-capped seat pay for them.

What the harvest log records (swaps 371-379), assign leg tagged [seat_refused]-less:

• freed bot41@blockcast.net, bot42@blockcast.net, bot43@blockcast.net, then
  bot41/42/43@blockcast.network — the accounts that were CARRYING the fleet; their live
  rows later read 5h=32/80/12, 7d=3/8/1 — then bot10, bot22, bot36, and
  bot44@blockcast.net minutes after an operator seated it (no captured credential →
  no_proven_credential, an unprovisioned donor: that part is by design).
• seated ally@blockcast.net four times and bot10/bot12 repeatedly (seated → freed next
  round as exhausted → re-seated as unmeasured), bot6@blockcast.net and bot9@blockcast.net
  into team_standard seats although their carry said last_util_7d=100 (aged past
  CARRY_MAX_AGE_SECS, so _carry_util_7d returned None, the 100 was ignored →
  carry_too_old → unmeasured → promoted), and bot11-team@n whose last reading — four hours
  old — was 5h=100.

Net effect: every seat held a capped account while the accounts with headroom sat unseated; only
one seat (bot44@blockcast.network) served.

A second observation at 11:30Z: the control plane's utilisation-equalising selector burns a
Team seat's whole 5h window in ~45 min, so every seated alias caps together, every bench alias
reads spent, and the loop holds while seats could be pre-positioned.

The rules, all in the pure decide()

1. An aged carry is still evidence of EXHAUSTION even when it is too old to prove HEADROOM.
seat_float.py: _carry_exhaustion_refusal — last_util_7d >= EXHAUSTED_AT (any age) with
reset_7d_at absent or ahead → target_carry_exhausted; last_util_5h >= 100 (any age) with a
known future reset_5h_at → target_carry_five_hour_spent. Nothing clears last_util_*
when a member is unseated (the vault carries it outside the seat gate, core vectorize-io#1682), so a carried
100 with no reset behind it is a live cap. The one-way "an unseated account only decays" argument
licenses trusting an old low number, never ignoring an old high one. _carry_util_7d /
_carry_util_5h keep their headroom semantics untouched; a live reading of a window still
wins over its carry; and the two established refusals target_already_exhausted (usable carry at
the ceiling) and target_five_hour_spent (reset-proved carry under the floor) keep their names —
the new ones cover only what those two had to discard.

2. Under UNMEASURED_ANY_EXHAUSTED, an unmeasured target may displace a merely-5h-capped donor
only with a positive serving signal. credential_layer == "seat_refused" (the provider answered
on the token and refused only the seat — the one class where a seat becomes serving capacity
immediately) and it passed rule 1. Every other unmeasured row is priced as
UNMEASURED_WORTHLESS_DONOR — it may still fill a free seat or displace a _worthless_seat
donor, both of which cost nothing — and is reported unmeasured_needs_worthless_donor. Scope
constant names and the CLI flag are unchanged, and the cold-start/dark-fleet deadlock the arm
exists to break stays broken.

3. Anti-thrash. New decide(recently_released=...), default empty — so every existing caller
keeps today's behaviour exactly. A member the ledger freed inside the release grace may not be
promoted again as an unmeasured target (recently_released); a measured target with proven
headroom is unaffected, because that is real capacity. Supporting wiring:
SeatLedger.recent_releases / recent_release_times (new, symmetric with recent_assignments,
reading the FREED leg and skipping fills / external records that free nothing);
harvest_released_aliases expands the ledger's one alias to the whole member (the
sibling-alias bypass the donor-side grace was fixed for on 2026-08-28);
HARVEST_RELEASE_GRACE_SECS = HARVEST_GRACE_MAX_SECS (the ceiling, not the floor — a freed member
discloses no utilisation, so no evidence can lift this grace early); read in _harvest from the
same ledger each invocation, so the loop driver's back-to-back changes see each other's frees.

4. Every new refusal is in the operator summary with a distinct reason string, documented in
the vocabulary block at the top of seat_float.py: target_carry_exhausted,
target_carry_five_hour_spent, unmeasured_needs_worthless_donor, recently_released,
target_reset_imminent, reset_imminent_donor_resets_sooner. _harvest also prints
withheld, freed recently : N (...) and the pre-position window, so a shrunk candidate pool is
never silent.

5. Pre-positioning onto an imminent recovery (the 11:30Z observation). A bench row with fresh
telemetry, a proven credential, weekly headroom, live util_5h >= 100 and a reset_5h_at inside
RESET_IMMINENT_TARGET_SECS (default 40 min — the upper bound of observed seat propagation;
env PENSTOCK_HARVEST_RESET_IMMINENT_TARGET_S, 0 disables the lane) is an eligible 5h-pass
target (target_reset_imminent) instead of being refused. It ranks behind every
measured-headroom target and behind stale_rejected, is ordered by soonest reset (the weekly
sort is skipped for that pool), and may only displace a donor recovering more than one window
later than itself — never one recovering sooner, never one that still has 5h headroom, never one
on an unknown clock (reset_imminent_donor_resets_sooner). A _worthless_seat donor is exempt
from the comparison; a vacancy needs no donor at all.

One vocabulary note for rule 5's (b) case: a live util_5h = 100 bench row is refused by
_bench_exhausted as target_already_exhausted — that is the gate this lane hooks, and it
predates this PR. target_five_hour_spent names the rows below the ceiling but under the floor
(99, or a reset-proved carry). Both are asserted, so the distinction cannot drift.

Connection shapes

retrieve_semantic_bm25_combined / retrieve_temporal_combined are called with either a DatabaseConnection wrapper or a raw asyncpg connection (acquire_with_retry accepts a DatabaseBackend or a raw asyncpg.Pool, and the recall path still passes the latter). The helper therefore:

• reads getattr(conn, "backend_type", "postgresql") rather than the attribute directly — a raw asyncpg connection has no backend_type and is Postgres by definition;
• ignores the value yielded by transaction() — the wrapper yields the connection, but raw asyncpg yields a Transaction, which has no query methods — and runs its statements on conn.

Both shapes are covered by tests; an earlier revision of this branch got both wrong and tests/test_hnsw_indexes.py::test_retrieve_semantic_bm25_grouped_by_fact_type caught it.

Tests

Added to scripts/seat-float.test.py (a dedicated THE 2026-09-03 INCIDENT section replaying all
four shapes with the incident's own numbers, plus a RULE 5 section), scripts/seat-ledger.test.py
(recent_releases on the FREED leg, org scoping, ageing, fills release nothing) and
scripts/claude-org.test.py (alias→member expansion, the seat_refused spelling agreeing across
both modules, the env knob's fail-safe fallbacks, and two end-to-end cmd_harvest runs whose only
difference is a FREED leg in the ledger / a capped candidate minutes from its reset).

Also required: a seat_refused target with clean carries does displace a 5h-capped donor, and
a measured target with proven headroom is unaffected by the release window.

Four existing fixtures that encoded the incident-causing behaviour were updated in place, each
with the reasoning written next to it: the "unmeasured target IS promoted against a donor whose
week is open" pair, the live 2026-08-25 fleet (now holds, and swaps once one seat_refused row is
on the bench), and the two donor-ranking fixtures, which are given the serving signal so the round
still reaches the donor sort they are about.

Mutation-checked: each new guard was reverted in turn (rule 1's two clauses, rule 2's signal
test, rule 3's release test, rule 5's eligibility / donor comparison / pool ordering, the ledger's
FREED leg, the member expansion) — every one failed its own test, and all suites were restored to
green afterwards.

Verification: the exact relogin job list from .github/workflows/ci.yml (40 invocations) —
exit 0; re-run under python 3.10 (the pod's interpreter) — exit 0;
python3 scripts/ci-change-filter.test.py — 58 invocations across 8 gated jobs, all reachable;
gitleaks detect --no-git --redact --source . — 1 finding, the pre-existing
oauthHcaptchaSitekey: line in charts/penstock-vault-node/environments/omar.values.yaml.

🤖 Generated with Claude Code