feat(aio): run cluster labeling on the flex service tier

[bernatixer]

Problem

• The cluster labeling agents run gpt-5.4 on the standard tier. They are daily batch jobs, so nobody waits on their latency.
• OpenAI's flex service tier bills the same synchronous calls at half the token rate.
• Flex is refused more often than standard (capacity), and a labeling run is a ReAct agent making tens of sequential LLM calls: one refused call must not throw away the run or silently degrade labels.

Changes

• The clustering labeling agents (trace and evaluation clusters) now request the flex tier, halving their token bill. The agents themselves are untouched — only client construction changes.
• A recoverable flex failure retries only the failing call on the standard tier, and latches the client to standard for the rest of that agent run — a flex brownout costs one 120s timeout per run, not one per call. FlexFirstChatOpenAI (in llm_endpoint.py) overrides _generate/_agenerate: the openai SDK's own retry loop re-sends a byte-identical request, so it can only re-roll the refused tier; the override is the seam where the tier can change per call. The agent keeps its completed turns.

flowchart TD
    A[agent turn N] --> B{{flex call<br/>120s, no SDK retries}}
    B -->|success| N[turn N+1]
    B -->|429 / 408 / 409 / 5xx / connection loss| C{{same call, standard tier}}
    C -->|success| L[latched: remaining turns go straight to standard] --> N
    C -->|failure| D[exception → existing default-label handling]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    class B,C phBlue;
    class D phRed;
    class A,N,L phYellow;

Loading

• What counts as recoverable is shared, not re-derived: is_flex_recoverable (429, connection loss and timeouts, 5xx, bare 408/409) moved from the summarization fallback (feat(aio): switch batch trace summarization to gpt-5-nano on flex #91639) to posthog/llm/openai_flex.py, next to the flex model allowlist both callers already share. 409 is new there: turning off the SDK's retry loop had silently dropped its 409 retry for both callers.
• Flex is gated by that explicit allowlist, not the gpt-5 family prefix: OpenAI decides eligibility per model (gpt-5.4-pro is excluded while gpt-5.4 is included, per the flex pricing table).
• Per-call budgets close on every path: flex 120s with max_retries=0 (the tier switch is the retry), standard clients 240s × 2. The previous shape (600s × 3 per call) let one parked call outlive the labeling activity's whole 600s budget.
• LLM_SCHEDULE_TO_CLOSE_TIMEOUT rises 900s → 1260s: a first Temporal attempt that times out at 600s now leaves the retry a full 600s instead of ~295s. The evaluation child workflow gets the 45min execution timeout its activity budgets already needed, replacing a hardcoded 30min that predates this PR.
• A call that fails on both tiers falls to the existing default-label handling and self-heals with the next day's batch. Rolling flex back stays a one-line revert plus a deploy (fix(aio): remove the summarization model env lever #94013 removed the summarization env lever for the same reason).

Note

Cost ingestion prices the standard tier, so $ai_total_cost_usd dashboards show no change from this PR (#94200 fixes that at ingestion). The discount is visible only on the provider invoice. Same caveat, mechanism, and rollout check (service_tier passthrough on the deployed gateway) as #91639.

How did you test this code?

• Fallback matrix, one row per recoverable error shape (429, bare-408, bare-409, 5xx, timeout): each asserts the second call carries service_tier="default" — removing any member of the except tuple fails its row. A 400 propagates with no fallback; a standard-tier client never falls back; the async path falls back too.
• The allowlist rows guard granting flex to a pro model by family prefix; the budget pins guard reverting to 600s × 3 per call, which starves the activity.
• The labeling agents, their tests, and their READMEs are byte-identical to master; repo-wide mypy --cache-fine-grained . is clean.
• Timeout sizing checked against production telemetry: the slowest standard-tier labeling call of the past week ran 37s (p99 9.5s), and under 0.005% of three days of internal flex summarization traffic exceeded 120s — those were parked requests that also died at summarization's 180s cap, which is the case the fallback rescues.
• Not run: a live labeling run through the deployed gateway; a failed flex call falls back to standard, and a misconfigured one fails to default labels as today.

Automatic notifications

• Publish to changelog?

Docs update

None: the labeling agent READMEs describe behavior that is unchanged by this PR.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Claude Code session directed by the assignee, carrying the review lessons from feat(aio): switch batch trace summarization to gpt-5-nano on flex #91639: flex allowlist over family prefix, retry × timeout accounting against the enclosing activity budget, one test row per except-tuple member.
• The per-call standard fallback lands on review feedback (retries belong on the non-flex tier). A whole-agent standard rerun was built first and replaced: with tens of calls per run, a failure at call N discarded and rebilled N−1 successful flex calls.
• Skills invoked: /writing-pr-descriptions, /writing-tests, /writing-code-comments.
• History note for reviewers: earlier revisions carried a standard-tier rerun with state isolation and partial-label salvage, removed as over-engineering; the per-call fallback is the narrow replacement. The --max-cached-workflows worker flag also lived here for a while and moves to a dedicated worker-tuning PR.

[trunk-io]

😎 Merged successfully - details.

[github-actions]

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Playwright — 1 flaky

🎭 Playwright report · View test results →

⚠️ 1 flaky test:

• Duplicating a dashboard preserves text cards, date filter, and variables (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

✅ ClickHouse migration SQL — none

No ClickHouse migrations in the latest push.

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[Radu-Raicea]

Let's do the retries on the non-flex version of the model, because it might be unavailable more often on flex than on non-flex.

The labeling agent also makes multiple multiple LLM calls, so if they all get significantly slower, I wonder if 10 min is still a good timeout.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-09-07 09:47 UTC	Run
prod-us	✅ Deployed	2026-09-07 10:04 UTC	Run
prod-eu	✅ Deployed	2026-09-07 10:07 UTC	Run