Support training with streaming harnesses - #2369
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
087942f to
80250d6
Compare
ApprovabilityVerdict: Needs human review This PR introduces a new streaming capability for training harnesses, adding abstract methods to dialect base classes, new serialization/streaming methods across three dialect implementations, and substantial new server-side SSE handling logic. There is also an unresolved High severity finding regarding missing No code changes detected at You can customize Macroscope's approvability policy. Learn more. |
80250d6 to
2f63e73
Compare
| *, | ||
| include_start: bool = True, | ||
| sequence_number: int = 0, | ||
| ) -> list[bytes]: |
There was a problem hiding this comment.
Streaming tool calls lack index
High Severity
Chat stream_events copies the completed message into delta, so tool_calls entries have no streaming index. OpenAI-compatible clients require index on each delta.tool_calls item and will reject or drop those chunks. Previously this helper only rewrote plain-text replies; the new training relay now sends unrewritten tool-call completions through it, so streaming tool-using harnesses fail.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 2f63e73. Configure here.
2f63e73 to
e46f3e2
Compare
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| if response.message.reasoning_content is not None: | ||
| output.append( | ||
| { | ||
| "id": "rs_vf_intercept", |
There was a problem hiding this comment.
🟡 Medium dialects/responses.py:610
Serialized reasoning items are removed on the next turn instead of being replayed, so multi-turn Responses requests lose the model's reasoning history and receive a capability-removal notice. blocked_content_path rejects any reasoning item with an id but no encrypted_content, which is exactly the shape emitted here; omit the synthetic id or serialize encrypted replay state.
- "id": "rs_vf_intercept",
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/dialects/responses.py around line 610:
Serialized reasoning items are removed on the next turn instead of being replayed, so multi-turn Responses requests lose the model's reasoning history and receive a capability-removal notice. `blocked_content_path` rejects any `reasoning` item with an `id` but no `encrypted_content`, which is exactly the shape emitted here; omit the synthetic `id` or serialize encrypted replay state.
There was a problem hiding this comment.
🟡 Medium
verifiers/verifiers/v1/dialects/responses.py
Line 666 in e46f3e2
Buffered Responses streams announce resp_vf_intercept in response.created but complete with response.id when a request_id is present, so one SSE stream contains two response identities. Clients correlating events by response ID can split or reject the result; keep the synthetic ID consistent during serialization.
- "id": response.id or "resp_vf_intercept",
+ "id": "resp_vf_intercept",🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/dialects/responses.py around line 666:
Buffered Responses streams announce `resp_vf_intercept` in `response.created` but complete with `response.id` when a `request_id` is present, so one SSE stream contains two response identities. Clients correlating events by response ID can split or reject the result; keep the synthetic ID consistent during serialization.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e46f3e2. Configure here.
| "sequence_number": sequence_number, | ||
| "response": response, | ||
| } | ||
| return f"data: {json.dumps(event)}\n\n".encode() |
There was a problem hiding this comment.
Stream start IDs diverge
Medium Severity
stream_start and stream_heartbeat hardcode response IDs such as resp_vf_intercept and vf-intercept, while serialize_response prefers the generate request_id when present. After keepalives, terminal events can carry a different ID than the lifecycle events already sent, breaking clients that correlate the stream by response ID.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit e46f3e2. Configure here.
|
still a work in progress. |
1 participant
Summary
stream=trueWhy
Streaming harness calls enter the interception server's relay path. Eval clients can relay provider SSE, but the training client previously only implemented non-streaming
get_response(), so those harnesses failed before generation.Renderer generation remains intentionally buffered. The engine's completed result is still the source of truth for tokens and logprobs; Verifiers only adapts its delivery to the streaming protocols expected by harness SDKs.
Ordinary SSE comments are insufficient for clients that reset liveness only after decoding an API event. During generation, Verifiers therefore sends native no-content events: an empty Chat delta, Responses lifecycle events, or an empty Anthropic message delta. No fake model text or tool arguments are introduced.
Grok Build's newer content watchdog intentionally ignores lifecycle and empty-delta events. The companion Grok harness PR, #2366, sets that independent watchdog beyond Verifiers' maximum remote runtime instead of fabricating model tokens.
Scope
Validation
uv run pytest tests/— 914 passed, 75 skippeduv run pytest tests/v1 -m 'not e2e'— 70 passedtyhooks passedLive Prime requests reached the inference service, but the currently available model deployments did not provide a complete training smoke: attempts encountered a removed model, an unsupported generate route, missing generate logprobs, and a temporary 502. The deterministic protocol and full repository suites above are green.
The all-files markdown hook reports 321 pre-existing MD033 findings in the untouched
verifiers/legacy/envs/experimental/composable/tasksets/swe/README.md.The commit is signed with the contributor's verified SSH signing key.