Skip to content

feat(v1): add trainable Prime Agent harness - #2355

Open
hallerite wants to merge 10 commits into
codex/training-streaming-harnessesfrom
codex/prime-agent-harness
Open

feat(v1): add trainable Prime Agent harness#2355
hallerite wants to merge 10 commits into
codex/training-streaming-harnessesfrom
codex/prime-agent-harness

Conversation

@hallerite

@hallerite hallerite commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

  • add a thin Prime Agent harness on top of the shared ACP harness
  • install a pinned Prime Agent release and route its OpenAI-compatible model calls through Verifiers interception
  • support ACP MCP servers, skills, resume, and appended system prompts
  • isolate per-trace Prime Agent state and daemon sockets while retaining reusable runtime caches
  • run Prime Agent evaluations in autonomous mode by default, with an explicit opt-out
  • optionally require Prime Agent's correlated ACP lifecycle contract and record terminal quiescence separately from benchmark rewards
  • isolate the main-agent answer from subagent, compaction, and refinement updates

Why

Prime Agent's native model API uses streaming Chat Completions. The harness itself can stay small because the generic renderer-backed streaming bridge now lives in the parent PR, #2369. This PR contains only the Prime Agent integration.

With Prime Agent #1378, ACP-provided MCP tools are translated into session-scoped Python programs by Prime Agent itself. Verifiers supplies the environment's MCP URLs while Prime Agent owns its native tool representation.

Stack and dependency

Validation

  • focused Ruff and formatting checks passed
  • uv run --frozen ty check verifiers passed
  • focused ACP lifecycle/scoring/config/import suite — 42 passed
  • uv run pytest tests/v1 -m 'not e2e' — 70 passed
  • isolated Docker MCP evaluation — reward 1.0
  • real Prime VM + Prime Inference MCP evaluation — reward 1.0; generated MCP Python program executed successfully, no trace errors, sandbox teardown verified

The lifecycle commits were added append-only. Local hooks that invoke uv --locked were skipped because the existing branch lockfile is stale; the equivalent direct frozen Ruff, formatting, type, and focused test commands above passed.

Note

Add trainable Prime Agent harness with ACP lifecycle enforcement

  • Adds PrimeAgentHarness and PrimeAgentHarnessConfig in harness.py, which installs Prime Agent, prepares per-trace state directories, emits models.json, and launches it in ACP mode with configurable autonomous/skills flags
  • Overhauls the ACP runner in runner.py: VerifiersACPClient now tracks lifecycle metadata per turn, filters visible replies to current-turn answer chunks, and exposes stop_reason, response_boundary, and terminal_quiescence; prompt returns a structured dict instead of a plain string and enforces correlated lifecycle semantics when configured
  • Adds lifecycle_meta_namespace to ACPConfig and _record_lifecycle_status in acp/init.py so lifecycle outcomes are recorded on Trace.info without affecting rewards; successful autonomous completions populate acp_answer_fallback
  • Updates read_answer_file_or_last_reply in score.py to prefer the ACP-recorded fallback reply over last_reply
  • Behavioral Change: prompt in runner.py now returns a dict (was a string); ACPSession.run and serve_stream propagate this structured type. In-tree callers are updated; out-of-tree consumers expecting a string return will break

Changes since #2355 opened

  • Added autonomous mode budget parameters to Prime Agent CLI arguments [e494cd9]
  • Implemented incomplete lifecycle status recording for cancelled and failed turns [e494cd9]
  • Implemented selective process stopping based on turn lock acquisition state [e494cd9]
  • Extended lifecycle status recording with terminal quiescence and phase tracking fields [e494cd9]

Macroscope summarized 8a869eb.


Note

Cursor Bugbot is generating a summary for commit 8a869eb. Configure here.

Comment thread verifiers/v1/harnesses/prime_agent/harness.py Outdated
@hallerite
hallerite force-pushed the codex/prime-agent-harness branch from 6f9ecc5 to f053b5b Compare August 15, 2026 01:47
@hallerite
hallerite changed the base branch from main to codex/training-streaming-harnesses August 15, 2026 01:47
Comment thread verifiers/v1/harnesses/prime_agent/harness.py Outdated
@hallerite
hallerite force-pushed the codex/training-streaming-harnesses branch from 2f63e73 to e46f3e2 Compare August 15, 2026 10:39
@hallerite
hallerite force-pushed the codex/prime-agent-harness branch from f053b5b to 2eed65b Compare August 15, 2026 10:39
@sethkarten

Copy link
Copy Markdown

Added the generic Prime Agent ACP lifecycle consumer onto this PR at 547abf168.

  • Lifecycle enforcement is explicit opt-in, so the pinned 0.7.3 default remains compatible.
  • Compatible agents correlate responseBoundary and terminalQuiescence across prompt turns with monotonic event sequencing.
  • Rejected prompt admissions are distinguished from accepted turns that still owe terminal settlement.
  • Main-agent answer fallback excludes subagent, compaction, refinement, and foreign-turn updates.
  • Lifecycle/infrastructure status is recorded independently from benchmark rewards.
  • No OOLONG-specific dataset, task selection, sandbox tuning, retry, or workflow changes are included.

Paired producer contract: Prime Agent #1494 at 095ba1e5d.

Validation: frozen Ruff/format and ty pass; focused lifecycle/scoring/config/import suite passes 42/42; GitHub Ruff, Ty, and Macroscope checks are green; zero unresolved review threads.

@sethkarten
sethkarten marked this pull request as ready for review August 20, 2026 00:43
@macroscopeapp

macroscopeapp Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a new Prime Agent harness with substantial new runtime behavior: lifecycle state management, terminal quiescence coordination, and modified answer resolution. The scope of new async logic and behavioral changes warrants human review.

You can add or adjust custom eligibility rules. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can remove this and instead add prime agent into e2e test suite

client.terminal_quiescence is not None
and client.terminal_quiescence["outcome"] == "error"
):
raise RuntimeError("Prime Agent reported a terminal lifecycle error")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cant have prime agent specific code in a generic acp runner

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels like a lot of addition to the acp runner but dont know this part of the code super well so will let @hallerite judge

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes feels excessive. I will take a deeper look.

autonomous: bool = True
"""Run Prime Agent in autonomous mode unless an evaluation opts out."""

require_terminal_quiescence: bool = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this? does not seem like smth users should set?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

autonomous mode forwards the budget given by verifiers so it is probably ok to only be true instead of user opt-out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants