prime-runs

[kcoopermiller]

wip!!!

Why

The eval CLI hand-rolled its own platform client. verifiers/v1/utils/platform.py was 317 lines of resolve env → create → post samples → finalize over raw httpx. PrimeIntellect-ai/prime#856 moves that lifecycle into prime-runs, a leaf SDK (httpx + prime-traces) where a run is an object held for the length of the run rather than three stateless calls at the end of it.

What

platform.py goes from 317 lines to 218, and what is left is wiring rather than transport:

run = open_run(config, push_state)     # before the first rollout
config.run.adopt_id(run.id)            # the platform's id is the run id

async def on_complete(episode):
    episode.record_run(EvalRunInfo(id=config.run.id, name=config.run.name))
    await append_episode(out, episode, write_lock)
    await asyncio.to_thread(run.log_traces, [episode])   # streams as rollouts land

await asyncio.to_thread(finish_run, run, episodes, push_state)

Gone: push_traces, credentials, json_bytes, run_metrics, the batching loop, the payload ceiling, the in-memory buffer. trace_to_sample and build_samples moved to prime_runs.projection — that is knowledge about a platform wire format, not about an eval framework. build_samples stays here as a re-export because prime-rl imports it across the repo boundary (prime_rl/monitors/prime.py) against a released verifiers, so dropping the name would break its next upgrade; nothing imports trace_to_sample, so that one is gone.

file	what changed
utils/platform.py	open_run / finish_run / abort_run + PushState; the SDK owns the rest
cli/eval/runner.py	both runners open the run before the first rollout, stream to it, and close it out on every exit path
cli/eval/main.py	drops the end-of-run push; records the @ file.toml the run was launched from
configs/cli/eval.py	RunConfig.adopt_id() and record_source(), both private attrs
cli/resolve.py	config_file_ref(argv) — the root-level @ <path>
cli/dashboard/eval.py	the footer reads the live run instead of an upload's outcome

Config tab

This ships the upload half only. A run now carries the fields somebody actually set (model_dump(exclude_unset=True)) plus, when it was launched from one, the @ eval.toml itself byte for byte under metadata.config_source — but the tab renders neither yet: frontend/src/app/_utils/eval/evaluationConfig.ts walks a hard-coded 29-key allowlist and drops everything else. Until it learns to render config_source verbatim and fall back to today's projection — a small additive frontend change — the tab shows environment and model alone, two fields fewer than the v0 blob it replaces, which happened to carry num_examples and rollouts_per_example under the names that allowlist reads.