feat(runs): add the prime-runs SDK for eval and training runs - #856
Open
kcoopermiller wants to merge 23 commits into
Open
feat(runs): add the prime-runs SDK for eval and training runs#856kcoopermiller wants to merge 23 commits into
kcoopermiller wants to merge 23 commits into
Conversation
prime-evals models a run as three stateless calls over Dict[str, Any], so
neither of its intended consumers uses it on their main path: verifiers
reimplemented create -> batch -> finalize inline in v1/utils/platform.py, and
prime-rl hits a different API family entirely from utils/monitor/prime.py.
Producers have objects, a long-running loop, steps, ranks, forks and crashes;
each of them solved that privately, twice.
prime-runs makes the run an object instead:
run = pr.init(name=..., environments=["gsm8k"], model=..., framework=...)
run.log_traces([episode]); run.log({"reward": r}, step=step)
run.finish(summary=pr.metrics.from_episodes(episodes))
Identity: init() is called before rollouts start and the ID it returns is the
run ID everywhere, local archive included. Producers already stamp the run onto
their traces, so nothing is re-stamped and no producer record is rewritten. The
join key is run.id inside the trace document -- an indexed ClickHouse column
with a delete-by-run path -- not an upload-scoped context key.
Backends and sinks are independent axes. Backends own lifecycle (EvalsBackend,
OfflineBackend); sinks own transport (TracesSink, plus EvalSamplesSink for the
viewer's flat table). Both sinks run during the transition, because Prime
Traces is gated to an account allowlist and a traces-only client would leave
everyone else with an empty dashboard. When the Viewer API reads traces
natively, the default sink list drops one entry and no producer changes.
The SDK owns the operational work: streaming instead of buffering, a bounded
upload queue that drops and counts rather than stalling a training run, fork
safety via register_at_fork, contained errors (on_error="warn" by default),
terminal status through the context manager / atexit / signals, rank awareness,
and an offline mode that is a real run -- which is what lets producers delete
their --no-push branching.
trace_to_sample / build_samples move here from verifiers: it is knowledge about
a platform wire format, and prime-rl currently reaches across a repo boundary
to import it from a module path that has already drifted.
Known platform gap: there is no producer-facing way to mark an evaluation
failed (finalize only goes PROCESSING -> COMPLETED, UpdateEvaluationRequest has
no status). EvalsBackend calls the status endpoint it needs, latches on 404 so
it probes once, and falls back to recording the terminal state in metadata
while warning that the run will keep showing as running. The fallback stops
firing on its own once the endpoint ships.
Leaf package by construction -- httpx, pydantic, tenacity, prime-traces and
nothing else -- because the prime CLI depends on verifiers, so verifiers can
never depend on prime. verifiers already takes prime-tunnel and
prime-sandboxes on the same terms.
107 tests, hermetic (httpx.MockTransport + tmp dirs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kcoopermiller
requested review from
DamianB-BitFlipper,
JannikSt,
JohannesHa,
burnpiro,
kennethnym and
willccbb
as code owners
August 19, 2026 20:53
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dcb62a816
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All six were real. Verified against the backend where the claim depended on
service behaviour.
Run identity (high). init() exported PRIME_RUN_ID and then read it back, so a
second init() in the same process silently attached to the first run and never
created or finalized one of its own. Exports now record the exporting PID, and
only an ID from a *different* PID counts as inherited — a forked child sees the
parent's PID and joins, a re-init sees its own and opens a fresh run. Lifecycle
ownership follows intent rather than "was an ID present": an explicit id= is a
deliberate resume and finalizes, an ID picked up from the environment belongs
to whoever exported it. finish() also stops advertising the run it owned.
Fork safety (high). The child inherited the parent's httpx pools and buffered
file handles. Two processes writing one socket interleave into a single HTTP
stream, and a duplicated write buffer gets flushed twice. Connection and file
holders now reset in the child through a single process-wide hook in _fork.py:
dropped, never closed (closing sends close_notify down a socket the parent is
still reading) and never flushed (the buffer holds records the parent will
write itself). The hook is registered once rather than per instance, because
os.register_at_fork cannot be undone — the old per-worker registration pinned
every run the process ever opened.
Metadata replacement (medium). The service writes metadata with
{"$set": {"metadata": ...}}, a document-level replace. The failure fallback
PUT carried only {"prime_runs": ...}, erasing the config finish() had just
written. finalize() now receives the run's full config and merges into it.
Abandoned uploads (medium). finish() waited 60s for a flush, ignored the
result, then close() joined 30s — while a single sample POST is allowed 300s.
The budget is now derived from the upload timeout, a flush that does not drain
warns instead of passing silently, and close() leaves sinks open when the
thread is still alive rather than pulling a client out from under a live
request.
Signal status (medium). The handler reported FAILED while atexit, RunStatus and
the README all said CRASHED. Signals and Ctrl-C now report CRASHED: the
producer never said the run failed, it was stopped from outside its control
flow. FAILED stays for what the producer itself reports.
Environment version pinning (medium). EnvironmentRef accepted version_id and
dropped it, even though the API's EnvironmentReference carries it — the run
attached to whatever version the hub resolved that day.
Found while testing the signal fix: _handle_signal read the displaced handler
*after* finish(), which restores and clears that table, so chaining always fell
back to SIG_DFL — re-raising the signal at default disposition and killing the
process instead of running the handler the application installed. Captured
before finish() now.
122 tests (up from 107), including a real os.fork() end-to-end check that the
child joins the parent's run and no record is written twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings the Bugbot pass did not cover. The other two Codex raised (environment version pinning, PRIME_RUN_ID re-init) were already fixed in 14bbac5, including its note that `setdefault` left descendants pointing at a stale ID when an explicit `id=` was supplied — `_announce` now assigns unconditionally. Non-idempotent retries. The client retried every method through ambiguous failures, so a lost response to POST /evaluations/ would create a second evaluation that the SDK never tracked, leaving an orphaned duplicate run. The module docstring even asserted this was safe "because run creation happens once per init()", which confuses the call site with the retry loop inside it. Retry safety is now decided per call. A failure is ambiguous when the request may already have been processed (gateway 502/504, read timeout, stream broken after the bytes went out); unambiguous when nothing reached the server (connect failure, 429 refused before any work). Unambiguous failures replay for every method. Ambiguous ones replay only when the caller declares `idempotent=True`, which defaults to `method != "POST"`. Run creation and sample appends keep the default; get-or-create, finalize and status writes declare themselves safe. Same classification prime-traces' client already uses. This also stops the samples sink duplicating rows on a lost response — duplicates silently skew every average on the dashboard, where a lost batch is at least recoverable. Uploader failures under on_error="raise". A sink fails on the uploader thread, where the raise went straight into the worker's own except and was discarded — so flush() and finish() returned success while records were being dropped, in exactly the mode documented as being for "tests and CI, where a silent upload failure is the bug". The failure is now held and re-raised at the next synchronization point the caller controls: flush(), or the very end of finish() so the run is still closed out properly first. The atexit and signal paths swallow it, since neither is a place to surface an exception. Signal handlers were never restored. `_restore_signal_handlers` compared `signal.getsignal(signum) is self._handle_signal`, but every access to a bound method builds a new object, so the identity check could never match. Handlers stayed installed for the life of the process: the finished Run was pinned, and the next run in that process saw a non-default handler and declined to install its own, leaving it unable to report signal termination. The bound method is created once in __init__ and compared against. 133 tests (up from 122). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two follow-up findings, both on the previous rounds' fixes. Offline records were buffered. reset_after_fork() dropped the inherited file handles, but on CPython the last reference going away closes them, and close() flushes — writing out the child's copy of the parent's buffer and duplicating every record still sitting in it. Dropping without closing is not expressible for a buffered writer, so the buffer is gone instead: records are written to an unbuffered append-mode handle, encoded here. Nothing is ever held in process memory, so a fork has nothing to copy and flush() has nothing to do. The earlier fork test passed on this path by luck. It forked before the uploader thread had opened the file, so no handle and no buffer existed in the child. Replaced with an assertion that records are readable through a separate handle with no flush and no close, which is what actually pins the property. Transient sample failures retired the sink. Making sample POSTs non-replayable (332fc21) was right on its own, but combined with the worker disabling a sink on any raise it meant a single 502 stopped every later batch — so one gateway blip could leave the rest of a run missing from the dashboard of exactly the accounts the v0 sample table exists to serve. That trade is worse than the duplicates it avoids. The worker now separates "this batch failed" from "this sink is finished". A permanent failure (gated account, rejected credential) will fail identically forever and still retires the sink immediately. A transient one gets three consecutive strikes, reset by any success, so a blip costs one batch and a sustained outage still stops the SDK re-attempting for hours. Dropped records are counted either way, so run.dropped_records reflects the loss. This applies to the traces sink too, which had the same all-or-nothing behaviour. 138 tests (up from 133). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit folded failed sink writes into `dropped`, which is documented and warned about as queue backpressure. Two things went wrong with that. A default online run writes to both the traces sink and the sample table, so one failed batch was counted twice. And a failure on one sink was counted at all even when the other sink stored the records, so `finish()` could warn about data missing from a run that has all of it. They are different losses and stay separate now. `dropped` counts records that reached no sink because the queue was full — the producer outran the uploader, and those records are stored nowhere. `failed_records` counts per sink, exposed as a mapping rather than a total, because summing it would recreate exactly the overstatement above. The finish warnings are phrased to match: one about records that reached nothing, one per sink about what that sink could not store. 139 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five points where prime-runs diverged from prime-traces, prime-evals, prime-sandboxes and prime-tunnel without meaning to: - Normalize an explicitly passed `base_url`. `Config` strips a trailing `/api/v1`; the `PlatformClient` constructor did not, so `pr.init( base_url=".../api/v1")` requested `/api/v1/api/v1` while the identical value in `PRIME_API_BASE_URL` worked. prime-traces carries the same helper for the same reason. - Map 403 to a typed `ForbiddenError`, matching `prime_traces.ForbiddenError` — which the traces sink already branches on for beta gating, while a 403 from the platform API collapsed into a generic `RunAPIError`. Behavior is unchanged (it was already classified permanent); callers can now branch. - Drop `pydantic` and `tenacity`. Neither was imported: this package models no response bodies and hand-rolls its retry loop. They do not belong in the dependency tree of a leaf package that lands inside verifiers. - Add the LICENSE file the other packages ship. - Declare 3.13, which CI has been testing all along. Also documents the three departures that are deliberate — private client instead of `core/`, dataclasses instead of pydantic models, no async client — and what to do about the three blocking calls when driving a run from async code, since every sibling SDK ships an async client and this one does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The platform's Config tab is unusable for both run kinds, and neither cause is the platform's: eval runs store a v0 blob of four keys, training runs store a fully-resolved dump where three chosen values sit under hundreds of defaults nobody picked. Both producers are now launched from one user-authored file — `uv run eval @ eval.toml`, `uv run rl @ train.toml` — and nothing captured it. Two changes, matching the two failures: `config_source=` takes the path to that file and stores it byte for byte, comments and section grouping intact. It rides inside the run's config under a reserved key, so every write that already carries the config carries the source too — create, the periodic update, finalize, the failure fallback, and the offline archive — with no extra plumbing and no chance of one path forgetting it. A str or Path is always a path, never inline text: guessing between them would turn a mistyped filename into a run whose config tab displays the filename. `config=` now accepts a pydantic model and dumps it with `exclude_unset=True`, so only fields somebody actually set are recorded. A mapping is still stored exactly as given, and a caller who wants every resolved default can still pass `cfg.model_dump()`. The asymmetry is deliberate — the shorter call should give the more useful answer. Note this is only the upload half. Both Config tabs are derived projections today (the eval tab filters through a 31-key allowlist, the training tab reconstructs TOML from stored fields), so rendering `metadata.config_source` verbatim is a separate frontend change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d2a97a6. Configure here.
config_source= was a second way to say the same thing. Collapse it into
config=, which now takes whichever form the caller has — the path to the
file the run was launched from, a mapping, or a pydantic model — the same
polymorphism environments= already has for slugs, dicts and
EnvironmentRefs.
The forms are distinguished by type, never by inspecting keys, so a
config that happens to carry a `text` field is not mistaken for a launch
file. Storage is unchanged: a path still lands under the reserved
config_source key inside the run's config, which stays the single key a
config-tab renderer has to know about.
A run launched from a file that also wants a derived value adds it with
run.update_config({...}) rather than the SDK carrying a second argument
for the uncommon case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four bits of surface that were each a second way to say something the SDK already had a first way to say. dataset= is always the environment under a different name, and no UI reads the column. The API field is still populated, derived from the first environment — producers just no longer repeat themselves to fill it. summary= on init() asked for a run's *outputs* at the moment it opens, before it has any. finish(summary=) and run.summary already cover the real case, so RunSpec loses the field too and create() stops sending an empty metrics blob. sinks= was a third way to configure transport next to traces= and samples=, with no caller outside one of my own tests. Run(sinks=) stays, so tests still inject fakes and a future custom sink has somewhere to land. Re-adding a keyword argument is non-breaking; removing one is not, which is the argument for cutting it now rather than later. log_episodes was a third name for log_traces. log_samples stays — it is the method name prime-rl's Monitor ABC already uses, so the adapter it exists for is real. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_normalize_config guessed why model_dump() failed: any TypeError was read as "this callable has a different signature" and retried bare. Inferring a cause from an exception type is the problem, because the only recovery available — dumping every field, defaults included — is exactly the outcome passing a model was meant to avoid. A wrong guess therefore degrades silently, in the one direction that matters. Now the signature is inspected before the call, so a failure during serialization surfaces as itself, and the fallback fires only when the keywords genuinely are not accepted. When it does fire it logs, because silently recording a hundred defaults nobody chose is the behaviour this path exists to prevent. Note for the record: Bugbot reported this as PydanticSerializationError being caught by `except TypeError`. That specific claim is wrong — PydanticSerializationError subclasses ValueError, and pydantic wraps even a serializer's own TypeError into it, so the handler could only ever catch a signature mismatch. The underlying concern about guessing was still worth acting on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Design review found ~40% of the package serving scenarios with no backend
and no consumer: training runs, multi-rank lifecycle ownership, PRIME_RUN_ID
join, signal handling, a probe for a /status endpoint the platform does not
have, and a dict-based reimplementation of the v0 projection. verifiers —
the one producer — uses seven init() arguments, log_traces() and finish().
Removed:
- training / multi-rank / join machinery: RANK_ENV_VARS, is_primary,
PRIME_RUN_ID publish/retract, id= resume, attach(), log()/log_samples()/
update_config()/commit=, MetricItem/RunUpdateItem, supports_step_metrics,
the summary timer, RunKind/kind, RunHandle.raw
- signal handling (install/relinquish/chain protocol, _pending_signal);
atexit and the context manager still report CRASHED/FAILED
- the /evaluations/{id}/status probe; finalize() writes metadata.prime_runs
directly for non-COMPLETED runs
- projection.py's serialized-mapping path and the bare-Trace branch of
EvalSamplesSink; the sink takes Episode objects or v0 sample dicts
- init() params kind, id, traces_url, traces, samples, handle_signals,
queue_size, finish_timeout, and the pydantic-model form of config=
- _build(); the second PlatformClient (backend and samples sink share one)
- 20 names from __all__ (Backend, Sink, RunSpec, RunHandle, sink and
backend classes, RUN_ID_ENV)
Consolidated: is_episode/stamp_run live once in sinks/base.py; line_format
and step no longer thread through log_traces -> WriteItem -> Sink.write.
Docstrings trimmed to contracts; README rewritten for the reduced surface.
Kept offline mode, the fork hooks and the prime_runs-local HTTP/config/
exception plumbing — each is a separate discussion.
Source 4,105 -> 2,524 lines; tests 3,201 -> 2,492 (165 passing); init()
21 -> 13 parameters.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
prime-runs gives eval jobs a single call —
pr.init(...)— that opens a run on the platform, uploads the exact TOML someone launched with, streams traces and samples to it while it executes, and marks it finished, failed, or crashed when it stops. Before this, verifiers hand-rolled its own version of that lifecycle against the raw API, which is why the Config tab shows a v0 JSON blob for evals — nobody was uploading the config people actually wrote. Training runs (prime-rl) are a later backend, designed against what prime-rl actually does today (/external-runs, parquet → R2), not pre-built here.What
A new leaf package,
prime-runs, where the run is an object:init()takes 13 keyword arguments;Runexposesid,url,log_traces(),flush(),finish(),fail(), a context manager, and loss counters (dropped_records,failed_records,errors). The consumer integration is PrimeIntellect-ai/verifiers#2415.Scope (latest commit)
7d3cb5ddcuts the package to what its consumer uses, following a design review. Source went 4,105 → 2,524 lines, tests 3,201 → 2,492,init()21 → 13 parameters. Removed, with the reason:kind="train",RANKdetection,PRIME_RUN_IDjoin,id=resume,log()/step metrics, summary timer)handle_signals=False(verifiers owns SIGINT/SIGTERM); atexit + the context manager still reportcrashed/failedPOST /evaluations/{id}/statusprobeprojection.py−290 lines)branchesderivation in duck-typed dict form; nothing sent itinit()paramskind, id, traces_url, traces, samples, handle_signals, queue_size, finish_timeout; pydantic-model form ofconfig=__all__exports (Backend,Sink,RunSpec,RunHandle, sink/backend classes)Still in, pending separate discussion: offline mode (
mode="offline",dir=), theregister_at_forkhooks, and the package-local_http.py/config.py/exceptions.pyrather than reusingprime_traces.core.Design decisions
Identity —
init()before rollouts, and its ID is the run ID. Including inside every trace document and the local archive. Producers already stamp the run at rollout time (verifiers doestrace.record_run(EvalRunInfo(id=config.run.id))), so nothing is re-stamped and no producer record is rewritten — only where that ID comes from changes.verifiers/v1/configs/cli/eval.py:43has the matching TODO today: "fetch the id from the Prime SDK once runs are registered there."The join key is
run.idinside the trace document, not an uploadcontextkey: the ingestion service extracts it into an indexed ClickHouse column with a delete-by-run path, whilecontextis an upload-scoped map.contextcarries provenance only. Bare dicts without arunkey are stamped; producer objects pass through untouched.A run records the config it was actually configured with.
config=takes the path to the file the run was launched from, or a mapping:config="eval.toml")config_sourcekeyBoth producers are launched from one user-authored file (
uv run eval @ eval.toml) and that file is the run's configuration, comments and section grouping included. It is stored, not parsed: parsing would buy a second representation of something the platform can already read, at the cost of a TOML dependency. A caller that wants structured values and the file puts the file underpr.CONFIG_SOURCE_KEYin the mapping — which is what verifiers does (model_dump(exclude_unset=True)+ the TOML).This is the direct fix for the unusable Config tab: eval runs currently store a v0 blob of four keys.
Run identity is separate from run config, on purpose.
environments,nameandmodelare typed columns the platform queries, not entries in an opaquemetadatablob:environment_id/environment_nameare the list-endpoint filters,name/model_nameare the search regex, and environments are resolved from a name orowner/nameslug to hub IDs before create.datasetis derived (always the environment under another name).Backends and sinks are independent axes. Backends own lifecycle (
EvalsBackend,OfflineBackend); sinks own transport (TracesSinkprimary,EvalSamplesSinkfor the flat table today's viewer reads). Both sinks run during the transition, because Prime Traces is gated to an account allowlist — a traces-only client would leave everyone else with an empty dashboard. When the Viewer API reads traces natively, the default sink list drops one entry and no producer changes.The SDK owns the operational work: streaming instead of buffering (verifiers currently holds every episode in RAM, then blasts); a bounded upload queue that drops and counts rather than stalling the run; contained errors (
on_error="warn"default,"raise"for CI, surfacing fromflush()/finish()); terminal status through the context manager andatexit; fork-safe queue, connection pool and file handles.trace_to_sample/build_samplesmove here from verifiers. It is knowledge about a platform wire format, not about any one eval framework. prime-rl currently doesfrom verifiers.v1.push import trace_to_sample(src/prime_rl/utils/monitor/prime.py:18) — a trainer importing an eval framework for a wire format neither owns, from a module path that no longer exists on verifiersmain. The fix is for prime-rl to import it fromprime_runs.projection; that is a prime-rl change.Leaf package by construction —
httpxandprime-traces, nothing else. Not pydantic (nothing is modelled), not tenacity. TheprimeCLI depends onverifiers, so verifiers can never depend onprime. verifiers already takesprime-tunnelandprime-sandboxeson exactly these terms.Known platform gaps
There is no producer-facing way to mark an evaluation FAILED.
finalizemoves a run PROCESSING → COMPLETED,UpdateEvaluationRequestcarries nostatus, and FAILED is written only when an internal Cloud Task trigger fails — so a crashed run stays RUNNING forever.EvalsBackend.finalizerecords the terminal state (status,finished_at, error) undermetadata.prime_runsand warns that the run will keep showing as running. A platform PR adding a status endpoint is the real fix; the SDK call is one line once it exists.The Config tab will not show
config_sourcewithout a frontend change.frontend/src/app/_utils/eval/evaluationConfig.tsfilters through a hard-coded allowlist and drops everything else. This PR ships the upload half; renderingmetadata.config_sourceverbatim when present, falling back to today's projection otherwise, is a small additive change on the frontend side.Behaviour change worth reviewing
An environment that will not resolve now fails the run instead of being silently skipped. The old
_resolve_environmentscontinued past failures, which produces a run attached to the wrong environments, looks like a successful upload, and is discovered wrong much later.Testing
httpx.MockTransport+ tmp dirs, no secrets, including a realos.fork()), ruff clean,tycleanopen_run/finish_run/abort_runin #2415) was exercised against this revision in disabled mode and the no-key fallbackpyproject.toml(pytest pythonpath +exclude-newer-package),uv.lock, atest-runsCI job,release-runs.yml, and the version-bump guardNot in this PR
prime-runsis on PyPI. It needs one follow-up for7d3cb5dd: drophandle_signals=Falsefromopen_run./api/v1/rft/external-runs, designed with prime-rl.🤖 Generated with Claude Code
https://claude.ai/code/session_01W4S4DNVRX5BNtsLva7TZz1
Note
Medium Risk
New public SDK that authenticates with API keys, talks to evaluations/traces APIs, and ships a PyPI release workflow. Logic is isolated in a new package with hermetic tests, but retry/idempotency and dual-sink upload behavior are easy to get wrong.
Overview
Adds
prime-runs, a leaf SDK (httpx+prime-tracesonly) so eval producers canpr.init()before rollouts, stream traces while the run is live, andfinish()with dashboard metrics.init()returns the canonical run ID (stamped onto bare dicts; producer objects pass through).config=stores a launch file verbatim underconfig_sourceor a mapping as given. Online mode creates evaluations, resolves hub environments (fail-hard, not skip), and writes both Prime Traces and the v0 sample table; offline/disabled modes keep the same handle shape. Uploads run on a bounded background queue with fork-safe reset, contained errors (on_error="warn"default), and atexit/context-manager terminal status. Failed evals are recorded inmetadata.prime_runsbecause the API cannot mark FAILED.Also wires workspace CI (
test-runson 3.11–3.13), version-bump guard, andrelease-runs.ymlfor PyPI. Training runs and producer migrations are out of scope.Reviewed by Cursor Bugbot for commit 7d3cb5d. Bugbot is set up for automated code reviews on this repo. Configure here.