Skip to content

feat(eval): check selected attempts in rollout health reports - #3881

Draft
yuhezhang-ai wants to merge 1 commit into
yuhez/2135-rollout-recoveryfrom
yuhez/2135-recovery-health
Draft

yuhezhang-ai wants to merge 1 commit into
yuhez/2135-rollout-recoveryfrom
yuhez/2135-recovery-health

Conversation

@yuhezhang-ai

@yuhezhang-ai yuhezhang-ai commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Make health reports inspect the same completed attempts that evaluation scoring selects. If a newer attempt failed or was interrupted, an older saved success must not reappear as a healthy completed sample.

Extracted from #3302 as a separate follow-up to #2135. Depends on #3302; this draft targets its branch. Judge-only recovery is the independent sibling draft #3879.

  • Read selected result locations from the store and let workers read those slices directly from the original files. Avoid copying or passing whole trajectories between processes.
  • Detect when a file was replaced after indexing, instead of reading unrelated bytes at the old position. Appending to the original file remains supported.
  • Apply the same selection to automatic, standalone, and multi-shard health checks, including inputs reached through file symlinks. Loose legacy files and explicitly supplied failure sidecars retain their existing line-by-line behavior.

Validation

Rebased onto #3302 5cd78777c. Final follow-up head: bc7eef604.

  • Python 3.13.14: pytest tests/unit_tests/test_{journal_health,rollout_health,rollout_journal,rollout_store,rollout_collection,rollout_outcomes,cli_eval}.py -q --tb=short — 671 passed.
  • pre-commit run --all-files and cd fern && npm run check — passed (Fern: zero errors, one warning).
  • Inherited feat(eval): persist failures and resume unfinished evaluations #3302 fixes passed 6,070 non-sandbox core tests and 1,364 focused collector/GDPVal/adapter tests. Its fresh real-model crash/resume smoke at 5cd78777c fetched server configuration over HTTP with no local server blocks.
  • The earlier combined-sibling inference smoke below was not repeated for this rebase. Full sandbox/server-shard CI remains pending; this remains a draft.

Rollouts

This changes artifact reading, not agent execution or verifier scoring. The earlier read smoke at 9eb9066f5 used actual Qwen2.5-0.5B outputs from Gym's production model adapters, Simple Agent, FrontierScience verifier, and legacy Environment Server adapter. Those runs exercised SIGKILL/resume with interrupted file tails, a judge outage, and whole-task retries.

Earlier combined evidence at 87048b331 enabled automatic health during SIGKILL/resume and judge-only recovery, with three selected records and no read or duplicate-identity errors. That combined inference smoke was not repeated for this rebase. Capture-dependent checks remain unobserved because these smoke runs have no trajectory captures; this is recovery evidence, not benchmark-quality or training evidence.

Compatibility and benchmark impact

Removes #3302's temporary guard against journal-backed health reads. Health sample counts and findings may change for histories containing superseded attempts; they now describe the selected completed results. Evaluation rewards, aggregation rules, saved trajectories, and training behavior are unchanged.

Checklist

  • I have read the contributing guidelines.
  • The change is focused on health reads over recovery histories.
  • Tests added or updated and pass locally for the tested scope; full CI is pending.
  • Documentation added or updated.
  • Pre-commit checks pass locally.
  • New source files include the required Apache-2.0 SPDX header.
  • Authored commits have DCO sign-off.

@yuhezhang-ai yuhezhang-ai added feature New capabilities, enhancements, or enablement work area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@yuhezhang-ai
yuhezhang-ai force-pushed the yuhez/2135-recovery-health branch from a391a53 to bc7eef6 Compare October 2, 2026 17:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant