fix(genrm): bound cohort deadlines and failure cleanup

[yuhezhang-ai]

This follows merged #2814, which established group, attempt, and member identities. It addresses #3181's immediate timeout and failure-cleanup requirements.

• While collecting answers: enforce a finite deadline. If a member never arrives, fail the group and release waiting requests without publishing partial rewards.
• Once all answers are available: retry transient judge HTTP failures, including interrupted response bodies, within a bounded budget using the same answers.
• If judging ultimately fails: preserve each generated answer and return an explicitly masked failure. The zero reward is a placeholder, excluded from Gym evaluation metrics by default.
• On failure, cancellation, or shutdown: release waiters and clean up pending work.

Recovery and compatibility

A judge failure returns HTTP 200 with mask_sample: true, failure_kind: judge_failed, a bounded failure_reason, and the existing failure-sidecar tags. The shared judge failsafe also preserves instance_config.mask_sample: true for older RL consumers. This shared response fix applies to every environment using judge_failsafe.

The existing evaluation opt-in count_failure_classes_as_zero: [judge_failed] still includes selected failures as zeros. Only the metric-input copy is unmasked; saved failure records and training masks remain unchanged. Online and offline aggregation apply the same policy, including when every rollout fails. With the opt-in, an all-judge_failed run reports mean/reward: 0.0; without eligible counted failures or successful results, collection still raises. The main rollout JSONL never receives synthetic successes.

A masked HTTP 200 result does not automatically make RL retry the group. It is a completed, unusable result for the current RL caller. If the caller chooses replacement:

• Explicit-ID groups: advance the shared group-attempt number for every member and submit a complete replacement group.
• Legacy groups without an ID: failed groups retain a failure record. Reusing their task/prompt key returns failure with migration guidance; recovery requires a fresh explicit group ID for the complete group. This prevents delayed old members from joining a replacement. Successful legacy groups still allow sequential new evaluations.

The judge-request default is now 1,800 seconds, matching the overall evaluation budget. This allows substantially more room for long generations and queueing than the previous 300-second default; deployments must still size both limits for their workload.

Required RL coordination: before NeMo RL adopts this Gym revision, coordinate with RL #4160 / #4061. Reading the mask alone is insufficient: TransferQueue must preserve it in the training loss mask, and unavailable judge rewards must be excluded from group statistics. #4160 includes these fixes with masked_reward_policy: exclude as the default. Top-level field ingestion remains separate migration work, so the nested compatibility field remains. Before updating RL's Gym revision, validate a real failed Gym response through RL's loss mask and group statistics. Gym can merge first if RL keeps its existing Gym revision until those fixes are available.

GenRM-specific usage and recovery documentation is now in the resources-server README, rather than a new Fern site page.

Closes #3181.

Validation

Current code: 6a7f30b707aee62c46478d757005f3fcc9fabe0d, incorporating main at d4c0f86c7. The latest fix is 059adb409.

• 5,110 core, GenRM, and OSWorld dependency-policy tests passed on Python 3.13.14: python -m pytest tests/unit_tests resources_servers/genrm_compare/tests responses_api_agents/osworld_agent/tests/test_dependency_policy.py -m 'not sandbox' -q --tb=short. There were 103 skips, 673 sandbox tests deselected, and 184 passing subtests. Sandbox tests were not run locally.
• The all-failure regression checks one and four masked judge failures, zero counting enabled/disabled, an unrelated counted class, and deferred aggregation. Four cases fail on the preceding code and pass with this fix; default and wrong-class error behavior remains intact. Existing mixed-success tests remain covered.
• Actual TCP collector → SimpleAgent → GenRM tests check both collection timeout and judge outage, with zero counting on/off. Online and offline metrics match, the failure sidecar remains byte-for-byte unchanged, and the main/merged rollout JSONL stays empty.
• Final all-files pre-commit, whitespace checks, and DCO sign-off passed. Exact-head CI passed, including core and all eight server shards. The preceding run passed core and seven server shards but failed OSWorld dependency resolution; merged fix(osworld): resolve Python 3.13 NumPy metadata conflict #3649 fixes that NumPy metadata conflict and is included in this head. Previous-head CI passed on be7e6353d, including core and all eight server shards.
• No new model run for the latest collector aggregation fix: it changes metric eligibility, not policy generation or judging. The earlier real-model evidence below is from be7e6353d.
• Real-model functional validation passed: Slurm job 19084624, exit 0 in 14m58s, on be7e6353d. Gym's native CLI/HTTP path used Qwen3-0.6B policy and Qwen3-Nemotron-235B-A22B-GenRM-2603 on eight H100s. Eight policy generations and four completed real judge calls were captured.
• The run preserved four masked failure answers during an injected judge outage, then scored a caller-coordinated complete replacement attempt. Four HTTP 503s and four interrupted HTTP response bodies recovered within the retry budget without extra policy generation. A second collector resume made no further model calls. Failed legacy groups rejected delayed members, explicit zero-count aggregation preserved saved masks, and rewards exactly matched the captured judgments. Saved artifacts were independently inspected.
• The model run enabled CUDA_LAUNCH_BLOCKING=1 for diagnostics after an earlier CUDA failure. That failure did not recur; this does not establish its root cause. The run's functional assertions passed before the separate launcher teardown issue described below.
• No full RL training, checkpoint restart, or comparative throughput claim.

Limits and follow-ups

The caller coordinates complete replacement attempts. There is no partial scoring, automatic group scheduling, durable answer reuse, or change to generation/judge overlap. Legacy identities cannot isolate overlapping runs or provide completed replay. Explicit-ID replay and failure records are process-local and bounded by retention; eviction or restart is not a recovery protocol.

Ordinary individual reverification is unsupported. The existing judge_failed_only guard bypass does not create groups or advance attempts: retained failures remain failures until the caller supplies a complete replacement with the required identity.

Active-group memory limits, retention optimization, caller-minted group identities, durable outcomes, richer telemetry, and improvements to retry backoff remain follow-up work. The inherited judge-failed-only mode-guard bypass needs a separate compatibility change; documenting it does not make individual GenRM reverification supported.

A Ray teardown abort was previously reproduced on both main and an earlier PR head; its native cause remains undiagnosed. In the passing job 19084624, the Gym launcher again exited with SIGABRT after the functional assertions completed. The harness stopped the remaining owned children and recorded no survivors; the separate policy server exited normally and Slurm released the allocation. This validates the GenRM failure/recovery behavior, not a clean Ray launcher shutdown.

Related PRs: what is covered and what could follow

Built on #2814 (Anish Mahishi), with deadline and cleanup requirements from #2385 (Teodor-Dumitru Ene) and #2903 (Jiacheng Xu). Contributor credit remains in commit trailers.

#2385: the timeout requirement is covered. Its partial-scoring policy is not adopted: if A/B/C arrive but D is missing, this PR fails the incomplete group. Whole-group failure is allowed by #3181. Close #2385 as superseded if that policy is accepted; otherwise retain/rework partial scoring as separate follow-up work with explicit missing-member and training/metric semantics.

#2903: together, #2814 and this PR cover request deduplication, duplicate-disconnect safety, and failure cleanup. Three policy differences remain:

• fix(genrm): make cohort retries failure-safe #2903 permits a fresh group under the same identity after failure. Here, replacement requires a newer shared explicit attempt, or a fresh explicit ID when migrating a failed legacy group.
• fix(genrm): make cohort retries failure-safe #2903 keeps completed rewards for the server's lifetime, including legacy groups. Here, completed replay is bounded and restricted to identical responses in explicit-ID groups; legacy groups have no completed replay.
• fix(genrm): make cohort retries failure-safe #2903 starts comparing available answer pairs while other answers are still generating. Here, we preserve fix(genrm): isolate prompt cohort retries #2814's wait-for-all schedule. Each judge call needs only two answers, so early judging remains a possible optimization; it does not require partial reward publication, and its performance benefit has not been benchmarked.

After this PR merges, maintainers can close #2903 as superseded if these choices are accepted, tracking desired early judging or broader replay/retry behavior separately. Alternatively, rework the existing PR against current main to focus on the remaining scope. Closing either older PR means its core problem is addressed and remaining choices are accepted or tracked, not that every proposed behavior was merged.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-yuhez-3181-genrm-cohort-lifecycle.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/evaluation/genrm-cohorts

[yuhezhang-ai]

/ok to test e4154b6

[yuhezhang-ai]

/ok to test cfc48b3

[yuhezhang-ai]

/ok to test 3491eed

[yuhezhang-ai]

Thanks @ananthsub — the updates are in the current head, be7e6353d. Summary of the changes following your review:

• The shared judge failsafe now returns the standard mask_sample, failure_kind, and bounded failure_reason, while retaining instance_config.mask_sample for older RL consumers. Tests cover the actual agent /run response.
• Failed legacy groups retain a failure record, preventing delayed old answers from joining a replacement while that record is retained. Recovery requires a fresh explicit group ID; explicit-ID callers advance the shared attempt for the complete group.
• The judge-request default is now 1,800 seconds. Transient HTTP errors and interrupted response bodies use bounded retries with the same answers.
• The docs now live in the resources-server README and explicitly describe a masked HTTP 200 as a terminal result, with replacement coordinated by the caller. It does not automatically trigger RL regeneration.
• A further regression fix preserves the evaluation opt-in to count judge failures as zero, without changing saved failure records or training masks.

Validation: CI passed, along with 4,845 local core tests and 192 GenRM tests. The real-model smoke passed on this head using Qwen3-0.6B and the supported 235B GenRM judge: eight policy generations and four completed judge calls. It verified preserved failed answers, complete replacement, recovery from HTTP/body interruptions, correct rewards, and no additional work on a second collector resume.

RL coordination remains required: the compatibility field alone does not fix the downstream loss-mask and group-statistics issues. RL #4160 contains those fixes and is still open. Before RL adopts this Gym revision, those fixes and a focused Gym-to-RL masking integration check are needed. Gym could merge first while RL retains its existing Gym revision; please confirm your preferred sequencing.

The known Ray launcher abort still occurred during teardown after the functional checks passed; cleanup stopped all owned processes. That limitation and the individual-reverification restrictions remain documented. Ready for your re-review.

[ananthsub]

looking really good! one comment inline

[ananthsub]

Consider an evaluation with one rollout and this configuration:

route_failures_to_sidecar: true
count_failure_classes_as_zero:
  - judge_failed

The rollout returns:

{
  "reward": 0.0,
  "mask_sample": true,
  "_ng_failure_class": "judge_failed"
}

Online collection writes the row to the failure sidecar, then reaches the no-persisted-results guard and raises:

RuntimeError: None of the 1 dispatched rollouts produced a result

This happens before the opted-in failure is converted into a metrics-only zero at lines 1660–1673.

Running offline aggregation over the same artifacts succeeds and reports:

{"mean/reward": 0.0}

Therefore online and offline behavior still differs when every rollout belongs to an explicitly counted failure class. This parity test covers three successes plus one failure, so it does not reach the all-failure boundary.

Please determine the eligible metrics-only failure rows before applying the no-results guard. The guard should raise only when both are empty:

not persisted_results and not counted_failures

Keep the current failure behavior when count_failure_classes_as_zero is empty. Please add an all-judge_failed regression asserting that:

1. online and offline aggregation both report mean/reward: 0.0;
2. the saved sidecar remains masked and unchanged; and
3. the main rollout JSONL remains free of synthetic results.

[yuhezhang-ai]

/ok to test 7ffb0ef

[yuhezhang-ai]

/ok to test 6a7f30b

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/35895013332