Skip to content

Add unified results for single-sbatch Slurm scenarios - #1041

Draft
podkidyshev wants to merge 1 commit into
ipod/unified-slurmfrom
ipod/slurm-api-ssbatch
Draft

podkidyshev wants to merge 1 commit into
ipod/unified-slurmfrom
ipod/slurm-api-ssbatch

Conversation

@podkidyshev

Copy link
Copy Markdown
Contributor

Summary

  • Include each single-sbatch case and sweep point in experiment.json, with workload status and canonical metrics. Previously single-sbatch output contained only scenario-level information.
  • Match accounting steps by their generated stdout path to report individual UTC timing. For workloads spanning multiple steps, use the earliest start and latest finish. Preserve unknown timing when accounting is unavailable.
  • Reuse DSE winner selection for single-sbatch sweeps and populate case metrics from the selected successful trial. Workload success continues to use was_run_successful; allocation-wide status and timing are not applied to every case.

Test Plan

Affected tests on macOS / Python 3.14.3:

uv run --locked --extra dev pytest tests/test_single_sbatch_runner.py tests/test_handlers.py tests/test_cloudaigym.py tests/test_agents.py tests/test_trajectory.py tests/test_gymnasium_adapter_contract.py tests/systems/slurm tests/test_base_runner.py tests/test_output.py tests/test_acceptance.py
325 passed in 3.62s

All pre-commit checks passed for the four changed files. Existing completion and trajectory tests were extended to cover per-case metrics/timing, failure/cancellation, missing execution evidence, and DSE selection; no new test functions were added.

A live single-sbatch test on one eight-H100 node ran a normal NCCL case and a two-point NCCL algorithm sweep. All three executions passed. Downloaded artifacts were analyzed locally using the workload success and canonical metric methods:

  • Three completed run records, each with 36 measurements and UTC timing matching its accounting step (31, 32, and 33 seconds).
  • Both cases have 36 case-level measurements.
  • DSE selected step 1 (Ring), matching the trajectory's highest configured inverse-latency reward.

Additional Notes

Stack: #1030 (ipod/unified-outputmain) → #1040 (ipod/unified-slurmipod/unified-output) → this PR (ipod/slurm-api-ssbatchipod/unified-slurm).

Live progress, iteration aggregation, and DSE with iterations remain separate work.

Signed-off-by: Ivan Podkidyshev <ipodkidyshev@nvidia.com>
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@podkidyshev
podkidyshev added this pull request to stack #1042 September 18, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant