Skip to content

Revert slurm.sim to the pre-2.0 template (fixes -l local launches) - #556

Merged
JRPan merged 1 commit into
accel-sim:devfrom
purdue-aalp:fix-slurm-sim-local-launcher
Aug 26, 2026
Merged

Revert slurm.sim to the pre-2.0 template (fixes -l local launches)#556
JRPan merged 1 commit into
accel-sim:devfrom
purdue-aalp:fix-slurm-sim-local-launcher

Conversation

@JRPan

@JRPan JRPan commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Problem

The Accel-Sim 2.0 merge (#553) rewrote util/job_launching/slurm.sim to stage the working directory in /tmp and rsync it back to NFS. That template is now hard-bound to a real Slurm + NFS cluster, but it is also the template used by the local launcher:

elif options.launcher == "local":
    job_submit_call = os.path.join(this_directory, "procman.py")
    job_template = "slurm.sim"

procman.py provides no $SLURM_JOB_ID, and a CI container has neither squeue nor (necessarily) rsync. Three things then go wrong at once:

  1. #SBATCH --output=/dev/null / --error=/dev/nullprocman.py parses exactly these lines to decide where to write a job's stdout/stderr:

    outFMatch = re.match(r"#SBATCH --output=(.*)", line.strip())
    if outFMatch: job.outF = outFMatch.group(1)

    So every byte the simulator prints is now discarded, leaving no diagnostics at all.

  2. The watchdog self-destructs. while squeue -j $SLURM_JOB_ID &>/dev/null; do ... done returns 127 immediately when squeue is absent, so the subshell falls straight through to sync_to_nfs && rm -rf "$TMP_DIR" at t≈0 — deleting the staging directory while the main script is still racing to rsync into it and cd there. With set -eE and trap copy_output EXIT, the job dies before the simulator produces anything.

  3. ${SLURM_JOB_ID} is not substituted. procman.py does re.sub(r"\$SLURM_JOB_ID", str(job.id), line), which matches the bare form on the exec > and squeue lines but not the braced form on TMP_DIR=/tmp/${SLURM_JOB_ID}_REPLACE_NAME. The staging directory and the log file inside it end up disagreeing about the job id.

Net effect: no <name>.o<jobid> ever lands in the run directory, so job_status.py reports NOT_RUNNING_NO_OUTPUT for every job, within seconds.

Impact

This is what is currently breaking gpgpu-sim CI, which clones this repo unpinned and launches with -l local. Every matrix entry — QV100, TITANV, TITANV-LOCALXBAR, RTX2060, RTX3070, H100 — fails identically on all 10 rodinia_2.0-ft apps, with zero output to diagnose from. Last green run was before #553 landed; everything after it fails.

Fix

Revert slurm.sim to the pre-2.0 template. It only needs coreutils mv, so it works under both sbatch and procman.

Follow-up (not in this PR): the tmp-staging behaviour is genuinely useful on the Slurm runners and can be reintroduced guarded on [ -n "$SLURM_JOB_ID" ] plus command -v squeue rsync, keeping #SBATCH --output=/--error= pointed at real paths so procman.py keeps working.

🤖 Generated with Claude Code

The 2.0 rewrite (accel-sim#553) made slurm.sim hard-bound to a real Slurm + NFS
cluster, which breaks every `run_simulations.py -l local` run. The local
launcher submits through procman.py but still uses slurm.sim as its job
template, and procman provides neither $SLURM_JOB_ID, squeue, nor a
guarantee that rsync exists:

  * `#SBATCH --output=/dev/null` / `--error=/dev/null` -- procman parses
    exactly these lines to choose where to write a job's stdout/stderr,
    so all simulator output was being discarded.
  * The background watchdog `while squeue -j $SLURM_JOB_ID; ...` exits
    immediately when squeue is absent (e.g. inside a CI container) and
    falls straight through to `sync_to_nfs && rm -rf "$TMP_DIR"`,
    deleting the working directory while the job is still starting up.
  * procman rewrites the bare `$SLURM_JOB_ID` but not the braced
    `${SLURM_JOB_ID}` on the TMP_DIR line, so the staging directory and
    the log file inside it disagree about the job id.

The net effect is that no `<name>.o<jobid>` ever lands in the run
directory and job_status.py reports NOT_RUNNING_NO_OUTPUT for every job.
This is what is currently breaking the gpgpu-sim CI, which clones this
repo unpinned and launches with `-l local`.

The old template only needs coreutils `mv`, so it works under both sbatch
and procman. Reverting restores the local launcher; the tmp-staging
behaviour can be reintroduced guarded on `[ -n "$SLURM_JOB_ID" ]` plus
`command -v squeue rsync`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@JRPan
JRPan merged commit d930ad6 into accel-sim:dev Aug 26, 2026
@JRPan
JRPan deleted the fix-slurm-sim-local-launcher branch August 26, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant