Revert slurm.sim to the pre-2.0 template (fixes -l local launches) - #556
Merged
Merged
Conversation
The 2.0 rewrite (accel-sim#553) made slurm.sim hard-bound to a real Slurm + NFS cluster, which breaks every `run_simulations.py -l local` run. The local launcher submits through procman.py but still uses slurm.sim as its job template, and procman provides neither $SLURM_JOB_ID, squeue, nor a guarantee that rsync exists: * `#SBATCH --output=/dev/null` / `--error=/dev/null` -- procman parses exactly these lines to choose where to write a job's stdout/stderr, so all simulator output was being discarded. * The background watchdog `while squeue -j $SLURM_JOB_ID; ...` exits immediately when squeue is absent (e.g. inside a CI container) and falls straight through to `sync_to_nfs && rm -rf "$TMP_DIR"`, deleting the working directory while the job is still starting up. * procman rewrites the bare `$SLURM_JOB_ID` but not the braced `${SLURM_JOB_ID}` on the TMP_DIR line, so the staging directory and the log file inside it disagree about the job id. The net effect is that no `<name>.o<jobid>` ever lands in the run directory and job_status.py reports NOT_RUNNING_NO_OUTPUT for every job. This is what is currently breaking the gpgpu-sim CI, which clones this repo unpinned and launches with `-l local`. The old template only needs coreutils `mv`, so it works under both sbatch and procman. Reverting restores the local launcher; the tmp-staging behaviour can be reintroduced guarded on `[ -n "$SLURM_JOB_ID" ]` plus `command -v squeue rsync`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Accel-Sim 2.0 merge (#553) rewrote
util/job_launching/slurm.simto stage the working directory in/tmpandrsyncit back to NFS. That template is now hard-bound to a real Slurm + NFS cluster, but it is also the template used by the local launcher:procman.pyprovides no$SLURM_JOB_ID, and a CI container has neithersqueuenor (necessarily)rsync. Three things then go wrong at once:#SBATCH --output=/dev/null/--error=/dev/null—procman.pyparses exactly these lines to decide where to write a job's stdout/stderr:So every byte the simulator prints is now discarded, leaving no diagnostics at all.
The watchdog self-destructs.
while squeue -j $SLURM_JOB_ID &>/dev/null; do ... donereturns 127 immediately whensqueueis absent, so the subshell falls straight through tosync_to_nfs && rm -rf "$TMP_DIR"at t≈0 — deleting the staging directory while the main script is still racing torsyncinto it andcdthere. Withset -eEandtrap copy_output EXIT, the job dies before the simulator produces anything.${SLURM_JOB_ID}is not substituted.procman.pydoesre.sub(r"\$SLURM_JOB_ID", str(job.id), line), which matches the bare form on theexec >andsqueuelines but not the braced form onTMP_DIR=/tmp/${SLURM_JOB_ID}_REPLACE_NAME. The staging directory and the log file inside it end up disagreeing about the job id.Net effect: no
<name>.o<jobid>ever lands in the run directory, sojob_status.pyreportsNOT_RUNNING_NO_OUTPUTfor every job, within seconds.Impact
This is what is currently breaking gpgpu-sim CI, which clones this repo unpinned and launches with
-l local. Every matrix entry — QV100, TITANV, TITANV-LOCALXBAR, RTX2060, RTX3070, H100 — fails identically on all 10rodinia_2.0-ftapps, with zero output to diagnose from. Last green run was before #553 landed; everything after it fails.Fix
Revert
slurm.simto the pre-2.0 template. It only needs coreutilsmv, so it works under bothsbatchandprocman.Follow-up (not in this PR): the tmp-staging behaviour is genuinely useful on the Slurm runners and can be reintroduced guarded on
[ -n "$SLURM_JOB_ID" ]pluscommand -v squeue rsync, keeping#SBATCH --output=/--error=pointed at real paths soprocman.pykeeps working.🤖 Generated with Claude Code