Add spur (Crusoe) scheduler support for multi-node runs

[mkuznet1]

Summary

Adds support for the spur (Crusoe) scheduler, whose SLURM-compatible CLI
breaks stock multi-node fan-out (srun does not dispatch tasks to other
nodes, empty SLURM_PROCID; scontrol show hostnames unsupported; Raft-based
eventually-consistent squeue/sacct).

• New SpurDeployment backend (target: spur) that fans out multi-node work
  via a job array (one single-node task per node); SLURM_ARRAY_TASK_ID
  is the node rank. Nodes self-form the cluster through the launcher's TCP
  rendezvous (rank 0 publishes its transport IP to a shared FS, peers read it
  as MASTER_ADDR).
• Marker-based completion detection in monitor() (per-rank done_rank<N>
  files) because spur sacct -j does not filter by job id; squeue used only
  as an eventually-consistent liveness guard.
• job.sh.j2: all spur logic gated behind {% if scheduler == 'spur' %};
  MAD_COLLECT_JOB_ID unifies log/artifact paths (== SLURM_JOB_ID on stock
  SLURM, SLURM_ARRAY_JOB_ID on spur).
• Orchestrator: an explicit non-local target now wins over structural
  inference (spur reuses the slurm config block).
• slurm_multi launcher path reworked to fan out via job array on spur.

Stock SLURM behavior is unchanged (spur paths are fully gated; render + import
smoke-checked).

Test plan

• Offline: template renders for both slurm and spur schedulers
• Stock SLURM multi-node run unaffected (regression)
• spur amd-spur 4-node run reaches container launch (blocked only by an
  unrelated docker image build issue)

[coketaste]

Code review

Found 16 issues. Ordered by severity; the "checked and cleared" section at the end lists what I verified as fine.

Blocking

1. Target-precedence flip regresses runtime overrides for all existing users

The manifest's build-time deployment_config["target"] now wins unconditionally over run-time --additional-context. This inverts the convention enforced 20 lines above (lines 232-234 only fill in keys missing from additional_context — runtime wins, manifest fills gaps) and the "Deployment Target Inference" section of CLAUDE.md.

Failure: build a manifest with {"slurm": {...}} (persists target: "slurm"), later run it with --additional-context '{"k8s": {...}}'. _infer_deployment_target would return "k8s", but it is never called — the job silently goes to SLURM.

Precedent: when the bare-metal backend hit the same "new backend reuses another's config shape" problem (bc0dd9e), the fix was to add a distinguishing key to _infer_deployment_target, not to invert precedence globally.

madengine/src/madengine/orchestration/run_orchestrator.py

Lines 252 to 268 in 625e608

	# An explicit non-local "target" in deployment_config wins. This is
	# required for schedulers that reuse another backend's config block:
	# e.g. the spur backend reuses the "slurm" block, so structural
	# inference alone would mis-detect it as plain "slurm".
	explicit_target = deployment_config.get("target")
	if explicit_target and explicit_target != "local":
	target = explicit_target
	else:
	# Infer deployment target from config structure (Convention over Configuration)
	# No explicit "deploy" field needed - presence of k8s/slurm indicates deployment type
	target = self._infer_deployment_target(self.additional_context)
	# Legacy support: check manifest for explicit target
	if not target or target == "local":
	target = deployment_config.get("target", "local")

2. target: "spur" is unreachable from madengine run

deployment_config["target"] is only ever written by build_orchestrator._save_deployment_config (line 1299), derived from additional_context["deploy"], and that runs only during build. _load_and_merge_manifest merges the key "deploy" — not "target". So madengine run --additional-context '{"deploy":"spur",...}' against an existing manifest silently falls through to plain SlurmDeployment.

Also missing: _infer_deployment_target (line 1113) and ConfigLoader.infer_and_validate_deploy_type (config_loader.py:228) have no spur awareness; there is no presets/spur (both presets/slurm and presets/k8s exist); and there is no doc, README, or CHANGELOG entry describing how to select spur.

madengine/src/madengine/orchestration/run_orchestrator.py

Lines 494 to 500 in 625e608

	if "deployment_config" in manifest:
	stored_config = manifest["deployment_config"]
	# Runtime --additional-context overrides stored config
	for key in ["deploy", "slurm", "k8s", "kubernetes", "distributed", "vllm", "env_vars", "debug"]:
	if key in self.additional_context:
	stored_config[key] = self.additional_context[key]
	manifest["deployment_config"] = stored_config

3. The _check_job_completion changes are dead code for spur and a regression for stock SLURM

SpurDeployment.monitor() fully overrides monitor() and never calls super().monitor() or _check_job_completion() (base.execute() -> _monitor_until_complete -> self.monitor()). So dropping -X and adding _ACTIVE_STATES — both commented as "Spur compatibility" — execute only on vanilla SLURM. All cost, zero benefit.

Cost: -X guaranteed a single allocations-only row; without it, _lines[0] is an order-dependent assumption over main + .batch + .extern rows. Unlike every other spur change in this PR, this one is not gated behind IS_SPUR.

Worse, the _ACTIVE_STATES early-return can hang forever: if squeue reports the job gone but sacct's terminal state contains REQUEUED/SUSPENDED, monitor() returns RUNNING every poll, and _monitor_until_complete is a bare while True: ... time.sleep(30) with no timeout — config.timeout is never consulted.

madengine/src/madengine/deployment/slurm.py

Lines 1618 to 1625 in 625e608

	# Spur compatibility: spur's sacct shim does NOT support the "-X"
	# (allocations-only) flag and errors out on it. Omit it. Without "-X",
	# sacct returns the main job row plus sub-steps (.batch/.extern); we take
	# the first (main job) row as the authoritative State below.
	sacct_cmd = ["sacct", "-j", job_id, "-n", "-o", "State"]
	try:

madengine/src/madengine/deployment/slurm.py

Lines 1653 to 1673 in 625e608

	# The queue check (squeue) can transiently report a job as gone on
	# eventually-consistent schedulers (e.g. spur's Raft control plane)
	# while it is still active. If sacct still reports an active state,
	# treat it as RUNNING instead of FAILED.
	_ACTIVE_STATES = (
	"RUNNING",
	"PENDING",
	"CONFIGURING",
	"COMPLETING",
	"REQUEUED",
	"RESIZING",
	"SUSPENDED",
	)
	if any(s in status for s in _ACTIVE_STATES):
	return DeploymentResult(
	status=DeploymentStatus.RUNNING,
	deployment_id=job_id,
	message=f"Job {job_id} is {status.lower()}",
	)

madengine/src/madengine/deployment/base.py

Lines 240 to 251 in 625e608

	while True:
	status = self.monitor(deployment_id)
	if status.status in [DeploymentStatus.SUCCESS, DeploymentStatus.FAILED, DeploymentStatus.UNKNOWN, DeploymentStatus.CANCELLED]:
	return status
	# Still running, wait and check again
	self.console.print(
	f" Status: {status.status.value} - {status.message}"
	)
	time.sleep(30) # Check every 30 seconds

4. A job array gives no gang-scheduling guarantee — the rendezvous can silently produce an empty MASTER_ADDR

--array=0-{{ nodes - 1 }} has no concurrency floor and no preflight check that N nodes are free. Combined with #SBATCH --exclusive (default true), array tasks can start serially. Peers wait only 180s for rank 0's master_addr, and rank 0 is not guaranteed to be scheduled first.

The timeout is silent: export MASTER_ADDR="$(cat ... 2>/dev/null)" does not trip set -e in bash (verified). The node continues with MASTER_ADDR="" and fails obscurely inside the launcher — no error message, no retry, no diagnostic. This is the central assumption of the design and it has no guard.

madengine/src/madengine/deployment/templates/slurm/job.sh.j2

Lines 95 to 110 in 625e608

	# spur may not set SLURM_SUBMIT_DIR; the manifest mounts it for /run_logs.
	export SLURM_SUBMIT_DIR="${SLURM_SUBMIT_DIR:-{{ manifest_file | dirname }}}"
	_MAD_REND_DIR="{{ rendezvous_dir }}/${SLURM_ARRAY_JOB_ID:-${SLURM_JOB_ID}}"
	mkdir -p "$_MAD_REND_DIR" 2>/dev/null || true
	_MAD_IFACE="${NCCL_SOCKET_IFNAME:-ens3}"; _MAD_IFACE="${_MAD_IFACE%%,*}"
	_MAD_MY_IP="$(ip -4 -o addr show "$_MAD_IFACE" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -n1)"
	[ -z "$_MAD_MY_IP" ] && _MAD_MY_IP="$(hostname -I | awk '{print $1}')"
	if [ "${SLURM_PROCID}" = "0" ]; then
	echo "$_MAD_MY_IP" > "$_MAD_REND_DIR/master_addr"
	export MASTER_ADDR="$_MAD_MY_IP"
	else
	for _i in $(seq 1 180); do [ -s "$_MAD_REND_DIR/master_addr" ] && break; sleep 1; done
	export MASTER_ADDR="$(cat "$_MAD_REND_DIR/master_addr" 2>/dev/null)"
	fi
	echo "[spur-rendezvous] rank=${SLURM_PROCID} node=$(hostname) my_ip=$_MAD_MY_IP MASTER_ADDR=${MASTER_ADDR}"
	export MASTER_PORT={{ master_port | default(29500) }}

madengine/src/madengine/deployment/slurm.py

Lines 550 to 560 in 625e608

	'[ -z "$_MAD_MY_IP" ] && _MAD_MY_IP="$(hostname -I | awk \'{print $1}\')"',
	'if [ "${NODE_RANK}" = "0" ]; then',
	' echo "$_MAD_MY_IP" > "$_MAD_REND_DIR/master_addr"',
	' export MASTER_ADDR="$_MAD_MY_IP"',
	'else',
	' for _i in $(seq 1 180); do [ -s "$_MAD_REND_DIR/master_addr" ] && break; sleep 1; done',
	' export MASTER_ADDR="$(cat "$_MAD_REND_DIR/master_addr" 2>/dev/null)"',
	'fi',
	'echo "[spur-rendezvous] rank=${NODE_RANK} node=$(hostname) my_ip=$_MAD_MY_IP MASTER_ADDR=${MASTER_ADDR}"',
	"",
	])

5. Log collection is broken on the spur slurm_multi path

#SBATCH --output=...%j_%t.out is unchanged. Under a job array, %j is the individual task's job id, not the array id returned by sbatch — arrays need %A_%a. Both _collect_slurm_multi_results (slurm.py:2257) and _show_log_summary (slurm.py:1571) glob madengine-*_{deployment_id}_*.out with deployment_id = array job id.

The template path survives because the task script writes _node_N logs keyed on MAD_COLLECT_JOB_ID. The slurm_multi path — which the PR description says spur uses — writes no node logs at all, so results["logs"] is always empty and any failure before the workload starts is undiagnosable.

madengine/src/madengine/deployment/slurm.py

Lines 481 to 484 in 625e608

	"#!/bin/bash",
	f"#SBATCH --job-name=madengine-{model_info['name']}",
	f"#SBATCH --output={self.output_dir}/madengine-{model_info['name']}_%j_%t.out",
	f"#SBATCH --error={self.output_dir}/madengine-{model_info['name']}_%j_%t.err",

madengine/src/madengine/deployment/slurm.py

Lines 2256 to 2259 in 625e608

	# Collect SLURM output logs for diagnostics
	flat_out_files = sorted(self.output_dir.glob(f"madengine-*_{deployment_id}_*.out"))
	results["logs"] = [str(f) for f in flat_out_files]

6. live_output silently does nothing on spur

SlurmDeployment.monitor() reads additional_context["live_output"] and calls _stream_job_output. SpurDeployment.monitor() does neither, and _stream_job_output globs the same broken _{job_id}_ pattern as issue 5. A user passing --live-output gets silence.

7. Infinite monitor hang when squeue is unavailable on spur

_live_task_count returns -1 on any exception or non-zero exit, which resets _spur_empty_polls = 0 every poll. If squeue is persistently broken, monitor() returns RUNNING forever — again with no timeout in the caller.

madengine/src/madengine/deployment/spur.py

Lines 160 to 164 in 625e608

	# after sbatch, spur's squeue does not yet list the array tasks, so a fresh
	# healthy run legitimately reports 0 live tasks for the first ~1-2 min.
	live = self._live_task_count(self._model_job_name())
	if live > 0:
	self._spur_seen_live = True

Should fix

8. Zero test coverage for 189 lines of new branchy state logic. No test references spur / SpurDeployment / IS_SPUR. Untested: monitor()'s _spur_seen_live / _spur_empty_polls state machine, marker parsing, _live_task_count whitespace parsing, _expand_nodelist, the sbatch retry loop, and the _check_job_completion changes. tests/unit/test_slurm_multi.py and test_slurm_job_template.py are the precedent, and reviewers requested exactly this on #149, #130, and #113.

9. mad_expand_nodelist is called from a child process where it is undefined. Defined at job.sh.j2:64 in the outer script, but the _generate_sglang_disagg_command / _generate_deepspeed_command snippets that call it are spliced at lines 503/512 — inside the cat > "$TASK_SCRIPT" << 'TASK_SCRIPT_EOF' heredoc (466-792), run as a separate bash "$TASK_SCRIPT". Shell functions are not inherited without export -f, so command -v mad_expand_nodelist is always false. Masked today only because the elif/else fallback duplicates the logic — which means the "prefer the helper" comment is wrong and the two copies will drift.

madengine/src/madengine/deployment/templates/slurm/job.sh.j2

Lines 63 to 66 in 625e608

	# unsupported. Try scontrol only for compressed forms, else split on commas.
	mad_expand_nodelist() {
	local nl="$1"
	if [ -z "$nl" ]; then return 0; fi

10. docker_image interpolated unquoted into generated bash. Reviewers have required shlex.quote for this exact class of issue three times in this file family (#99, #122, and #130 on this very function). The sibling elif branch has the same flaw but is pre-existing; this is new code copying it.

madengine/src/madengine/deployment/slurm.py

Lines 577 to 594 in 625e608

	if is_registry_image and self.IS_SPUR:
	# spur: each array task is its own node, so pull locally (no srun fan-out).
	script_lines.extend([
	"",
	"# Pull Docker image on this node (one array task per node)",
	"echo '=========================================='",
	f"echo \"[$(hostname)] Pulling {docker_image}...\"",
	"echo '=========================================='",
	f"docker pull {docker_image}",
	"PULL_EXIT=$?",
	"if [ $PULL_EXIT -ne 0 ]; then",
	f" echo \"[$(hostname)] Docker pull failed for {docker_image}\"",
	" exit $PULL_EXIT",
	"fi",
	"echo ''",
	])
	elif is_registry_image:

11. Rendezvous logic duplicated in two already-drifting copies. slurm.py:531-559 uses NODE_RANK; job.sh.j2:78-114 uses SLURM_PROCID for the identical check. About 20 lines of near-identical bash, two maintenance sites. CLAUDE.md, Simplicity First: "Minimum code that solves the problem."

madengine/src/madengine/deployment/slurm.py

Lines 530 to 560 in 625e608

	if self.IS_SPUR:
	# spur job-array rank + shared-filesystem rendezvous. Each array task
	# is one node: SLURM_ARRAY_TASK_ID is the node rank. Pin SLURM_JOB_ID to
	# the shared SLURM_ARRAY_JOB_ID so the launcher's rendezvous port and
	# /run_logs/<id> dir match across nodes. rank 0 publishes its transport
	# IP; peers read it as MASTER_ADDR (see also job.sh.j2 spur branch).
	rendezvous_dir = getattr(self, "rendezvous_dir", str(self.output_dir.resolve() / "spur_rendezvous"))
	script_lines.extend([
	"# --- spur job-array rank + rendezvous ---",
	'export NODE_RANK="${SLURM_ARRAY_TASK_ID:-0}"',
	'export SLURM_PROCID="${SLURM_ARRAY_TASK_ID:-0}"',
	f"export NNODES={self.nodes}",
	f"export SLURM_NNODES={self.nodes}",
	f"export WORLD_SIZE={self.nodes}",
	'export SLURM_JOB_ID="${SLURM_ARRAY_JOB_ID:-$SLURM_JOB_ID}"',
	f'export SLURM_SUBMIT_DIR="${{SLURM_SUBMIT_DIR:-{manifest_dir}}}"',
	f'_MAD_REND_DIR="{rendezvous_dir}/${{SLURM_ARRAY_JOB_ID:-$SLURM_JOB_ID}}"',
	'mkdir -p "$_MAD_REND_DIR" 2>/dev/null || true',
	'_MAD_IFACE="${NCCL_SOCKET_IFNAME:-ens3}"; _MAD_IFACE="${_MAD_IFACE%%,*}"',
	'_MAD_MY_IP="$(ip -4 -o addr show "$_MAD_IFACE" 2>/dev/null | awk \'{print $4}\' | cut -d/ -f1 | head -n1)"',
	'[ -z "$_MAD_MY_IP" ] && _MAD_MY_IP="$(hostname -I | awk \'{print $1}\')"',
	'if [ "${NODE_RANK}" = "0" ]; then',
	' echo "$_MAD_MY_IP" > "$_MAD_REND_DIR/master_addr"',
	' export MASTER_ADDR="$_MAD_MY_IP"',
	'else',
	' for _i in $(seq 1 180); do [ -s "$_MAD_REND_DIR/master_addr" ] && break; sleep 1; done',
	' export MASTER_ADDR="$(cat "$_MAD_REND_DIR/master_addr" 2>/dev/null)"',
	'fi',
	'echo "[spur-rendezvous] rank=${NODE_RANK} node=$(hostname) my_ip=$_MAD_MY_IP MASTER_ADDR=${MASTER_ADDR}"',
	"",
	])

12. All N array tasks write the same completion-marker file. The path is namespaced by ${SLURM_JOB_ID}, which the spur block pins to the shared SLURM_ARRAY_JOB_ID — so every rank races on madengine_<model>_<arrayid>.complete on the shared filesystem. Harmless only because SpurDeployment.monitor() ignores it; the line's own comment says it exists to prevent exactly this collision.

madengine/src/madengine/deployment/slurm.py

Lines 635 to 637 in 625e608

	completion_marker_dir
	/ f"madengine_{model_info['name']}_${{SLURM_JOB_ID:-local}}.complete"
	)

Minor

13. export MAD_NODE_RANK=$SLURM_NODEID — the spur block sets SLURM_PROCID / NNODES / NTASKS / LOCALID / JOB_ID / SUBMIT_DIR but not SLURM_NODEID. Each array task is its own single-node allocation, so this is 0 on every node. Nothing in-repo reads it, but it is exported for model scripts.

madengine/src/madengine/deployment/templates/slurm/job.sh.j2

Lines 179 to 181 in 625e608

	export MAD_SLURM_JOB_ID=$SLURM_JOB_ID
	export MAD_NODE_RANK=$SLURM_NODEID
	export MAD_TOTAL_NODES={{ nodes }}

14. _live_task_count matches by job name only, so a second concurrent run of the same model by the same user inflates the count and defeats the liveness guard. os.environ.get("USER", "") also yields squeue -u "" when USER is unset.

15. -> list should be List[str]; the file uses parameterized List[...] throughout.

madengine/src/madengine/deployment/slurm.py

Lines 107 to 109 in 625e608

	@staticmethod
	def _expand_nodelist(nodelist: str) -> list:
	"""Expand a SLURM nodelist string into a list of hostnames.

16. None of the 7 commits use conventional-commit prefixes (spur: ... rather than feat(spur): ...), against CLAUDE.md Code Style and the repo's recent history.

Checked and cleared

MAD_COLLECT_JOB_ID's ${SLURM_ARRAY_JOB_ID:-$SLURM_JOB_ID} fallback is behavior-preserving for stock SLURM. _MAD_REND_DIR is in scope everywhere it is used. The sbatch retry loop cannot leave result as None. srun is invoked without --nodes/--ntasks, so overriding SLURM_NNODES does not cause unintended fan-out. node_<rank>/ artifact collection lines up between Python and template. The non-spur srun path is correctly gated. The nodes == 1 spur path works (the done_rank marker is emitted outside the nodes > 1 gate).

---

The two worth fixing first are 1 (silently misroutes existing non-spur users) and 3 (a spur-motivated change that only affects stock SLURM, and can hang the monitor). 4 is the one most likely to make real spur runs fail unexplainably.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}