fix(slurm): resolve node IPs to non-loopback addresses

[i-kosarev]

Problem

The generated SLURM job script builds the peer list with:

SLURM_NODE_IPS=$(scontrol show hostname ${SLURM_JOB_NODELIST} | while read node; do
    getent hosts "$node" | awk '{print $1}'
done | ...)

On Ubuntu, /etc/hosts maps the machine's own hostname to 127.0.1.1, and
getent hosts returns that entry first. Since the loop runs on each node, every
node resolves itself to 127.0.1.1 while resolving its peers correctly — so
each node publishes a peer list with a loopback address in its own slot.

Anything that waits on all peers then hangs forever: no node listens on the
loopback address its peers were told to connect to. We hit this with the
sglang-disagg socket barrier, where two nodes sat in wait_for_all_ports() and
never launched their servers, which in turn collapsed the 32-rank rendezvous
(TCPStore ... shut down too early, client socket has timed out ... 5757).

The symptom is a multi-node run that hangs at bring-up with no error until the
SLURM time limit. It is load-dependent: the larger the allocation, the likelier
at least one node carries the /etc/hosts self-mapping, which makes it look
intermittent.

Fix

Resolve each node with getent ahostsv4, skipping 127.*, and fall back to the
primary non-loopback interface address for the local node.

Verification

• Reproduced on 8-node runs: peer lists contained 127.0.1.1 in the local slot;
  after the fix every node resolves to a routable address and all nodes reach the
  barrier.
• tests/unit/test_slurm_job_template.py — 21/21 pass.
• The rendered script passes bash -n.

Made with Cursor

[Copilot]

Pull request overview

This PR updates the SLURM-generated SGLang-disaggregated job script to avoid advertising loopback (127.*) addresses in the per-node peer list, which can break multi-node rendezvous/barriers and cause silent hangs during bring-up.

Changes:

• Switch node resolution from getent hosts to getent ahostsv4, selecting the first non-127.* IPv4.
• Add a fallback that derives an address from local interfaces when name resolution yields only loopback output.

Suppressed comments (1)

src/madengine/deployment/slurm.py:964

• The fallback (hostname -I ...) is applied for any node that fails to resolve via getent ahostsv4, but the comment says it should only be used for the local node. As written, an unresolvable peer hostname would silently get replaced with the current node’s IP, producing an incorrect peer list and hard-to-debug hangs. Also, because this uses a pipeline, exit 1 inside the while body wouldn’t reliably fail the overall command unless you avoid the pipeline or enable pipefail.

Consider restructuring the loop to avoid scontrol | while ... | tr | sed so you can reliably error out on unresolved non-local nodes, and only use the interface-address fallback when $node matches the local host.

SLURM_NODE_IPS=$(scontrol show hostname ${{SLURM_JOB_NODELIST}} | while read node; do
    node_ip=$(getent ahostsv4 "$node" | awk '$1 !~ /^127\\./ {{print $1; exit}}')
    if [ -z "$node_ip" ]; then
        node_ip=$(hostname -I | tr ' ' '\\n' | grep -vE '^127\\.' | head -n1)
    fi

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/madengine/deployment/slurm.py:964

• The fallback is applied to every unresolved hostname, not just the local node as the comment states. If a remote node has a transient NSS/DNS failure (or only an IPv6 record), each task inserts its own interface address into that remote slot, producing different/duplicate peer lists and recreating the barrier hang. Identify the local entry (for example via SLURM_NODEID/SLURMD_NODENAME) before using the local fallback, and fail the task if a remote entry—or the fallback itself—cannot be resolved; make sure that failure is not swallowed by the surrounding pipeline.

    if [ -z "$node_ip" ]; then
        node_ip=$(hostname -I | tr ' ' '\\n' | grep -vE '^127\\.' | head -n1)
    fi

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Copilot]

🟢 Approval recommended

The fix is focused and covers the critical resolution and container handoff paths with regression tests.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[coketaste]

LGTM

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

🟡 Changes recommended

A failed node enumeration can still produce an empty peer list and bypass the new fail-fast guard.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The implementation addresses the failure modes with focused end-to-end shell and environment handoff coverage.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Balanced