Conversation
block_digits * 10^10 overflows 1e14 when Slurm block names strip to more than four digits, which kills the head node before any actor starts. Bound each half so the quantity stays near 1e10. Signed-off-by: Liang Yan <lyan@coreweave.com>
Digit-stripping SLURM_TOPOLOGY_ADDR into a 10^10 polynomial overflows Ray's 1e14 custom-resource cap, and clamping the digits only delays that. Rank the allocation 1..N by Slurm block name then hostname so the value stays unique, bounded by node count, and collision-free across names that share a numeric substring. Signed-off-by: Liang Yan <lyan@coreweave.com>
Dense 1..N ranks stay unique; wrapping would collide. Abort if N is above Ray's custom-resource limit, and drop hostname digit-stripping which could still overflow that limit. Signed-off-by: Liang Yan <lyan@coreweave.com>
Signed-off-by: Liang Yan <lyan@coreweave.com>
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ray.sub builds topo_rank as
block_digits * 1e10 + node_digits. Ray rejects custom resources above 1e14, so any block name with more than four digits (region/zone/site codes) kills the head before any actor starts. The ValueError is only in ray-head.log; stdout looks like GCS never came up.Don't strip digits. Sort the allocation by Slurm block name, then hostname, and assign 1..N. Fail if N > 1e14; wrapping would collide. Dropped the hostname-digit fallback for the same reason.