Skip to content

fix: Only bind device id when needed, Fixes #8248 - #8269

Open
pengdurice wants to merge 3 commits into
deepspeedai:masterfrom
pengdurice:peng-fix-device-id-v1
Open

fix: Only bind device id when needed, Fixes #8248#8269
pengdurice wants to merge 3 commits into
deepspeedai:masterfrom
pengdurice:peng-fix-device-id-v1

Conversation

@pengdurice

Copy link
Copy Markdown
Contributor

Fixes #8248.

The change

init_process_group now asks a small resolver instead of inlining the condition:

device_id = get_init_process_group_device_id(world_size)
if device_id is not None:
    kwargs.update(device_id=device_id)

DEEPSPEED_SET_DEVICE_ID overrides the decision, parsed by a new get_env_flag helper:

Value Effect
1 / true / yes / on always bind
0 / false / no / off never bind
unset the default above: bind iff world_size > 1
anything else warn and ignore, so the default still decides

Multi-rank behaviour is unchanged.

Tests

tests/unit/comm/test_dist.py

Signed-off-by: pengdurice <pengduhit@gmail.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>
@pengdurice
pengdurice marked this pull request as ready for review August 18, 2026 21:33
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]DeepSpeed latest version fails during NCCL process-group initialization with CUDA error 600, while 0.17.2 works

1 participant