Skip to content

Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity - #8286

Merged
tohtana merged 1 commit into
deepspeedai:masterfrom
ebarkhordar:fix/8156-elastic-microbatch-world-size
Aug 23, 2026
Merged

Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity#8286
tohtana merged 1 commit into
deepspeedai:masterfrom
ebarkhordar:fix/8156-elastic-microbatch-world-size

Conversation

@ebarkhordar

Copy link
Copy Markdown
Contributor

Root cause

With return_microbatch=True, compute_elastic_config takes the else at elasticity.py:372 for any elasticity version other than 0.2, and that branch divides final_batch_size by world_size. In practice that means version 0.1: 0.3 is rejected at line 301 and any other value raises NotImplementedError at line 348, both before this point. The branch is only reachable when world_size is unset, because the if world_size > 0: block above it returns for every positive value. So on master the division is always by zero and the caller gets a bare ZeroDivisionError at line 375 rather than a configuration error.

Version 0.2 avoids this by resolving world_size from the WORLD_SIZE environment variable at lines 321-334, and raising ElasticityConfigError naming that variable when it cannot. Version 0.1 never reads the environment, so a caller who follows the 0.2 message's own advice, "set it as an environment variable", still crashes:

config WORLD_SIZE in env master
0.2 yes returns (9792, [...], 17)
0.2 no ElasticityConfigError naming WORLD_SIZE
0.1 yes ZeroDivisionError
0.1 no ZeroDivisionError

Fix

Resolve world_size from WORLD_SIZE in the non-0.2 branch the way 0.2 already does, and raise ElasticityConfigError with the same guidance when it cannot be resolved. Then check the resolved value against valid_gpus before dividing, matching the sibling block at lines 352-355; without that check an out-of-range WORLD_SIZE would reach the loop and fail on the micro_batch_size is not None assertion instead of ElasticityIncompatibleWorldSize.

Both divisions by world_size in this function now run only on a value that is positive and a member of valid_gpus. Nothing that works today changes: on master this branch raised ZeroDivisionError for every input, and the 0.2 path is untouched.

Verification

  • Three new tests in tests/unit/elasticity/test_elastic.py cover the unset case, resolution from WORLD_SIZE, and an out-of-range WORLD_SIZE. All three fail on master with ZeroDivisionError at elasticity.py:375 and pass here.
  • pytest unit/elasticity/ gives 23 passed, 3 skipped, on Python 3.12 with torch==2.10.0+cpu to match the cpu-torch-latest leg. The 3 skips are the DistributedTest classes that need FusedLambBuilder, and they skip on master too.
  • pre-commit run --files passes on both changed files, yapf, flake8, codespell and check-torchdist included.
  • Not checked: the GPU legs, and the 0.2 return_microbatch return at line 371, which no test in the repo reaches either before or after this change.

#8162 proposed the same resolution in July, and its author closed it unmerged on 2026-08-12 without a review.

Fixes #8156

…non-0.2 elasticity

With return_microbatch=True and an elasticity version other than 0.2,
compute_elastic_config divides final_batch_size by world_size. That branch is
reached only when world_size is unset, because the block above it returns for
every positive value, so the division is always by zero.

Version 0.2 resolves world_size from the WORLD_SIZE environment variable and
raises ElasticityConfigError when it cannot. Do the same here, then check the
resolved value against valid_gpus the way the sibling block already does.

Fixes deepspeedai#8156

Signed-off-by: Ehsan Barkhordar <realbarkhordar@gmail.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e37bb0fe06

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

if float(elastic_config.version) == 0.2:
return final_batch_size, valid_gpus, candidate_microbatch_size
else:
# Only an unset world_size reaches here, since the block above returns

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required sign-off trailer

This is a non-merge commit, but its message has no Signed-off-by trailer, so it does not satisfy the repository's commit requirements and may fail the DCO/CI gate. Recreate the commit with --signoff using the configured author name and email.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @ebarkhordar,
Thank you for your contribution! Looks good to me.

@tohtana
tohtana enabled auto-merge August 23, 2026 06:42
@tohtana
tohtana added this pull request to the merge queue Aug 23, 2026
Merged via the queue into deepspeedai:master with commit ad1c516 Aug 23, 2026
13 of 15 checks passed
@ebarkhordar

Copy link
Copy Markdown
Contributor Author

Thanks for the review and the merge, Masahiro. Glad this one was useful. I will keep an eye on the elasticity config paths and send anything else I hit there your way.

@ebarkhordar
ebarkhordar deleted the fix/8156-elastic-microbatch-world-size branch August 23, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compute_elastic_config raises bare ZeroDivisionError for return_microbatch=True on non-0.2 elasticity versions without explicit world_size

2 participants