Fix ZeroDivisionError in compute_elastic_config return_microbatch on non-0.2 elasticity - #8286
Conversation
…non-0.2 elasticity With return_microbatch=True and an elasticity version other than 0.2, compute_elastic_config divides final_batch_size by world_size. That branch is reached only when world_size is unset, because the block above it returns for every positive value, so the division is always by zero. Version 0.2 resolves world_size from the WORLD_SIZE environment variable and raises ElasticityConfigError when it cannot. Do the same here, then check the resolved value against valid_gpus the way the sibling block already does. Fixes deepspeedai#8156 Signed-off-by: Ehsan Barkhordar <realbarkhordar@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e37bb0fe06
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if float(elastic_config.version) == 0.2: | ||
| return final_batch_size, valid_gpus, candidate_microbatch_size | ||
| else: | ||
| # Only an unset world_size reaches here, since the block above returns |
There was a problem hiding this comment.
Add the required sign-off trailer
This is a non-merge commit, but its message has no Signed-off-by trailer, so it does not satisfy the repository's commit requirements and may fail the DCO/CI gate. Recreate the commit with --signoff using the configured author name and email.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
tohtana
left a comment
There was a problem hiding this comment.
Hi @ebarkhordar,
Thank you for your contribution! Looks good to me.
ad1c516
|
Thanks for the review and the merge, Masahiro. Glad this one was useful. I will keep an eye on the elasticity config paths and send anything else I hit there your way. |
Root cause
With
return_microbatch=True,compute_elastic_configtakes theelseatelasticity.py:372for any elasticity version other than0.2, and that branch dividesfinal_batch_sizebyworld_size. In practice that means version0.1:0.3is rejected at line 301 and any other value raisesNotImplementedErrorat line 348, both before this point. The branch is only reachable whenworld_sizeis unset, because theif world_size > 0:block above it returns for every positive value. So on master the division is always by zero and the caller gets a bareZeroDivisionErrorat line 375 rather than a configuration error.Version
0.2avoids this by resolvingworld_sizefrom theWORLD_SIZEenvironment variable at lines 321-334, and raisingElasticityConfigErrornaming that variable when it cannot. Version0.1never reads the environment, so a caller who follows the0.2message's own advice, "set it as an environment variable", still crashes:WORLD_SIZEin env(9792, [...], 17)ElasticityConfigErrornamingWORLD_SIZEZeroDivisionErrorZeroDivisionErrorFix
Resolve
world_sizefromWORLD_SIZEin the non-0.2branch the way0.2already does, and raiseElasticityConfigErrorwith the same guidance when it cannot be resolved. Then check the resolved value againstvalid_gpusbefore dividing, matching the sibling block at lines 352-355; without that check an out-of-rangeWORLD_SIZEwould reach the loop and fail on themicro_batch_size is not Noneassertion instead ofElasticityIncompatibleWorldSize.Both divisions by
world_sizein this function now run only on a value that is positive and a member ofvalid_gpus. Nothing that works today changes: on master this branch raisedZeroDivisionErrorfor every input, and the0.2path is untouched.Verification
tests/unit/elasticity/test_elastic.pycover the unset case, resolution fromWORLD_SIZE, and an out-of-rangeWORLD_SIZE. All three fail on master withZeroDivisionErroratelasticity.py:375and pass here.pytest unit/elasticity/gives 23 passed, 3 skipped, on Python 3.12 withtorch==2.10.0+cputo match thecpu-torch-latestleg. The 3 skips are theDistributedTestclasses that needFusedLambBuilder, and they skip on master too.pre-commit run --filespasses on both changed files, yapf, flake8, codespell andcheck-torchdistincluded.0.2return_microbatchreturn at line 371, which no test in the repo reaches either before or after this change.#8162 proposed the same resolution in July, and its author closed it unmerged on 2026-08-12 without a review.
Fixes #8156