Conversation
Signed-off-by: seonjinn <sna@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: seonjinn <sna@nvidia.com>
|
GPU regression update: on GB200 with vLLM 0.25.1, the integration reproducer failed at pipeline ranks 0, 1, and 3 before the fix with the same missing The run used integration commit Read-only lifecycle review confirms model parallel state is initialized before metadata preparation, while the communicator-owned rank is initialized later. This removes that initialization-order dependency without changing weight conversion. It does not establish arbitrary distributed rank-order correctness or full 20-step numerical parity; those remain validation work. |
|
/ok to test 4f53b46 |
Signed-off-by: seonjinn <sna@nvidia.com>
Problem
Async NCCL reshard setup builds source metadata before initializing the refit communicator. The metadata builder reads
self.my_pp_stage, which is assigned only during communicator initialization, and fails withAttributeError.Change
Read the pipeline rank from the already-initialized Megatron model parallel state. This preserves rank ownership without relying on transport initialization or assuming rank zero. No weight conversion or transfer logic changes.
Validation
Added regression cases for pipeline ranks 0, 1, and 3 without a communicator-owned rank attribute. GPU regression execution is pending; this is a draft, not a validated 20-step result.
Observed in a GB200 Qwen3-30B-A3B Async run before the first refit. No documentation or configuration changes are required.