feat(nnx): fix checkpoint restore ShapeDtypeStruct crash under native LoRA#4416
Merged
Conversation
4703dce to
80e1325
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
80e1325 to
ccc9f4f
Compare
054531c to
7fa9df0
Compare
ccc9f4f to
ebd5fa8
Compare
…checkpointer restoration
ebd5fa8 to
d7d1d66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes a JAX
ShapeDtypeStructcrash during checkpoint restoration under Flax NNX native LoRA (and QLoRA).IMPORTANT NOTE: This pull request is blocked by PR #4346 (
feat(nnx): support native Flax NNX PEFT/LoRA training loop). It targetsmaindirectly to keep its commit history clean, but should only be merged after PR #4346 has been successfully integrated intomain.Key Changes
ShapeDtypeStructs from Orbax partial restores are treated as absent/None to prevent them from overwriting concrete initialized weights.init_state_fnbefore overwriting restored weights.ici_fsdp_parallelism=1andici_data_parallelism=num_devicesfor downscaled test models, eliminating any compilation FSDP partitioning divisibility constraints.FIXES: #4400
Tests
The complete sequential native LoRA integration test suite covering Gemma4 and Qwen3 SFT, and LLaMA2 pre-training save/restore loops was successfully run and verified on both CPU/CI and real TPU VM host environments:
All 6 integration tests successfully pass:
test_native_sft_gemma4_lora(passed)test_native_sft_gemma4_qlora(passed)test_native_sft_qwen3_lora(passed)test_native_sft_qwen3_qlora(passed)test_pretrain_lora_checkpoint_save_and_restore(passed)test_pretrain_qlora_checkpoint_save_and_restore(passed)Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.