Skip to content

feat(nnx): fix checkpoint restore ShapeDtypeStruct crash under native LoRA#4416

Merged
RexBearIU merged 6 commits into
nnx-lora-supportfrom
jackyf/nnx-lora-native-fix
Jul 10, 2026
Merged

feat(nnx): fix checkpoint restore ShapeDtypeStruct crash under native LoRA#4416
RexBearIU merged 6 commits into
nnx-lora-supportfrom
jackyf/nnx-lora-native-fix

Conversation

@RexBearIU

@RexBearIU RexBearIU commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes a JAX ShapeDtypeStruct crash during checkpoint restoration under Flax NNX native LoRA (and QLoRA).

IMPORTANT NOTE: This pull request is blocked by PR #4346 (feat(nnx): support native Flax NNX PEFT/LoRA training loop). It targets main directly to keep its commit history clean, but should only be merged after PR #4346 has been successfully integrated into main.

Key Changes

  • Treats Fallback SDS as Absent: Fallback ShapeDtypeStructs from Orbax partial restores are treated as absent/None to prevent them from overwriting concrete initialized weights.
  • Materializes Template State: Properly materializes and shards the concrete template state using init_state_fn before overwriting restored weights.
  • Refactored Integration Tests: Transitions the PEFT integration tests to use sequential isolated Python subprocesses. This completely avoids JAX exclusive-locking conflicts in multi-device/TPU VM host environments, allowing the test suite to execute successfully on actual accelerator hardware.
  • Dynamic Multi-Device TPU Support: Dynamically detects the TPU VM device count and overrides ici_fsdp_parallelism=1 and ici_data_parallelism=num_devices for downscaled test models, eliminating any compilation FSDP partitioning divisibility constraints.

FIXES: #4400

Tests

The complete sequential native LoRA integration test suite covering Gemma4 and Qwen3 SFT, and LLaMA2 pre-training save/restore loops was successfully run and verified on both CPU/CI and real TPU VM host environments:

PYTHONPATH=src pytest tests/integration/peft_integration_test.py

All 6 integration tests successfully pass:

  • test_native_sft_gemma4_lora (passed)
  • test_native_sft_gemma4_qlora (passed)
  • test_native_sft_qwen3_lora (passed)
  • test_native_sft_qwen3_qlora (passed)
  • test_pretrain_lora_checkpoint_save_and_restore (passed)
  • test_pretrain_qlora_checkpoint_save_and_restore (passed)

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@RexBearIU RexBearIU force-pushed the jackyf/nnx-lora-native-fix branch from 4703dce to 80e1325 Compare July 10, 2026 10:16
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

@RexBearIU RexBearIU force-pushed the jackyf/nnx-lora-native-fix branch from 80e1325 to ccc9f4f Compare July 10, 2026 11:39
@RexBearIU RexBearIU force-pushed the nnx-lora-support branch 2 times, most recently from 054531c to 7fa9df0 Compare July 10, 2026 11:54
@RexBearIU RexBearIU force-pushed the jackyf/nnx-lora-native-fix branch from ccc9f4f to ebd5fa8 Compare July 10, 2026 12:15
@RexBearIU RexBearIU changed the base branch from nnx-lora-support to main July 10, 2026 12:16
@RexBearIU RexBearIU marked this pull request as ready for review July 10, 2026 12:20
@RexBearIU RexBearIU changed the base branch from main to nnx-lora-support July 10, 2026 17:57
@RexBearIU RexBearIU force-pushed the jackyf/nnx-lora-native-fix branch from ebd5fa8 to d7d1d66 Compare July 10, 2026 18:02
@RexBearIU RexBearIU merged commit 68a3834 into nnx-lora-support Jul 10, 2026
5 checks passed
@RexBearIU RexBearIU deleted the jackyf/nnx-lora-native-fix branch July 10, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants