Skip to content

fix: derive text-encoder alpha_pattern keys from the LoRA module name - #14972

Open
akhil9tiet wants to merge 3 commits into
huggingface:mainfrom
akhil9tiet:fix/14970-text-encoder-alpha-pattern
Open

akhil9tiet wants to merge 3 commits into
huggingface:mainfrom
akhil9tiet:fix/14970-text-encoder-alpha-pattern

Conversation

@akhil9tiet

Copy link
Copy Markdown

Fixes #14970

In get_peft_kwargs, the text-encoder alpha_pattern key derivation dropped the LoRA module segment (to_k_lora), producing the parent block (...self_attn) instead of the module (...self_attn.k_proj). Since PEFT matches alpha_pattern keys against the end of module names, the key never matched and every text-encoder module fell back to the most common alpha. The derivation now translates the diffusers LoRA weight suffix (.to_{q,k,v,out}_lora, .lora_linear_layer) to the PEFT module name, and drops the text_model. prefix so the key matches whether or not the loader stripped it for flattened text encoders. Adds two regression tests to tests/lora/test_peft_utils.py.

@github-actions github-actions Bot added fixes-issue size/M PR with diff < 200 LOC tests utils and removed size/M PR with diff < 200 LOC fixes-issue labels Oct 6, 2026
@andre-menutole

Copy link
Copy Markdown

Thanks for the quick fix, @akhil9tiet! I checked this PR (at e1e02ab) against the reproduction from #14970 on diffusers 0.41.0, transformers 5.16.1 and peft 0.20.0:

  • CLIPTextModel (flattened, no text_model level): k_proj scaling is now 1.0 (was 0.25), and alpha_pattern is {'encoder.layers.1.self_attn.k_proj': 4.0}.
  • CLIPTextModelWithProjection (still has text_model): k_proj scaling is 1.0 as well.
  • The modules with the most common alpha keep 0.25 (alpha 1 / rank 4), as expected.

@github-actions github-actions Bot added size/M PR with diff < 200 LOC fixes-issue labels Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-module alphas of kohya text encoder LoRAs collapse to the most common alpha (alpha_pattern keys stop at the parent module)

2 participants