[LoRA] Text-encoder LoRA loading assumes CLIP module names and fails for T5 / SmolLM3 / ChatGLM text encoders #15020

Describe the bug

The text-encoder LoRA loader assumes CLIP module names (q_proj, k_proj, v_proj, out_proj, fc1, fc2), so loading breaks for text encoders that name their modules differently: T5 (Chroma, DeepFloyd IF, Kandinsky 3), SmolLM3 (Bria FIBO) and ChatGLM (Kolors). Diffusers can't even load back a T5 text-encoder LoRA it saved itself with save_lora_weights (reproduction below).

There are two causes in _load_lora_into_text_encoder:

1. convert_state_dict_to_diffusers() only converts CLIP-named keys (PEFT_TO_DIFFUSERS), and convert_state_dict_to_peft() then can't infer the format of the rest (L221-L228). If no key has a CLIP name (T5, or a LLaMA-style LoRA on o_proj only), loading fails with ValueError: Could not automatically infer state dict type. Adapter metadata doesn't help, because the conversion runs before it is used.
2. Without metadata, ranks are only read from CLIP-named modules (L391-L395). Any other module gets the most frequent rank, so an o_proj with r=8 next to a q_proj with r=4 fails with a size mismatch. This is why test_simple_inference_with_partial_text_lora is skipped for Bria FIBO. The model-level path (loaders/peft.py#L250-L259) already reads the rank from every lora_B key.

I verified the T5 case with Chroma; DeepFloyd IF, Kandinsky 3 and Kolors go through the same code.

Proposed fix

1. Don't run convert_state_dict_to_peft on keys already in the PEFT format: either call it only when there are lora_linear_layer keys, or make it return a PEFT state dict as is.
2. Without metadata, read the rank from every lora_B.weight key, as loaders/peft.py does.
3. Re-enable the two skipped Bria FIBO tests and add a T5 regression test next to test_load_lora_into_flattened_clip_text_encoder.

If this approach is welcome, I'd like to open the PR myself.

Reproduction

import tempfile

from peft import LoraConfig
from peft.utils import get_peft_model_state_dict
from transformers import T5Config, T5EncoderModel

from diffusers import ChromaPipeline, ChromaTransformer2DModel


def tiny_pipe():
    text_encoder = T5EncoderModel(T5Config(vocab_size=32, d_model=16, d_kv=4, d_ff=32, num_layers=2, num_heads=2))
    transformer = ChromaTransformer2DModel(
        patch_size=1, in_channels=4, num_layers=1, num_single_layers=1, attention_head_dim=16, num_attention_heads=2,
        joint_attention_dim=16, axes_dims_rope=[4, 4, 8], approximator_hidden_dim=32, approximator_layers=1,
        approximator_num_channels=16,
    )
    return ChromaPipeline(scheduler=None, text_encoder=text_encoder, tokenizer=None, transformer=transformer, vae=None)


# A LoRA on Chroma's T5 text encoder, saved with diffusers' own API
pipe = tiny_pipe()
pipe.text_encoder.add_adapter(LoraConfig(r=4, target_modules=["q", "k", "v", "o"]))
lora_layers = get_peft_model_state_dict(pipe.text_encoder)

with tempfile.TemporaryDirectory() as tmpdir:
    ChromaPipeline.save_lora_weights(tmpdir, text_encoder_lora_layers=lora_layers)
    tiny_pipe().load_lora_weights(tmpdir)  # ValueError: Could not automatically infer state dict type

Logs

Traceback (most recent call last):
  File "bug1_lora.py", line 27, in <module>
    tiny_pipe().load_lora_weights(tmpdir)  # ValueError: Could not automatically infer state dict type
  File "src/diffusers/utils/peft_utils.py", line 347, in wrapper
    return fn(*args, **kwargs)
  File "src/diffusers/loaders/lora_pipeline.py", line 1661, in load_lora_weights
    self.load_lora_into_text_encoder(
  File "src/diffusers/loaders/lora_pipeline.py", line 1800, in load_lora_into_text_encoder
    _load_lora_into_text_encoder(
  File "src/diffusers/loaders/lora_base.py", line 384, in _load_lora_into_text_encoder
    state_dict = convert_state_dict_to_peft(state_dict)
  File "src/diffusers/utils/state_dict_utils.py", line 228, in convert_state_dict_to_peft
    raise ValueError("Could not automatically infer state dict type")
ValueError: Could not automatically infer state dict type

System Info

• 🤗 Diffusers version: 0.41.0.dev0
• Platform: Linux-7.0.0-38-generic-x86_64-with-glibc2.39
• Running on Google Colab?: No
• Python version: 3.12.3
• PyTorch version (GPU?): 2.14.1+cu130 (True)
• Huggingface_hub version: 2.2.0
• Transformers version: 5.19.0
• Accelerate version: 1.15.0
• PEFT version: 0.21.2
• Safetensors version: 0.8.0
• xFormers version: not installed
• Accelerator: NVIDIA GeForce RTX 5070 Ti, 16303 MiB
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@sayakpaul @BenjaminBossan