fix(tuners/lora): re-apply lora_dtype after peft _cast_adapter_dtype (#9694)

[Anai-Guo]

What this fixes

Fixes #9694.

--lora_dtype bfloat16 (or float16) has no effect when using the peft backend: LoRA adapter weights are always saved as float32.

Root cause

swift/tuners/peft.py monkey-patches LoraModel.__init__ (__new_init__), which converts the adapter to lora_dtype right after peft builds it. But peft's PeftModel.__init__ then calls LoraModel._cast_adapter_dtype(autocast_adapter_dtype=True) after LoraModel.__init__ returns, and that helper upcasts float16/bfloat16 adapter params back to float32:

PeftModel.__init__
 ├─ LoraModel.__init__ (= swift __new_init__)
 │    ├─ inject_adapter        → float32   (peft default)
 │    └─ _convert_dtype        → bfloat16  ✅ (swift)
 └─ base_model._cast_adapter_dtype(autocast_adapter_dtype=True)
                                → float32  ❌ overrides swift

So swift's conversion runs too early and is silently discarded.

Fix

Patch LoraModel._cast_adapter_dtype so that, after peft's cast, swift re-applies lora_dtype. This runs at the correct point (after peft's upcast) and is a no-op for the default lora_dtype=None, so existing fp32-adapter behavior is unchanged.

Verification (CPU, no training, from the issue repro)

import torch
from swift.tuners.peft import LoraConfig as SwiftLoraConfig
from peft import get_peft_model

class MiniModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(32, 64, bias=False)
    def forward(self, x): return self.linear(x)

pm = get_peft_model(MiniModel(),
                    SwiftLoraConfig(r=8, lora_alpha=16, target_modules=["linear"], lora_dtype="bfloat16"),
                    autocast_adapter_dtype=True)  # swift's default
for n, p in pm.named_parameters():
    if "lora" in n:
        print(n, p.dtype)
# before: torch.float32 (bug)
# after:  torch.bfloat16 ✅

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request modifies swift/tuners/peft.py to re-apply lora_dtype after PEFT's _cast_adapter_dtype is called, which prevents float16/bfloat16 adapters from being silently upcast back to float32. The review feedback suggests optimizing this change by checking if autocast_adapter_dtype is False and returning early to avoid redundant iterations over modules.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

If autocast_adapter_dtype is False, the original _cast_adapter_dtype method returns early without performing any upcasting. In this case, the lora_dtype applied during initialization remains intact, and re-running the conversion is redundant. We should check autocast_adapter_dtype and return early to avoid unnecessary iteration over all modules.

        def _cast_adapter_dtype(self, adapter_name: str, autocast_adapter_dtype: bool = True):
            self._cast_adapter_dtype_origin(
                adapter_name=adapter_name, autocast_adapter_dtype=autocast_adapter_dtype)
            if not autocast_adapter_dtype:
                return
            active_config = self.peft_config.get(adapter_name)