Skip to content

[bugfix] Fix Qwen3.5 non-packing causal conv routing - #9812

Open
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-qwen35-causal-conv-routing
Open

[bugfix] Fix Qwen3.5 non-packing causal conv routing#9812
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-qwen35-causal-conv-routing

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

What changed

  • Treat cu_seq_lens_q=None as a non-packing Qwen3.5 forward instead of selecting the FLA packing/sequence-parallel implementation merely because the optional kwarg is present.

Root cause

In transformers==5.12.1, the failing line reported in #9810 is after the causal-convolution call, not immediately after in_proj_qkv. The native causal_conv1d_fn returns a tensor, while FLA's causal_conv1d returns an (output, state) tuple.

Older ms-swift code could install the FLA callable into mod.causal_conv1d_fn when the packed path was selected. Although #9643 isolated that callable on current main, the wrapper still selected the packed path whenever the cu_seq_lens_q key existed, even when its value was None. Optional FlashAttention kwargs can be propagated with None in a regular non-packing forward.

This change uses the value of cu_seq_lens_q to select the packed path, so a normal Qwen3.5 SFT forward stays on the Transformers implementation.

Impact

Qwen3.5 non-packing SFT, including LoRA, gradient checkpointing, DeepSpeed, and resume flows, no longer enters the tuple-returning FLA causal-convolution path solely because an empty optional kwarg was propagated.

Fixes #9810.

Validation

  • flake8 swift/model/models/qwen.py
  • isort --check-only swift/model/models/qwen.py
  • yapf --diff swift/model/models/qwen.py
  • git diff --check

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@taking-lying-flat
taking-lying-flat marked this pull request as ready for review July 29, 2026 02:58
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen3.5 SFT LoRA all-linear 断点续训在 GatedDeltaNet 中报错:in_proj_qkvtranspose 前返回 tuple

1 participant