[bugfix] Fix Qwen3.5 non-packing causal conv routing - #9812
Open
taking-lying-flat wants to merge 1 commit into
Open
[bugfix] Fix Qwen3.5 non-packing causal conv routing#9812taking-lying-flat wants to merge 1 commit into
taking-lying-flat wants to merge 1 commit into
Conversation
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
taking-lying-flat
requested a deployment
to
qoder-review
July 29, 2026 00:13 — with
GitHub Actions
Waiting
taking-lying-flat
marked this pull request as ready for review
July 29, 2026 02:58
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
cu_seq_lens_q=Noneas a non-packing Qwen3.5 forward instead of selecting the FLA packing/sequence-parallel implementation merely because the optional kwarg is present.Root cause
In
transformers==5.12.1, the failing line reported in #9810 is after the causal-convolution call, not immediately afterin_proj_qkv. The nativecausal_conv1d_fnreturns a tensor, while FLA'scausal_conv1dreturns an(output, state)tuple.Older ms-swift code could install the FLA callable into
mod.causal_conv1d_fnwhen the packed path was selected. Although #9643 isolated that callable on currentmain, the wrapper still selected the packed path whenever thecu_seq_lens_qkey existed, even when its value wasNone. Optional FlashAttention kwargs can be propagated withNonein a regular non-packing forward.This change uses the value of
cu_seq_lens_qto select the packed path, so a normal Qwen3.5 SFT forward stays on the Transformers implementation.Impact
Qwen3.5 non-packing SFT, including LoRA, gradient checkpointing, DeepSpeed, and resume flows, no longer enters the tuple-returning FLA causal-convolution path solely because an empty optional kwarg was propagated.
Fixes #9810.
Validation
flake8 swift/model/models/qwen.pyisort --check-only swift/model/models/qwen.pyyapf --diff swift/model/models/qwen.pygit diff --check