[Qwen] avoid creating attention masks when there is no padding

[kashif]

What does this PR do?

This pull request sets all ones masks to None.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Thanks!

Could you update with the following things?

• Shed light into what caused the speed regression
• Add a test with masks in the compilation tests here
• Do a before and after comparison in the outputs with the PR

[kashif]

will do!

[sayakpaul]

Does it lead to graph breaks? If not, then we should add additional contexts:

diffusers/tests/models/test_modeling_common.py

Line 2145 in 3996788

torch._dynamo.config.patch(error_on_recompile=True),

[dxqb]

does this PR lack a merge, or is the amount of code changes intentional and really part of this PR only? (+318 - 170)
if intentional, could you explain what it does and why all these changes are necessary?

avoiding masks if they're not necessary has been one line before
attention_mask = attention_mask if not torch.all(text_attention_mask) else None
(on a boolean mask)

[kashif]

@dxqb i intially mis-calculated and was adding support for the pipeline to be compiled, i will revert and simplify

[dxqb]

@dxqb i intially mis-calculated and was adding support for the pipeline to be compiled, i will revert and simplify

thanks!
regarding compile,

• Regional compilation is usually as efficient as full compilation (even if it was possible). Regional compilation compiles the transformer blocks, but not the entire pipeline.
• whenever you want to branch depending on GPU data, that's a graph break for torch.compile. Either inefficient, or fails depending on fullgraph. Or, in less abstract terms:
  you want to set your attention mask to None if the entire attention mask is True. But the attention mask lives on GPU. Checking whether all values of a tensor on GPU are True requires a transfer back to CPU - that's always a graph break and cannot be compiled (efficiently).

Therefore, I'd suggest to

• check for an all-True mask in the pipeline
• pass a None-Mask to the transformer block in that case - but don't check any GPU data in the transformer block, so the transformer block can be compiled.

[dxqb]

Thanks!

Could you update with the following things?

* Shed light into what caused the speed regression

Here is a benchmark of the impact of using a mask unnecessarily (second graph): #12870 (comment)
torch SDPA falls back to a non-flash algorithm if a mask is used.

[kashif]

yes i also benchmarked the mask and got:

[kashif]

thanks @dxqb please check now

[DefTruth]

I tested this #12987 on L20, and there was an 11% performance improvement in my test.

[sayakpaul]

And do you find that to resolve your concerns?

[DefTruth]

I think this has solved my problem. Based on this PR, the performance in my test cases has been working as expected.