Fix the Qwen GQA test failure.

[SandSnip3r]

Description

Fix the Qwen GQA test failure.

The Qwen GQA test failed in the decomposed variant due to the intemediate between the matmuls being downcast to fp16. Plumb use_fp32_acc through the dynamo backend.
Pick a specific seed for the GQA test.
Reenable GQA test for fp16.

Type of change

Please delete options that are not relevant and/or add your own.

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes
• I have added the relevant labels to my PR in so that relevant reviewers are notified

[zewenli98]

@SandSnip3r The idea looks good! The decomposed attention path needs the entire computation in FP32, not just matmul.

My only concern is that the definition of use_fp32_acc currently suggests FP32 applies only to matmul, whereas this implementation extends it to the whole attention path. It may help to clarify that in the definition, for example by noting that [op1, op2, ...] will use FP32 for the full computation rather than matmul alone for accuracy.

[SandSnip3r]

My only concern is that the definition of use_fp32_acc currently suggests FP32 applies only to matmul, whereas this implementation extends it to the whole attention path. It may help to clarify that in the definition, for example by noting that [op1, op2, ...] will use FP32 for the full computation rather than matmul alone for accuracy.

Yeah, I understand that. I don't necessarily love my overloading of the fp32 accumulation flag for this exact reason.

I have a comment inside scaled_dot_product_attention_decomposition specifically for this, as it's a behavior that's specific to this decomposition.

Are you proposing something different? Possibly commenting about the option's behavior with regard to sdpa in CompilationSettings?

[zewenli98]

My only concern is that the definition of use_fp32_acc currently suggests FP32 applies only to matmul, whereas this implementation extends it to the whole attention path. It may help to clarify that in the definition, for example by noting that [op1, op2, ...] will use FP32 for the full computation rather than matmul alone for accuracy.

Yeah, I understand that. I don't necessarily love my overloading of the fp32 accumulation flag for this exact reason.

I have a comment inside scaled_dot_product_attention_decomposition specifically for this, as it's a behavior that's specific to this decomposition.

Are you proposing something different? Possibly commenting about the option's behavior with regard to sdpa in CompilationSettings?

Yeah it's very clear for developers to see your comments inside scaled_dot_product_attention_decomposition. For users, I think it would be clear if we add the op-specific explanations in public API docs like CompilationSettings, compile(), cross_compile_for_windows(), convert_exported_program_to_serialized_trt_engine() and anywhere it's documented. Eventually they will be shown in the doc.

[SandSnip3r]

Ok, I've gone through and added comments about attention, on the fp32_acc flag, when decompose_attention is also set

[zewenli98]

LGTM