[executorch][nvidia][tensorrt][22/n] Add correctness tests

[shoumikhin]

Stack from ghstack (oldest at bottom):

• [executorch][nvidia][tensorrt][25/n] Share CUDA stream across TRT delegates #17936
• [executorch][nvidia][tensorrt][24/n] Add power and unary converters with model improvements #17935
• [executorch][nvidia][tensorrt][23/n] Add benchmark infrastructure #17934
• -> [executorch][nvidia][tensorrt][22/n] Add correctness tests #17933
• [executorch][nvidia][tensorrt][21/n] Add comparison and slice converters #17932
• [executorch][nvidia][tensorrt][20/n] Add bmm converter #17931
• [executorch][nvidia][tensorrt][19/n] Add SDPA converter #17930
• [executorch][nvidia][tensorrt][18/n] Add layer_norm and pixel_shuffle converters #17929
• [executorch][nvidia][tensorrt][17/n] Add embedding, expand, upsample converters #17928
• [executorch][nvidia][tensorrt][16/n] Enable MobileNetV3 model support #17927
• [executorch][nvidia][tensorrt][15/n] Add linear model support with addmm and permute_copy converters #17926
• [executorch][nvidia][tensorrt][14/n] Add additional converters (mul, sub, div, mm, relu) #17925
• [executorch][nvidia][tensorrt][13/n] Add examples, C++ runner and CI workflow #17924
• [executorch][nvidia][tensorrt][12/n] Add C++ runtime infrastructure #17923
• [executorch][nvidia][tensorrt][11/n] Complete preprocess integration with serialization #17922
• [executorch][nvidia][tensorrt][10/n] Add blob serialization format with I/O metadata #17921
• [executorch][nvidia][tensorrt][9/n] Implement preprocess basic implementation #17920
• [executorch][nvidia][tensorrt][8/n] Add TensorRTCompileSpec dataclass #17919
• [executorch][nvidia][tensorrt][7/n] Add converter utils and add converter #17918
• [executorch][nvidia][tensorrt][6/n] Add converter registry infrastructure #17917
• [executorch][nvidia][tensorrt][5/n] Integrate partitioner #17916
• [executorch][nvidia][tensorrt][4/n] Add operator support base class #17915
• [executorch][nvidia][tensorrt][3/n] Add TensorRT partitioner stub #17914
• [executorch][nvidia][tensorrt][2/n] Add TensorRT backend stub #17913
• [executorch][nvidia][tensorrt][1/n] Add directory structure and initialization #17912

Add comprehensive correctness tests for TensorRT backend including model export and inference validation.

Differential Revision: D93275042

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17933

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 48 New Failures, 7 Unrelated Failures

As of commit 9af9e09 with merge base 01d21fa ():

NEW FAILURES - The following jobs have failed:

• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command docker exec -t 873feed0a5e0feded27ebd0e9e50ec521efcbabea0139f1232b53a74e7105ef8 /exec failed with exit code 1
• Test CUDA Builds / test-models-cuda (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t 0b020a5adf2c72e60fbfed5bcc9ede8316e37f2c4919a14538365068c4bf9dc8 /exec failed with exit code 1
• Test CUDA Builds / test-models-cuda (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 49656a7857dcf23ca89ff45a57092609344809b30ff4fe6a10387b581d195ed1 /exec failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (nvidia, parakeet-tdt, quantized-int4-metal) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, quantized-int4-metal) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test TensorRT Backend / check-all-tensorrt-tests (gh)
  ERROR: TensorRT build test failed!
• Test TensorRT Backend / test-export / linux-job (gh)
  RuntimeError: Command docker exec -t c8d1b669ede8d1d6081959d65040ec1d18fe34628f9964a772848e52b3d7a8a6 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t e7958adbcc46d07b5a9a6849ba6c2ed291eb6b7287b7f53b30e376d7a13e6b4c /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add) / linux-job (gh)
  RuntimeError: Command docker exec -t 1be6ef73c8d9c5555022344af537985dd86bf4740ed05abe927b46c6bce8b749 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t 60f6dc29d8f49287cf579d4b86657b24692b39f5211f34106650eedb50d76aec /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t 99e3448631ec004c79cd1b499ec054456257fcd63e9aaae27d743bf47a9febf1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t bbdd541c43e773e06f04a6fcf71408f5d75cc955b541b73ca81fc99b41cea49b /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t 1c02aed7f2220a0258de46f49292fe86aa14ceba5076194c335b0a35ecd459f5 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t 5d1bade5e21d1214df6063c98cbc3de5fa65b812ab84770ff81e1e6045042543 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t bc80ca291a5f8410f232f536e53a6969a0d7c0b422d88eaf207a57a45e6043da /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 01bc28801a2e9aff9688c0e8cbf467e8a93b237dffed9e637a2e5e4719c9fffa /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t f6e6a248ad4d12ce0f2aa79cd279911b3571e33b64b30de5bc0955d5912b90e3 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 5b69e307bfbfcf4d0d22669b48b5b58324809483d1c42d90a5c07738e8fb6d07 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t a17e08053ee4cf97c9d7c5ecda931f4641fa08fe745f263d0b272aeb115a83c8 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t 9ca96ef3ab5621989cf528539fa15be840502f20f1bce5b2c74d415eaa9dc5c5 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t c3fa4bb41b6f3b35578d5ca051290b86a215afe29c276150936417ddfa403097 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t bc26db6d47172d97398f120015bdca0134e5301510351b4610becf17c40f76c4 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t 3f013ae216634085d4736e1f4c328f5b6270ba3058f83135f4f5f4c1f1480e97 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 5c20bcd0a7048ebbe0ef4f70028a462015c638e3e9c6dcd986ce15ac171d16b9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t 7aa3bb1d59e9536f8dbb23a53f5cf72b6bc29b077f8126f675c66087a87ad7ca /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t 60fcb0dcae8417ab9ee0c49d25547a643dbeeb9815d79c34890b7124d719b9c1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 8663d02055b91897e7ea101969e59599fa5ebf409df4945979fd68e5df19b947 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add) / linux-job (gh)
  RuntimeError: Command docker exec -t 01b36f7d165542b3b32ad159dfb69e1c0922b601d1804b1e1f1efebaa4a89635 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t 466cd8886a6abb0e767ebe174d1f0ebb775aa757593d098baca768f9bcc23732 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t a678ec3bda3a16c6b4679e15dceaea365d4c8a41ef56b9b81cc4b87b4fd4a90f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t 60791929829fd31a9663f3c1a53f213624c8dbda21a6618291b499fee22fb843 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t 90a41f7913fdb257fd1b8cd6ff0d522c51818c813559fc2d2b0402b107cf715f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t 39d7e1cb4ec2607433e0f9e99f76965943c1d5ba55d81f7d00e165f8b43c9b38 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t c9efb54aa3e179154daf93575d9187619001469647b307600791aa19f7595306 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 032ca3855646ee3a2188efc135f5dd4637e8df55c66c46fbd30996c5856bbb24 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t e7617249a6b0f6afdd820650408851f586d786ef98cac66993a537ee2e75fe33 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 0698331d111414a528beaa626f4309a6b1f0e1ba3d88f328cd788195cd725ec0 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 958c78d07ddca8be6af493701345ff4210920dc9a4ad712a99c13cfddde070c6 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t 67dcc40aea7dbb2e44bb4e59406b2dd12edd1467d78b6a8c12e36eb280681c9d /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t d1db3b630e88ad2a50da92282d6a05a04f433539799dcd75d3152f44e1bd66e9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t 8a8cc7dffd8a16ccd7d34fb778432d1f58f2ba36c0004cf4a43a8fd7ec800fab /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t 754511d8bd6e20fbe1fba97aea424476b6b95ca874096d7094f9b870c1bc595f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 95b84cfd4ed12d777c32c3fb63621a27705a14d082540ebeff8ad7c1b88b0ec1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t 1a41ac7febbeefede167a4dd002379457266d5719423bd3043c288c9b16eeaee /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t c08e6f465fe9433799752bf99cb034764f818b7b5b017fadbc2c46d0729861a4 /exec failed with exit code 1
• Test TensorRT Backend / test-tensorrt-build / linux-job (gh)
  RuntimeError: Command docker exec -t be117fc65effcb675735459e1a9cc16a451c39e62e4207b3374f5cd8dc855f92 /exec failed with exit code 1
• Test TensorRT Backend / unittest-tensorrt / linux-job (gh)
  RuntimeError: Command docker exec -t 520b6c066c4520d2941c2eebaee637fe7fd4a75242cb7aea558084b0c0f803f5 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, non-quantized) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, non-quantized) / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, quantized-int4-weight-only) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.