[executorch][nvidia][tensorrt][23/n] Add benchmark infrastructure

[shoumikhin]

Stack from ghstack (oldest at bottom):

• [executorch][nvidia][tensorrt][25/n] Share CUDA stream across TRT delegates #17936
• [executorch][nvidia][tensorrt][24/n] Add power and unary converters with model improvements #17935
• -> [executorch][nvidia][tensorrt][23/n] Add benchmark infrastructure #17934
• [executorch][nvidia][tensorrt][22/n] Add correctness tests #17933
• [executorch][nvidia][tensorrt][21/n] Add comparison and slice converters #17932
• [executorch][nvidia][tensorrt][20/n] Add bmm converter #17931
• [executorch][nvidia][tensorrt][19/n] Add SDPA converter #17930
• [executorch][nvidia][tensorrt][18/n] Add layer_norm and pixel_shuffle converters #17929
• [executorch][nvidia][tensorrt][17/n] Add embedding, expand, upsample converters #17928
• [executorch][nvidia][tensorrt][16/n] Enable MobileNetV3 model support #17927
• [executorch][nvidia][tensorrt][15/n] Add linear model support with addmm and permute_copy converters #17926
• [executorch][nvidia][tensorrt][14/n] Add additional converters (mul, sub, div, mm, relu) #17925
• [executorch][nvidia][tensorrt][13/n] Add examples, C++ runner and CI workflow #17924
• [executorch][nvidia][tensorrt][12/n] Add C++ runtime infrastructure #17923
• [executorch][nvidia][tensorrt][11/n] Complete preprocess integration with serialization #17922
• [executorch][nvidia][tensorrt][10/n] Add blob serialization format with I/O metadata #17921
• [executorch][nvidia][tensorrt][9/n] Implement preprocess basic implementation #17920
• [executorch][nvidia][tensorrt][8/n] Add TensorRTCompileSpec dataclass #17919
• [executorch][nvidia][tensorrt][7/n] Add converter utils and add converter #17918
• [executorch][nvidia][tensorrt][6/n] Add converter registry infrastructure #17917
• [executorch][nvidia][tensorrt][5/n] Integrate partitioner #17916
• [executorch][nvidia][tensorrt][4/n] Add operator support base class #17915
• [executorch][nvidia][tensorrt][3/n] Add TensorRT partitioner stub #17914
• [executorch][nvidia][tensorrt][2/n] Add TensorRT backend stub #17913
• [executorch][nvidia][tensorrt][1/n] Add directory structure and initialization #17912

Add benchmark runners and infrastructure for measuring TensorRT inference performance including latency and throughput metrics.

Differential Revision: D93275038

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17934

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 56 New Failures, 2 Unrelated Failures

As of commit b7b5e9e with merge base 01d21fa ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner (gh)
  >>> Lint for examples/nvidia/tensorrt/tests/test_export.py:
• Lint / lintrunner-mypy (gh)
  >>> Lint for install_utils.py:
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• pull / unittest / macos / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• pull / unittest-buck / macos / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• pull / unittest-editable / macos / macos-job (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/Users/ec2-user/runner/_work/executorch/executorch/test-infra/.github/actions/check-disk-space'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Builds / check-all-cuda-builds (gh)
  Process completed with exit code 1.
• Test CUDA Builds / test-executorch-cuda-build-12.9 / linux-job (gh)
  RuntimeError: Command docker exec -t ae227bc4810ad99691189198dbe5c6093de9205f363d3a0bedd89be3e7e3878a /exec failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (nvidia, parakeet-tdt, quantized-int4-metal) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-small, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-small, quantized-int4-metal) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / test-metal-backend-modules / macos-job (gh)
  test_sdpa_strided_float32_output_consistency
• Test TensorRT Backend / check-all-tensorrt-tests (gh)
  ERROR: TensorRT build test failed!
• Test TensorRT Backend / test-export / linux-job (gh)
  RuntimeError: Command docker exec -t 721304dc53706b8147d97e6ffd92c4ffcbeccb4e9234d5ca280bebba86ad4e8b /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 1dac26a0f5752fdb348d428335d5320ca5c08203bb7eb85706ccef05962df211 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add) / linux-job (gh)
  RuntimeError: Command docker exec -t 4687adcb42ce168e43832051644c0eac88711163e19d3953ccd8f7c2bf8c88c9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t 35150754b8e60770d57e68053d14eb5b96a63bd94758a12b3d98e5b8a452c2da /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t 7829f7388b918ea4cd071db20ac74176a9911495a358d25cf804a307d3470eba /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t 23ec3f5737dfa17bf465db3060cf73fc0f98f2fb89f11590e4616bd77ac11673 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t 76ba6a43d908e3cec7fa73bdb8deaab8d35de8b10b7232e2378d4b5d4b099433 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t cbe192d575e0649834262fa72d8f4b849863e5a3ff158c0f9430fc950ace66ae /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t b4be9a2777cfccbfcdd324b66bedb7987b7a19d3204941b5d51eb7e6a2116d27 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 4e50b1a7becc37636c957d11b27f9c6f0e7e667f4f44dcd43b07fe9aee45743a /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t 415d7258a53ced187b53a62e8e9a66726d7b394f2630363356c681b4c9621e16 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 3acf526c84ace8a86f446e6995fe4078cc895da6a95f3dddab1d3565358740eb /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 9cbfb0d6b3e4ad8a6cb6e88059dcc86c64db1cd473d8a58bf954c3917591e000 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t 30d6bbce5c30a0819ff953582c4c78acd0a9f7242fd2921807a81ba2a8d000f0 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t eb1ff8efebd6e781fec8fb1e5198f568d2e2268e7e754ffcae5a38a02f97af8e /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t de6d8e697ddc70b943cf2015bd7c2da645d48e185fa67885d97ad5aa6984da0b /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t 0551af0f2a214337d74ec9ecc717c9deecb0370fdc27cf97d564e11d07042f85 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t f31f4a6ad7ecb33342996d850a0d1251ceb408e02264b07b9a4d371a412bd952 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t b049b9332cfbcdf4e4fa43e2a1e6c05a9a7b49dfc9777235ad722a4c2f790e89 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t 6a99f728931c5227e69009e616410181506914eaf4acbd3a0b1a89f9a055d17a /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 6335ad25b839eb6fc7f359c7cd4232145625a07515f4afda918345fb88c7fc79 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add) / linux-job (gh)
  RuntimeError: Command docker exec -t 13425f12c941f87bf93dacec517ea4a640ec5aae5a5eb427ec1520531b3a5094 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t 9a7404ea9ce9b120ab8e00a69a6339b2af4835ba802d8b3bb1101213cf826e99 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t 2463895a7c2a1387e643c5b14b62dfa69e6a696db45c9c33eccecac8d84534c0 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t 0705eee73354d36cbfc43b40867ce9ff534e3fbe9072551a425c333599f8d092 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t 5787d308f3e1a85c861d6064121ff5ad77333ec753158b765814429ad30926e1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t 7a4831503d3e7ee6335906d7f500612804c3d3fc9ce3737d0da282da1a4df6d2 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t 2583f922bc8210871dc0ef29c572b68625dcabb7f592e58ab731200044f97f3e /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 21970357f783e09ff9953bc521946831793ad0ee7fababda6f52468c29478d97 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t 4aca066a44bc1191bbee995c36f6b5948b4fa73ff4228257a64eab666cca5525 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 3777c71542004ca27cc6becedebb8d409c73fee62abd99061b4a8f369dc826fe /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 503414c511f3f3b802ead4bf80324789ca3b8a3bfbba396ae8d70b647870fda7 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t dc5a179cd01139f09affb6962bbf440f831e681048cbab18e269f05b9b07d4e9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t f6cac241f4115ed89d5484565dc91b665d14cfccfb890ff71874a68df5c2397f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t 0536857046c40b1dbef2dbd008fdf23a75b7fb4485a21504def2e1e2785e769f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t b2ca58321e5626f8942f4c7c91eec04ea1df69fe765a93ca27f7d8f60a9f4b3f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 60a8598e7700afa21e2e7c5b094c201a2d97ba724d5cb752557fd6b565812688 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t 117f380babee19b68dae406240903dca4c9e0777475a66d616838bf1a5e710ef /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t 0ce199219ba96a5454f2f6a6e4c116e690faa19de24688b6a56cc13e9f153209 /exec failed with exit code 1
• Test TensorRT Backend / test-tensorrt-build / linux-job (gh)
  RuntimeError: Command docker exec -t 41d0647024b2b7a33ec8f1463047a8bbad5f78b470820f87cfcba73d61a4afe0 /exec failed with exit code 1
• Test TensorRT Backend / unittest-tensorrt / linux-job (gh)
  RuntimeError: Command docker exec -t 7b61c78e6b523ac5ce3b5d9842a78e0cb6a44e216d1490273373ba500c99b376 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, quantized-int4-weight-only) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.