[executorch][nvidia][tensorrt][25/n] Share CUDA stream across TRT delegates

[shoumikhin]

Stack from ghstack (oldest at bottom):

• -> [executorch][nvidia][tensorrt][25/n] Share CUDA stream across TRT delegates #17936
• [executorch][nvidia][tensorrt][24/n] Add power and unary converters with model improvements #17935
• [executorch][nvidia][tensorrt][23/n] Add benchmark infrastructure #17934
• [executorch][nvidia][tensorrt][22/n] Add correctness tests #17933
• [executorch][nvidia][tensorrt][21/n] Add comparison and slice converters #17932
• [executorch][nvidia][tensorrt][20/n] Add bmm converter #17931
• [executorch][nvidia][tensorrt][19/n] Add SDPA converter #17930
• [executorch][nvidia][tensorrt][18/n] Add layer_norm and pixel_shuffle converters #17929
• [executorch][nvidia][tensorrt][17/n] Add embedding, expand, upsample converters #17928
• [executorch][nvidia][tensorrt][16/n] Enable MobileNetV3 model support #17927
• [executorch][nvidia][tensorrt][15/n] Add linear model support with addmm and permute_copy converters #17926
• [executorch][nvidia][tensorrt][14/n] Add additional converters (mul, sub, div, mm, relu) #17925
• [executorch][nvidia][tensorrt][13/n] Add examples, C++ runner and CI workflow #17924
• [executorch][nvidia][tensorrt][12/n] Add C++ runtime infrastructure #17923
• [executorch][nvidia][tensorrt][11/n] Complete preprocess integration with serialization #17922
• [executorch][nvidia][tensorrt][10/n] Add blob serialization format with I/O metadata #17921
• [executorch][nvidia][tensorrt][9/n] Implement preprocess basic implementation #17920
• [executorch][nvidia][tensorrt][8/n] Add TensorRTCompileSpec dataclass #17919
• [executorch][nvidia][tensorrt][7/n] Add converter utils and add converter #17918
• [executorch][nvidia][tensorrt][6/n] Add converter registry infrastructure #17917
• [executorch][nvidia][tensorrt][5/n] Integrate partitioner #17916
• [executorch][nvidia][tensorrt][4/n] Add operator support base class #17915
• [executorch][nvidia][tensorrt][3/n] Add TensorRT partitioner stub #17914
• [executorch][nvidia][tensorrt][2/n] Add TensorRT backend stub #17913
• [executorch][nvidia][tensorrt][1/n] Add directory structure and initialization #17912

Share a single CUDA stream across all TensorRT delegate instances instead of creating a per-delegate stream. This improves performance for serialized execution (the common case) by eliminating synchronization overhead between subgraphs.

Internal:
Addresses feedback from D93275039.

Differential Revision: D93778115

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17936

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 51 New Failures, 7 Unrelated Failures

As of commit 8938dec with merge base 01d21fa ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner (gh)
  >>> Lint for backends/nvidia/tensorrt/runtime/TensorRTExecutor.h:
• Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, non-quantized) / linux-job (gh)
  RuntimeError: Command docker exec -t 2239f7ff88109bfb6d66c9b3da906f6e2b7d5812adf09e999cfbe349e5c09a64 /exec failed with exit code 1
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command docker exec -t 2d83e8ebf3b29f6d448ea33b96dda328334b53eee2e45e35995196dbd1c9932e /exec failed with exit code 1
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-small, non-quantized) / linux-job (gh)
  RuntimeError: Command docker exec -t 7924610af3c1d235f73a75f012a9131f3a7808514d797a0beeca257fc71d38b6 /exec failed with exit code 1
• Test CUDA Builds / test-models-cuda (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t ca7a8c0179386764acb16359431fa689f5f95b7d3ae6c08a6285505f8c2fb27d /exec failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (nvidia, parakeet-tdt, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / export-model-metal-artifact (openai, whisper-small, non-quantized) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• Test Metal Backend / test-metal-backend-modules / macos-job (gh)
  test_sdpa_strided_float32_output_consistency
• Test TensorRT Backend / check-all-tensorrt-tests (gh)
  ERROR: TensorRT build test failed!
• Test TensorRT Backend / test-export / linux-job (gh)
  RuntimeError: Command docker exec -t 3b9172f2c8c042717aaaef1c2775b91461fe5aa93e7cd2b82214d662daf0e0c4 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 7ab55f7f4111267b87ac3b065ec76ea592db22e2f596fac7b00c42d27bc53b3c /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (add) / linux-job (gh)
  RuntimeError: Command docker exec -t 455d090956ff6462ede85c911c0b9fed3a4abddce93747a0c7c657c305ed6707 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t f933247bef0f7f44fbca5a523e2606e6c3e4414abf8599f4dcc2a6bbb710dbcf /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t b77c6edbf0864d0f20ed8df58321de804bc6fa7facf5f8485ec7b502450c2001 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t a72c82260b5e08556c086c05b61e3eae30c362711ca0d0286e22ad43f08e94eb /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t af77f5391c5aefccaa0f8062df3ec201986c239e5d1572fbc9ef1a15f9a697ec /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t 91250c3ce128838fd2ea16b6fd8e52a974972999ce2d6a394fa0da304fa654f4 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t ea3f20db1856e56a4c28c184f021665a94ff20ebe1b6143b40713cecfe22e0cf /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 0618b96f45d59b567caef97dd076181b9d841f49d8c253370050d629af0a8f77 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t a89dac24a5b8d243f1ff1bc4231692ce6aea0fbb7d386469f2b9659995c0074b /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 5301342dfd40e2982bdfdaa67791b72eab9f85765175b3173bf1a21d3c2ae73a /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 0fa6b02b42d7d96560a38c80d4829177c4fc0bb723aa29b6f9374aa8444abda4 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t 7354af4504c388c13f9ffd16d99695b4b303b5fe0114cdf189a9085f998ba6d1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t 72df92f7e168c8dfd782a9e0b3f36da9ecf8d3e5ccdbbfd1c05133c23ca0b7f3 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t f2d3bb9c5c9a7c1260a8a5bbeef2048105443878a00bc52197ff5a536eb8523b /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t 36fe63b5d30e63ec9219822a4b0e5e46bb36356d8a0c59d785ee732f8f56f12e /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 88515746207dcf994e5d299959010fee692e20af6267494cfcddbcf77d3f2e0a /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t 78265b9f3389010f35519a8cf7c5ca3be7c844b70a7fdcf558de635ed97d5006 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-cpp (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t 21c41ea69742a51ffd73078be9dad7bf29084e220a539d03b45c5218ecd8fda8 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add_mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 6a4e8afa2b23f1420521ab2311faae76ca8f638f21e3f4788d192203b598e6fe /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (add) / linux-job (gh)
  RuntimeError: Command docker exec -t aba39f86bbe5e0773781bd26c9af47869ddf87072014c5a871a762490be45265 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (conv1d) / linux-job (gh)
  RuntimeError: Command docker exec -t 511aadcf807e39c02a1d8a44e205b69a9aa62b235d30f93ae958c43a63d0c52c /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (dl3) / linux-job (gh)
  RuntimeError: Command docker exec -t c39719496086d1d7a57269fa03510a536a932d92c4e12d645a52519f2210d9f1 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (edsr) / linux-job (gh)
  RuntimeError: Command docker exec -t 0da794e03986b2ba7b4bdae9378f55f17c5a2663f683a41ec8c546d0ee92da0f /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (efficient_sam) / linux-job (gh)
  RuntimeError: Command docker exec -t d3c54d8e1aba38c407364d555ce02c927deea0f245c863e73070d8608ef4d5ca /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_join) / linux-job (gh)
  RuntimeError: Command docker exec -t df18d722a6d01717dbf9b4499b4eba26c12e51389a6253fbd787e97811ddc4d7 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (emformer_transcribe) / linux-job (gh)
  RuntimeError: Command docker exec -t f1a681a2c5c3c4ba409ea0a6f97366e52ab66a724447e32c9c515e5da2edcbbf /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic3) / linux-job (gh)
  RuntimeError: Command docker exec -t 6c2d104bc534a55d071e0fb9ae527ca5361b5097d4dbf5a6c5f5bcebf557f1ec /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (ic4) / linux-job (gh)
  RuntimeError: Command docker exec -t 7edb66caad8daa16e98b073c1df66548c25959def76b5d864991f5149f66d927 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (linear) / linux-job (gh)
  RuntimeError: Command docker exec -t 69b2fac912419febd18510df688ac461b3fd3a4542c594ea45ab8eaf3b40bea9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mul) / linux-job (gh)
  RuntimeError: Command docker exec -t 9bb42b0347eb4a089801d6a202511b0cc3887b1243d625d110e3c6e0e25a2dbd /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv2) / linux-job (gh)
  RuntimeError: Command docker exec -t b52457ebd7ec727b742761d8fb42916b3c2f522224768e0a6150fecae9a75929 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (mv3) / linux-job (gh)
  RuntimeError: Command docker exec -t ded41b343e55612858ef27b572d4fea4c9a01d8415b59c7e68ef8466f590fd84 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet18) / linux-job (gh)
  RuntimeError: Command docker exec -t b117ff38b8ed43dbc74a826851758811a88a2f263eed3b94074f32d075444cdc /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (resnet50) / linux-job (gh)
  RuntimeError: Command docker exec -t a650af4f4371b4d41ffc2f660a1cd69ada0018a032312613f85cb4a36ef73267 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (sdpa) / linux-job (gh)
  RuntimeError: Command docker exec -t 0264e8c68638e33de04191408c73beae00b4d34b51275296afe1211b79e344a9 /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (softmax) / linux-job (gh)
  RuntimeError: Command docker exec -t 7ab50252bd891ec145d2ac0388ba4c2b7a5f2c91a3be1727edccdb7b388d40df /exec failed with exit code 1
• Test TensorRT Backend / test-models-tensorrt-python (w2l) / linux-job (gh)
  RuntimeError: Command docker exec -t 5d84e8e1ade94c389f0f14032ca18a8ecd774598a4fd87f30daa57db57156bad /exec failed with exit code 1
• Test TensorRT Backend / test-tensorrt-build / linux-job (gh)
  RuntimeError: Command docker exec -t 9c77fb7daaa22bd544e76464cd102fd29fc4de45c25081034153e70bcb2294be /exec failed with exit code 1
• Test TensorRT Backend / unittest-tensorrt / linux-job (gh)
  RuntimeError: Command docker exec -t 79915b86758571774cae12850818c42adf339eb2fdd935a77a0b2c2bce1a01dd /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, non-quantized) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, non-quantized) / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, quantized-int4-weight-only) / windows-job (gh) (matched win rule in flaky-rules.json)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 'C:\actions-runner\_work\executorch\executorch\test-infra\.github\actions\teardown-windows'. Did you forget to run actions/checkout before running your local action?

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.