Skip to content

Build Automodel compiled dependencies in CI image#15737

Open
pzelasko wants to merge 30 commits into
mainfrom
codex/automodel-compiled-deps
Open

Build Automodel compiled dependencies in CI image#15737
pzelasko wants to merge 30 commits into
mainfrom
codex/automodel-compiled-deps

Conversation

@pzelasko
Copy link
Copy Markdown
Collaborator

@pzelasko pzelasko commented May 29, 2026

Summary

  • Add a compiled-deps wheel stage to docker/Dockerfile.ci for NeMo Automodel packages.
  • Build and install TransformerEngine, DeepEP V1, nv-grouped-gemm, causal-conv1d, mamba-ssm, and flash-attn into CI Dockerfile images.
  • Add GPU_TARGET with default h100plus and a100 option; A100 applies a DeepEP V1 patch to disable NVSHMEM.
  • Keep CI resource settings and Docker runtime resource limits unchanged.

Validation

  • bash -n over Dockerfile heredoc snippets
  • git diff --check
  • docker buildx build --call=check -f docker/Dockerfile.ci .
  • Local constrained builds/import smoke tests were run before the final review cleanup for both GPU_TARGET=h100plus and GPU_TARGET=a100.

chtruong814 and others added 28 commits May 13, 2026 13:40
@chtruong814
Move requirements metadata to pyproject
882084f 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update CI Dockerfile to install with uv
2e93ef9 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Remove legacy requirements files
0d53a71 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Remove legacy Docker dependency installer
fb05691 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix container build
611aa64 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix nemo install
fd0cce2 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update lhotse minimum version
419ad1c 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Omit PyTorch synthetic module from coverage
5eb646d 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Mark test_speaker_target_setting as broken
8c5a48e 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Revert "Mark test_speaker_target_setting as broken"
44e56e4 
This reverts commit 8c5a48e.

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Use cu12 in docker CI
7f3a9f9 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Set back to cuda 13 install
327708f 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update to pytorch 2.12 and cuda dl base image
4ec47c9 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Merge remote-tracking branch 'origin/main' into chtruong/uv-lock
7612de7 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Try torch 2.10
697544c 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Add deps
186033e 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update uv lock file
085a343 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix install
69d4607 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Add speech CI build ID label
bcc4980 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@pzelasko
Fix SpeechLM2 automodel data parallel rank
f72d5ab 
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
@pzelasko
Fix lint failures after dependency migration
88de033 
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
@pzelasko
Merge branch 'main' into chtruong/uv-lock
fa5e5f9 
Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
@pzelasko
Fix ASR CUDA graph transcribe workers
074a90a
@pzelasko
Fix functional model support tests
5034b01
@pzelasko
Use CUDA 13.2 PyTorch wheels
eca699c
@pzelasko
Fix ASR CUDA graph fallback errors
1eca063
@pzelasko
Fix TDT model support vocab targets
a360c2f 
Use decoder vocab size when generating synthetic TDT transcript labels so duration outputs from the joint are not sampled as labels.

Move CUDA graph compile exception types into cuda_python_utils per review feedback.

Signed-off-by: Piotr Żelasko <pzelasko@nvidia.com>
@pzelasko
Build Automodel compiled dependencies in CI image
aecace9
@pzelasko pzelasko requested a review from a team as a code owner May 29, 2026 16:57
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 29, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Base automatically changed from chtruong/uv-lock to main May 29, 2026 19:35
@pzelasko
Merge remote-tracking branch 'origin/main' into codex/automodel-compi…
972a8ef 
…led-deps

# Conflicts:
#	docker/Dockerfile.ci
@github-actions github-actions Bot added the ASR label May 29, 2026
@pzelasko
Copy link
Copy Markdown
Collaborator Author

pzelasko commented May 30, 2026

/ok to test 14c7818

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 30, 2026

[🤖]: Hi @pzelasko 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ASR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pzelasko @chtruong814