config: switch MI355X DSv4.1 Flash to the ROCm nightly image / MI355X DSv4.1 Flash 切换到 ROCm nightly 镜像 - #3035
config: switch MI355X DSv4.1 Flash to the ROCm nightly image / MI355X DSv4.1 Flash 切换到 ROCm nightly 镜像#3035Fangzhou-Ai wants to merge 5 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
thanks fangzhou! can u change the docs such that it isnt pinned to the custom image before we merge to nightly
The deepseekv41-flash-0909 tag predates vllm-project/vllm#56503, which moves the mHC delayed pre block off the eager Torch reference and onto AITER. That block is 85% of the decoder's kernel launches for this model, so the release tag leaves most of the decode cost unaddressed. Tag is nightly-TBD until the first nightly containing that commit is published; it is then pinned to nightly-<sha> like the other ROCm entries.
AITER's Triton MoE GEMM logs "GLUON backend not available. Using TRITON backend!!!" on every call. Gluon supports only gfx1250, so on gfx950 the message reports a fixed property of the hardware rather than a condition, and it dominates the server log: 410,984 of 417,504 lines in a gsm8k run, 29.8 MiB of 32 MiB. Set AITER_TRITON_LOG_LEVEL=ERROR, the knob aiter.ops.triton.utils.logger reads. Every warning that module emits is about Gluon availability -- the two Gluon GEMM depth fallbacks are inside gfx1250-only paths, and fused_mxfp4_quant warns only when Gluon is requested explicitly -- so raising the threshold hides nothing actionable on this architecture. Log hygiene rather than performance: the emits run at 61/s per worker and 4.33 us each, or 0.026% of wall time. Signed-off-by: Fangzhou Ai <fangzhou@semianalysis.ai> Co-authored-by: Cursor <cursoragent@cursor.com>
The 2*CONC cap sat below AgentX's subagent fan-out. At CONC=1 the engine admitted 2 requests and queued the rest on scheduling capacity, with vLLM reporting queue times up to 16.2 s. Removing the cap takes queue time to zero at identical output.
d904975 to
420322c
Compare
The nightly containing vllm-project/vllm#56503 is out: nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 (sha256:960228cf..., published 2026-09-12). Replace the nightly-TBD placeholder in the config, recipe header, and both docs, and record the upstream recipe PR that moves vllm-project/recipes to the same image. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Nightly published — Upstream companion: vllm-project/recipes#962 moves the AMD image in Placeholder replaced in nightly 已发布，占位符已替换为真实标签；上游配方同步 PR 见 vllm-project/recipes#962。退出 draft 前仍需在该镜像上完成 GPU 验证。 |
|
Claude finished @Fangzhou-Ai's task in 1m 50s —— View job Review of PR #3035
LGTM - no blocking issues found. Checks performed: Two things I could not verify from this sandbox (no network): that the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because configs/amd-master.yaml is CODEOWNERS-protected, the PR description itself says it's still draft pending GPU validation on this exact image, and an earlier CHANGES_REQUESTED review from functionstackx has no visible re-review resolving it, a human look would still be worthwhile.
What was reviewed: the image tag switch in configs/amd-master.yaml and the benchmark script (vllm/vllm-openai-rocm:deepseekv41-flash-0909 → nightly-eed1f3d0c6043bd494424a22443ee198dd56f657), the new AITER_TRITON_LOG_LEVEL=ERROR env var and the MAX_NUM_SEQS=2*CONC → fixed 128 change (both cross-checked against the ruled-out candidates already listed), and the matching docs/configuration-procedures*.md / perf-changelog.yaml updates for consistency with the config.
Extended reasoning...
Overview
This PR is a config-only change: it swaps the MI355X DeepSeek-V4.1-Flash vLLM image from a release tag (deepseekv41-flash-0909) to a pinned ROCm nightly (nightly-eed1f3d0c6043bd494424a22443ee198dd56f657) in configs/amd-master.yaml and the corresponding benchmark script comment, adds a new AITER_TRITON_LOG_LEVEL=ERROR env var to suppress a known-noisy warning, replaces the MAX_NUM_SEQS=2*CONC scheduler cap with a fixed upstream default of 128, and updates the English/Chinese docs and perf-changelog.yaml to match. I diffed all five files against the merge base and cross-checked the two candidate issues already surfaced (the AITER_TRITON_LOG_LEVEL vs. AITER_LOG_LEVEL naming used elsewhere in the repo, and the CAPTURE_SIZE growth from the now-fixed MAX_NUM_SEQS), both of which the run's bug hunter already investigated and ruled out.
Security risks
None. This is entirely benchmark/config plumbing — an image tag, an env var, a numeric scheduler cap, and prose documentation. No auth, crypto, injection, or data-exposure surface is touched.
Level of scrutiny
Two factors push this above a rubber-stamp approval despite the change being mechanical. First, configs/amd-master.yaml is listed in .github/CODEOWNERS under a specific owner group (billishyahao, chunfangamd, seungrokj, yctseng0211, 1am9trash), and per the approval guidelines a CODEOWNERS-protected path should get a human look rather than an automated approval. Second, the PR's own description states it is "Still draft pending GPU validation on this exact image" — the quoted performance numbers were measured against an approximation of the final image (nightly + two specific patches applied by hand), not the exact pinned digest, so the core claim of the PR (that this image works and is faster) has not yet been independently confirmed on the pinned tag. There is also a CHANGES_REQUESTED review from functionstackx in the timeline with no subsequent re-review visible; while five commits followed it (plausibly addressing it), I cannot confirm from the metadata alone that the specific requested changes were satisfied.
Other factors
No test suite covers config YAML or benchmark shell scripts beyond the shape checks already implied by the repo's tooling, so there's little automated safety net beyond human eyes. Given the combination of CODEOWNERS ownership, explicit draft/pending-validation status, and an unresolved third-party review, I'm deferring rather than approving, even though the bug hunt itself found nothing.
Follow-up to #3035, which pins the ROCm nightly image. This one is independent of that pin and targets main directly; the two can merge in either order. The recipe pinned --moe-backend aiter_triton_mxfp4_bf16, which names a single W4A16 entry and dispatched _moe_gemm_a16w4. Passing plain aiter opens vLLM's priority list, whose head is the CK backend, and CK quantizes activations to FP8 internally: the decode trace switches to mfma_moe1_silu_mul_afp8_wfp4_bf16 and mfma_moe2_afp8_wfp4_bf16, the same a8w4 expert family the DSV4-Pro MI355X recipe already runs. The BF16 in the backend name and this checkpoint's activation_scheme=dynamic both suggest otherwise, hence the comment at the flag. Measured on one MI355X half, TP4, 131k context, CUDA graphs on, both arms captured on the same four GPUs under identical profiling overhead: conc MoE GEMM ms/step whole step ms 1 2.72 -> 2.30 19.11 -> 18.31 (4.2%) 4 4.91 -> 4.27 19.42 -> 18.40 (5.3%) 16 9.06 -> 7.18 28.38 -> 25.92 (8.7%) AgentX replay at conc 1, each arm solo on a quiet node, improves mean ITL from 4.01 to 3.91 ms and mean TTFT from 681 to 639 ms. gsm8k is unchanged: 0.9719 +/- 0.0045 strict-match, 0.9712 +/- 0.0046 flexible, identical to the Triton arm. Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34676892125 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34676892125 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34676892125 |
2 participants
Switch the MI355X DeepSeek-V4.1-Flash recipe from the
deepseekv41-flash-0909release tag to the upstream ROCm nightly, so it picks up vllm-project/vllm#56503 (merged 2026-09-12, commit46d2b23). That PR moves the mHC delayed pre block off the eager Torch reference and onto AITER; on this model the mHC block is 85% of the decoder's kernel launches, so the0909tag leaves most of the decode cost unaddressed.The nightly is now published, so the earlier
nightly-TBDplaceholder is replaced with the real pin:vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657sha256:960228cfcb5de9f4cd22d28998d1125be62b546c3d220a884f570343e99ffceeeed1f3dis 11 commits ahead of, and 0 behind, #56503's merge commit, so the change is included. The upstream vLLM recipe is being moved to the same image in vllm-project/recipes#962, which keepsrecipes.vllm.aiandamd-master.yamlpointing at one tag.Still draft pending GPU validation on this exact image. Rebased onto
mainnow that #2962 has merged.Measured on MI355X (8x gfx950, TP4, fp4, five-token DSpark) against the 2026-09-11 ROCm nightly with #56214 and #56503 applied, which is what the new image contains:
deepseekv41-flash-0909Decode is 6.6x faster at an identical pinned
synthetic_acceptance_lengthof 3.51, and accuracy is within one standard error, so this is a kernel-cost change rather than a numerics change. Uncached prefill measures 27,273 tok/s at 16k context and 22,012 tok/s at 128k.Also raises
max_num_seqsto the upstream default of 128. The old2*CONCcap was below AgentX's subagent fan-out, so requests queued instead of being admitted — vLLM reported queue times up to 16.2 s at CONC=1, and removing the cap takes them to zero. Verified by an A/B over the same 393 traces, same seed and duration, one arm per GPU half: identical output token counts, TTFT max 9,003 ms to 4,263 ms, no regression elsewhere.将 MI355X 的 DeepSeek-V4.1-Flash 配方镜像从
deepseekv41-flash-0909发布标签切换到上游 ROCm nightly，以纳入 vllm-project/vllm#56503（2026-09-12 合入，提交46d2b23）。该 PR 将 mHC delayed pre 块从 eager Torch 参考实现切换到 AITER；该模型中 mHC 块占解码器内核启动数的 85%，因此0909标签会浪费大部分解码开销。nightly 已发布，此前的
nightly-TBD占位符已替换为真实标签vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657（摘要sha256:960228cf...，发布于 2026-09-12T05:27:48Z）。eed1f3d比 #56503 的合并提交领先 11 个提交、落后 0 个，因此已包含该改动。上游 vLLM 配方正在 vllm-project/recipes#962 中切换到同一镜像，使recipes.vllm.ai与amd-master.yaml指向同一标签。仍保持 draft，等待在该镜像上完成 GPU 验证。 #2962 已合入，本 PR 已 rebase 到
main。在 MI355X（8x gfx950、TP4、fp4、五 token DSpark）上实测：并发 1 时 token 间延迟 p50 从 25.58 ms 降至 3.85 ms，单用户输出吞吐从 39.1 提升到 260.0 tokens/s，聚合输出吞吐从 35.9 提升到 132.9 tokens/s，冷前缀 TTFT 中位数从 10,236 ms 降至 1,440 ms，gsm8k strict-match 为 0.9719 ± 0.0045（基线 0.9704）。在相同固定的
synthetic_acceptance_length3.51 下解码快 6.6 倍，准确率变化在一个标准误内，说明这是内核开销变化而非数值变化。未缓存 prefill 在 16k 上下文为 27,273 tok/s，128k 为 22,012 tok/s。同时将
max_num_seqs提升到上游默认值 128。此前的2*CONC上限低于 AgentX 的子代理扇出，导致请求排队而非被接纳——vLLM 在 CONC=1 时报告排队时间最高达 16.2 秒，取消该上限后归零。已通过 A/B 验证：相同的 393 条 trace、相同随机种子与时长，每个 GPU 半区一组，输出 token 数完全一致，TTFT 最大值从 9,003 ms 降至 4,263 ms，其余指标无回退。Note
Medium Risk
Pins a new serving image and changes vLLM admission limits for a benchmark recipe, which materially affects measured throughput/latency and sweep comparability until GPU validation completes.
Overview
Updates the MI355X DeepSeek-V4.1-Flash AgentX DSpark recipe (
dsv41flash-fp4-mi355x-vllm-agentic-dspark) fromvllm/vllm-openai-rocm:deepseekv41-flash-0909to pinned upstream ROCm nightlynightly-eed1f3d0c6043bd494424a22443ee198dd56f657, so runs pick up vllm#56503 (mHC delayed pre on AITER instead of eager Torch).The benchmark script sets
MAX_NUM_SEQS=128(replacing2*CONC) so vLLM can admit AgentX subagent fan-out without multi-second queue times at low concurrency, and exportsAITER_TRITON_LOG_LEVEL=ERRORto suppress repetitive Gluon-unavailable Triton warnings on gfx950 (log hygiene only).amd-master.yaml, EN/ZH configuration docs, andperf-changelog.yamlare aligned with the new image pin and scheduler change. GPU validation on this exact image remains pending.Reviewed by Cursor Bugbot for commit 9fbd0ab. Bugbot is set up for automated code reviews on this repo. Configure here.