perf: MI355X DSv4.1 Flash on the ROCm nightly image with the CK a8w4 MoE experts / MI355X DSv4.1 Flash 切换 ROCm nightly 镜像并选择 CK a8w4 MoE 专家内核 - #3058
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Switch --moe-backend from aiter_triton_mxfp4_bf16 to aiter, which selects the CK a8w4 experts (mfma_moe1_silu_mul_afp8_wfp4_bf16 / mfma_moe2_afp8_wfp4_bf16) instead of the W4A16 _moe_gemm_a16w4 Triton kernel, matching the DSV4-Pro MI355X recipe. Decode traces at 131k context show MoE GEMM falling from 2.72 to 2.30 ms/step at conc 1, 4.91 to 4.27 at conc 4 and 9.06 to 7.18 at conc 16, for whole-step gains of 4.2%, 5.3% and 8.7%. AgentX conc 1 improves mean ITL 4.01 to 3.91 ms and mean TTFT 681 to 639 ms; gsm8k is unchanged at 0.9719 strict-match." | ||
| - "将 --moe-backend 从 aiter_triton_mxfp4_bf16 改为 aiter，从而选择 CK a8w4 专家内核（mfma_moe1_silu_mul_afp8_wfp4_bf16 / mfma_moe2_afp8_wfp4_bf16），取代 W4A16 的 _moe_gemm_a16w4 Triton 内核，与 DSV4-Pro MI355X 配方保持一致。131k 上下文的解码 trace 显示 MoE GEMM 在并发 1 时从 2.72 降至 2.30 ms/step，并发 4 从 4.91 降至 4.27，并发 16 从 9.06 降至 7.18，整步分别加速 4.2%、5.3% 和 8.7%。AgentX 并发 1 的平均 ITL 从 4.01 降至 3.91 ms，平均 TTFT 从 681 降至 639 ms；gsm8k strict-match 保持 0.9719 不变。" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING |
There was a problem hiding this comment.
🔴 pr-link uses pull/PENDING, which is neither the canonical pull/<number> pattern nor a recognized placeholder (XXX / pull/XXX), so validate_added_pr_link in infx/workflows/validate_perf_changelog.py will raise ChangelogValidationError and fail the changelog-gate CI check, blocking merge until fixed. Fix: use the XXX placeholder convention (or pull/XXX) that the validator and prepare_perf_changelog_merge.py actually recognize and auto-canonicalize on merge.
Extended reasoning...
infx/workflows/validate_perf_changelog.py defines PR_LINK_PLACEHOLDERS = {"XXX", ".../pull/XXX"} and CANONICAL_PR_LINK requiring \d+; validate_added_pr_link (called from the CI changelog gate) raises ChangelogValidationError for any link that is neither the canonical numeric link nor one of those exact placeholders. pull/PENDING matches neither, so the gate fails with 'new PR entry must use ... or an XXX placeholder; found ...'. This must be fixed before merge, and prepare_perf_changelog_merge.py's canonicalize_appended_links also only rewrites entries whose link is in PR_LINK_PLACEHOLDERS, so PENDING would also break the merge-time link substitution.
Verification: normal — perf-changelog.yaml:7422 adds pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PENDING. This is neither the canonical numeric link (CANONICAL_PR_LINK at validate_perf_changelog.py:21-23 requires /pull/\d+) nor a recognized placeholder (PR_LINK_PLACEHOLDERS at :24-27 is exactly {"XXX", ".../pull/XXX"}). validate_added_pr_link (:144-160) raises ChangelogValidationError…
6677759 to
637c3f9
Compare
The deepseekv41-flash-0909 tag predates vllm-project/vllm#56503, which moves the mHC delayed pre block off the eager Torch reference and onto AITER. That block is 85% of the decoder's kernel launches for this model, so the release tag leaves most of the decode cost unaddressed. Tag is nightly-TBD until the first nightly containing that commit is published; it is then pinned to nightly-<sha> like the other ROCm entries.
AITER's Triton MoE GEMM logs "GLUON backend not available. Using TRITON backend!!!" on every call. Gluon supports only gfx1250, so on gfx950 the message reports a fixed property of the hardware rather than a condition, and it dominates the server log: 410,984 of 417,504 lines in a gsm8k run, 29.8 MiB of 32 MiB. Set AITER_TRITON_LOG_LEVEL=ERROR, the knob aiter.ops.triton.utils.logger reads. Every warning that module emits is about Gluon availability -- the two Gluon GEMM depth fallbacks are inside gfx1250-only paths, and fused_mxfp4_quant warns only when Gluon is requested explicitly -- so raising the threshold hides nothing actionable on this architecture. Log hygiene rather than performance: the emits run at 61/s per worker and 4.33 us each, or 0.026% of wall time. Signed-off-by: Fangzhou Ai <fangzhou@semianalysis.ai> Co-authored-by: Cursor <cursoragent@cursor.com>
The 2*CONC cap sat below AgentX's subagent fan-out. At CONC=1 the engine admitted 2 requests and queued the rest on scheduling capacity, with vLLM reporting queue times up to 16.2 s. Removing the cap takes queue time to zero at identical output.
The nightly containing vllm-project/vllm#56503 is out: nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 (sha256:960228cf..., published 2026-09-12). Replace the nightly-TBD placeholder in the config, recipe header, and both docs, and record the upstream recipe PR that moves vllm-project/recipes to the same image. Co-authored-by: Cursor <cursoragent@cursor.com>
637c3f9 to
4346a82
Compare
Fangzhou-Ai
changed the title
4346a82 to
1c62aee
Compare
The recipe pinned --moe-backend aiter_triton_mxfp4_bf16, which names a single W4A16 entry and dispatched _moe_gemm_a16w4. Passing plain aiter opens vLLM's priority list, whose head is the Composable Kernel backend, and CK quantizes activations to FP8 internally: the decode trace switches to mfma_moe1_silu_mul_afp8_wfp4_bf16 and mfma_moe2_afp8_wfp4_bf16, the same a8w4 expert family the DSV4-Pro MI355X recipe already runs. The BF16 in the backend name and this checkpoint's activation_scheme=dynamic both suggest otherwise, hence the comment at the flag. Measured on one MI355X half, TP4, 131k context, CUDA graphs on, both arms captured on the same four GPUs under identical profiling overhead: conc MoE GEMM ms/step whole step ms 1 2.72 -> 2.30 19.11 -> 18.31 (4.2%) 4 4.91 -> 4.27 19.42 -> 18.40 (5.3%) 16 9.06 -> 7.18 28.38 -> 25.92 (8.7%) AgentX replay at conc 1, each arm solo on a quiet node, improves mean ITL from 4.01 to 3.91 ms and mean TTFT from 681 to 639 ms. gsm8k is unchanged: 0.9719 +/- 0.0045 strict-match, 0.9712 +/- 0.0046 flexible, identical to the Triton arm. Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
1c62aee to
ab76c13
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34702236566 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34702236566 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34706621929 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34702236566 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34710937012 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34710937012 |
|
/stage-results 34710937012 |
|
@chunfangamd staged run 34710937012: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-12~r34710937012 This run remains available across future |
|
/reuse-sweep-run 34710937012 |
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
2 participants
Supersedes #3035 and targets
maindirectly, so the MI355X DSv4.1 Flash recipe lands in one self-consistent state. #3035 is frozen and its five commits are carried here unchanged (rebased onto currentmain); this PR adds the MoE backend change on top. #3035 can be closed once this merges.Carried from #3035:
configs/amd-master.yaml: imagevllm/vllm-openai-rocm:deepseekv41-flash-0909→nightly-eed1f3d0c6043bd494424a22443ee198dd56f657, matching [DeepSeek] V4.1-Flash: point the AMD image at a ROCm nightly and select the CK a8w4 MoE experts vllm-project/recipes#962AITER_TRITON_LOG_LEVEL=ERROR, which removed 98% of the server log (411k of 417k lines) with no throughput effectMAX_NUM_SEQSfrom2 * CONCto the upstream default of 128, which took queue times at CONC=1 from 16.2 s to zeroNew here: the MoE backend change described below.
Summary
The recipe pinned
--moe-backend aiter_triton_mxfp4_bf16. That name maps to exactly one entry, the Triton W4A16 kernel, so decode ran_moe_gemm_a16w4. Passing plainaiteropens vLLM's full priority list, whose head is the Composable Kernel backend, and the experts become the a8w4 family the DSV4-Pro MI355X recipe already uses:Two things made this non-obvious, so the flag carries a comment: the backend is named
..._BF16, and this checkpoint declaresactivation_scheme: dynamicwith noinput_scaletensors. vLLM's explicit W4A8 path does require static FP8 activation scales and correctly refuses, but CK quantizes activations to FP8 inside the kernel and needs no checkpoint scales.Decode traces
Both arms captured on the same four GPUs, TP4, 131k context, CUDA graphs active, 50 engine steps, identical profiler settings. Per-rank busy spread was under 1% on both.
CK replaces one fused Triton kernel with two CK GEMMs plus its own sorting and gating kernels, so the expert-GEMM win (29-46%) is larger than the whole-step win. The whole-step column is the one that matters.
A separate steady-state decode benchmark run on the opposite GPU half agreed in direction and size (ITL 2.1% / 4.8% / 6.3% better at conc 1 / 4 / 16), which rules out GPU-half asymmetry as the explanation.
AgentX replay, conc 1
Each arm ran solo on a quiet node. A first attempt ran the two arms concurrently on the two halves and the CK lane died on a 600 s NCCL all-gather timeout inside
capture_model— two simultaneous CUDA-graph captures contend and desync the ranks. Serial is both safer and the right baseline for a latency comparison.Both arms used
AIPERF_EXPERIMENTAL_FAST=1(20-minute profiling window), so these absolute numbers are not comparable to a full-duration reference run; the delta is.Accuracy
gsm8k 5-shot at conc 8, same lane configuration as the baseline run, is unchanged to four decimal places:
Notes
Using AITER OPUS for large sparse MLA prefill on gfx950is already in every server log on this image.aiter::mhc_pre_big_fuse_rmsnorm_kernel) is not reachable here and is not part of this PR. AITER ships no delayed mHC kernel at all, so vLLM composes the delayed pre block frommhc_pre_gemm_sqrsum+mhc_pre_big_fuse+ a Triton collapse, and only the non-delayedmhc_pretakes anorm_weight. Closing that gap needs a new AITER kernel; the ceiling looks like 2-3% of decode.Note
Medium Risk
Touches serving image, MoE kernel selection, and scheduler limits for a production AgentX recipe; accuracy was reported unchanged on gsm8k but broader GPU validation is still marked pending.
Overview
Updates
dsv41flash-fp4-mi355x-vllm-agentic-dsparkso MI355X AgentX serving matches the intended upstream ROCm stack and avoids AgentX scheduling bottlenecks.Container and docs:
configs/amd-master.yamland the benchmark script now pinvllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657instead ofdeepseekv41-flash-0909, because the release tag predates vLLM #56503 (mHC delayed pre on AITER). English and Chinese configuration docs reflect the new image and MoE flag.Benchmark script (
dsv41flash_fp4_mi355x_vllm_mtp.sh): SetsAITER_TRITON_LOG_LEVEL=ERRORto cut Gluon fallback noise from logs. RaisesMAX_NUM_SEQSto 128 (from2*CONC) so AgentX subagent fan-out is not capped at two in-flight sequences at CONC=1. Changes--moe-backendfromaiter_triton_mxfp4_bf16toaiterso vLLM selects CK a8w4 expert kernels (aligned with DSV4-Pro MI355X) rather than the Triton W4A16 path.Changelog:
perf-changelog.yamladds three append-only entries for this config key documenting the nightly bump, scheduler change, and MoE backend switch (with measured perf notes in the descriptions).Reviewed by Cursor Bugbot for commit ab76c13. Bugbot is set up for automated code reviews on this repo. Configure here.