Add GB300 DeepSeek V4.1 Flash P/D without KV offload / 新增无 KV 卸载的 GB300 P/D - #3036
Add GB300 DeepSeek V4.1 Flash P/D without KV offload / 新增无 KV 卸载的 GB300 P/D#3036Oseltamivir wants to merge 5 commits into
Conversation
新增 GB300 DeepSeek-V4.1-Flash 1P/1D AgentX 配方，使用 NIXL 传输和 GPU 驻留 KV，并覆盖并发 64、128、256。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| engram-config: '{"cpu_offload":true}' | ||
| gpu-memory-utilization: 0.90 | ||
| no-disable-hybrid-kv-cache-manager: true | ||
| speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' |
There was a problem hiding this comment.
Adaptive verification left on throughput
Medium Severity
Prefill and decode speculative-config keep enable_adaptive_verification true. The vLLM injector only flips rejection_sample_method to synthetic and adds synthetic_acceptance_length, so throughput still runs adaptive verification. The cited AL 3.51 and the single-node DSpark recipe both disable it for synthetic throughput.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9fea618. Configure here.
在性能变更记录中填写 PR 3036 链接。
让 GB300 P/D 使用 DeepSeek V4.1 Flash 模型别名。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34670625751 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$MODEL_PREFIX" == "dsv41flash" && "$PRECISION" == "fp4" && "$FRAMEWORK" == "vllm" ]]; then | ||
| # Both direct and Dynamo-vLLM paths resolve the V4.1 Flash checkpoint from | ||
| # the persistent shared HF cache. The alias must match model.path in the | ||
| # checked-in P/D recipe. | ||
| export MODEL_PATH="$MODEL" | ||
| export SRT_SLURM_MODEL_PREFIX="deepseek-v4.1-flash" |
There was a problem hiding this comment.
🔴 The comment claims this branch now covers "both direct and Dynamo-vLLM paths", but the condition still checks FRAMEWORK=="vllm"; the new nvidia-master.yaml entry sets framework: dynamo-vllm, so for that recipe this branch is skipped and MODEL_PREFIX/PRECISION/FRAMEWORK fall through to the hard else) exit 1 at the bottom of the chain, failing the job before it launches. Fix: also match FRAMEWORK=="dynamo-vllm" (mirroring the dsv4/minimaxm3/kimik2.5 dynamo-vllm branches later in the script) so dsv41flash+dynamo-vllm resolves MODEL_PATH/SRT_SLURM_MODEL_PREFIX instead of hitting the unsupported-model exit.
Extended reasoning...
configs/nvidia-master.yaml's new dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg entry sets framework: dynamo-vllm, and benchmark-multinode-tmpl.yml passes that straight through as FRAMEWORK (FRAMEWORK: ${{ inputs.framework }}, no normalization). In launch_gb300-nv.sh the very first if/elif chain (lines 40-119) is the only place SRT_SLURM_MODEL_PREFIX/MODEL_PATH get set, and it ends in else echo "Unsupported model: $MODEL_PREFIX-$PRECISION..."; exit 1; fi (line ~118-119). Only line 40's branch matches MODEL_PREFIX==dsv41flash, and it requires FRAMEWORK=="vllm"; dynamo-vllm never matches any dsv41flash branch (the other elifs match dsr1/dsv4/glm5/minimaxm2.5/minimaxm3/kimik2.5/kimik3/qwen3.5), so the script exits 1 immediately for the newly added recipe, before any srtctl/srun invocation. This is the first dsv41flash+dynamo-vllm entry added, so the path is newly exercised and was never covered before.
Verification: normal. The new configs/nvidia-master.yaml entry dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg sets framework: dynamo-vllm (line 10419), model-prefix: dsv41flash, precision: fp4. This framework value passes through unchanged: run-sweep.yml -> benchmark-multinode-tmpl.yml FRAMEWORK: ${{ inputs.framework }} (line 248), no normalization; the launch script itself confirms literal handling…
| description: | ||
| - "Add DeepSeek-V4.1-Flash GB300 AgentX 1P/1D with NIXL P/D transfer, GPU-resident KV, Engram UVA, DSpark5 synthetic AL 3.51 and concurrency 64, 128, 256" | ||
| - "新增 DeepSeek-V4.1-Flash GB300 AgentX 1P/1D：使用 NIXL P/D 传输、GPU 驻留 KV、Engram UVA、DSpark5 合成 AL 3.51 和并发 64、128、256" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0 |
There was a problem hiding this comment.
🟡 (optional) The new changelog entry's pr-link is the placeholder https://github.com/SemiAnalysisAI/InferenceX/pull/0, unlike every other entry in the file which points at the real merged PR number, breaking traceability from the recipe change back to its PR. Fix: replace pull/0 with the actual PR number before merge.
Extended reasoning...
perf-changelog.yaml is append-only and used to trace which PR introduced a given config-key/behavior change; grepping the file shows this is the only pull/0 occurrence, confirming it's an unfilled placeholder rather than an established convention. Once merged, anyone auditing why dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg was added via this changelog is sent to a nonexistent PR #0 instead of the real PR.
Verification: nit. perf-changelog.yaml line 7370 sets pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0 for the newly appended entry (config-key dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg). grep confirms this is the ONLY pull/0 occurrence in the file, while all 882 other pr-links reference real PR numbers (e.g. the immediately preceding entry uses .../pull/2962). AGENTS.md:32…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34670911561 |
为 GB300 P/D 解析具体的缓存模型快照，并移除 worker 不支持的前端参数。
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8c419d5. Configure here.
为 GB300 P/D 挂载缓存快照引用的权重 blob。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34674339904 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34674778771 |
1 participant
Add DeepSeek V4.1 Flash AgentX on GB300 as 1P/1D DEP4/DEP16 at concurrency 64, 128, and 256. It uses the PR #3017 image/runtime, Engram UVA, DSpark5 synthetic AL 3.51, NIXL P/D transfer, and GPU-resident KV with no MooncakeStore tier.
新增 GB300 DeepSeek V4.1 Flash AgentX 1P/1D DEP4/DEP16 配方，并发为 64、128、256。使用 PR #3017 的镜像与运行时、Engram UVA、DSpark5 合成 AL 3.51、NIXL P/D 传输；KV 驻留 GPU，不启用 MooncakeStore。
Note
Low Risk
Changes are benchmark recipes, registry docs, and GB300 launcher model-path/mount plumbing; no production serving or auth paths.
Overview
Adds a disaggregated GB300 AgentX benchmark for DeepSeek-V4.1-Flash: five-node 1P/1D (DEP4 prefill, DEP16 decode) with NIXL KV transfer, GPU-resident KV (no MooncakeStore / external prefix cache), Engram UVA, and DSpark5 at concurrency 64, 128, and 256. Throughput uses synthetic AL 3.51; eval keeps real block verification.
Registers
dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disaggin the master config, documents it (EN/ZH), and records it inperf-changelog.yaml. The new srt-slurm recipe lives underbenchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4.1-flash/agentic/.launch_gb300-nv.shnow resolves all fp4dsv41flashruns (including Dynamo multinode) from the shared HF cache snapshot with aliasdeepseek-v4.1-flash, bind-mounts HF blobs, and overlays v4.1-flash agentic recipes into the pinned srt-slurm checkout.test_dsv41flash_gb300_pd.pyasserts NIXL roles and absence of Mooncake / external KV store wiring.Reviewed by Cursor Bugbot for commit 350ac5a. Bugbot is set up for automated code reviews on this repo. Configure here.