Add GB200 DeepSeek V4.1 Flash P/D without KV offload / 新增无 KV 卸载的 GB200 P/D - #3037
Add GB200 DeepSeek V4.1 Flash P/D without KV offload / 新增无 KV 卸载的 GB200 P/D#3037Oseltamivir wants to merge 5 commits into
Conversation
新增 GB200 DeepSeek-V4.1-Flash 1P/1D AgentX 配方，使用 NIXL 传输和 GPU 驻留 KV，并覆盖并发 64、128、256。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
在性能变更记录中填写 PR 3037 链接。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ac2df17. Configure here.
| export MODEL_PATH="$MODEL" | ||
| export SRT_SLURM_MODEL_PREFIX="deepseek-v4.1-flash" | ||
| fi | ||
|
|
There was a problem hiding this comment.
Watchtower shared FS omits new model
Medium Severity
uses_watchtower_shared_fs still treats only dsv4 dynamo-vllm as needing a compute-visible shared workspace. The new dsv41flash dynamo-vllm P/D arm runs on the same cluster:gb200-nv runner, so the srt-slurm checkout and outputs can stay on a login-node path that compute nodes cannot see.
Reviewed by Cursor Bugbot for commit ac2df17. Configure here.
| engram-config: '{"cpu_offload":true}' | ||
| gpu-memory-utilization: 0.90 | ||
| no-disable-hybrid-kv-cache-manager: true | ||
| speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' |
There was a problem hiding this comment.
Throughput leaves adaptive verification enabled
Medium Severity
The checked-in DSpark config keeps enable_adaptive_verification true. Throughput jobs only inject synthetic rejection and synthetic_acceptance_length; they do not turn adaptive verification off. Existing DeepSeek V4.1 Flash AgentX arms disable it for golden AL 3.51 throughput and keep it only for eval.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ac2df17. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34670628201 |
让 GB200 P/D 使用 DeepSeek V4.1 Flash 模型别名。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # DeepSeek V4.1 Flash resolves through the persistent shared HF cache for both | ||
| # direct serving and Dynamo P/D. | ||
| if [[ "$MODEL_PREFIX" == "dsv41flash" && "$FRAMEWORK" == "vllm" ]]; then | ||
| export MODEL_PATH="$MODEL" | ||
| export SRT_SLURM_MODEL_PREFIX="deepseek-v4.1-flash" |
There was a problem hiding this comment.
🔴 New MODEL_PATH/SRT_SLURM_MODEL_PREFIX resolution for dsv41flash only fires when FRAMEWORK=="vllm", but the new recipe added in configs/nvidia-master.yaml (dsv41flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg) uses framework: dynamo-vllm. That combination is not in the dynamo-vllm case's supported list, so the job hits the else branch, prints "Unsupported model prefix/precision combination" and exits 1 before ever launching. Fix: add dsv41flash/fp4 as a handled case in the elif [[ $FRAMEWORK == "dynamo-vllm" ]] block (around line 264-284), setting MODEL_PATH/SRT_SLURM_MODEL_PREFIX there, not only in the vllm-only guard.
Extended reasoning...
launch_gb200-nv.sh sets MODEL_PREFIX=dsv41flash, FRAMEWORK=dynamo-vllm from the new config key. The new guard at line 102 is [[ "$MODEL_PREFIX" == "dsv41flash" && "$FRAMEWORK" == "vllm" ]], which is false for dynamo-vllm. Execution falls through to the elif [[ $FRAMEWORK == "dynamo-vllm" ]] block (~line 264), whose only branches are kimik2.5/fp4, kimik3/fp4, dsv4/fp4, minimaxm2.5/{fp4,fp8}, minimaxm3/{fp4,fp8}; dsv41flash/fp4 falls to else: echo Unsupported...; exit 1. So the newly added disagg benchmark job added by this PR cannot run at all on GB200 CI/slurm.
Verification: normal. The PR adds config dsv41flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg with framework: dynamo-vllm, model-prefix: dsv41flash, precision: fp4 (configs/nvidia-master.yaml). In runners/launch_gb200-nv.sh the new dsv41flash MODEL_PATH resolution at line 104 ([[ "$MODEL_PREFIX" == "dsv41flash" && "$FRAMEWORK" == "vllm" ]]) is gated on FRAMEWORK=="vllm", so it is skipped for…
| description: | ||
| - "Add GB200 DeepSeek-V4.1-Flash AgentX 1P/1D with NIXL transfer, GPU-resident KV and concurrency 64, 128, 256" | ||
| - "新增 GB200 DeepSeek-V4.1-Flash AgentX 1P/1D：使用 NIXL 传输、GPU 驻留 KV 和并发 64、128、256" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0 |
There was a problem hiding this comment.
🔴 The new changelog entry's pr-link is a literal .../pull/0, which is neither the canonical pull/<real PR number> form nor an accepted placeholder (XXX / pull/XXX), so it will fail the changelog validation gate in infx/workflows/validate_perf_changelog.py (validate_added_pr_link, called from compare_entries) and block this PR from merging until fixed. Fix: use the XXX placeholder (auto-canonicalized to the real PR number by prepare_perf_changelog_merge.py) instead of a fabricated pull/0 link.
Extended reasoning...
validate_added_pr_link(link, pr_number) computes expected = f'https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}' where pr_number is the actual PR's number (not 0); since the checked-in link is literally '.../pull/0', it is not in PR_LINK_PLACEHOLDERS = {'XXX', '.../pull/XXX'} and not equal to expected, so ChangelogValidationError is raised: 'new PR entry must use ... or an XXX placeholder; found .../pull/0'. This runs in CI (validate_perf_changelog main / prepare_perf_changelog_merge) on every added entry, so the PR's changelog-gate check fails until the author replaces the literal 0 with the XXX placeholder or the real PR number.
Verification: normal (CI-blocking): The added entry at perf-changelog.yaml:7370 uses pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0. In infx/workflows/validate_perf_changelog.py, PR_LINK_PLACEHOLDERS (lines 24-27) contains only "XXX" and ".../pull/XXX"; pull/0 is not in that set. validate_added_pr_link (lines 144-160) computes expected = .../pull/{pr_number} with the real PR…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34670911627 |
为 GB200 P/D 解析具体的缓存模型快照，并移除 worker 不支持的前端参数。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34674339893 |
让 GB200 P/D 使用支持当前配方字段的已发布 srt-slurm schema，并挂载缓存权重 blob。
8ed1c80 to
2aee357
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34674793890 |
1 participant
Add DeepSeek V4.1 Flash AgentX on GB200 as 1P/1D DEP4/DEP16 at concurrency 64, 128, and 256. Uses Engram UVA, DSpark5 synthetic AL 3.51, NIXL P/D transfer, and GPU-resident KV without MooncakeStore.
新增 GB200 DeepSeek V4.1 Flash AgentX 1P/1D DEP4/DEP16 配方，并发为 64、128、256。使用 Engram UVA、DSpark5 合成 AL 3.51、NIXL P/D 传输；KV 驻留 GPU，不启用 MooncakeStore。
Note
Medium Risk
Adds a five-node multinode Slurm/disagg benchmark path with launcher and cache-mount changes on a shared GB200 cluster; misconfiguration could waste GPU time but does not touch auth or data handling.
Overview
Adds a GB200 disaggregated AgentX lane for DeepSeek-V4.1-Flash: one DEP4 prefill worker and one DEP16 decode worker on five nodes, swept at concurrency 64, 128, and 256. KV stays GPU-resident; prefill/decode exchange KV via NIXL only (no MooncakeStore tier), with Engram CPU offload and DSpark5 plus synthetic acceptance length 3.51 for throughput.
A new srt-slurm recipe (
disagg-gb200-1p1d-dep4-dep16-c64-c256-dspark5-agentic.yaml) drivesagentic_srt.shwith Dynamo router 1.4.0 and the pinned vLLM flash image.Registry and ops wiring:
nvidia-master.yamlkeydsv41flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg, perf changelog, and EN/ZH procedure docs.launch_gb200-nv.shresolves weights from the shared HF hub snapshot, clones srt-slurm v1.0.36 for this AgentX P/D path, overlays v4.1-flash recipes, and mounts HF blobs for agentic containers.A small
test_dsv41flash_gb200_pd.pyguards the recipe shape (NIXL roles, no Mooncake/external KV store).Reviewed by Cursor Bugbot for commit 2aee357. Bugbot is set up for automated code reviews on this repo. Configure here.