[Klaud Cold] Try TEP4 and DEP4 for DeepSeek V4.1 Flash AgentX on B300 / 在 B300 上尝试 DeepSeek V4.1 Flash AgentX 的 TEP4 与 DEP4 - #3028
Conversation
Add expert-parallel (TEP4) and data-parallel-attention (DEP4) rows at concurrency 32/64/128 to dsv41flash-fp4-b300-vllm-agentic-dspark. The shared DSpark script gains EP_SIZE / DP_ATTENTION gating: TEP adds --enable-expert-parallel; DEP serves --data-parallel-size TP behind vllm-router 0.1.14 consistent-hash session affinity with the engine on PORT+1. Existing TP4 rows on every SKU serve the same command as before. 为 dsv41flash-fp4-b300-vllm-agentic-dspark 新增并发 32/64/128 的专家并行 （TEP4）与数据并行注意力（DEP4）行。共享 DSpark 脚本按 EP_SIZE / DP_ATTENTION 门控：TEP 追加 --enable-expert-parallel；DEP 以 --data-parallel-size TP 运行， 由 vllm-router 0.1.14 一致性哈希保证会话亲和，引擎监听 PORT+1。各 SKU 现有 TP4 行的服务命令与以前完全一致。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
将 perf-changelog 条目链接到 PR #3028。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3d68f98. Configure here.
| # the same rank by mapping AIPerf's correlation ID to X-Session-ID. | ||
| VLLM_BACKEND_PORT=$((PORT + 1)) | ||
| VLLM_ROUTER_VERSION=0.1.14 | ||
| VLLM_ROUTER_METRICS_PORT=$((PORT + 10000)) |
There was a problem hiding this comment.
DEP ports skip availability checks
Medium Severity
DEP assigns the engine to PORT+1 and router metrics to PORT+10000 after select_available_server_port only probes PORT. On the shared Pyxis host network this recipe already treats as contested, PORT+1 can be occupied, and an ephemeral PORT above 55535 makes PORT+10000 an invalid TCP port, so the engine or vllm-router fails to bind.
Reviewed by Cursor Bugbot for commit 3d68f98. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This automated review ran and the inline comments flag the server-state overwrite and unchecked derived-port issues in the new DEP4 path, plus the placeholder PR link in perf-changelog.yaml. Beyond those, I also checked whether stacking the new TEP4/DEP4 rows onto the existing TP4 grid disrupts automatic eval-entry selection in generate.py's ag_sn_groups grouping (keyed by model/runner/framework/precision, not by parallelism/topology) — that group-collapsing-to-one-representative behavior predates this PR, so it is not a new regression introduced here.
Extended reasoning...
Three findings were already confirmed and will be posted inline: a server-watch PID/state overwrite in the DEP4 router startup path that could blind the crash-watch used by the AgentX replay client, unchecked derived ports (PORT+1, PORT+10000) that skip the bind-probe validation the base PORT goes through, and a literal pull/XXX placeholder left in the new perf-changelog.yaml entry. Given the first two are plausible functional bugs in a new, process-lifecycle/port-binding code path rather than mere nits, approval is not warranted here regardless of the config-only nature of most of the diff. I additionally looked at whether adding TEP4/DEP4 rows for the same model/runner/framework/precision key would break the mark_eval_entries eval-selection logic used for smoke/subset/all sweeps — the ag_sn_groups branch groups by that 4-tuple (not parallelism topology) and always collapses to a single highest-concurrency representative via max(), so only one of TP4/TEP4/DEP4 gets an accuracy eval selected in those modes today; however this collapsing behavior already existed for multiple TP4 concurrency rows before this PR, so it is a pre-existing limitation rather than something this change newly introduces, and it only affects which row gets an accuracy check, not whether throughput results are collected for all rows.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| wait_for_server_ready --port "$VLLM_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| if [[ "$DP_ATTENTION" == "true" ]]; then | ||
| echo "Starting vllm-router on port $PORT for $TP DP ranks..." | ||
| vllm-router \ | ||
| --worker-urls "http://localhost:$VLLM_BACKEND_PORT" \ | ||
| --policy consistent_hash \ | ||
| --intra-node-data-parallel-size "$TP" \ | ||
| --host 0.0.0.0 --port "$PORT" \ | ||
| --prometheus-host 127.0.0.1 --prometheus-port "$VLLM_ROUTER_METRICS_PORT" \ | ||
| --request-timeout-secs 14400 \ | ||
| --disable-retries > "$ROUTER_LOG" 2>&1 & | ||
| ROUTER_PID=$! | ||
| wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID" |
There was a problem hiding this comment.
🔴 In the new DEP4 path, wait_for_server_ready is called a second time for the router (line 114) after the first call already registered the vllm engine (line 101); the second call overwrites the global INFERENCEX_SERVER_PID/STATE used by run_server_client's crash-watch, so the AgentX replay client is now only guarded against the router dying, not the actual vLLM engine. Fix: preserve monitoring of the engine PID (and its VllmWorker/EngineCore descendants) alongside the router, e.g. snapshot both PIDs or union their required-process sets, so an engine crash still stops the client under DEP as it does under TP4/TEP4.
Extended reasoning...
wait_for_server_ready() (benchmark_lib.sh:499-528) does server_watch capture --pid $server_pid and stores the result in INFERENCEX_SERVER_STATE/INFERENCEX_SERVER_PID, unconditionally overwriting any prior value. server_watch.snapshot() walks only the given PID's process-tree descendants for VllmWorker/EngineCore/etc; the router process (started independently via vllm-router ... &) is not a parent of the vllm engine, so snapshotting ROUTER_PID yields a required-set containing only the router itself. run_agentic_replay_and_write_outputs -> run_server_client then watches only that state. If the vllm engine OOMs or crashes while the lightweight router keeps running, healthy() in server_watch.run() still returns true, so the replay client is never stopped early -- it just keeps hitting a backend returning errors/timeouts until the benchmark's own timeout, instead of failing fast the way TP4/TEP4 runs (and the base branch) do. The same double-call pattern already exists in benchmarks/single_node/agentic/dsv4_fp4_b300_vllm_mtp.sh, so this diff extends a pre-existing gap to a new…
Verification: normal. In the new DEP4 path the script calls wait_for_server_ready twice: first for the engine (dsv41flash_fp4_vllm_mtp.sh:101, --server-pid $SERVER_PID) and then for the router (line 114, --server-pid
| VLLM_BACKEND_PORT=$((PORT + 1)) | ||
| VLLM_ROUTER_VERSION=0.1.14 | ||
| VLLM_ROUTER_METRICS_PORT=$((PORT + 10000)) |
There was a problem hiding this comment.
🔴 VLLM_BACKEND_PORT (PORT+1) and VLLM_ROUTER_METRICS_PORT (PORT+10000) are derived by pure arithmetic from PORT and never checked for availability or validity, unlike PORT itself which goes through select_available_server_port's bind-probe/fallback. When select_available_server_port falls back to an OS-assigned ephemeral port (its documented purpose, since "port 8888 can already belong to a host service"), that port can already be in the 50000-60999 range, pushing PORT+10000 past 65535 into an invalid port number, or PORT+1 into a port already bound by another process on the shared host. Fix: allocate/validate VLLM_BACKEND_PORT and VLLM_ROUTER_METRICS_PORT the same way PORT is (probe-and-bind or clamp+retry), not by fixed offset arithmetic on an already-randomized base port.
Extended reasoning...
select_available_server_port (benchmark_lib.sh:28-45) binds to 8888 or, on EADDRINUSE, to sock.bind(("0.0.0.0",0)) which the kernel satisfies from its local ephemeral range (commonly up to ~60999 on Linux). The new DEP4 code (lines 52-59) then sets VLLM_BACKEND_PORT=PORT+1 and VLLM_ROUTER_METRICS_PORT=PORT+10000 with no further check. If the OS assigns e.g. PORT=58000, VLLM_ROUTER_METRICS_PORT becomes 68000, an out-of-range port; vllm-router's --prometheus-port argument then fails to bind, crashing the router (and thus the whole DEP4 job) purely because PORT happened to fall back from 8888. Separately, PORT+1 can collide with any other process already listening there on the Pyxis-shared host network, which select_available_server_port would have caught for PORT itself but never checks for the derived port.
Verification: normal (conditional). New DEP4-only code derives ports by pure arithmetic with no validation: dsv41flash_fp4_vllm_mtp.sh:57 VLLM_ROUTER_METRICS_PORT=$((PORT + 10000)) (and line 55 VLLM_BACKEND_PORT=$((PORT + 1))), used at line 108 --prometheus-port "$VLLM_ROUTER_METRICS_PORT". PORT comes from select_available_server_port (benchmark_lib.sh:28-45), which binds the default 8888 and, on…
| description: | ||
| - "Add B300 TEP4 and DEP4 AgentX arms at concurrency 32, 64 and 128 alongside the unchanged TP4 grid: TEP4 shards the routed experts with --enable-expert-parallel; DEP4 runs data-parallel attention (--data-parallel-size 4) behind vllm-router 0.1.14 consistent-hash session affinity. The shared DSpark script gates both on EP_SIZE and DP_ATTENTION, so the existing TP4 rows on every SKU serve the same command as before." | ||
| - "为 B300 新增 TEP4 与 DEP4 AgentX 分支（并发 32、64、128），TP4 网格保持不变：TEP4 通过 --enable-expert-parallel 切分路由专家；DEP4 以 --data-parallel-size 4 运行数据并行注意力，并由 vllm-router 0.1.14 一致性哈希保证会话亲和。共享 DSpark 脚本按 EP_SIZE 与 DP_ATTENTION 门控，各 SKU 现有 TP4 行的服务命令与以前完全一致。" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 nit (optional): the new changelog entry's pr-link is left as a literal placeholder pull/XXX instead of the real PR number, unlike every one of the ~7340 preceding entries in this file which all point to an actual PR. Fix: replace with the real PR number/URL before merge so the changelog link resolves instead of 404ing.
Extended reasoning...
perf-changelog.yaml is a flat append-only log of dated entries with a pr-link field readers use to trace a perf change back to its PR; every existing entry (e.g. lines 7214-7340) has a concrete PR number filled in. The new entry added by this diff at the end of the file uses https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, which is not a valid PR URL. Anyone following this link from the changelog after merge gets a broken/nonexistent page, and any tooling that parses pr-link to cross-reference PRs will fail to resolve this entry.
Verification: nit: The new perf-changelog.yaml entry ends with a literal placeholder pr-link. The diff's last line is pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, whereas the preceding entry (and every prior entry, e.g. line 7340 pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3020) uses a concrete PR number. pull/XXX is not a valid GitHub PR URL, so following the link…
Under DEP every rank's expert layer receives the tokens dispatched from all DP ranks, so the TRT-LLM FP4 MoE autotuner needs a larger workspace than under TP. At the image default gpu-memory-utilization of 0.92 the warmup after KV-cache sizing died 4.45 GiB short on all four ranks (run 34655656558, DEP4 c64). Pass --gpu-memory-utilization 0.85 and enable expandable allocator segments for the DP arm only; TP4 and TEP4 are unchanged. DEP 下每个 rank 的专家层都会接收所有 DP rank 分发的 token，因此 TRT-LLM FP4 MoE 自动调优所需的工作区比 TP 更大。在镜像默认 gpu-memory-utilization 0.92 下，KV cache 定容后的预热在四个 rank 上均差 4.45 GiB 而失败（运行 34655656558，DEP4 c64）。仅对 DP 分支传入 --gpu-memory-utilization 0.85 并 启用可扩展分配器段；TP4 与 TEP4 不变。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34655656558 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34660106061 |
1 participant
Summary
Adds two experimental parallelism arms to
dsv41flash-fp4-b300-vllm-agentic-dsparkfor mid-to-high concurrency, alongside the unchanged TP4 grid that passed 1–128 in the merged #2958 sweep:tp: 4, ep: 4): TP attention, routed experts sharded across the 4 GPUs via--enable-expert-parallel. Concurrency 32, 64, 128.tp: 4, ep: 4, dp-attn: true): one attention rank per GPU (--tensor-parallel-size 1 --data-parallel-size 4) plus expert parallel, fronted byvllm-router0.1.14 withconsistent_hashsession affinity (AIPerf correlation ID mapped toX-Session-ID); engine onPORT+1, router onPORT. Concurrency 32, 64, 128.Both arms come from the upstream recipe's
compatible_strategies(single_node_tep,single_node_dep). The shareddsv41flash_fp4_vllm_mtp.shgates the new behaviour on the matrix-suppliedEP_SIZE/DP_ATTENTIONenv vars, so every existing TP4 row on B200/B300/GB200/GB300/H200 still serves the exact same command. DEP graph capture is sized per rank (twiceCONC/TPverification blocks).Changes: shared DSpark script, B300 master-config rows, perf-changelog entry, and the EN/ZH configuration docs.
Validation
bash -non the script; matrix generation for the key yields 14 jobs (8 TP4 + 3 TEP4 + 3 DEP4) with the expectedep/dp-attn/routerfields.pytest runners/test_dsv41flash_*.py utils/matrix_logic: 306 passed.full-sweep-fail-fastsweep (pending).Risks
cpu_offloadunder DP is untested upstream: each DP rank may pin its own copy of the ~189 GiB Engram tables (~756 GiB host DRAM at DEP4). The B300 nodes have the headroom, but this is what the sweep is for.摘要
为
dsv41flash-fp4-b300-vllm-agentic-dspark新增两条面向中高并发的实验性并行分支，TP4 网格（已在 #2958 的 sweep 中通过并发 1–128）保持不变：tp: 4, ep: 4）：TP 注意力，路由专家通过--enable-expert-parallel切分到 4 张 GPU。并发 32、64、128。tp: 4, ep: 4, dp-attn: true）：每张 GPU 一个注意力 rank（--tensor-parallel-size 1 --data-parallel-size 4）并启用专家并行，前置vllm-router0.1.14 的consistent_hash会话亲和（AIPerf correlation ID 映射为X-Session-ID）；引擎监听PORT+1，router 监听PORT。并发 32、64、128。两条分支均来自上游配方的
compatible_strategies（single_node_tep、single_node_dep）。共享脚本dsv41flash_fp4_vllm_mtp.sh按矩阵提供的EP_SIZE/DP_ATTENTION环境变量门控新行为，因此 B200/B300/GB200/GB300/H200 上现有的 TP4 行仍执行完全相同的命令。DEP 的 CUDA graph capture 按每个 rank 计算（CONC/TP验证块的两倍）。改动：共享 DSpark 脚本、B300 master-config 行、perf-changelog 条目，以及中英文配置文档。
验证
bash -n；该 key 的矩阵生成得到 14 个 job（8 个 TP4 + 3 个 TEP4 + 3 个 DEP4），ep/dp-attn/router字段符合预期。pytest runners/test_dsv41flash_*.py utils/matrix_logic：306 通过。full-sweep-fail-fastsweep（进行中）。风险
cpu_offload上游未验证：每个 DP rank 可能各自固定一份约 189 GiB 的 Engram 表（DEP4 约 756 GiB 主机内存）。B300 节点有余量，但这正是需要 sweep 验证的地方。🤖 Generated with Claude Code
Note
Medium Risk
Introduces new vLLM parallelism and vllm-router serving paths with DEP-specific GPU memory limits; Engram CPU offload per DP rank is experimental and could stress host DRAM during sweeps.
Overview
Adds TEP4 and DEP4 experimental AgentX sweep arms for
dsv41flash-fp4-b300-vllm-agentic-dsparkat concurrency 32–128, while the existing TP4 grid (1–128) stays unchanged.The shared
dsv41flash_fp4_vllm_mtp.shnow branches on matrixEP_SIZE/DP_ATTENTION: TEP4 keeps TP attention and adds--enable-expert-parallel; DEP4 runs--tensor-parallel-size 1 --data-parallel-size 4(plus EP when configured), lowers--gpu-memory-utilizationto 0.85 with expandable PyTorch segments, sizes CUDA graph capture per DP rank, and fronts the engine with vllm-router 0.1.14 (consistent_hash, AIPerf correlation →X-Session-ID, engine onPORT+1). Metrics scraping targets the engine endpoint explicitly under DEP.configs/nvidia-master.yaml,perf-changelog.yaml, and EN/ZH configuration docs document the new search-space rows and router metadata.Reviewed by Cursor Bugbot for commit c8f4da1. Bugbot is set up for automated code reviews on this repo. Configure here.