Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试#2158
Conversation
xinli-sw
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
xinli-sw
added
full-sweep-fail-fast
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| image: vllm/vllm-openai:nightly-09663abde0f50944a8d5ea30120666024b503faa | ||
| model: Kimi-K2.6-NVFP4 | ||
| model-prefix: kimik2.5 | ||
| runner: cluster:b300 |
There was a problem hiding this comment.
🔴 The new kimik2.5-fp4-b300-vllm-agentic-mtp config sets runner: cluster:b300 at configs/nvidia-master.yaml:2985, but that label doesn't exist in configs/runners.yaml — only cluster:b300-nv and cluster:b300-cw are defined. Because one search-space entry uses kv-offloading: dram, matrix generation calls agentic_dram_offload_gb → runner_available_cpu_dram_mib('cluster:b300', ...) which raises ValueError (no hardware entry), aborting sweep-config generation before any benchmark runs. Change to cluster:b300-nv to match the sibling non-mtp config at line 2970.
Extended reasoning...
What the bug is
The new kimik2.5-fp4-b300-vllm-agentic-mtp recipe added in this PR (configs/nvidia-master.yaml:2981-2996) declares runner: cluster:b300 at line 2985. This label does not exist anywhere in configs/runners.yaml. Grepping the runners file shows only two B300 cluster labels are defined: cluster:b300-cw (line 216) and cluster:b300-nv (line 222), and only cluster:b300-nv has a hardware: entry (line 306). The sibling non-mtp recipe kimik2.5-fp4-b300-vllm-agentic at line 2970 correctly uses cluster:b300-nv on what is clearly the same physical fleet.
How it manifests
The recipe contains a search-space entry with kv-offloading: dram, kv-offload-backend: native (line 2995). In utils/matrix_logic/generate_sweep_configs.py, generate_full_sweep calls agentic_dram_offload_gb(agentic_config, bmk, runner, runner_data) for that entry. That function calls runner_available_cpu_dram_mib('cluster:b300', runner_data) → runner_hardware_int(...) which looks up hardware['cluster:b300']. Since there is no such key (only cluster:b300-nv is registered under hardware:), runner_hardware_int raises:
ValueError: Runner 'cluster:b300' requires 'available-cpu-dram-mib' in runner hardware metadata.
Available hardware keys: cluster:b300-nv, ...
That exception propagates out of generate_full_sweep, so the entire sweep-config generation step (the get-jobs/setup job in the workflow) fails with a non-zero exit code. No benchmark job in the matrix ever dispatches — not even the non-dram entries at lines 2993/2994/2996.
Why existing code doesn't catch it
AgenticCodingConfig.validate_dram_offload_capacity in validation.py only checks that dram-utilization is set when kv-offloading == 'dram'; it does not cross-check that the recipe's runner: label actually appears in runners.yaml. Master-config validation is purely structural, so an arbitrary string is accepted for runner. The mistake only surfaces at matrix-generation time.
Impact
This is the whole point of the PR: the sweep for the new MTP recipe cannot generate configs, so no runs will occur. Even for the non-dram entries where the missing-hardware code path is not exercised, GitHub Actions would try to schedule jobs with runs-on: cluster:b300, which no self-hosted runner group carries — so they would sit in "waiting for a runner" forever and eventually time out.
Fix
One-character change: replace runner: cluster:b300 with runner: cluster:b300-nv at configs/nvidia-master.yaml:2985, matching the sibling recipe at line 2970. If the CW fleet was intended instead, use cluster:b300-cw — but note that cluster:b300-cw has no hardware: entry either, so the dram search-space entry would still fail; only cluster:b300-nv currently supports DRAM offload matrix generation.
Step-by-step proof
configs/runners.yamlline 306:hardware:only containscluster:b300-nv(nocluster:b300).run-sweep.yml/e2e-tests.yml'setup' job runspython3 utils/matrix_logic/generate_sweep_configs.py ....generate_full_sweepiterates configs; reacheskimik2.5-fp4-b300-vllm-agentic-mtp,is_multinode = False,runner = "cluster:b300".- Enters the agentic loop; for the dram entry
{tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: native, conc-list: [2, 4, 16]}. total_cpu_dram_gb = agentic_dram_offload_gb(agentic_config, bmk, "cluster:b300", runner_data).agentic_dram_offload_gb→runner_available_cpu_dram_mib("cluster:b300", runner_data)→runner_hardware_int("cluster:b300", runner_data, "available-cpu-dram-mib").runner_hardware(runner_data).get("cluster:b300", {})returns{};value = None→ raisesValueError.- Python exits non-zero;
CONFIG_JSON=$(python3 ...)fails; theget-jobs/setupstep fails; no downstream benchmark job dispatches.
| "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":$CPU_OFFLOAD_BYTES,\"lazy_offload\":false}}" | ||
| ) | ||
| ;; | ||
| *) echo "Error: unsupported KV_OFFLOAD_BACKEND value '$KV_OFFLOAD_BACKEND' with EAGLE3 (expected: native)" >&2; exit 1 ;; | ||
| esac | ||
| fi | ||
|
|
||
| DCP_SIZE="${DCP_SIZE:-1}" | ||
| DCP_ARGS=() | ||
| if [[ "$DCP_SIZE" -gt 1 ]]; then | ||
| DCP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE") | ||
| NUM_SPEC_TOKENS=3 | ||
| SYNTHETIC_ACCEPT_LEN=2.88 | ||
| else | ||
| NUM_SPEC_TOKENS=4 | ||
| SYNTHETIC_ACCEPT_LEN=3.24 | ||
| fi |
There was a problem hiding this comment.
thanks for the contribution @xinli-sw
can u please update recipes.vllm.ai with ur new configs?
vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes
as in sglang cookbook recipes , they also have support for cpu offloading recipes
similar applies to DCP, doubt the inferact folks won't mind contributions to add DCP support too
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29121027372 |
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
9f46169 to
3b42122
Compare
xinli-sw
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29137748102 |
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
3b42122 to
e5923ca
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29138755696 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29139024185 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29140751554 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29157674951 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29158176591 |
|
Opened this PR for the recipe. Will work with the vLLM team to get it in. Feel free to comment and review the PR, and help me get the right recipe in. |
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
41f7b72 to
979b077
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29338763676 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29339039057 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29341422010 |
xinli-sw
force-pushed
the
kimik2.6-fp4-b300-mtp
branch
from
e5846cd to
79012e8
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29351457139 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29352348984 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29359203708 |
|
/reuse-sweep-run |
|
@cquil11 @functionstackx this PR is good to ship |
|
/stage-results 29359203708 |
|
@cquil11 staged run 29359203708: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-14~r29359203708 This shared staging slot remains available until the next |
|
@claude Add simple, succinct, no fluff PR description for posterity. |
|
Claude finished @cquil11's task in 1m 47s —— View job PR description added ✅
Added a succinct description to the PR body covering: the new |
中文：通过复用扫描产物恢复 PR #2158 的数据库摄取。
…数据库摄取 (#2222) * feat(kimik2.6-b300-mtp): add Kimi-K2.6 NVFP4 B300 EAGLE3 AgentX benchmark - Add kimik2.5-fp4-b300-vllm-agentic-mtp config (runner: cluster:b300-nv) - Add benchmark script with EAGLE3 speculative decoding (draft: lightseekorg/kimi-k2.6-eagle3-mla, num_speculative_tokens=4, synthetic_acceptance_length=3.24) - TOKENSPEED_MLA attention backend + TRTLLM_RAGGED prefill backend - Search space: TP8 no-offload, TP4 no-offload, TP4 native CPU-offload, TP4 DCP4 no-offload, TP4 DCP4 native CPU-offload - Add cluster:b300-nv runner label and hardware entry - Add spec-decoding field propagation for single-node agentic matrix - Add perf changelog entry for PR 2158 * fix(kimik2.6-b300-mtp): use spec3/AL2.88 for DCP configs * fix(kimik2.6-b300-mtp): align DCP vllm args with working command * fix(changelog): omit null optional metadata * chore: prepare PR 2158 ingest recovery 中文：准备恢复 PR #2158 的数据库摄取。 * fix: recover PR 2158 ingest 中文：通过复用扫描产物恢复 PR #2158 的数据库摄取。 --------- Co-authored-by: Xin Li <xinli@nvidia.com>
6 participants
Adds an EAGLE3 speculative-decoding AgentX (agentic-coding) benchmark for Kimi K2.6 NVFP4 on B300, served with upstream vLLM (
vllm/vllm-openai:nightly-94c0ef30…).Changes
kimik2.5-fp4-b300-vllm-agentic-mtp(configs/nvidia-master.yaml), runnercluster:b300-nv. Search space: TP8, TP4, TP4 + native CPU KV offload (SimpleCPUOffloadConnector,dram-utilization: 0.80), and TP4/DCP4 with and without offload at high concurrency.benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh: EAGLE3 draftlightseekorg/kimi-k2.6-eagle3-mlawith simulated synthetic acceptance — non-DCP arms usenum_speculative_tokens=4/ AL 3.24; DCP4 arms usenum_speculative_tokens=3/ AL 2.88.TOKENSPEED_MLAattention backend,TRTLLM_RAGGEDMLA prefill (non-DCP), fp8 KV cache.Kimi-K2.6-NVFP4to the b300-nv model allowlist.process_changelog.pynow emits JSON withexclude_none=True; perf-changelog entry added.Validation
/stage-results)为 B300 上的 Kimi K2.6 NVFP4 新增 EAGLE3 投机解码 AgentX（agentic-coding）基准测试，使用上游 vLLM 镜像。新增
kimik2.5-fp4-b300-vllm-agentic-mtp配置（TP8 / TP4 / TP4+CPU KV offload / TP4+DCP4 组合）、基准脚本（EAGLE3 草稿模型lightseekorg/kimi-k2.6-eagle3-mla，非 DCP 使用 spec4/AL 3.24，DCP4 使用 spec3/AL 2.88）、b300-nv 模型白名单及 perf-changelog 条目。