[AgentX GB300] enable Kimi-K3 disaggregated power / 启用分离部署功耗 - #3047
[AgentX GB300] enable Kimi-K3 disaggregated power / 启用分离部署功耗#3047edwingao28 wants to merge 6 commits into
Conversation
a72c17f to
a6ed56d
Compare
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 如需进行 PR 验证,请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
a6ed56d to
702a76d
Compare
c213b50 to
2d4b118
Compare
|
Claude finished @edwingao28's task in 1m 56s —— View job Review of PR #3047
LGTM - no blocking issues found Checked against the base branch: the four recipes carry the same Note: the sandbox did not permit running |
There was a problem hiding this comment.
Looks good, straightforward config change. Reviewed the four new telemetry: blocks and benchmark: additions against a sibling GB300 agentic recipe in the same directory (agg-gb300-dcp8-nospec-mooncake-agentic.yaml) — byte-for-byte identical pattern. Verified the perf-changelog.yaml entry is appended at the tail without altering prior bytes, and that its conc-lists match configs/nvidia-master.yaml's kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg search-space (48/52/56, 32/48/64, 1, 32/48) which already references all four recipe files via CONFIG_FILE. The docs/ci-procedures.md and _zh.md additions are matched bilingual sections.
Extended reasoning...
Overview
This PR adds an identical telemetry: block (DCGM power provider, required, dcgm-exporter on port 9401) plus client_placement: head, concurrencies, and ENABLE_AGENTX_POWER/REQUIRE_POWER env vars to four GB300 Kimi-K3 disaggregated recipe YAMLs, appends bilingual documentation sections to docs/ci-procedures.md/_zh.md, and appends one new entry to perf-changelog.yaml. No application/launcher code is touched in this diff (that lives in the referenced parent PR #3046).
Security risks
None identified. This is declarative YAML configuration for internal benchmark infrastructure — no user input handling, auth, or crypto paths are touched. The dcgm-exporter port and container image values match the existing convention used elsewhere in the repo.
Level of scrutiny
Low-to-moderate is appropriate: the change is purely additive, mechanical, and repeats an exact pattern already in production use on sibling GB200/GB300 recipes in the same directory. I verified this by diffing the new blocks against agg-gb300-dcp8-nospec-mooncake-agentic.yaml's existing telemetry block, which is byte-identical apart from concurrency values. I also cross-checked the four recipes' concurrency lists against the corresponding search-space entries in configs/nvidia-master.yaml for the kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg config-key, confirming all four conc-lists match exactly, and confirmed the perf-changelog append preserves all prior bytes (only additions after line 7438, at EOF).
Other factors
The PR description itself notes the feature is currently blocked (parent dependency, unresolved port-9401 conflict, no validated disaggregated power measurement yet) — but that is a rollout/operational concern for the author's team, not a code-correctness issue in this diff. The bug hunter reported no findings, and I found nothing beyond that on my own pass through the diff, master config, and doc changes.
2d4b118 to
58f476e
Compare
702a76d to
f56f70a
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f56f70a. Configure here.
| collector_join_timeout_seconds: 12 | ||
| dcgm_exporter: | ||
| container_image: dcgm-exporter | ||
| port: 9401 |
There was a problem hiding this comment.
GB300 recipes use conflicting exporter port
High Severity
The four GB300 disaggregated recipes start the required DCGM exporter on 9401, which the parent GB300 aggregate work already showed is occupied by the node system service. With required: true and REQUIRE_POWER: "1", collector startup fails and the jobs cannot produce valid power results.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit f56f70a. Configure here.
f56f70a to
b44e63a
Compare
58f476e to
78dde37
Compare
基于 main 独立启用 GB300 聚合部署实测功耗;仅复用已合入的共享采集代码,保留配方和端口修复。
b44e63a to
9a7f353
Compare
78dde37 to
49c49b0
Compare
测试执行实际 recipe 遥测开关与模型、框架、精度、场景限制,避免预设功耗标志绕过生产判断。
启用 GB300 分离部署实测功耗,并仅依赖 GB300 公共启动接线。
GB300 分离部署改用已验证的 19401 端口;验证 prefill 与 decode 分组均传递该端口,并保留完整硬件验证待办。
9a7f353 to
f72e2ca
Compare
中文:同步已合并的主分支,保留分离部署功耗改动及主分支历史更新记录。
中文:同步最新 main,完整保留 GB300 分离部署配方、功耗配置及两条历史 changelog。


Description
Enable required power for four GB300 Kimi-K3 disaggregated recipes. Move exporters to 19401 to avoid the system listener on 9401.
Testing: Local replay of all four recipes confirms exporter ports and prefill/decode Slurm groups; the old configurations fail this check. Changelog checks pass. Shared lifecycle evidence is linked in #3046.
Pending: #3046 is merged and included. Disaggregated hardware/request accounting, full sweep and eval validation still require a new authorized GPU budget.
中文
为四个 GB300 Kimi-K3 分离部署 recipe 启用必需功耗采集,exporter 改用 19401,避免与 9401 系统服务冲突。
测试: 四个 recipe 的本地回放确认 exporter 端口与 prefill/decode Slurm 分组正确;旧配置会失败。Changelog 校验通过;共享生命周期证据见 #3046。
待完成: #3046 已合入,本分支已包含其代码。分离部署硬件、请求账目、完整 sweep 与 eval 仍待验证,需新的 GPU 预算授权。
Related Issue
Scope index and shared policy / 范围索引与共同规则: #3030.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
Medium Risk
Benchmark recipe and CI doc changes only, but required telemetry and mandatory power env can fail runs if collectors or exporter port 9401 conflict; no new validated measurements are claimed in the PR description.
Overview
Enables required DCGM power telemetry and AgentX power measurement on four GB300 Kimi-K3 disaggregated agentic Slurm recipes, aligning them with the launcher/collector work from #3046.
Each recipe gains a
telemetryblock (dcgm-power,required: true, DCGM exporter on port 9401),client_placement: head, recipe-specificconcurrencies, and benchmark envENABLE_AGENTX_POWER/REQUIRE_POWER.Adds bilingual CI backfill guidance for Kimi-K3 AgentX power (missing-point generation,
require-power, pairing perf and energy per run) and aperf-changelog.yamlentry forkimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg.Reviewed by Cursor Bugbot for commit 9a7f353. Bugbot is set up for automated code reviews on this repo. Configure here.