Skip to content

[AgentX GB300] enable Kimi-K3 disaggregated power / 启用分离部署功耗 - #3047

Open
edwingao28 wants to merge 6 commits into
mainfrom
feat/kimik3-gb300-disaggregated-power
Open

[AgentX GB300] enable Kimi-K3 disaggregated power / 启用分离部署功耗#3047
edwingao28 wants to merge 6 commits into
mainfrom
feat/kimik3-gb300-disaggregated-power

Conversation

@edwingao28

@edwingao28 edwingao28 commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Description

Enable required power for four GB300 Kimi-K3 disaggregated recipes. Move exporters to 19401 to avoid the system listener on 9401.

Testing: Local replay of all four recipes confirms exporter ports and prefill/decode Slurm groups; the old configurations fail this check. Changelog checks pass. Shared lifecycle evidence is linked in #3046.

Pending: #3046 is merged and included. Disaggregated hardware/request accounting, full sweep and eval validation still require a new authorized GPU budget.

中文

为四个 GB300 Kimi-K3 分离部署 recipe 启用必需功耗采集,exporter 改用 19401,避免与 9401 系统服务冲突。

测试: 四个 recipe 的本地回放确认 exporter 端口与 prefill/decode Slurm 分组正确;旧配置会失败。Changelog 校验通过;共享生命周期证据见 #3046

待完成: #3046 已合入,本分支已包含其代码。分离部署硬件、请求账目、完整 sweep 与 eval 仍待验证,需新的 GPU 预算授权。

Related Issue

Scope index and shared policy / 范围索引与共同规则: #3030.

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have tested my changes locally
  • I have updated documentation if necessary
  • For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.

Note

Medium Risk
Benchmark recipe and CI doc changes only, but required telemetry and mandatory power env can fail runs if collectors or exporter port 9401 conflict; no new validated measurements are claimed in the PR description.

Overview
Enables required DCGM power telemetry and AgentX power measurement on four GB300 Kimi-K3 disaggregated agentic Slurm recipes, aligning them with the launcher/collector work from #3046.

Each recipe gains a telemetry block (dcgm-power, required: true, DCGM exporter on port 9401), client_placement: head, recipe-specific concurrencies, and benchmark env ENABLE_AGENTX_POWER / REQUIRE_POWER.

Adds bilingual CI backfill guidance for Kimi-K3 AgentX power (missing-point generation, require-power, pairing perf and energy per run) and a perf-changelog.yaml entry for kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg.

Reviewed by Cursor Bugbot for commit 9a7f353. Bugbot is set up for automated code reviews on this repo. Configure here.

@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-disaggregated-power branch from a72c17f to a6ed56d Compare September 12, 2026 03:36
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-disaggregated-power branch from a6ed56d to 702a76d Compare September 12, 2026 08:02
@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-aggregate-power branch from c213b50 to 2d4b118 Compare September 12, 2026 08:02
@edwingao28
edwingao28 marked this pull request as ready for review September 12, 2026 08:02
@claude

claude Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Claude finished @edwingao28's task in 1m 56s —— View job


Review of PR #3047

  • Gather context (diff vs feat/kimik3-gb300-aggregate-power, launcher/collector contract)
  • Review the four disaggregated recipe changes
  • Review docs and perf-changelog.yaml entry placement
  • Post review

LGTM - no blocking issues found

Checked against the base branch: the four recipes carry the same telemetry block, client_placement: head, and ENABLE_AGENTX_POWER/REQUIRE_POWER env as the aggregate recipes in #3046, so the launch_gb300-nv.sh lane detection (enabled: true + provider: dcgm-power scoped to the telemetry: block) and the AgentX power route will pick them up. The static concurrencies in each recipe match the corresponding conc-list in configs/nvidia-master.yaml (1p1d [48,52,56], 1p2d [32,48,64], 1p3d-dspark4 [32,48], 1p3d-dspark7 [1]), and the launcher overwrites them at runtime from CONC_LIST anyway. The perf-changelog.yaml entry is appended at the physical end of the file, no master config was edited, and the power adapter already handles disagg: true aggregates with split prefill/decode GPU counts.

Note: the sandbox did not permit running pytest in this session, so the changelog/contract tests were not re-executed here; the PR description reports they pass in CI. The port-9401 exporter conflict is already acknowledged in the PR body and is not new to this change.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, straightforward config change. Reviewed the four new telemetry: blocks and benchmark: additions against a sibling GB300 agentic recipe in the same directory (agg-gb300-dcp8-nospec-mooncake-agentic.yaml) — byte-for-byte identical pattern. Verified the perf-changelog.yaml entry is appended at the tail without altering prior bytes, and that its conc-lists match configs/nvidia-master.yaml's kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg search-space (48/52/56, 32/48/64, 1, 32/48) which already references all four recipe files via CONFIG_FILE. The docs/ci-procedures.md and _zh.md additions are matched bilingual sections.

Extended reasoning...

Overview

This PR adds an identical telemetry: block (DCGM power provider, required, dcgm-exporter on port 9401) plus client_placement: head, concurrencies, and ENABLE_AGENTX_POWER/REQUIRE_POWER env vars to four GB300 Kimi-K3 disaggregated recipe YAMLs, appends bilingual documentation sections to docs/ci-procedures.md/_zh.md, and appends one new entry to perf-changelog.yaml. No application/launcher code is touched in this diff (that lives in the referenced parent PR #3046).

Security risks

None identified. This is declarative YAML configuration for internal benchmark infrastructure — no user input handling, auth, or crypto paths are touched. The dcgm-exporter port and container image values match the existing convention used elsewhere in the repo.

Level of scrutiny

Low-to-moderate is appropriate: the change is purely additive, mechanical, and repeats an exact pattern already in production use on sibling GB200/GB300 recipes in the same directory. I verified this by diffing the new blocks against agg-gb300-dcp8-nospec-mooncake-agentic.yaml's existing telemetry block, which is byte-identical apart from concurrency values. I also cross-checked the four recipes' concurrency lists against the corresponding search-space entries in configs/nvidia-master.yaml for the kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg config-key, confirming all four conc-lists match exactly, and confirmed the perf-changelog append preserves all prior bytes (only additions after line 7438, at EOF).

Other factors

The PR description itself notes the feature is currently blocked (parent dependency, unresolved port-9401 conflict, no validated disaggregated power measurement yet) — but that is a rollout/operational concern for the author's team, not a code-correctness issue in this diff. The bug hunter reported no findings, and I found nothing beyond that on my own pass through the diff, master config, and doc changes.

@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-aggregate-power branch from 2d4b118 to 58f476e Compare September 12, 2026 09:33
@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-disaggregated-power branch from 702a76d to f56f70a Compare September 12, 2026 09:33

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f56f70a. Configure here.

collector_join_timeout_seconds: 12
dcgm_exporter:
container_image: dcgm-exporter
port: 9401

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GB300 recipes use conflicting exporter port

High Severity

The four GB300 disaggregated recipes start the required DCGM exporter on 9401, which the parent GB300 aggregate work already showed is occupied by the node system service. With required: true and REQUIRE_POWER: "1", collector startup fails and the jobs cannot produce valid power results.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit f56f70a. Configure here.

基于 main 独立启用 GB300 聚合部署实测功耗;仅复用已合入的共享采集代码,保留配方和端口修复。
@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-disaggregated-power branch from b44e63a to 9a7f353 Compare September 13, 2026 00:13
@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-aggregate-power branch from 78dde37 to 49c49b0 Compare September 13, 2026 00:13
测试执行实际 recipe 遥测开关与模型、框架、精度、场景限制,避免预设功耗标志绕过生产判断。
启用 GB300 分离部署实测功耗,并仅依赖 GB300 公共启动接线。
GB300 分离部署改用已验证的 19401 端口;验证 prefill 与 decode 分组均传递该端口,并保留完整硬件验证待办。
@edwingao28
edwingao28 force-pushed the feat/kimik3-gb300-disaggregated-power branch from 9a7f353 to f72e2ca Compare September 13, 2026 00:49
Base automatically changed from feat/kimik3-gb300-aggregate-power to main September 13, 2026 09:37
@edwingao28
edwingao28 requested a review from a team September 13, 2026 09:37
中文:同步已合并的主分支,保留分离部署功耗改动及主分支历史更新记录。
中文:同步最新 main,完整保留 GB300 分离部署配方、功耗配置及两条历史 changelog。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant