[AgentX GB300] enable Kimi-K3 aggregate power / 启用聚合部署功耗

[edwingao28]

Description

Enable required GPU power collection for two GB300 Kimi-K3 aggregate recipes, using exporter port 19401.

Testing: 24 local checks pass. Full sweep passed four benchmarks and four selected tool-call evals; eight-GPU power reintegration and successful-client token accounting validated. Request errors, warmup failures and deadline cancellations remain recorded.

Published: Four validated power points. #3047 requires separate disaggregated validation.

中文

为两个 GB300 Kimi-K3 聚合 recipe 启用必需 GPU 功耗采集，exporter 使用 19401 端口。

测试： 24 项本地检查通过；完整 sweep 的四个基准点和四组所选工具调用 eval 通过。已验证八张 GPU 的功耗积分及成功请求的客户端 token 计数；请求错误、预热失败和窗口结束时取消的请求均保留。

发布： 四个有效功耗点已入库并通过公开 API 核验。#3047 仍需单独验证分离部署。

Related Issue

Scope index and shared policy / 范围索引与共同规则: #3030.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /reuse-sweep-run on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.

---

Note

Medium Risk
Changes GB300 multinode launcher clone routing, Slurm job monitoring, and strict power result collection for large Kimi-K3 agentic runs; mis-routing or collection failures would block benchmark publication.

Overview
Turns on required DCGM power telemetry (exporter port 19401) for the two GB300 Kimi-K3 aggregate agentic srt-slurm recipes, with head client placement, recipe-specific concurrency sweeps, and ENABLE_AGENTX_POWER / REQUIRE_POWER benchmark env.

launch_gb300-nv.sh adds a USES_AGENTX_POWER path for kimik3 + fp4 + dynamo-vllm agentic recipes: clones a pinned AgentX power producer (AGENTX_POWER_SRT_SLURM_PIN), overlays workspace recipes, injects concurrencies from the runner, streams Slurm logs via stream_slurm_job_log, and post-job collect_agentic_power_results (non–eval-only), failing the run on job/power validation errors. DCGM-power is no longer limited to dynamo-sglang when this route applies.

perf-changelog.yaml documents the two config keys and the 19401 port change. test_gb300_power_official_contract.py adds routing/commit checks for the Kimi AgentX clone path.

^{Reviewed by Cursor Bugbot for commit 49c49b0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @edwingao28's task in 3m 44s —— View job

---

Review of PR #3046

• Gather context (diff vs feat/kimik3-gb200-power, AGENTS.md)
• Review launcher, recipe, changelog, and test changes
• Run CPU tests / syntax checks (blocked locally, see note)
• Post review

LGTM - no blocking issues found.

What I checked in runners/launch_gb300-nv.sh:

• The USES_AGENTX_POWER gate (lines 196-206) mirrors the GB200 launcher exactly, and the GB300 Kimi-K3 disagg recipes have no telemetry: block, so they still fall through to the generic IS_AGENTIC v1.0.36 route.
• The pinned-producer clone verifies HEAD before writing power-producer-sha.txt, and the --no-preflight flag is already covered by IS_AGENTIC.
• CONC_LIST is exported by benchmark-multinode-tmpl.yml, so the concurrency injection and collect_agentic_power_results call have their input. Since INFMAX_WORKSPACE equals GITHUB_WORKSPACE on this cluster, copy_agentic_results hits the -ef same-file path and does not error.
• The failure path exits after the LOGS_DIR check so the EXIT trap still bundles diagnostics. The real run in 34658869181 confirms this end to end: route selected, pin checked out, inject_srt_power_concurrencies.py applied, collector ran, AGENTX_POWER_RC=1, provenance markers copied. The failure there is the exporter task exiting on both nodes within 4s of start (exporter_exited, exporter_startup_timeout), which is the acknowledged port-9401 issue, not launcher logic.

Recipes: concurrencies match the master-config conc-list values, the telemetry block is byte-identical to the GB200/B200 siblings, and model.container still matches image. perf-changelog.yaml entry is appended at the tail and both config keys exist in configs/nvidia-master.yaml. The test-marker change to if [[ "$USES_AGENTX_POWER" == "1" ]] correctly anchors on the real first branch of the routing chain.

Note: pytest utils/test_gb300_power_official_contract.py, bash -n, and shellcheck were blocked by the action's sandbox permissions, so I could not re-run them here. The CI "Test Changelog Gate" and "Test Process Result" checks on this head SHA are green. Hardware power validation remains pending on the exporter port fix, as the PR description states.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2d4b118. Configure here.}

[claude]

I reviewed this PR and found no bugs. Because it touches multi-node SLURM launcher job-monitoring/exit-code propagation and adds a second power-telemetry producer pin, and the author notes hardware power validation is still pending, a human look is still worthwhile.

What was reviewed:

• AGENTX_POWER_RC propagation through stream_slurm_job_log/collect_agentic_power_results: collect_agentic_power_results independently checks sacct job status, so a job that dies early still ends up with a nonzero final rc even though the variable gets reassigned.
• Whether the new AgentX power route needs runners/patch_srt_vllm_dp_ranks.py like gb200's kimik3 route does: gb300's existing (pre-diff) non-power kimik3 path already skips that patch, so this isn't a new gap introduced here.
• Confirmed stream_slurm_job_log/collect_agentic_power_results already exist and are sourced via runners/slurm_utils.sh.
• perf-changelog.yaml is only appended to, and the two recipe YAML diffs add only telemetry/benchmark fields (no model.container/image change), so no master-config sync is needed.

Extended reasoning...

Overview

This PR extends runners/launch_gb300-nv.sh with a new "AgentX power" route for Kimi-K3 fp4 dynamo-vllm agentic recipes: a USES_AGENTX_POWER gate, a separate srt-slurm producer pin (AGENTX_POWER_SRT_SLURM_PIN), a clone/checkout/copy of that pin's recipe tree, an inject_srt_power_concurrencies.py call, and a replacement of the old inline tail/poll job-monitoring loop with calls to stream_slurm_job_log/collect_agentic_power_results plus an AGENTX_POWER_RC exit-code propagation path. Two recipe YAMLs gain telemetry/benchmark blocks, perf-changelog.yaml gets one appended entry, and the contract test is updated to match the new routing marker and parametrized over both gb200 and gb300 launchers.

Security risks

No injection, auth-bypass, or data-exposure concerns. The pinned-commit checkout is verified with git rev-parse HEAD equality checks before use, mirroring the existing pattern for other pins in this file.

Level of scrutiny

This is CI/benchmark infrastructure rather than a user-facing security boundary, but it is non-trivial: it changes exit-code/error-propagation semantics for a production benchmark launcher and introduces a second, framework-specific producer pin alongside the existing one. I verified stream_slurm_job_log/collect_agentic_power_results are pre-existing, already-sourced helpers (from runners/slurm_utils.sh) reused verbatim from launch_gb200-nv.sh's equivalent route, and that collect_agentic_power_results independently validates job completion via sacct, so the apparent "overwrite, not OR" of AGENTX_POWER_RC does not silently swallow a failed job in practice. I also checked the missing patch_srt_vllm_dp_ranks.py call raised as a candidate concern: gb300's pre-existing (unmodified by this diff) non-power kimik3 path already omits that patch, so the new AgentX branch is consistent with gb300's existing behavior rather than introducing a fresh regression — this is a divergence from gb200's kimik3 route that predates this PR. Both candidates check out as non-issues on direct code reading, matching what the earlier bug-hunt round concluded.

Other factors

The PR description itself states hardware power validation is pending and that the change is blocked on a dependency (#3045) and an unresolved exporter port conflict, which is a reasonable signal that a human maintainer familiar with the GB300/AgentX power rollout should look this over before merge even though no bug was found. perf-changelog.yaml was only appended to (byte-preserving), and the two recipe YAML changes add only telemetry/benchmark fields with no model.container/image change, so no master-config sync was needed for this diff.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34744429340
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34744429340

[edwingao28]

/reuse-sweep-run 34744429340