[AgentX Power] collect measured power across B200 K3 recipes / 为 B200 K3 配方采集实测功耗 - #2926
[AgentX Power] collect measured power across B200 K3 recipes / 为 B200 K3 配方采集实测功耗#2926edwingao28 wants to merge 7 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
将新增 K3 功耗配方记录关联到独立交付 PR #2926，保留所有历史 changelog 内容。
edwingao28
changed the title
复用共享 DCGM 与功耗适配器，为现有 K3 B200 C1 双节点运行时采集并保留部署功耗证据。配方启用单独交付，保留后续硬件验收与扫描调度要求。
将新增 K3 功耗配方记录关联到独立交付 PR #2926，保留所有历史 changelog 内容。
将七个 B200 K3 配置的元数据修正为 DCP8、KV offload none；保留现有服务行为、Mooncake connector 和合成接受率，历史测量工件不变。
复用 C1 已验证的 16 卡功耗采集路径，为其余六个并发配方声明相同遥测和各自的测量窗口。保留现有服务、模型及合成接受率设置，明确区分共享路径验收与缺失的各并发实测数据。
移除七个 B200 K3 配方的 localhost 覆盖值，使用 producer 发现的逻辑 leader 地址，使固定版本 AIPerf 正确去重。实际渲染和指标重放确认服务器计数不再重复，保留原始请求与能耗证据及历史测量限制。
移除 DCGM 镜像名的提前转换，交由现有导入工具统一生成 Enroot 地址。真实启动器代码的缓存缺失测试先复现双井号错误，修复后 B200、H200 和 GB300 的 registry 检查全部通过。已有缓存路径不变。
中文：精简 B200 K3 功耗中英文说明，保留自动采集行为、验证限制及证据链接。
9dc9423 to
db9aaff
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it's a large, multi-file change touching multi-node SLURM launch/power-validation logic and CI artifact conditions, a human look would still be worthwhile.
What was reviewed: the new AGENTX_POWER code path and pin selection in runners/launch_b200-nscale-slurm.sh (including the per-concurrency power_adapter loop and exit-code propagation), the new --multinode-contract-missing failure path in power_adapter.py, the DCGM exporter enroot-ref fix, and the widened GitHub Actions artifact upload globs. Checked the per-loop AGENTX_POWER_RC handling for silently dropping an earlier concurrency's failure — a later success does not reset it, so overall failure detection still triggers correctly.
Extended reasoning...
Overview
This PR adds official GPU power telemetry collection for seven B200 Kimi-K3 AgentX multi-node recipes. It touches runners/launch_b200-nscale-slurm.sh (new AGENTX_POWER code path, pin selection, per-concurrency power_adapter invocation loop, and an unrelated DCGM exporter enroot-ref fix), utils/agentic/aggregation/power_adapter.py (new --multinode-contract-missing explicit-invalid-verdict path via a refactored _write_multinode_failure_validation helper), process_agentic_result.py (adds num_gpus), benchmark_lib.sh (GPU identity CSV sidecar, routing to the new power_adapter call), seven recipe YAMLs plus configs/nvidia-master.yaml (metadata corrections to TP8/PP2/DCP8, kv-offloading none), GitHub Actions workflow templates (widened artifact upload conditions/globs), docs (bilingual), and perf-changelog.yaml (appended entry). Several new/modified test files accompany the adapter and result-processing changes.
Security risks
No injection, auth, or data-exposure concerns were identified. The changes are limited to internal CI/benchmark infrastructure (SLURM job launching, local file I/O for power metrics, YAML config). No secrets or credentials are introduced or handled differently.
Level of scrutiny
This warrants a moderate-to-high level of scrutiny: it is a fairly large diff (21 files) spanning shell script control flow, a Python validation adapter, GitHub Actions workflow conditions, and seven paired recipe/master-config edits that must stay consistent with each other. The correctness of the power-validation exit-code propagation and the new failure-path semantics in power_adapter.py are subtle enough that a maintainer familiar with the srt-slurm power contract should double check the intended behavior, especially since the PR description itself flags this as a draft pending further rollout gates and evaluation evidence.
Other factors
The bug-hunting run itself reported zero findings. One candidate concern (AGENTX_POWER_RC being overwritten each loop iteration, potentially masking an earlier concurrency's power validation failure) was examined directly in the diff: the pattern is (...) || AGENTX_POWER_RC=$?, which only assigns on failure, so a later successful iteration cannot silently clear a prior failure — the aggregate failure detection (if [[ "$AGENTX_POWER_RC" != "0" ]]) still correctly triggers on any failure in the loop, so this was correctly ruled out as not a functional bug (though the specific surviving exit code may not correspond to the first failure, which is a minor diagnostic nit, not a correctness issue). Given the size and cross-cutting nature of the change plus its own "keep draft" framing, human review remains warranted despite no bugs being found.
This review covers commit 9dc9423, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
close as this is included in #3030 |
1 participant
Description
Collect and validate measured GPU power automatically for all seven existing B200 Kimi-K3 AgentX recipes (C1/4/8/14/24/48/96). Reuse the shared monitor, measurement-window helper and adapter with compatible two-node vLLM producer
80d7203e424f903c9017de4608ee2044afce9574; retain diagnostics before strict validation failures. Correct their metadata to TP8/PP2/DCP8 andkv-offloading: none, matching the unchanged serving recipes and disabled Mooncake offload.Stacked on #2923. C1/4/8/14 retain DSpark7 and synthetic acceptance length 3.84; C24/48 retain DSpark4 and 3.36; C96 retains no speculation. Actual renderer comparisons confirm all six extensions preserve serving/model/settings and use the C1-qualified telemetry path. All seven schemas, exact matrix equality, shell syntax and diff checks pass; the shared producer passed 548 tests and shared result processing passed 221 tests.
The ordinary 16-B200 C1 pilot, immutable source
11240cafbcf3c0786f1ace2283e622ff5ce63f0c, completed and uploaded all four native artifact bundles. Independent validation matched all 16 GPU UUIDs, the full 3611.995-second window, raw request accounting and 21,637,117.619 GPU-board joules. Of 236 profiling requests, 235 succeeded; one parser error plus 11 warmup errors reject a clean comparison despite valid power. Six other concurrency measurements remain missing. Native pilot metadata keeps its older DCP1/DRAM labels; the source correction does not rewrite evidence. The pilot scraped the same logical server by IP and localhost, doubling server cumulative counters. The seven recipes now use the discovered IP; actual pinned-producer/AIPerf checks select one endpoint with unchanged serving commands, and 47 focused tests pass. An offline single-endpoint replay halves duplicated server counters while raw request and energy denominators remain unchanged.Keep draft pending canonical rollout gates. The current changelog selects all seven recipes. Merge/sweep requires its approved budget/window, applicable evaluation evidence and Core/CODEOWNER sign-off. Manual e2e artifacts do not qualify for canonical run-sweep reuse and have not created production data.
The shared B200 exporter cold-import path also passes its plain image name through the existing URI helper exactly once. A real-launcher regression reproduces the former double-registry-separator failure; all three hardware URI cases pass after the fix. The current pilot reused the cached exporter; no new image import or GPU experiment was needed.
Related Issue
No linked GitHub issue. / 暂无关联 GitHub issue。
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Historical changelog bytes are preserved; only this PR's appended entry changed.
中文说明
为现有七个 B200 Kimi-K3 AgentX 配方自动采集并校验 GPU 功耗，复用共享监控、测量窗口和适配器，固定兼容双节点 vLLM 的 producer。元数据修正为 TP8/PP2/DCP8、KV offload none，与未改变的服务配置一致。全部七个配方架构检查及矩阵核对通过；六个扩展的实际服务命令与原版本完全一致，保留各自模型和合成接受率设置。
C1 普通 CI 试验已完成并上传全部四类原生工件。独立核对全部 16 个 GPU UUID、3611.995 秒窗口、请求核算与 21,637,117.619 焦耳 GPU 板卡能耗。236 个 profiling 请求中 235 个成功，另有 1 个解析错误和 11 个 warmup 错误，因此功耗有效但不能用于无错误对比。其余六个并发仍缺少实测数据；旧工件 DCP1/DRAM 标签保持原样。该试验通过 IP/localhost 重复采集同一服务器，导致累计计数加倍；七个配方现改用发现的 IP，固定版本 producer/AIPerf 验证仅保留一个地址，服务命令不变，47 项针对性测试通过。离线单地址重放修正重复累计计数，原始请求和能耗分母保持不变。
保持草稿，等待七点规范 sweep 的资源窗口、评估证据及 Core/CODEOWNER 签核；手动 e2e 工件不能作为规范 sweep 复用，也未新增生产数据。历史 changelog 内容保持不变。
共享 B200 exporter 缓存缺失路径现仅转换一次 registry 分隔符。真实启动器测试先复现双井号错误，修复后 B200、H200、GB300 三个 URI 检查全部通过。当前试验复用了缓存 exporter，没有新增镜像导入或 GPU 试验。
Note
Medium Risk
Changes benchmark launch, post-run validation, and metric collection for seven production AgentX configs; serving commands are intended unchanged but failed power validation now fails the job after artifacts are staged.
Overview
Enables required DCGM deployment GPU power telemetry for all seven B200 Kimi-K3 AgentX srt-slurm recipes (C1/4/8/14/24/48/96). Each recipe now declares
telemetry(dcgm-power), per-runbenchmark.concurrencies, and drops theAIPERF_SERVER_METRICS_URLSlocalhost override so server metrics use the producer-discovered endpoint and avoid double-counting the same vLLM engine.The B200 nscale launcher gains an AgentX power lane: it pins a dedicated
srt-slurmproducer SHA for Kimi-K3 AgentX vLLM, overlays the Kimi-K3 agentic recipes, injects power concurrencies into the applied config, runspower_adapterwith--require-powerafter the job (staging diagnostics before failing), and fixes DCGM exporter cold-import to pass the image reference through the URI helper once.Master config rows for
kimik3-fp4-b200-dynamo-vllm-agentic-dsparkare aligned with actual serving:kv-offloading: none,dcp-size: 8, and Mooncake offload-backend labels removed (Mooncake connector with offload disabled in recipes is unchanged). Docs, perf-changelog, and launcher routing tests are updated accordingly.Reviewed by Cursor Bugbot for commit db9aaff. Bugbot is set up for automated code reviews on this repo. Configure here.