[PowerX] collect native llm-d 8k1k power / 采集 llm-d 8k1k 原生功耗 - #3056
[PowerX] collect native llm-d 8k1k power / 采集 llm-d 8k1k 原生功耗#3056edwingao28 wants to merge 47 commits into
Conversation
保留完整请求结果状态与诊断批次，拆分 Slurm 生命周期和原生功耗采集。
为固定序列场景增加默认关闭的功耗要求，并贯通矩阵及工作流；保持 AgentX 和评估隔离。
拆分原生多节点功耗采集契约、UTC 上下文和生命周期屏障，不启用新路径。
验证 Slurm 完成状态并让 llm-d 工作进程正常退出，先保存诊断证据。
整合 llm-d 所需的正常完成契约。 # Conflicts: # docs/results-and-ingestion.md # docs/results-and-ingestion_zh.md # perf-changelog.yaml
整合按场景启用功耗的矩阵契约。
验证 Slurm 完成状态并让 llm-d 工作进程正常退出，先保存诊断证据。
拆分原生多节点功耗采集契约、UTC 上下文和生命周期屏障，不启用新路径。
为 llm-d 8k1k 收集原生功耗，并协调节点 readiness、排空与结果证据保存。
同步原生采集器前置交付。 # Conflicts: # docs/results-and-ingestion.md # docs/results-and-ingestion_zh.md # perf-changelog.yaml # runners/test_native_collector_barriers.py
同步 Slurm 前置交付引用。 # Conflicts: # benchmarks/multi_node/llm-d/job.slurm # benchmarks/multi_node/llm-d/server.sh # docs/results-and-ingestion.md # docs/results-and-ingestion_zh.md # perf-changelog.yaml
保留每个测量窗口的 SMI 上下文工件，覆盖原 #3027 的共享上传规则。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
未启用功耗时跳过 llm-d 原生采集准备和挂载。
同步功耗上下文产物保留规则。
按完整字段解析 Slurm 分配退出码，避免误用 DerivedExitCode。
独立生成 prefill 与 decode 角色指标，并在 AMD 采集器被回收前完成取消状态记录。
整合原生采集器退出时限和单角色功耗修复。
整合严格匹配 Slurm ExitCode 字段的修复。
在现有 CPU 工作流中执行原生采集契约回归，并跟踪对应文件变更。
在现有 CPU 工作流中执行 llm-d 生命周期回归，并跟踪相关脚本变更。
将新增功耗回归测试与相关路径接入现有 CPU 工作流。
保留共享原生功耗与运行时测试命令的并集。
同步共享 Slurm 生命周期 CPU 测试接线。
矩阵生成同时识别校验允许的字段名与别名，避免静默丢失功耗门槛。
同步矩阵字段别名的回归修复。
将已验证的信号就绪屏障移至共享原生采集前置，避免重放 exec 前的竞态。
复用共享原生分支的信号就绪屏障，消除测试进程 exec 竞态。
保留主分支已发布的 Kimi-K3 B200 改动，将当前契约声明追加至 changelog 末尾。
保留原生采集测试并纳入主分支 Kimi-K3 测试，复用已验证的业务改动。
合并 Kimi-K3 B200 前置，并保留所有路径在失败返回前的诊断采集。
保留普通固定序列与 AgentX 路由隔离、失败结果保留和 CPU 测试并集。
同步 Slurm 前置并保留各 scope 独立 changelog 条目。
将已验证的信号就绪屏障移至共享原生采集前置，避免重放 exec 前的竞态。
同步 Slurm 独立前置的已验证信号就绪测试修复。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked whether the log-copy step's || main_rc=$? at job.slurm:96 could flip a genuinely successful run into a reported failure whenever a node's raw-power directory is missing. native_power_collect.sh runs mkdir -p "$power_dir" unconditionally as its very first action whenever POWERX_NATIVE_ENABLED=1, so every node that starts the collector will have its node-$RANK directory present — making the "missing directory on an otherwise successful run" path effectively unreachable in practice.
Extended reasoning...
A CONFIRMED finding (job.slurm:96, POWERX_RAW_ROOT never cleaned up under /tmp) is already posted inline and is a real disk-leak concern worth a human's attention, so this run should not approve. I independently traced the related concern about main_rc being overwritten by the log-copy step when a node's raw-power source directory is missing: reading benchmarks/native_power_lifecycle.sh and benchmarks/native_power_collect.sh (both unchanged by this diff, so out of review scope but relevant context) shows mkdir -p "$power_dir" executes unconditionally as the first statement of the collector script for every node whenever native power is enabled, before any of the later failure branches. That means the specific failure mode described in the ruled-out candidate list — a successful run (main_rc=0) getting overwritten because a node's directory never existed — requires the collector process to never have started at all on that node, which would itself typically correlate with a non-zero overall job outcome rather than a "genuinely successful" one. This is a narrow, concrete point beyond the already-reported findings, so I'm noting it per the defer exception rather than staying fully silent, without restating or diminishing the CONFIRMED finding itself.
| export POWERX_RAW_ROOT="${POWERX_RAW_ROOT:-/tmp/inferencex-native-$SLURM_JOB_ID}" | ||
| srun --nodes="$NUM_NODES" --ntasks-per-node=1 mkdir -p "$POWERX_RAW_ROOT" | ||
| if [[ "$POWERX_NATIVE_ENABLED" == 1 ]]; then | ||
| mkdir -p "$BENCHMARK_LOGS_DIR/power_control-$SLURM_JOB_ID" "$DI_REPO_DIR/LOGS/native_power" | ||
| chmod 777 "$BENCHMARK_LOGS_DIR/power_control-$SLURM_JOB_ID" | ||
| srun --nodes="$NUM_NODES" --ntasks-per-node=1 chmod 777 "$POWERX_RAW_ROOT" | ||
| srun --nodes="$NUM_NODES" --ntasks-per-node=1 bash -c ' | ||
| { hostname; timedatectl show -p NTPSynchronized --value 2>/dev/null || echo false; } \ | ||
| > "$BENCHMARK_LOGS_DIR/power_control-$SLURM_JOB_ID/host-$SLURM_PROCID" | ||
| ' | ||
| fi | ||
| main_rc=0 | ||
| finish_llmd_job() { | ||
| local rc=$? | ||
| trap - EXIT | ||
| [[ "$rc" == 0 ]] || main_rc=$rc | ||
|
|
||
| if [[ "$POWERX_NATIVE_ENABLED" == 1 ]]; then | ||
| # Copy as the host user only after collectors and serving containers drain. | ||
| srun --nodes="$NUM_NODES" --ntasks-per-node=1 bash -c ' | ||
| source_dir="$POWERX_RAW_ROOT/node-$SLURM_PROCID" | ||
| [[ -d "$source_dir" ]] || exit 1 | ||
| cp -R "$source_dir" "$DI_REPO_DIR/LOGS/native_power/" | ||
| ' || main_rc=$? |
There was a problem hiding this comment.
🟡 (optional) POWERX_RAW_ROOT (/tmp/inferencex-native-$SLURM_JOB_ID) is mkdir'd on every node for every llm-d job (line 74, unconditional) and, when native power is enabled, filled with per-GPU raw power samples for the whole run, but it is never removed: finish_llmd_job (lines 90-97) only cp -R's it into LOGS/native_power/ and leaves the /tmp source behind. Every job gets a unique dir (keyed by SLURM_JOB_ID), so repeated sweeps permanently accumulate one leftover directory per node per job on local /tmp (often size-limited or tmpfs) until an operator manually cleans it up. Fix: rm -rf "$POWERX_RAW_ROOT" per node (e.g. an extra srun step in finish_llmd_job, or an EXIT-time cleanup) after the copy succeeds or the job ends, for both the native-power-enabled and disabled cases.
Extended reasoning...
job.slurm:73-74 unconditionally sets POWERX_RAW_ROOT under /tmp and mkdir's it via srun on every node, regardless of POWERX_NATIVE_ENABLED; it is bind-mounted into every container (lines 140/238) either way. When POWERX_NATIVE_ENABLED=1 (llm-d 8192/1024 require-power runs), native_power_collect.sh (via powerx_start_collector) writes per-GPU power samples there for the full benchmark duration. finish_llmd_job's EXIT trap (lines 84-99) only does cp -R "$POWERX_RAW_ROOT/node-$SLURM_PROCID" "$DI_REPO_DIR/LOGS/native_power/" when native power was enabled; there is no rm -rf of the /tmp source in that block, nor anywhere else in the repo (confirmed via grep for POWERX_RAW_ROOT). Since the directory name embeds SLURM_JOB_ID, no job ever reuses or overwrites another job's leftover, so every sweep run (canary, full sweep, or manual dispatch) adds one more stale directory per node, with real per-GPU sample data when power collection is on, on a filesystem that on many SLURM clusters is either capacity-limited or backed by RAM (tmpfs).
Verification: nit. Real /tmp leak introduced by this change, low impact. job.slurm:73-74 sets POWERX_RAW_ROOT=/tmp/inferencex-native-$SLURM_JOB_ID and mkdir -p's it via srun on every node UNCONDITIONALLY (before the POWERX_NATIVE_ENABLED gate at line 75). It is bind-mounted into every container either way (line 140 docker, line 238 pyxis). When native power is enabled, server.sh:152 (powerx_start_collector…
只忽略已定义的诊断 sidecar，保留损坏与缺失点门禁；旧处理器逐点处理并累计失败，新版导入错误不回退。
继承已验证的 sidecar 分类与旧版本逐点结果处理，并保留 UTC 产物测试。
恢复合并时意外改动的测试注释分隔线与空行。
同步结果前置的 sidecar 排除及旧版处理器失败传播。
独立核对每种角色的 GPU 数量，并记录窗口外异常样本，窗口内异常仍阻止功耗有效性。
同步共享原生契约对角色身份与边界行审计的修复。
发布就绪收据前检查采集进程仍存活，并将 AMD 结束身份采集失败写入失败收据。补充真实进程与清单回归测试。
同步原生采集器就绪及结束身份失败收据修复。
同步主分支的原生采集器 UTC 边界和诊断修复，保留 llm-d 功耗集成并明确尚未合并的 Slurm 生命周期依赖。
中文：合入主分支 GLM MI355X 更新，保留 llm-d 功耗功能、采集器局部 UTC 边界及独立性能日志。
中文：合入主分支 TileRT 更新，同时保留 llm-d 运行逻辑、双语说明和两条生命周期测试路径。
中文：合入 GLM AgentX 功耗采集，保留 llm-d 与原生作业状态优先级，以及失败时的诊断产物。
中文：同步主分支 GB300 功耗收集并保留 llm-d 全节点监控，保持原生失败状态优先。
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
1 participant
Description
Require native measured power only for llm-d 8192/1024. Coordinate per-node readiness, collector drain and normal completion while retaining results and audit artifacts.
Testing: CPU CI passed at
ce06e2708, including four lifecycle cases. Hardware sweep/evals and CODEOWNER approval remain pending.中文
仅 llm-d 8192/1024 要求原生实测功耗；协调节点就绪、采集器排空与正常退出，并保留结果和审计产物。
测试：
ce06e2708的 CPU CI 通过，含四个生命周期用例。硬件 sweep/eval 与 CODEOWNER 审批仍待完成。拆自 #3027，依赖 #3051、#3052、#3050。
Related Issue
Split from #3027; depends on #3051, #3052 and #3050.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
High Risk
Changes multinode llm-d Slurm shutdown, coordinator topology, and benchmark job success semantics while enabling mandatory native power for a production GB200 config.
Overview
Adds an opt-in
require-powermatrix flag (only 8192/1024 fixed-seq-len; stripped from eval-only rows) and turns it on fordsv4-fp4-gb200-llmd-vllm, wiring sweeps/E2E to passrequire-powerinto benchmark workflows when the matrix row requests it.For that llm-d path,
submit.shsetsPOWERX_NATIVE_ENABLEDandjob.slurm/server.shintegratenative_power_lifecycle: per-node collector mounts, coordinator readiness/drain, staging underLOGS/native_power, and normal Slurm completion via a status-bearing.bench_donemarker (noscancelon success). The benchmark coordinator role moves to the prefill leader node; workers exit from the published exit code.verify_slurm_job_completioninslurm_utils.shrecordsslurm_job_*_outcome.txtfromsacct/scontrol; launchers collect artifacts before failing on non-zero allocation status, and power-audit uploads include those receipts. Docs,perf-changelog, and lifecycle/pytest coverage accompany the change.Reviewed by Cursor Bugbot for commit ebdb138. Bugbot is set up for automated code reviews on this repo. Configure here.