[AgentX B300] recover Kimi-K3 image and draft storage / 恢复镜像与草稿存储 - #3040
[AgentX B300] recover Kimi-K3 image and draft storage / 恢复镜像与草稿存储#3040edwingao28 wants to merge 8 commits into
Conversation
恢复 Kimi-K3 B300 镜像与集群路由。
d6701b0 to
33cdf99
Compare
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
为 Kimi-K3 的 DSpark 权重导出并挂载持久化可写目录，避免预置目标模型路径遗漏草稿模型存储。将无关 B200 注释改动留给对应 PR。
|
Claude finished @edwingao28's task in 3m 8s —— View job Review: PR #3040
Summary: No blocking issues found. One 🟡 warning posted inline on Checked and OK:
Not verified (sandbox limits):
The acknowledged blocker (mlx5 rail discovery in |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Replace unavailable vLLM build 5894fdf with 3696c77 and route the existing Kimi-K3 recipe to B300 DSXE. This establishes a new runtime baseline." | ||
| - "恢复 Kimi-K3 B300 镜像与集群路由。" |
There was a problem hiding this comment.
🟡 WARNING: This entry says the PR routes the recipe to B300 DSXE, but the runner was already cluster:b300-dsxe on origin/main (changed in #2927, commit 5a473ca). The only config change in this PR is the image tag.
Why it matters: Changelog entries are immutable history once merged (the checklist forbids editing them later), and the reuse/ingest tooling and readers use them to explain result discontinuities. Attributing the cluster move to this PR misdates the runtime baseline change.
Fix:
| - "Replace unavailable vLLM build 5894fdf with 3696c77 and route the existing Kimi-K3 recipe to B300 DSXE. This establishes a new runtime baseline." | |
| - "恢复 Kimi-K3 B300 镜像与集群路由。" | |
| - "Replace the unavailable vLLM build 5894fdf with 3696c77 for the existing Kimi-K3 recipe on B300 DSXE (already routed there in #2927). This establishes a new runtime baseline." | |
| - "为已在 B300 DSXE 上运行的 Kimi-K3 配方，将不可用的 vLLM 镜像 5894fdf 更换为 3696c77，建立新的运行时基线。" |
The same claim appears in the PR title/description ("恢复镜像与集群路由"); worth aligning those too.
修正当前 PR 尚未合并的 changelog，明确 DSXE 路由已经存在，仅镜像建立新的运行时基线。保留基线历史字节。
edwingao28
changed the title
There was a problem hiding this comment.
This automated review ran and one finding is posted inline. Beyond that, I also checked the new image tag vllm/vllm-openai:nightly-dev-x86_64-cu13-3696c77 in configs/nvidia-master.yaml against the other nightly-dev-x86_64-cu13* entries in the same file (which all keep a .0.1 CUDA patch segment) — this looks like the tag as actually published for this specific build rather than a typo, so I did not raise it separately. Given the inline finding, a human look is still worthwhile before merging.
Extended reasoning...
The PR swaps the Kimi-K3 vLLM image tag in configs/nvidia-master.yaml, adds a WRITABLE_MODELS_DIR mount/export for the kimik3/vllm/agentic path in runners/launch_b300-dsxe.sh, and updates docs and perf-changelog.yaml to match. An inline finding (CONFIRMED) was already surfaced on runners/launch_b300-dsxe.sh:493 regarding the shared writable draft-model directory being reachable across the sweep's parallel per-concurrency matrix jobs, which is a real concurrency/data-race concern worth a maintainer's attention given the recipe's 11-way conc-list fan-out on cluster:b300-dsxe.
I independently checked the image tag change (configs/nvidia-master.yaml:1490) against the pattern used by other nightly-dev-x86_64-cu13.0.1-<hash> entries in the same file (lines 965, 1703, 10004) and confirmed the new tag nightly-dev-x86_64-cu13-3696c77 omits the .0.1 segment those entries carry. This is a real deviation from the file's established naming convention, but without registry access to confirm whether Docker Hub actually publishes a .0.1-less tag for this specific commit, I treat it as ruled out rather than a fresh finding — it was already investigated and dropped, and I have no new evidence to escalate it.
Given the CONFIRMED inline finding touches shared-storage semantics across concurrent CI matrix jobs (a correctness/race concern, not a simple typo), this is not the kind of small, mechanical, unambiguous change that warrants outright approval. The change also affects a runner script with cluster-wide launch semantics (runners/launch_b300-dsxe.sh), which merits a maintainer's judgment on whether the shared mount is acceptable given the PR author's own stated caveat that hardware validation is still pending.
No security-sensitive code (auth, crypto, permissions) is touched; the risk here is operational (shared writable storage under concurrent access) rather than an exploitable vulnerability. Test coverage per the PR description is limited to CPU/matrix checks, with GPU validation explicitly blocked/pending, which further supports deferring to a human rather than approving.
| if [[ "$MODEL_PREFIX" == "kimik3" && "$FRAMEWORK" == "vllm" && "${IS_AGENTIC:-0}" == "1" ]]; then | ||
| # The pre-staged target is read-only; DSpark needs the writable, | ||
| # persistent model root as a separate mount. | ||
| mkdir -p "$WRITABLE_MODELS_DIR" | ||
| export WRITABLE_MODELS_DIR | ||
| if [[ "$MODEL_MOUNT_DIR" != "$WRITABLE_MODELS_DIR" ]]; then | ||
| CONTAINER_MOUNTS+=("$WRITABLE_MODELS_DIR:$WRITABLE_MODELS_DIR") | ||
| fi | ||
| fi |
There was a problem hiding this comment.
🔴 Exporting+mounting WRITABLE_MODELS_DIR for kimik3/vllm/agentic makes the DSpark draft-model dir a shared, persistent path across the sweep's parallel per-concurrency matrix jobs (kimik3-fp4-b300-vllm-agentic-dspark's 11-way conc-list); before this diff the var was never exported, so the draft fell back to an unmounted, job-local /data/models with no cross-job interference. Now concurrent jobs race the non-atomic download-check in kimik3_fp4_b300_vllm_mtp.sh, so a job can see another's partially-downloaded (non-empty) dir and skip its own download, then serve vllm against incomplete draft shards. …
Extended reasoning...
…Fix: serialize first-download of the shared draft dir across concurrent jobs (e.g. flock a per-model lock before the exists/ls-A check, or atomic rename after a fully-completed download) so no racer observes a partial directory.
kimik3-fp4-b300-vllm-agentic-dspark's search-space has one entry with conc-list [1,2,4,8,16,24,32,40,48,56,70]; the sweep workflow (run-sweep.yml sweep-agentic) expands this into separate matrix.config entries each carrying a single conc, dispatched as parallel GH Actions jobs on cluster:b300-dsxe. Each job sources launch_b300-dsxe.sh, which (new code, lines 485-493) exports WRITABLE_MODELS_DIR and mounts it read-write into the container. kimik3_fp4_b300_vllm_mtp.sh:58-65 (unchanged) then does: DRAFT_MODEL_PATH=$WRITABLE_MODELS_DIR/Kimi-K3-DSpark; if [[ ! -d
Verification: normal. The new block at runners/launch_b300-dsxe.sh:485-493 (export WRITABLE_MODELS_DIR + RW mount of the shared Lustre root /data/home/sa-gha-runner/models, defined line 22) is reached whenever MODEL_PREFIX=kimik3, FRAMEWORK=vllm, IS_AGENTIC=1 — exactly the kimik3-fp4-b300-vllm-agentic-dspark arm. Behavior change vs base: on the base branch WRITABLE_MODELS_DIR is only assigned at line 22,…
使用模型专用锁串行下载共享 DSpark 权重，交由 HF 校验并续传缓存，避免并发任务把未完成目录当作完整模型。并发下载和失败后续传测试通过；未运行 GPU。
保留 B300 镜像与草稿缓存修复，同步主分支并按追加规则解决变更日志冲突。
同步主分支并保留 B300 恢复改动，按追加规则解决变更日志冲突。
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
中文：合并 main 的 TileRT 原生支持，保留 B300 运行逻辑及各自的文档和变更记录。
中文：合并已验证的 GLM GB200 功耗支持，保持 B300 行为并追加本 PR 的独立变更记录。
1 participant
Description
Replace unavailable Kimi-K3 vLLM build
5894fdfwith3696c77, establishing a new B300 runtime baseline. Persist DSpark weights and serialize cached downloads so concurrent cells cannot serve partial drafts.Testing: Four launcher/download tests passed, including concurrent staging and failed-download propagation; regressions failed before fixes. Matrix, changelog, and Bash checks passed.
Blocked: DSXE EFA conflicts with launcher RoCE assumptions; related same-image CI reports
IBVERBS_PRIVATE_34. Hardware qualification remains pending.中文
中文说明
将不可用的 Kimi-K3 vLLM 镜像
5894fdf替换为3696c77，建立新的 B300 运行时基线。持久保存 DSpark 权重并串行校验或续传缓存，避免并发任务使用未完成的草稿模型。测试： 四项启动器与下载测试通过，包括并发准备和下载失败传播；回归测试在修复前失败。矩阵、changelog 和 Bash 检查通过。
阻塞： DSXE EFA 与启动器的 RoCE 假设冲突；同镜像的相关 CI 报错
IBVERBS_PRIVATE_34。硬件资格验证仍待完成。Related Issue
Scope index and shared policy / 范围索引与共同规则: #3030.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.Note
Medium Risk
Changes benchmark container image baseline and B300 launcher mounts for a live recipe; incorrect mounts could break Kimi-K3 agentic runs but scope is limited to that path.
Overview
Restores the Kimi-K3 B300 DSXE agentic-dspark recipe by swapping the unavailable vLLM image
5894fdffor3696c77innvidia-master.yaml, with matching perf-changelog entries noting a new runtime baseline.For agentic Kimi-K3 + vLLM on B300 DSXE,
launch_b300-dsxe.shnow exportsWRITABLE_MODELS_DIRand adds a second container mount when the read-only staged target (/scratch/models) differs from the persistent draft root, so DSpark weights can survive container reuse. English and Chinese host-setup docs describe this split.Adds
runners/test_kimik3_b300.pyto assert kimik3 gets the writable mount and draft env while other prefixes (e.g. dsv4) do not.Reviewed by Cursor Bugbot for commit 5e6eee7. Bugbot is set up for automated code reviews on this repo. Configure here.