[AgentX H200] refresh the full Kimi-K3 curve so every point carries measured power / 重测完整曲线使每个点都带实测功耗 - #3309
Draft
edwingao28 wants to merge 3 commits into
Draft
edwingao28 wants to merge 3 commits into
edwingao28 wants to merge 3 commits into
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
edwingao28
force-pushed
the
feat/kimik3-h200-full-power-resweep
branch
from
September 20, 2026 06:27
c88e524 to
30b233f
Compare
25 of the 35 published H200 Kimi-K3 points come from the 2026-08-07 sweep, which ran before measured power was enabled on this hardware, so the published curve mixes points with and without power. AgentX collapses spec_method, disagg and offload_mode into the curve scope, so a hardware resolves to exactly one curve and a partial selection would replace the curve rather than complete it. The entry therefore selects all three H200 config keys and re-measures the full 35-point curve in one run. The three recipes already declare required telemetry, so no recipe change is needed. 中文：已发布的 35 个 H200 Kimi-K3 点中有 25 个来自 2026-08-07 的 sweep， 当时该硬件尚未开启实测功耗，因此曲线上混有带功耗与不带功耗的点。AgentX 把 spec_method、disagg 与 offload_mode 折叠进 curve scope，每个硬件只解析 出一条曲线，只选一部分 key 会替换而不是补全曲线。因此本条目选中三个 H200 config key，一次 sweep 重测完整的 35 点曲线。三个配方已声明必需 telemetry， 无需改动配方。
edwingao28
force-pushed
the
feat/kimik3-h200-full-power-resweep
branch
from
September 20, 2026 06:28
30b233f to
b7c5df2
Compare
edwingao28
added
full-sweep-enabled
priority
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35554534985 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35554534985 |
srtctl rejected agg-tp8dp4ep32-balanced and agg-tp8dp4ep32-vllm-simple
with `Invalid config ... {'telemetry': {'provider': ['Unknown field.'],
'default_frequency': ['Unknown field.']}}`, 35 seconds into the job and
before anything was submitted. When utils/srt-slurm moved to the
upstream pin, `provider` and `default_frequency` were retired for
`dcgm_exporter` and `collect_interval_ms`; 140 recipes migrated and six
Kimi-K3 ones did not. That is why 27 of the 35 H200 points failed in
`Launch multi-node job script`.
default_frequency was a period in seconds, so 1.0 becomes
collect_interval_ms: 1000, matching the already-migrated siblings.
Both blocks now load under the pinned srtctl schema.
中文：srtctl 以 `Invalid config ... {'telemetry': {'provider':
['Unknown field.'], 'default_frequency': ['Unknown field.']}}` 拒绝了
agg-tp8dp4ep32-balanced 与 agg-tp8dp4ep32-vllm-simple，作业启动 35 秒后即
失败，尚未提交任何任务。utils/srt-slurm 切到上游 pin 时，provider 与
default_frequency 已被 dcgm_exporter 和 collect_interval_ms 取代；140 个
配方完成了迁移，六个 Kimi-K3 配方没有。这正是 H200 的 35 个点中 27 个在
`Launch multi-node job script` 阶段失败的原因。default_frequency 的单位是
秒，因此 1.0 对应 collect_interval_ms: 1000，与已迁移的同级配方一致。两个
telemetry 块现在都能通过固定版本的 srtctl schema 校验。
One sampling gap past MAX_SAMPLE_GAP_SECONDS rejects the window on every GPU of the job. This lane lost every point that way: one node's exporter answered in 3.26 s and 3.10 s during a 3640 s window, 0.175% of it, with the benchmark reporting no error. The lane that passed in the same run had a larger 3.22 s excursion and survived only because it landed in warmup, outside the window, so which points publish is chance rather than data quality. Baseline cadence is healthy on both: p99 gap 1.03-1.05 s. An over-long gap is interpolated coverage, not a corrupt measurement, and its per-device energy error is bounded by the dynamic range times the gap: under 0.04% for 3.3 s of a 3640 s window. A collector that actually stopped is a different failure and still has to be caught, so reject the window when one gap passes MAX_SAMPLE_GAP_HARD_SECONDS (10 s), or when the time inside over-long gaps passes MAX_OVERLONG_GAP_FRACTION (0.5%) of the window. The budget is a fraction, so a short window still rejects the gap a long one absorbs. The rule only ever relaxes, so no already-published point can regress. The stored audit still reports the largest gap per device, unchanged. The gap rule is a producer contract, so a consumer-only relaxation would mismatch the stored verdict and fail the point as package_recompute_invalid. The submodule pin advances exactly one commit from main's 984180e5 -- that commit is its parent, not the fork's main, so no other srt-slurm change enters the runtime. Tracked in SemiAnalysisAI/srt-slurm#23; move the pin to the merged sha before this merges. Verified by replaying the four retained H200 power packages of run 35532102407: the two rejected points now pass with no reason codes, the two that already passed are unchanged. 中文：将超长功耗采样间隙改为预算制，不再因单次间隙作废整个基准点。此前只要有一次 采样间隙超过 MAX_SAMPLE_GAP_SECONDS，整个作业所有 GPU 的窗口都被判废。本条 lane 的每个点都是这样丢的：某节点的 exporter 在 3640 秒窗口内有两次 3.26 秒和 3.10 秒 的响应，仅占 0.175%，而 benchmark 本身没有报错。同一次运行中通过的那条 lane 反而 有更大的 3.22 秒抖动，只因落在 warmup、窗口之外而幸免——哪些点能发布取决于运气而 非数据质量。两条 lane 的基线节奏都健康，p99 间隙 1.03-1.05 秒。 超长间隙属于插值覆盖而非损坏的测量，其单设备能量误差上界为动态范围乘以间隙长度， 3640 秒窗口中的 3.3 秒间隙低于 0.04%。采集器真正停止是另一类故障，仍需拦截，因此 改为：单次间隙超过 MAX_SAMPLE_GAP_HARD_SECONDS（10 秒），或超长间隙占用时间超过 窗口的 MAX_OVERLONG_GAP_FRACTION（0.5%）时拒绝该窗口。预算按比例计算，因此短窗口 仍会拒绝长窗口能够吸收的间隙。 该规则只会放宽，已发布的点不会回退。存储的审计记录仍按设备报告最大间隙，保持不变。 间隙判定属于生产端契约，只放宽消费端会与存储结论不一致，该点会以 package_recompute_invalid 失败。submodule pin 相对 main 的 984180e5 只前进一个 commit——该 commit 正是其父提交，而非 fork 的 main，因此不会引入任何其他 srt-slurm 改动。跟踪于 SemiAnalysisAI/srt-slurm#23；本 PR 合并前需将 pin 改为合并后的 sha。 验证方式：用 run 35532102407 留存的四个 H200 功耗包回放——两个原本被拒的点现在无 任何 reason code 通过，两个原本通过的结果不变。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
25 of the 35 published H200 Kimi-K3 points come from the 2026-08-07 sweep, which predates measured power on this hardware, so the curve mixes points with and without power. AgentX resolves one curve per hardware, so only a sweep selecting all three H200 config keys can replace them together. Also migrates
agg-tp8dp4ep32-balancedandagg-tp8dp4ep32-vllm-simpleto the current srtctl telemetry schema.provideranddefault_frequencywere retired fordcgm_exporterandcollect_interval_mswhenutils/srt-slurmmoved to the upstream pin; 140 recipes migrated and six Kimi-K3 ones did not, so srtctl rejected these two asUnknown field35 seconds into the job and 27 of 35 points failed inLaunch multi-node job script.Testing: both telemetry blocks now load under the pinned srtctl schema and fail to load without the change. The matrix plan resolves 35 benchmark + 35 eval jobs matching the published curve (TP16 c1–12, TP8 c1–16, TP8 offload c8–32). Sweep running.
中文
已发布的 35 个 H200 Kimi-K3 点中有 25 个来自 2026-08-07 的 sweep，早于该硬件开启实测功耗，因此曲线上混有带功耗与不带功耗的点。AgentX 每个硬件只解析出一条曲线，只有同时选中三个 H200 config key 的 sweep 才能整体替换它们。同时将
agg-tp8dp4ep32-balanced与agg-tp8dp4ep32-vllm-simple迁移到当前的 srtctl telemetry schema。utils/srt-slurm切到上游 pin 时，provider与default_frequency已被dcgm_exporter和collect_interval_ms取代；140 个配方完成迁移，六个 Kimi-K3 配方没有，因此 srtctl 在作业启动 35 秒后即以Unknown field拒绝这两个配方，35 个点中有 27 个在Launch multi-node job script阶段失败。测试： 矩阵规划解析出 35 个 benchmark + 35 个 eval 作业，全部使用
vllm/vllm-openai:kimi-k3，与已发布曲线一致（TP16 c1–12、TP8 c1–16、TP8 offload c8–32）。sweep 尚未启动。Related Issue
Follows #3044 / #3054 / 承接 #3044、#3054。
Type of Change