[PowerX] measure Grace CPU-side power on NVL72 power lanes / 在 NVL72 功耗 lane 上实测 Grace CPU 侧功耗

.github/AGENT_OPERATIONS.md

@@ -94,6 +94,8 @@ Single-node fixed-sequence results may include `power_valid`, `avg_power_w`, `av
 
 Multinode disaggregated results add `prefill_gpu_energy_j`, `decode_gpu_energy_j`, `prefill_avg_power_w`, `decode_avg_power_w`, `prefill_joules_per_input_token`, and `decode_joules_per_output_token`. Role energy covers the full formal benchmark window, not kernel-level phases, and the role watts are that energy divided by the same window and by the role's GPU count.
 
+NVL72 packages that also carry srt-slurm's `power/cpu/` sub-package add `cpu_power_valid`, `avg_cpu_socket_power_w`, `avg_total_cpu_power_w`, `total_cpu_energy_j`, and, when the module sensor is exposed on every socket, `avg_total_module_power_w` and `total_module_energy_j`. That verdict is independent of `power_valid` and of `REQUIRE_POWER`; see [`docs/results-and-ingestion.md`](../docs/results-and-ingestion.md#measured-grace-cpu-side-power-nvl72).
+
 Every power result — valid or invalid, single-node or multinode — carries `power_metric_schema_version`. Version 2 defines each unprefixed `joules_per_*` field as whole-deployment GPU-board energy over the named denominator; role-scoped energy uses the explicit `prefill_*` / `decode_*` keys. Rows without the field predate the whole-deployment switch and their unprefixed joules are not comparable across topologies.
 
 For srt-slurm recipes, `telemetry.enabled: true` with `telemetry.dcgm_exporter` enables official energy collection. The Git submodule pointer at `utils/srt-slurm` is the source of truth for the shared srt-slurm commit, used by both power and non-power NVIDIA lanes. TileRT is the single documented fork exception. CI derives `POWER_PRODUCER_SHA` from the launcher stamp. The aggregate-power and AgentX power tests validate telemetry and provenance. These local tests do not prove hardware power collection. Eligible recipe-gated `dynamo-sglang` dcgm-power lanes are validated.

benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb200-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml

@@ -132,3 +132,6 @@ telemetry:
   dcgm_exporter:
     container_image: dcgm-exporter
     port: 9401
+  cpu_power_exporter:
+    port: 9405
+    source: acpi

benchmarks/multi_node/srt-slurm-recipes/qwen3.5/sglang/gb300-fp8/8k1k/disagg-1p1d-tp4-tp4-stp.yaml

@@ -134,3 +134,6 @@ telemetry:
     # 9401 is already bound by the cluster-level exporter on im-gb300 nodes;
     # use a port outside that range.
     port: 19401
+  cpu_power_exporter:
+    port: 9405
+    source: acpi

docs/results-and-ingestion.md

@@ -101,7 +101,7 @@ InferenceX-app treats routing fields as columns or config dimensions and stores
 
 The serving client records `benchmark_outcome` before saving its raw result. It retains the existing maximum request-failure rate of 5%, including the requested/completed/failed counts. The processor verifies this record, copies it to the aggregate, and returns failure even when telemetry is valid. Zero successful requests retain a diagnostic aggregate without fabricated reciprocal latency. Invalid request counts retain a failed diagnostic outcome with the raw `requested`/`completed` values and an `error`, without a fabricated failed count or rate; the client saves the raw JSON before exiting and the processor still rejects it. Legacy results without outcome metadata remain distinguishable; power validity alone never establishes benchmark success or answer quality.
 
-`power_invalid_reasons` and `power_audit` carry a bounded summary alongside numeric metrics. The summary includes the available measurement window, expected/observed GPU counts, sampling diagnostics, observed device identifiers and producer pin. Its `source` names the retained `power_validation_*.json` sidecar. Device identifiers retain the collector's semantics; local SMI indices are not physical UUID proof.
+`power_invalid_reasons` and `power_audit` carry a bounded summary alongside numeric metrics. The summary includes the available measurement window, expected/observed GPU counts, sampling diagnostics, observed device identifiers and producer pin. Its `source` names the retained `power_validation_*.json` sidecar. Device identifiers retain the collector's semantics; local SMI indices are not physical UUID proof. When the package carries the Grace CPU-side leg, `power_audit.cpu` adds its sensor kind, source, socket counts, row count, and reason codes; see [Measured Grace CPU-side power (NVL72)](#measured-grace-cpu-side-power-nvl72).
 
 For multinode fixed-sequence jobs, `utils/process_result.py --all` processes every available result before returning failure. It accepts `_c<N>_gpus_...`, `_conc<N>_gpus_...`, and AMD `_concurrency_<N>_req_rate_<R>_gpus_...` filenames, including `inf` request rates. It compares result concurrencies with `CONC_LIST`, rejects duplicate or contradictory point identities, and records omissions/errors in `result_processing_<RESULT_FILENAME>.json`. Aggregate workers pass `AGGREGATE_GPUS` with zero role GPU counts to telemetry validation; separate prefill/decode energy remains absent. For a `DISAGG=true` group with zero decode workers, the aggregate row intentionally sets `disagg: false` and reports `num_aggregate_gpu`; the filename, artifact name, and workflow inputs retain the group identity. Downstream consumers should use the row topology to interpret the measurement.
 
@@ -437,6 +437,55 @@ All four values are withheld when telemetry validation fails. Older results rema
 missing until their original raw traces can be replayed; average watts cannot
 supply P75 or P90. The validation sidecar records `power_percentile_method`.
 
+## Measured Grace CPU-side power (NVL72)
+
+GB200 and GB300 NVL72 recipes that enable srt-slurm's `telemetry.cpu_power_exporter` leg write
+`LOGS/power/cpu/samples.csv` and a non-authoritative `cpu_manifest.json` beside the GPU DCGM
+package. The multinode validator accepts the pinned v2.2.1 long format (one row per sensor
+reading, header `schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`)
+and the later wide format (one row per scrape, host, and socket, with `cpu_rail_w`, `soc_w`, and
+`dram_w` reference columns). It classifies each row by its `sensor` cell: `Module Power Socket N`
+is the whole-module reading (Grace, both Blackwell GPUs, HBM, LPDDR5X, and regulator loss);
+`Grace Power Socket N` or `CPU<n>:cpuSidePowerUsageW` is the Grace-side socket total (CPU, SoC,
+and LPDDR5X); `CPU<n>:cpuPowerUsageW` is DCGM field 1130, the CPU rail only. Component rails
+(`CPU Power Socket N`, `SysIO Power Socket N`, DRAM) never feed a published metric.
+
+Per socket the headline series is chosen in the order module, Grace socket total, DCGM CPU rail,
+and one kind must be present for every socket. Every fed series is integrated over the same bound
+formal window as GPU energy with the same trapezoid, linear boundary interpolation, and 3.0 s
+maximum sample gap. Expected sockets are two per distinct worker host in the manifest topology
+(one compute tray per host). The additive keys, rounded like the GPU keys and stripped on re-run,
+are:
+
+| Key | Meaning |
+| --- | --- |
+| `cpu_power_valid` | `1` or `0` for the CPU-side leg; absent when the package has no `cpu/` |
+| `avg_cpu_socket_power_w` | Mean over sockets of each socket's window-mean Grace-side watts |
+| `avg_total_cpu_power_w` | Sum over sockets of window-mean Grace-side watts |
+| `total_cpu_energy_j` | Grace-side energy over the window, all sockets |
+| `avg_total_module_power_w`, `total_module_energy_j` | Module watts and energy; only when a module series exists for every socket |
+
+The Grace-side keys come from the Grace socket total (or, without ACPI, the DCGM CPU rail) even
+when the module sensor is preferred, so a module reading is never published as a Grace-side one.
+The sidecar's `cpu` block and the aggregate's `power_audit.cpu` record `sensor_kind` (`module`,
+`grace_socket`, or `dcgm_cpu_rail`), `source` (`acpi` or `dcgm`), expected and observed socket
+counts, the parsed row count, and reason codes. `power_metric_schema_version` stays `2`.
+
+The leg is best effort and its verdict is independent of `power_valid`: it borrows only the bound
+formal window and the worker-host topology from the GPU package, and no GPU verdict reaches it, so
+an unpinned producer or failed GPU coverage withholds GPU energy while `cpu_power_valid` still
+judges the CPU samples on their own. Any CPU-leg failure records `cpu_power_valid: 0` with no CPU
+keys, leaves every GPU field byte-identical, and never fails `REQUIRE_POWER=1`. Reason codes:
+`cpu_samples_missing`, `cpu_samples_header_mismatch`, `cpu_samples_malformed`,
+`cpu_manifest_invalid`, `cpu_socket_count_mismatch`, `cpu_sensor_kind_mixed`,
+`cpu_sample_gap_exceeded`, `cpu_window_not_bracketed`, and `cpu_window_unavailable` (no completed
+window binds to the result, or the window's own contract checks failed). An overflowed CPU
+integration reuses the GPU leg's `non_finite_power_metric`, and `aggregate_result_missing` or
+`aggregate_result_unwritable` appears in both audits when the aggregate itself cannot be patched.
+At the pinned v2.2.1 the exporter does not classify the Module label,
+so current packages yield the Grace socket total; module keys appear once the upstream exporter
+classifies it. A package without `cpu/` produces an aggregate and sidecar identical to today.
+
 ## Verification and stop conditions
 
 A handoff is verified only when all applicable checks pass.

docs/results-and-ingestion_zh.md

@@ -101,7 +101,7 @@ InferenceX-app 将路由字段作为列或配置维度，并把数值测量存
 
 服务客户端在保存原始结果前写入 `benchmark_outcome`，保留现有的 5% 最大请求失败率，以及请求总数、完成数和失败数。处理器检查该记录并复制到聚合结果中；即使遥测有效，请求失败率超限仍返回失败。零成功请求会保留诊断聚合结果，但不会生成不存在的延迟倒数。请求计数无效时，失败诊断状态保留原始 `requested`/`completed` 值和 `error`，不生成无依据的失败数或失败率；客户端先保存原始 JSON 再退出，处理器仍拒绝该结果。没有状态元数据的历史结果仍可区分；功耗有效不能证明基准成功或答案质量。
 
-`power_invalid_reasons` 和 `power_audit` 在数值指标旁携带有界摘要，包括可用的测量窗口、预期与观测 GPU 数、采样诊断、观测设备标识和生产者版本。`source` 指向保留的 `power_validation_*.json` 工件名称。设备标识保留采集器原有语义，本地 SMI 序号不是物理 UUID 的证明。
+`power_invalid_reasons` 和 `power_audit` 在数值指标旁携带有界摘要，包括可用的测量窗口、预期与观测 GPU 数、采样诊断、观测设备标识和生产者版本。`source` 指向保留的 `power_validation_*.json` 工件名称。设备标识保留采集器原有语义，本地 SMI 序号不是物理 UUID 的证明。当产物包包含 Grace CPU 侧测量环节时，`power_audit.cpu` 会附带其传感器类型、来源、socket 数、行数和原因码；见 [NVL72 Grace CPU 侧实测功耗](#nvl72-grace-cpu-侧实测功耗)。
 
 对于多节点固定序列任务，`utils/process_result.py --all` 先处理所有已有结果，再返回失败。它接受 `_c<N>_gpus_...`、`_conc<N>_gpus_...` 和 AMD 的 `_concurrency_<N>_req_rate_<R>_gpus_...` 文件名，也支持 `inf` 请求速率。它将结果并发度与 `CONC_LIST` 比较，拒绝重复或矛盾的点身份，并将遗漏和错误记录到 `result_processing_<RESULT_FILENAME>.json`。共享工作池通过 `AGGREGATE_GPUS` 及零值角色 GPU 数进行遥测验证；独立的 prefill/decode 能耗保持缺失。当 `DISAGG=true` 的配置组中某个点没有 decode worker 时，聚合行会有意设置 `disagg: false` 并记录 `num_aggregate_gpu`；文件名、工件名和工作流输入仍保留配置组身份。下游应按聚合行的拓扑解释测量结果。
 
@@ -423,6 +423,49 @@ rm -rf -- "$tmp"
 各设备分位数的平均值。遥测验证失败时，四项指标都不发布。旧结果需要使用原始
 遥测重新计算；不能从平均功耗推算 P75 或 P90。验证 sidecar 会记录 `power_percentile_method`。
 
+## NVL72 Grace CPU 侧实测功耗
+
+启用了 srt-slurm `telemetry.cpu_power_exporter` 的 GB200 和 GB300 NVL72 recipe 会在 GPU DCGM
+产物包旁写入 `LOGS/power/cpu/samples.csv` 和一份非权威的 `cpu_manifest.json`。多节点校验器接受
+当前固定版本 v2.2.1 写出的长格式（每个传感器读数一行，表头为
+`schema_version,timestamp_unix,hostname,source,sensor,socket_id,power_w,total_power_w`），以及后续
+版本的宽格式（每次采集、每台主机、每个 socket 一行，另带 `cpu_rail_w`、`soc_w`、`dram_w` 参考列）。
+它按 `sensor` 单元格对每行分类：`Module Power Socket N` 是整模块读数（Grace、两颗 Blackwell GPU、
+HBM、LPDDR5X 及稳压损耗）；`Grace Power Socket N` 或 `CPU<n>:cpuSidePowerUsageW` 是 Grace 侧 socket
+总功耗（CPU、SoC 和 LPDDR5X）；`CPU<n>:cpuPowerUsageW` 是 DCGM 字段 1130，仅含 CPU 供电轨。组件供电轨
+（`CPU Power Socket N`、`SysIO Power Socket N`、DRAM）永远不会进入发布指标。
+
+每个 socket 的主序列按模块、Grace socket 总功耗、DCGM CPU 供电轨的顺序选取，且所有 socket 必须具备
+同一种传感器类型。每条参与计算的序列都在与 GPU 能耗相同的正式测量窗口内积分，采用同样的梯形法、
+线性边界插值和 3.0 秒最大采样间隔。预期 socket 数为 manifest 拓扑中每台不同 worker 主机两个（每台
+主机一个 compute tray）。新增字段与 GPU 字段采用相同的取整规则，重新运行时会先清除旧值：
+
+| 字段 | 含义 |
+| --- | --- |
+| `cpu_power_valid` | CPU 侧测量的有效性，`1` 或 `0`；产物包没有 `cpu/` 时不出现 |
+| `avg_cpu_socket_power_w` | 各 socket 窗口平均 Grace 侧功耗的均值 |
+| `avg_total_cpu_power_w` | 各 socket 窗口平均 Grace 侧功耗之和 |
+| `total_cpu_energy_j` | 窗口内所有 socket 的 Grace 侧能耗 |
+| `avg_total_module_power_w`、`total_module_energy_j` | 模块功耗与能耗；仅当每个 socket 都有模块序列时输出 |
+
+即使优先选用模块传感器，Grace 侧字段仍来自 Grace socket 总功耗（没有 ACPI 时来自 DCGM CPU 供电轨），
+因此模块读数永远不会被当作 Grace 侧读数发布。sidecar 的 `cpu` 块和聚合结果的 `power_audit.cpu`
+记录 `sensor_kind`（`module`、`grace_socket` 或 `dcgm_cpu_rail`）、`source`（`acpi` 或 `dcgm`）、
+预期与观测 socket 数、解析行数和原因码。`power_metric_schema_version` 保持为 `2`。
+
+该测量环节尽力而为，其结论与 `power_valid` 相互独立：它只从 GPU 产物包借用已绑定的正式测量窗口和
+worker 主机拓扑，GPU 侧的任何结论都不会传导过来，因此 producer 固定版本校验失败或 GPU 覆盖不足只会
+使 GPU 能耗不予发布，`cpu_power_valid` 仍按 CPU 采样自身给出结论。CPU 侧的任何失败都会记录
+`cpu_power_valid: 0` 且不输出 CPU 字段，所有 GPU 字段保持逐字节不变，也不会使 `REQUIRE_POWER=1` 失败。
+原因码包括 `cpu_samples_missing`、`cpu_samples_header_mismatch`、`cpu_samples_malformed`、
+`cpu_manifest_invalid`、`cpu_socket_count_mismatch`、`cpu_sensor_kind_mixed`、
+`cpu_sample_gap_exceeded`、`cpu_window_not_bracketed` 以及 `cpu_window_unavailable`（没有已完成的窗口
+与该结果绑定，或窗口自身的契约检查失败）。CPU 积分溢出沿用 GPU 侧的 `non_finite_power_metric`；聚合
+结果本身无法写入时，`aggregate_result_missing` 或 `aggregate_result_unwritable` 会同时出现在两侧的审计
+中。在当前固定的 v2.2.1 版本中，exporter 尚不识别 Module 标签，因此现有产物包只会得到 Grace
+socket 总功耗；上游 exporter 支持该标签后才会出现模块字段。没有 `cpu/` 的产物包产生的聚合结果和
+sidecar 与现在完全一致。
+
 ## 验证和停止条件
 
 只有全部适用检查通过，交接才算验证完成。

infx/results/power/__init__.py

@@ -34,6 +34,21 @@
     "decode_joules_per_output_token",
 )
 ALL_POWER_METRIC_KEYS = WHOLE_METRIC_KEYS + ROLE_METRIC_KEYS
+# Additive Grace CPU-side metrics from the multinode cpu/ sub-package. They
+# share schema version 2 and are governed by cpu_power_valid, not power_valid.
+CPU_METRIC_KEYS = (
+    "avg_cpu_socket_power_w",
+    "avg_total_cpu_power_w",
+    "total_cpu_energy_j",
+    "avg_total_module_power_w",
+    "total_module_energy_j",
+)
+
+
+def _rounded_metric(key: str, value: float | None) -> float:
+    if value is None or not math.isfinite(value):
+        raise ValueError(f"non-finite power metric: {key}")
+    return round(value, 3 if key.endswith(("_w", "_j")) else 6)
 
 
 def with_power_metrics(
@@ -43,26 +58,34 @@ def with_power_metrics(
     schema_version: int,
     power_valid: bool,
     metrics: Mapping[str, float],
+    cpu_power_valid: bool | None = None,
+    cpu_metrics: Mapping[str, float] | None = None,
 ) -> dict[str, Any]:
     """Return a result with stale metrics replaced by one validated metric set.
 
     The caller supplies the metric family and schema version. Detailed invalid
     reasons belong in its validation sidecar, not the numeric metric payload.
     Neither the input result nor the supplied metrics is mutated.
+
+    The CPU-side leg is tri-state: ``cpu_power_valid=None`` means the package
+    carried no ``cpu/`` sub-package, so only stale CPU keys are stripped and the
+    output stays byte-identical to a run without the leg.
     """
     # Do not coerce malformed JSON (such as a list of pairs) into an object.
     # Leave non-mappings to fail on the same operations as the existing adapters.
     data = dict(result) if isinstance(result, Mapping) else result
-    for key in metric_keys:
+    for key in (*metric_keys, *CPU_METRIC_KEYS, "cpu_power_valid"):
         data.pop(key, None)
     data["power_metric_schema_version"] = schema_version
     data["power_valid"] = int(power_valid)
     data.pop("power_invalid_reasons", None)
     data.pop("power_audit", None)
     if power_valid:
         for key, value in metrics.items():
-            if value is None or not math.isfinite(value):
-                raise ValueError(f"non-finite power metric: {key}")
-            precision = 3 if key.endswith(("_w", "_j")) else 6
-            data[key] = round(value, precision)
+            data[key] = _rounded_metric(key, value)
+    if cpu_power_valid is not None:
+        data["cpu_power_valid"] = int(cpu_power_valid)
+        if cpu_power_valid:
+            for key, value in (cpu_metrics or {}).items():
+                data[key] = _rounded_metric(key, value)
     return data