Repository navigation
Fix TRT reuse counters and P/D prompt totals / 修正 TRT 复用计数与 P/D 输入总量 - #4163
Open
chunfangamd wants to merge 23 commits into
Open
chunfangamd wants to merge 23 commits into
chunfangamd wants to merge 23 commits into
Conversation
… / 修正 ATOM 缓存命中率口径并标注有效性 The ATOM agentic result adapter summed prefix_cache_full_tokens over prefill and decode endpoints, reported lmcache_loaded_tokens (transfer volume) as CPU reuse, and set overall equal to GPU. The AIPerf metrics section also spans warmup. - Take endpoint roles from the launcher inputs (SRT_PREFILL/DECODE/AGG_ENDPOINTS). - Count prefill (or unified) endpoints over timeslices inside the profiling window. - CPU = prefix_cache_offload_tokens / full; external aliases CPU; overall = GPU + CPU. - Emit server_metrics.cache_validity so consumers know whether the rates are a complete, same-scope actual observation; refuse invalid or unprovable inputs. Refs: run 36734492117, #3543, #3158
Brings in 22 upstream commits up to 5107ebe. - backends/atom.py: upstream only removed a comment inside the old cache-rate block that this branch replaces; this branch's accounting is kept. - tests/results/agentic/test_process_agentic_result.py: upstream's comment removal is kept. - server_metrics.py and tests/bench/test_agentic_command.py merged cleanly. 合并 origin/main 的 22 个上游提交（至 5107ebe）。atom.py 中上游只删除了本分支已替换的旧缓存命中率代码块里的一条注释，保留本分支的计数逻辑；test_process_agentic_result.py 保留上游删除的注释；server_metrics.py 与 tests/bench/test_agentic_command.py 自动合并。
records_error_dropped counts failed requests in every phase, and the ATOM request-scope check refused pricing whenever it was nonzero. A failed warmup request therefore invalidated a profiling window whose admissions and prompt tokens conserved exactly. - load_records_with_accounting adds records_profiling_error_dropped for errors outside warmup; records without a phase still count as profiling. records_error_dropped stays the all-phase total. - request_scope_from_records passes the profiling count as error_records and falls back to the all-phase total for accounting without the split. - A processor-entry regression covers clean, warmup-only, profiling, both-phase and legacy missing-phase errors. Review: #3964 (andyluo7) records_error_dropped 统计所有阶段的失败请求，而 ATOM 请求范围检查只要它非零就拒绝计价，因此一次 warmup 失败也会让请求数和 prompt token 完全守恒的 profiling 窗口失去计价资格。load_records_with_accounting 新增 records_profiling_error_dropped，只统计 warmup 之外的错误，缺少 phase 的记录仍按 profiling 计；records_error_dropped 保持全阶段总数。request_scope_from_records 以 profiling 错误数作为 error_records，没有该字段的旧记账回退为全阶段总数。新增经处理器入口的回归测试，覆盖正常、仅 warmup 错误、profiling 错误、两个阶段都有错误，以及缺少 phase 的旧记录。
Drops a comment that restated the control flow and a descriptive block above a test fixture, and shortens the decode re-count note to one line. Comment-only. 按 AGENTS.md 的注释规则精简本 PR 自己的注释：删除一条复述控制流的注释和一段测试夹具上方的描述性注释，并把 decode 重复计数的说明压缩为一行。仅改注释。
The TRT-LLM adapter averaged the per-worker trtllm_kv_cache_hit_rate gauges, decode workers included at 0, and skipped the counters because AIPerf 0.12 exports them without _total. On GB300 dynamo-trt run 38198820297 it published 0.401; the prefill counters give 0.970 once the warmup increment is removed. No adapter could say whether a published rate covers the profiling window. - phase_validity.py: per-series window validity for every adapter, from summary.phase_time_ranges, the timeslices and warmup_metrics, with no fixed time tolerance. - TRT-LLM: the combined rate comes from the counters per counting domain (prefill or combined role) and is written to overall_cache_hit_rate only as interval telemetry (CONDITIONAL_INTERVAL). Raw counts, the raw ratio and the reasons stay in validity.cache_rate; gpu_cache_hit_rate stays null and exact eligibility stays false. - cache_validity carries one core for every adapter: status_class, published, pricing_eligible and basis. ATOM keeps its own fields. - This branch's adapter context reaches ATOM as adapter_context, and apply_validity runs after populate for every adapter. Refs: #3964 TRT-LLM 适配器把各 worker 的 trtllm_kv_cache_hit_rate gauge 做不加权平均（decode 的 0 也算在内），又因为 AIPerf 0.12 导出的计数器名不带 _total 而跳过了计数器：GB300 dynamo-trt 运行 38198820297 发布 0.401，去掉 warmup 增量后 prefill 计数器得到 0.970。新增 phase_validity.py，为所有适配器给出逐序列的窗口有效性；TRT 改用计数器按计数域计算 combined 命中率，只作为区间遥测（CONDITIONAL_INTERVAL）写入 overall，原始计数、原始比值和原因保留在 validity.cache_rate。cache_validity 为所有适配器提供统一核心（status_class、published、pricing_eligible、basis），ATOM 保留原有字段。
This was referenced Oct 16, 2026
Collaborator
Author
|
Updated to 中文已同步当前 main 与 #4405，修复不完整 frontend 阻断 prefill 回退，以及混合 TRT alias 漏计。保存输入的实际 CLI 重放仍得到受控 TRT 60% interval 遥测，消费端 guard 拒绝其用于精确实测计价；新 head CI 通过。这不代表真实 TRT P/D 或 MiniMax 已知偏差已完成验收。 |
Contributor
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
Contributor
|
Claude finished @chunfangamd's task in 2m 9s —— View job Review: PR #4163I found one warning and nothing blocking. I posted it as an inline comment.
The rest of the code read as consistent:
This PR only changes result post-processing, so it doesn't need a I didn't run the test suite: running commands in this environment needed approval. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
迁移共享资格与 TRT 差异，保留现有审阅历史并复用 ATOM 兼容层。
合入 ATOM 端点覆盖状态兼容修复。
移除重复覆盖和历史代号，保留更强的窗口及别名测试。
chunfangamd
requested review from
Ankur-singh,
RohitNagraj,
billishyahao,
kedarpotdar-nv,
nvpohanh,
seungrokj and
wzhao18
as code owners
October 22, 2026 17:08
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
chunfangamd
changed the base branch from
chun/fix-atom-kv-cache-accounting
to
chun/atom-cache-observation-validity
October 22, 2026 17:08
统一 P/D 的成功 profiling 输入总量，修复 TRT 聚合角色判断及 source 命中率校验，保留阶段诊断和条件区间合同。
….fang@amd.com/chun/scripts/lp_fxi_cpu_kvcache/codex/worktrees/atom-compat-20261022 into chun/compat-migrate-4163
… path P/D 总量使用完整合法的客户端或前端计数，覆盖缺失 metrics 的早退路径，并拒绝部分、负数抵消、布尔和非有限输入。
显式非法 disaggregation_mode 不再按聚合部署回退为 combined；无标签且可信聚合模式仍可接受。
按 counter total 或 histogram sum 严格读取前端总量；客户端替换 P/D worker 范围时清除三个标准化分量并保留 raw 诊断。
中文：将 TRT 校验收窄为现代 profiling 配对 counter，删除通用阶段分类与历史窗口恢复，保留 P/D 总量守恒。
中文：继承精简后的 ATOM 资格与共享 core，保留 TRT 独立差异。 # Conflicts: # inferencex-e2e/infx/results/agentic/cache_validity.py # inferencex-e2e/infx/results/agentic/server_metrics.py
中文：继承现代 ATOM 诊断独立性回归，保持本层产品差异。
中文：明确标为 router 或非法 component 的计数不能回退为聚合 worker。
中文：非法 frontend 条目返回未知总量，避免 P/D 聚合崩溃。
chunfangamd
changed the title
修复不完整 frontend 计数阻断有效 prefill 总量的问题，并按 endpoint 与 labels 合并 TRT counter 别名，避免漏记或重复计数。补充实际行为回归。
中文：同步 ATOM counter 与资格层维护，保持 TRT 独立增量。
同步最新 main 与 ATOM 资格前置修复，保留 TRT 分域计数、有效 prefill 回退和非有限值防护，并保留上游文档更新。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TRT reuse must pair cached and prompt counters over the same profiling and worker scope. This layer publishes their ratio as interval telemetry, without claiming exact-pricing eligibility. Counter aliases are combined per endpoint and complete labels: identical aliases count once, conflicting values remain unknown.
For P/D prompt totals, use complete successful client usage first, then a complete frontend total, then a validated prefill total. An incomplete frontend does not block a valid prefill fallback, and decode's repeated input is not added. Per-worker rates use validated pairs; transfer counters remain diagnostics.
Base: #4405. Next: #4164. Saved collector inputs were replayed through the producer CLI and actual app ETL/cache selection. This is CPU software validation; real TRT P/D serving and the known MiniMax/TRT bias are not claimed resolved.
中文
TRT 的 cached/prompt 必须在相同 profiling 与 worker 范围配对；本层发布 interval 遥测，不授予精确计价资格。alias 按 endpoint 与完整 labels 合并，相同值去重，冲突保持 unknown。
P/D 输入总量依次选完整成功 client usage、完整 frontend、已验证 prefill；不完整 frontend 不再阻断 prefill 回退，也不累加 decode 重复输入。per-worker 比率来自已验证配对，传输指标仍为诊断。依赖 #4405，后续为 #4164。保存输入经过实际 CLI 与 app ETL/缓存选择重放；这是 CPU 软件验证，不代表真实 TRT P/D 服务验收或 MiniMax/TRT 已知偏差已消除。
AI model disclosure
GPT-6 (Codex), including delegated implementation and independent review agents, prepared and checked these revisions. A finer runtime model identifier is not exposed. Earlier producer/app work was prepared by an agent reporting Claude Opus 5.5; that historical runtime identifier could not be independently verified.