docs: correct stale Codex telemetry claims in observability whitepaper - #379
Merged
Conversation
Two claims in the whitepaper cited openai/codex#12913 as evidence that Codex telemetry coverage is uneven across entrypoints, with codex exec emitting no metrics and codex mcp-server emitting none at all. That issue was closed as completed on 2026-02-28 with a fix committed for the following release, so the headless entrypoint is no longer dark and both claims are out of date. Corrections: - Section 5.3 now carries the actual Codex metric inventory (turn latency, time to first token, token usage by type including cached_input, tool calls, approvals, MCP, skills, hooks, memory phases), which is broader than the section previously implied. The "thinner than Claude Code" reading was wrong on metrics specifically; Claude Code leads on event breadth and span structure, Codex leads on metrics. - Section 5.3 replaces the closed-issue claim with the gaps that are currently open: no custom OTel resource attributes (#30987) and OTLP log records exported with timeUnixNano=0 causing collectors to drop them (#30936). - Section 9 replaces "where Codex telemetry is weakest" with the accurate compounding factor for unattended runs: the vendor signals that encode a human verdict (codex.approval.requested, code_edit_tool.decision) stay present but go inert when nobody is deciding. - Reference [8] annotated as closed; new references added for the Codex metric reference and the two open issues. Issue states verified 2026-08-05.
There was a problem hiding this comment.
Pull request overview
Updates the agent observability whitepaper to correct outdated statements about OpenAI Codex telemetry coverage and to replace them with an up-to-date metric inventory plus currently open telemetry gaps.
Changes:
- Rewrites §5.3 to enumerate Codex’s current metrics and dimensions and removes the stale “headless entrypoint is dark” claim.
- Updates §9 to replace the “weakest coverage” framing with the CI/unattended “signals go inert without a human” argument.
- Updates references: annotates the previously-cited issue as closed and adds new references for the metric reference and two open issues.
Comment on lines
+555
to
+557
| | Tools | `codex.tool.call` and `.duration_ms` (tool, success), `codex.approval.requested` (tool, approved/denied/amended/session/abort) | | ||
| | Transport | `codex.api_request`, `codex.sse_event`, `codex.websocket.request`/`.event`, each with a paired duration histogram | | ||
| | MCP / Skills / Hooks | `codex.mcp.call` (+`.duration_ms`), `codex.skill.injected`, `codex.hooks.run` (+`.duration_ms`, tagged `hook_name`/`source`/`status`) | |
Comment on lines
+562
to
+565
| `codex.turn.token_usage` breaks out `cached_input`, which yields prompt-cache effectiveness | ||
| directly -- the quantity that usually dominates marginal cost. And `codex.approval.requested`, | ||
| tagged approved or denied, exposes a human-judgment signal as a first-class metric -- an | ||
| outcome signal in the sense of Section 3.3, shipped as a counter rather than left to be built |
Comment on lines
+925
to
+926
| [15] OpenAI, *otel: exported OTLP logs have timeUnixNano=0, causing collectors to drop them*, | ||
| Issue #30936. Open as of 2026-08-05. https://github.com/openai/codex/issues/30936 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two claims in the whitepaper merged in #378 rest on openai/codex#12913, cited as evidence that Codex telemetry coverage is uneven across entrypoints (
codex execemitting no metrics,codex mcp-serveremitting none at all).That issue was closed as completed on 2026-02-28, with a fix committed for the following release. The headless entrypoint is not dark, and both claims are out of date. Docs only.
What changed
§5.3 now carries the actual Codex metric inventory. The section previously described Codex telemetry only through its config keys, which understated it considerably:
turn.e2e_duration_ms,turn.ttft.duration_ms,turn.ttfm.duration_ms,turn.token_usage(bytoken_type, incl.cached_input)tool.call+.duration_ms,approval.requested(approved/denied/amended/abort)api_request,sse_event,websocket.request/.event, each with a paired duration histogrammcp.call,skill.injected,hooks.runmemory.phase1/phase2with token-usage and e2e histogramsAll carrying
auth_mode,model,app.version, plusoriginator,session_source, andconversation_id.The implied "Codex is thinner than Claude Code" reading was wrong on metrics specifically: Codex emits roughly twice as many, with richer dimensions. Claude Code leads on event breadth, hook coverage, and the nested span tree. The section now says the two are not ordered.
§5.3 replaces the closed-issue claim with the gaps that are actually open: no custom OTel resource attributes, so telemetry cannot be tagged by team, tier, or cost center (#30987), and OTLP log records exported with
timeUnixNano = 0causing collectors to drop them while traces and metrics from the same exporter arrive normally (#30936).§9 replaces "where Codex's telemetry coverage is weakest" with the compounding factor that actually holds for unattended runs. The vendor signals that best approximate a quality judgment (
codex.approval.requested,code_edit_tool.decision) do not disappear in CI. They stay present and go inert, because nobody is deciding. The instrumentation remains; the judgment it encoded does not. That is a sharper version of the paper's own argument than the claim it replaces.References: [8] annotated as closed; [13] added for the Codex metric reference, [14] and [15] for the two open issues.
Notes
turn.token_usagebreaks outcached_input, which gives prompt-cache effectiveness directly, and cache behavior typically dominates marginal cost.approval.requestedis an outcome signal in the sense of §3.3, shipped as a counter rather than left to be built by hand.