test(qa): cover the Copilot runtime and fix the six telemetry bugs it found - #233
Open
bertschneider wants to merge 8 commits into
Open
test(qa): cover the Copilot runtime and fix the six telemetry bugs it found#233bertschneider wants to merge 8 commits into
bertschneider wants to merge 8 commits into
Conversation
Every Copilot chat span carried stopReason ("end_turn") as a raw,
unnamespaced attribute. The span's own status already says whether the
turn ended well, so the key added nothing and looked like a leak.
eventAttributes copies every payload field it does not recognize, and
attrSkipKeys is spelled in the field names Claude Code and Codex use.
Copilot's payloads are camelCase, so stop_hook_active on the same
agentStop payload was already denied while its camelCase sibling walked
straight through. Deny it by its own spelling.
Found by qa/tools/qa-attrs.py on the first Copilot QA run, which is the
same check that caught the equivalent Codex leak (turn_id).
The e2e guard is the stronger one: it feeds a real agentStop payload
through the built binary and asserts no emitted span carries the key.
Every Copilot chat span carried github.copilot.cost, Copilot's own accounting in AI credits, copied through from the native-OTel chat spans the turn is summed from. It collides with the figure next to it. Dash0 derives dash0.gen_ai.usage.cost from tokens at ingest, in money. Shipping both puts two attributes ending in "cost" on one span, in two units, one of them a vendor's internal accounting unit, and nothing on the span says which is which. A dashboard summing them is wrong without ever looking wrong. Usage.Cost and attrFloat go with it. A field whose only purpose is to be discarded at the last step is the shape the export grew out of, and leaving it invites the same commit back. No deny-list entry: the key reaches the plugin only through the native-OTel file, never through a hook payload, so nothing can copy it onto a span by accident. The guard is the e2e test, whose staged native chat span carries the attribute and whose emitted spans must not. The credits figure stays recoverable from the native-OTel file, and qa/tools/qa-otel.py still reports it as a channel figure.
|
Darkplane auto-approval is enabled for this repository (mode: Dry run). An evaluation will run once this pull request's CI checks have completed — no action needed. |
bertschneider
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
Darkplane would not approve this pull request. It scored 51.73 of 100, and 75 was required.
The other criterion
An AI evaluator produced these scores from the diff, the repository history, and the pull request text. The pull request text is author-controlled, so treat the scores as signals rather than proof. Reviewed for commit 608e359. |
bertschneider
force-pushed
the
qa-copilot
branch
from
August 28, 2026 14:30
f0aa947 to
c29c8ca
Compare
A resumed session double-counted the launch before it. Measured on a two-turn QA run: turn 2's chat span carried 59068 input tokens for a turn of 29655, having summed turn 1 a second time, and turn 1's tool span was emitted again under turn 2's trace. The cursor — the id of the last native span consumed — lived in the per-session directory that pipeline.Process removes on SessionEnd. But a Copilot session id outlives its session: `copilot --resume` comes back under the same id, and the second launch found no cursor and re-read the file from the start. The cursor is precisely the state that has to survive the end of a launch. It now lives beside the native-OTel files, keyed by conversation, and the stale-file sweep skips it. "Stale" there means "untouched for a day", which an idle conversation also is, while the shared native-OTel file stays fresh under other sessions' writes. Sweeping the cursor would send the next ReadTurn back to the top of that file and re-count everything it already reported, which is the defect this commit fixes. Only reachable when both launches write to one native-OTel file. The launch function the dash0-configure skill installs gives each launch its own file and deletes it at exit, which made the lost cursor harmless; a fixed COPILOT_OTEL_FILE_EXPORTER_PATH — the documented alternative to that function — does not. TestReadTurn_cursorSurvivesSessionEnd asserts the cursor's location rather than simulating the wipe: there is no session directory left in that path to wipe, which is the property that makes the wipe survivable. The wipe itself is driven end to end by TestE2ECopilotDefersTurnWhenTraceContextMissing.
A Copilot sub-agent's identity was carried on dash0.gen_ai.tool.task.name,
a custom key on the spawning task span. It reaches Dash0 and no backend
feature reads it, so the sub-agent was effectively anonymous.
The standard attributes were available all along and being thrown away.
Copilot's native OpenTelemetry describes the sub-agent with a proper
invoke_agent span carrying gen_ai.agent.name and gen_ai.agent.id, and the
plugin collapsed that layer, re-parented the inner tools onto the task
span, and invented a key for the part it had discarded.
The layer is now re-emitted, so Copilot produces the same shape as Claude
and Codex:
chat → execute_tool task → invoke_agent task → execute_tool bash
Only the layers nothing is emitted for are collapsed: the native chat
spans, and the turn's root invoke_agent, which the pipeline's own chat
span already represents. Parenting resolves each span to its nearest
ancestor that is also emitted, which does both jobs at once.
gen_ai.agent.id is the spawning tool call's id, not the native
"builtin:<kind>". The native value is shared by every sub-agent of a kind,
so it is a type filter wearing an id's name; the call id is unique per
invocation and is also the session id Copilot gives that sub-agent's own
hooks, so the dropped hook session and the emitted span name the same
thing.
No usage on the agent span. Attribution stays flat — a sub-agent's chat
spans fold into the parent turn — so repeating those tokens here would
double any sum across the trace. The native span carries none either.
dash0.gen_ai.tool.task.name is gone. The instance label the model chose
is still in the task span's gen_ai.tool.call.arguments, where Copilot
puts it.
bertschneider
force-pushed
the
qa-copilot
branch
from
August 28, 2026 14:45
c29c8ca to
6bf0183
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6bf0183. Configure here.
A delegating turn produced two conversations in Dash0: the real one, and a second holding a single `chat ` span with no model and no usage — the sub-agent's own hook session. Normalize drops a sub-agent session on a `call_` prefix, and that only holds in prompt mode. `copilot -p` names a sub-agent session call_<toolCallId>; an interactive session gives it a plain UUID, the prefix misses it, and the sub-agent's userPromptSubmitted/agentStop pair mints a standalone, token-less conversation. Its native-OTel spans carry the PARENT's conversation id, so there is nothing for it to report. Suppress on what holds in both modes instead: a sub-agent session receives no sessionStart. Each hook is its own short-lived process, so "this session never had a sessionStart" is a claim about something that happened elsewhere, and a file is the only way agentStop can learn it. A missing marker is not on its own enough, though, so the native-OTel file has to agree. Only sessionStart writes a marker, and that hook has upstream ways to fail that say nothing about sub-agents: the bootstrap downloads the binary inside it, under Copilot's 10s timeout, and every fail-open path exits before the binary runs. The first session after a version bump on a slow link is a real session with no marker, and suppressing on the marker alone silenced all of it, permanently. The file settles it. A sub-agent has no spans under its own conversation id — its chat spans carry the parent's, which is the whole reason its turn has nothing to report — so ReadTurn returns nil for a sub-agent and a turn for a real session. One gap is left, the smaller one: with native OTel off there is no file to ask, so a real session that also lost its marker is still suppressed. Such a session emits only bare chat spans anyway. The marker lives beside the per-session directories, not in one, and nothing ever deletes it. Both of those were paid for. Inside a session directory, two things removed it: SessionEnd handling, so a turn arriving afterwards — which recreates the directory and a fresh trace context, and would otherwise export normally — was read as a sub-agent's and dropped, which is the last turn of every session, since agentStop and sessionEnd fire together as separate processes and agentStop is the slower, scanning the native-OTel file; and the sweep below, which cannot tell a killed session from one idle for a day and so cost a live session its marker for good. Moving it out fixed both, and reclaiming it on a timer instead reintroduced exactly the same failure one level up: measured against the built binary, an idle session's marker went to another session's sweep, and every later turn of it was dropped and its state deleted, with no way back because sessionStart never fires again. So nothing reclaims a marker. It is an empty file named by a session id. The suppression is fail-closed on missing state by construction — a lost marker ends a session's telemetry silently, while a spurious export costs one token-less conversation — so the state has to be permanent, and a stale empty file is the smaller problem by a wide margin. Read at agentStop rather than at userPromptSubmitted: Copilot fires sessionStart and userPromptSubmitted in a nondeterministic order, so a real session may not be marked yet when its first prompt arrives. It is also written before the sweeps rather than after, since a concurrent agentStop can be reading it and unbounded directory I/O in between only widens that window. The prefix guard stays as the cheap path: it drops a prompt-mode sub-agent at its first event, before any scratch state is created. Found by running an interactive session by hand — every test and QA driver in this repo uses prompt mode, where the old guard works. The e2e tests replay the interactive payload shape (UUID session id, traceparent, no sessionStart) and the post-SessionEnd turn through the built binary, and reverting either half fails them. Verified against a live interactive session too. The session id is validated before it is used to resolve a directory. pipeline.Process substitutes a random id for one that is missing or unsafe, but it does that inside itself, and this suppression removes a directory before handing the event over. An agentStop with no sessionId resolved SessionDir to the data root ITSELF and deleted it — the bootstrap's cached binary and every concurrent session's state — while "../victim" walked out of the root entirely. Both reproduced against the built binary, exit 0, no diagnostic. A usable path segment is not enough either: "started" and "bin" are both usable and both name a directory this plugin owns. An id the guard will not vouch for means no marker, no suppression and no sweep; the event takes the normal path. Keeping the entrypoint's own paths off those names does not settle it. Process joins the id onto the data root itself and its SessionEnd removes what it joined, so a sessionEnd carrying "bin" deleted the bootstrap's binary cache and one carrying "started" deleted every live session's marker — both reproduced against the built binary. A reserved id is now renamed in the payload before Process reads it, by a prefix rather than a random id so every hook of the session lands on the same directory. The sweep skips those names for the same reason, since an events.jsonl written into one would otherwise date it as an ordinary stale session. Copilot's ids are UUIDs, so this bounds what a malformed payload can do rather than describing an observed failure. Also collects the session directories killed runs leave behind. A session that ends deletes its own; one that is killed — SIGKILL, a crash, a closed terminal — delivers no sessionEnd, and nothing else was collecting them. Measured on a developer's machine: 20 directories, the oldest six weeks old. They are small, but each holds an events.jsonl carrying that session's prompts, so it is user content sitting on disk indefinitely. SweepOldSessionDirs runs on sessionStart beside SweepOldOtelFiles, which has covered the native-OTel directory this way all along, and shares its TTL. It dates a session by its events.jsonl rather than by the directory, whose mtime only moves when an entry is added or removed and so stays frozen at the first event — that would date a long-running session by when it started and sweep it out from under itself. Only directories carrying an events.jsonl are considered, which keeps the sweep off the bootstrap's binary cache. Sweeping a live-but-idle session's directory costs it a trace context, which its next prompt mints again; the marker it would once have taken is no longer there. The turn's own invoke_agent span is now recognised by not hanging under an execute_tool call of this conversation, rather than by having no parent id. Copilot injects a traceparent into an interactive session's payloads, so a turn whose trace continues one from outside would carry a parent id naming a span the file does not hold, and an emptiness test would read that root as a sub-agent, minting a span that duplicates the turn and re-parenting the whole tool tree beneath it. Asking merely whether the parent is some span of this conversation is looser than the rule and admits the same duplication from inside one file, for a turn whose root continues an earlier turn's trace. The system-notification test grew a sessionStart: its fixture drove a turn in a session that never started, which is now exactly the shape that is dropped. Claude and Codex leak session directories the same way on a kill; this fixes the runtime the evidence came from.
…ends Copilot continues its own trace into a hook by putting a W3C traceparent on an interactive session's payloads. It is a propagation header, not a span attribute, and eventAttributes copied it onto every chat span the same way it copied stopReason: unnamespaced, undeclared, by the generic copy-what-nobody-denied path. On an emitted span it is worse than redundant. The span belongs to the plugin's own trace, derived from the session id; the header names Copilot's native trace and a span id that is not the span's parent. Measured 2026-08-28: a chat span in trace 3b361e94… carrying 00-558ca38a…-adc5bef1061afc70-01, which resolves to a trace that exists only in a local file and reaches no backend. It also landed on one span of four, since the other three are synthesised and never saw the payload. Correlating the plugin's spans with Copilot's native ones is worth doing for anyone who exports both, but as a span link on every span of the turn rather than whichever one happened to carry the header. That is its own piece of work. Prompt-mode payloads carry no traceparent, so neither the QA driver nor any existing test could see this; it was found by reading the attributes of a real interactive session's span. TestE2ECopilotPerTurnSpans now drives the interactive payload shape through the built binary, and reverting the deny fails it and the unit test.
FEATURE_MATRIX still described the behaviour this branch replaced: it rated Copilot "Partial" on sub-agent spans and said their tool calls re-parent under the spawning task span. A sub-agent now gets its own invoke_agent span, so the tree matches the other runtimes and the rating does too. Also names the one case where gen_ai.agent.id is absent. An execute_tool task encloses the invoke_agent it spawned and is therefore written to the native-OTel file later, so a read falling between the two flushes puts the agent in one turn and its spawning tool in the next — and that tool call is where the id comes from. It is the late flush the README already tolerates, but DEVELOPMENT.md and the README both stated the id unconditionally.
Adds the third arm of the QA suite. `qa/tools/qa-session-copilot.sh` provisions the shipped package into a throwaway home through the real marketplace path — `copilot plugin marketplace add` then `plugin install` — enables Copilot's native OpenTelemetry the way the dash0-configure launch function does, registers the QA recorder alongside the plugin's hooks, and runs `copilot -p --output-format json`. Nothing outside the throwaway home is written, as on the Codex arm. Copilot inverts the shape of the evidence, and the harness has to say so. Its hooks carry no numbers and no tool events the plugin consumes: tokens, cost, model, tool spans and sub-agent spans all come from Copilot's own OpenTelemetry file, which qa/tools/qa-otel.py re-reads as the second channel. That file is also the plugin's input, so agreement proves a faithful copy and not a correct measurement — every spec here names which of its assertions rest on the independent hook record and which do not. qa-compare.py grows a copilot branch. Its `hooks` column claims a chat span and prints `-` for the rest, because a None expectation is not zero and comparing tool spans against 0 would report every correct span as a surplus. Sub-agent `call_` hook sessions are counted apart from a reused run id, and an absent OTel file reports nothing rather than zero so the documented no-OTel degradation reads as a pass. The recorder gains two concessions: Copilot's camelCase payloads carry no event-name field, so it takes the name from argv as the plugin's own bootstrap does, and they name the session `sessionId`. Both keep record/index.jsonl one shape across all three runtimes. Also here: five preflight checks, six specs across four topics, five learnings, and the Settling note that Copilot's ingest lag is 25 seconds rather than the 8 the other runtimes use — at 8 a two-turn run shows one turn's spans and reads exactly like a product bug. The first runs of this suite found four of the six defects fixed in the commits before it — three from qa-attrs.py on the first probe, one from the two-turn probe. The other two came from driving an interactive session by hand, which is the one thing this driver cannot do: a sub-agent's session id is a plain UUID there rather than call_-prefixed, and only there does a payload carry a traceparent. The attribute spec and its coverage map now say that a clean result is a statement about prompt mode alone — naming what a check cannot see is worth as much as what it can. One learning is theirs as much as the suite's: copilot-subagent-sessions-are-dropped-on-purpose first recorded a sub-agent as running its own sessionStart and sessionEnd, which is the opposite of the invariant the suppression rests on, and now records the partial lifecycle that was measured. lifecycle that was measured.
bertschneider
force-pushed
the
qa-copilot
branch
from
August 28, 2026 15:02
6bf0183 to
608e359
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes SIG-323
The QA suite covered
claudeandcodex. This addscopilot, and fixes six product defects that running it turned up. Each defect gets its own commit, and the suite comes last.What makes this runtime different
Copilot's hooks carry no numbers, and none of the tool events the plugin consumes. Tokens, model, tool spans and sub-agent spans all come out of Copilot's own OpenTelemetry file, which the plugin reads at each turn boundary.
So the evidence works differently here. The
hookscolumn claims onechatspan per turn and says nothing about the rest. The tool and token comparison runs against a file that is also the plugin's input, so agreement there shows the plugin copied its input faithfully and says nothing about whether Copilot measured the session correctly. Each spec states which of its assertions rest on the hook record, the one input that stays independent.The fixes
stopReasonchatspan. The deny list is spelled in Claude's snake_case, so Copilot's camelCase twin passed through it.github.copilot.costexported at all. AI credits land one attribute away from the money figure Dash0 derives at ingest, and nothing on the span says which is which.SessionEnddeletes, and a Copilot session id outlives its session.invoke_agentcall_prefix guard holds in prompt mode only; interactive sessions give sub-agents a plain UUID.traceparentA delegating turn now produces the same tree the other runtimes produce:
Four of the six came from
qa-attrs.pyand the two-turn probe. The other two appear only in interactive sessions, which the driver cannot run, so I found them by driving Copilot by hand. The attribute spec now records that limit, so a clean result there reads as a statement about prompt mode.What the QA arm is
qa/tools/qa-session-copilot.shprovisions the shipped package into a throwaway home through the real marketplace path, enables native OTel the way thedash0-configurelaunch function does, registers the QA recorder beside the plugin's hooks, and runscopilot -p --output-format json.qa/tools/qa-otel.pyre-reads the native-OTel file as the second channel. With those come acopilotbranch inqa-compare.py, six specs across four topics, five learnings, and five preflight checks.Also here
Session directories left by killed runs are collected now. A session that ends deletes its own; one that is killed delivers no
sessionEnd, and nothing was collecting those. My machine held 20 of them, the oldest six weeks old, each with anevents.jsonlholding that session's prompts.Verification
Every commit builds, vets, and passes the unit and tagged
TestE2ECopilotsuites on its own.make lintandmake test(race plus shell) are green. The live canaryTestE2EFullFlowWithCopilotpasses against a real session with a PAT.Every fix is pinned by a test that fails when the fix is reverted, which I checked by reverting each one. Three review rounds ran against the branch. Several tests turned out unable to fail for the reason they claimed, and those were repaired.
Known limits
sessionStartmarker, a real session is still suppressed. Such a session emits bare chat spans anyway, and a marker goes missing only when thesessionStarthook failed outright.