reportDocsReach in apps/framework/scripts/report-results.ts buckets every
documentation call by whether it carries a page:
if ((call.pages ?? []).length > 0) arm.reached += 1;
else arm.empty += 1;
For Codex's web_search there is no result to bucket. The search is hosted:
the model receives the hits on the provider's side and the CLI never sees them.
docs-results.ts knows this and says so —
No page to attribute: the hits never reach the client. hasContent stays
unknown rather than false, because those hits carry snippet text the model may
well have read, and we can't see it either way.
— and sets pages: [] with hasContent left undefined. The reporter then reads
that absence as a measurement.
It is wrong in both directions
Every Codex web_search call in the 14 September snapshot (run 34813087863)
has no observed result at all — resultChars is absent on all 254 of them:
| shape |
calls |
counted as |
pages: [], no result |
216 |
empty — "the search returned nothing" |
pages: [url], no result |
38 |
reached a page — "the agent read the docs" |
The 38 come from the fallback that treats a url-shaped query as a page open. The
url is the query, not something we saw returned. So we under-credit 216 calls and
over-credit 38, and neither number is an observation.
By contrast shell_fetch carries a real result — 22 calls, resultChars 4208 and
similar — and is fine.
Why it matters
The printed note draws the conclusion explicitly:
A high empty rate is an external search index returning nothing, not a
documentation or skills gap.
That asserts the search failed. We do not know that it failed. Read against the
transcripts, it plainly did not: between these "empty" searches the agent narrates
"I found the versioned Hookdeck API base from the docs" and "I found the manual
retry API", then calls the correct endpoint first time.
#61 rests its headline on this bucket — "the search path mostly returns nothing",
58% of baseline calls. That table needs re-deriving once the distinction exists.
Its other half, that the two arms differ in tool mix (web_fetch against
web_search), is unaffected and still stands.
Done when
reached / empty / unobserved are three buckets rather than two, a call with
no result we ever saw lands in the third, and the printed note says what the third
means. The discriminator is already in the data: a web_search call with no
resultChars is unobserved, whichever way pages happens to have been filled.
Nothing about the scoreboard changes — docs.calls is diagnostic only.
reportDocsReachinapps/framework/scripts/report-results.tsbuckets everydocumentation call by whether it carries a page:
For Codex's
web_searchthere is no result to bucket. The search is hosted:the model receives the hits on the provider's side and the CLI never sees them.
docs-results.tsknows this and says so —— and sets
pages: []withhasContentleft undefined. The reporter then readsthat absence as a measurement.
It is wrong in both directions
Every Codex
web_searchcall in the 14 September snapshot (run34813087863)has no observed result at all —
resultCharsis absent on all 254 of them:pages: [], no resultpages: [url], no resultThe 38 come from the fallback that treats a url-shaped query as a page open. The
url is the query, not something we saw returned. So we under-credit 216 calls and
over-credit 38, and neither number is an observation.
By contrast
shell_fetchcarries a real result — 22 calls,resultChars4208 andsimilar — and is fine.
Why it matters
The printed note draws the conclusion explicitly:
That asserts the search failed. We do not know that it failed. Read against the
transcripts, it plainly did not: between these "empty" searches the agent narrates
"I found the versioned Hookdeck API base from the docs" and "I found the manual
retry API", then calls the correct endpoint first time.
#61 rests its headline on this bucket — "the search path mostly returns nothing",
58% of baseline calls. That table needs re-deriving once the distinction exists.
Its other half, that the two arms differ in tool mix (
web_fetchagainstweb_search), is unaffected and still stands.Done when
reached/empty/unobservedare three buckets rather than two, a call withno result we ever saw lands in the third, and the printed note says what the third
means. The discriminator is already in the data: a
web_searchcall with noresultCharsis unobserved, whichever waypageshappens to have been filled.Nothing about the scoreboard changes —
docs.callsis diagnostic only.