Skip to content

Docs-call reporting counts unobserved search results as evidence, in both directions #83

Description

@leggetter

reportDocsReach in apps/framework/scripts/report-results.ts buckets every
documentation call by whether it carries a page:

if ((call.pages ?? []).length > 0) arm.reached += 1;
else arm.empty += 1;

For Codex's web_search there is no result to bucket. The search is hosted:
the model receives the hits on the provider's side and the CLI never sees them.
docs-results.ts knows this and says so —

No page to attribute: the hits never reach the client. hasContent stays
unknown rather than false, because those hits carry snippet text the model may
well have read, and we can't see it either way.

— and sets pages: [] with hasContent left undefined. The reporter then reads
that absence as a measurement.

It is wrong in both directions

Every Codex web_search call in the 14 September snapshot (run 34813087863)
has no observed result at all — resultChars is absent on all 254 of them:

shape calls counted as
pages: [], no result 216 empty — "the search returned nothing"
pages: [url], no result 38 reached a page — "the agent read the docs"

The 38 come from the fallback that treats a url-shaped query as a page open. The
url is the query, not something we saw returned. So we under-credit 216 calls and
over-credit 38, and neither number is an observation.

By contrast shell_fetch carries a real result — 22 calls, resultChars 4208 and
similar — and is fine.

Why it matters

The printed note draws the conclusion explicitly:

A high empty rate is an external search index returning nothing, not a
documentation or skills gap.

That asserts the search failed. We do not know that it failed. Read against the
transcripts, it plainly did not: between these "empty" searches the agent narrates
"I found the versioned Hookdeck API base from the docs" and "I found the manual
retry API"
, then calls the correct endpoint first time.

#61 rests its headline on this bucket — "the search path mostly returns nothing",
58% of baseline calls. That table needs re-deriving once the distinction exists.
Its other half, that the two arms differ in tool mix (web_fetch against
web_search), is unaffected and still stands.

Done when

reached / empty / unobserved are three buckets rather than two, a call with
no result we ever saw lands in the third, and the printed note says what the third
means. The discriminator is already in the data: a web_search call with no
resultChars is unobserved, whichever way pages happens to have been filled.

Nothing about the scoreboard changes — docs.calls is diagnostic only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    harnessRunner, scorers, provisioner and CI — work that can be done in this repo

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions