[jeremycapps]

Part of #588: the measurement cut 2 waits on. Does leaving known out of the extraction prompt move the output beyond run-to-run variance? Bench only, no server change.

What changes in scripts/bench/recall.mjs

• --known shown|empty labels a run. The switch itself is UTOPIA_EXTRACT_KNOWN_IN_PROMPT (refactor(extraction): split run_open loop into call/parse + apply helpers (#588 cut 1) #980), set when the server starts, which the bench can neither set nor read. So each group runs against a server started to match, and --out refuses to run without a label.
• Per document: chunks (extracted), facts, entities, entities carried across chunks (an entity in facts from two or more chunks of the document), name_vector| pairs proposed to the adjudicator whose pieces appear in the document (Recall a mention by its name vector so a short form or another script meets its entity #877), drops by reason from extraction_drops (including chunk_unextracted), and seconds from queueing to the last chunk, measured on the database clock.
• Knowledge-base-wide: the name_vector| pair total, split into pending / merged / kept. A pair can touch two documents, so this is the number to add up.
• --out <dir>: one JSON record per run (commit, label, score, misses, per-document stats, pairs, the deduplicated subject/predicate/object set per document).
• --table <dir>: groups the records by label, with mean ± sd for each number, plus the fact-set overlap (Jaccard, averaged over documents) within each group (the run-to-run variance) and across the two groups (the effect of known).

The pair count is there because of #877: without known, a later chunk that names an earlier thing differently becomes a pair for the adjudicator, not a silent merge. The recall score can hold still while the pairs grow.

scripts/bench/README.md gains the commands and what the table means, in the #588 paragraph of the recall section.

Verified, and not

• node --check passes. --table has been run on made-up records, and the label checks reject what they should.
• Not yet run against a database. I don't have a model endpoint, so the SQL in stats() and nearPairs() has been reviewed but never executed. It uses starts_with (PG 11+). The first real round is the test; I'll fix anything it shows.

Running the measurement

node scripts/bench/recall.mjs --kb <id> --known shown --out runs/588   # ×3
# restart the server with UTOPIA_EXTRACT_KNOWN_IN_PROMPT=false
node scripts/bench/recall.mjs --kb <id> --known empty --out runs/588   # ×3
node scripts/bench/recall.mjs --table runs/588

🤖 Generated with Claude Code

[WaylandYang]

Thanks @jeremycapps, and welcome! This is a careful first PR. The --known label with --out refusing to run without it, measuring on the database clock, and the within-group vs cross-group Jaccard are exactly the design #588 needs. I checked the SQL against the schema (chunks, fact_evidence, extraction_drops, resolution_reviews), and the columns and joins line up.

One thing to fix before the numbers can be trusted:

Near pairs build up across runs. extract and reprocess don't clear resolution_reviews. Only a full graph rebuild (purge_graph) does. On one --kb, run N's name_vector| count therefore includes the pairs from runs 1…N-1. Since the empty group runs after shown, it would look worse from build-up alone. The simplest fix is to scope both nearPairs() and the per-document subquery to r.created_at >= queuedAt. For --score there's no queuedAt, so either skip pairs there or say in the README that the graph must be rebuilt between runs.

A small one: the DCO check is failing. git commit --amend -s and a force-push will fix it.

Since the SQL hasn't run against a database yet, it would help to paste the table from your first real round here. Happy to merge once the pair scoping is in.

[WaylandYang]

Two updates to my comment above, @jeremycapps:

• DCO is green now. Your signed push landed just before my comment, so please ignore that note.
• I saw your message in The chunks of one document are extracted one at a time #588 about the model budget. No need to run the rounds yourself: once the pair scoping is in, we'll run them on our bench setup (3 × shown, 3 × empty) and post the table here.

So the near-pair scoping is the only thing left on your side.

[jeremycapps]

Thanks @WaylandYang, that's generous. Two things before you run it, both found by one real round on gemini-3.5-flash (4 SEC filings, 55 chunks, 46/52):

1. The pair count can't filter on name_vector|. Once a pair is decided, its reason is overwritten: close_review_auto writes governed|…, escalate_review writes its own, merges write superseded by merge. The origin isn't kept anywhere else (the audit payload has names, score and confidence). In my round all 228 pairs had been decided by the end (196 governed, 32 superseded by merge), so the name_vector| count read 0. It would have read about 0 in all six of your rounds.

So the bench now counts every pair proposed this round (scoped to created_at >= queuedAt as you suggested), split pending / merged / kept, with the current reason prefixes alongside. It mixes the literal and vector channels, but it's what an empty known would inflate, and it's honest. If you want the channel back, a column set at insert and never overwritten would do it; happy to do that as its own PR.

2. The records are snapshots taken while governance is still running. The script stops waiting when extraction ends, but alignment and governance keep adding facts and deciding pairs afterwards (552 → 565 facts on the earnings release an hour later). I'll fix that in a follow-up PR that waits for the queue to drain before scoring. If you run before it lands, running --score --known <label> --out runs/588-settled once the queue is empty records the settled numbers without any model calls. Use a separate directory, or --table counts the round twice. The seconds column isn't affected, since it's read from extracted_at.

DCO is sorted, thanks for the note.

[WaylandYang]

Approving. Thanks @jeremycapps, and thank you for running a real round before we did.

Reviewed at 4308753 (your 251a032 plus the merge of dev):

• Dropping the name_vector| filter is right. I checked the code: governance rewrites reason to governed|… and a merge writes superseded by merge, so the filter would have read about 0 after every round. Counting every pair proposed in the round, with the current reason prefixes next to it, is the honest number.
• A correction to my first comment. I said pairs build up across runs. They don't on a normal run: the bench deletes the base's entities at the start of each round and the pairs go with them by cascade. I had checked the server's extract / reprocess and missed the script's own cleanup. The created_at >= queuedAt scope is still a good guard, so thanks for adding it.
• The SQL matches the schema, node --check passes, and CI is green.

Your second point (scoring while governance is still running) is a real gap and fine as a follow-up PR. Until it lands we'll record the settled numbers with --score into a separate directory, as you describe. A column that keeps the recall channel at insert would also be welcome as its own PR.

We'll run the six rounds and post the table on #588.