Skip to content

The recall bench waits for alignment and governance to settle before it scores a round (#588) - #1005

Merged
WaylandYang merged 1 commit into
deeplethe:devfrom
jeremycapps:bench/recall-settle
Sep 29, 2026
Merged

WaylandYang merged 1 commit into
deeplethe:devfrom
jeremycapps:bench/recall-settle

Conversation

@jeremycapps

Copy link
Copy Markdown
Contributor

Follow-up to #1002 (#588).

Why

recall.mjs stopped waiting once extract_document / process_document were done and scored straight away. But alignment, governance, time resolution and adjudication keep running after extraction, adding facts and deciding pairs. On a local round (gemini-3.5-flash, 4 SEC filings) the earnings release had 552 facts when the round was recorded and 565 an hour later, and the pair counts moved too. So each record was a snapshot at whatever moment extraction happened to end, which adds noise to the known shown/empty comparison that has nothing to do with known.

What changes

  • After extraction, the bench waits until no jobs are left to run: running, or queued with run_at due within a minute. That covers alignment's few-second debounce and jobs that re-queue themselves with a short delay; a job scheduled further out isn't part of this round.
  • Progress is judged the same way as the extraction wait: slow isn't a timeout, but no job finishing for 15 minutes is. In that case it records anyway and marks the run settled: false.
  • Each record gains settled and settleSeconds. --table shows the wait as a row and names any run that didn't settle, so a snapshot can't hide inside a mean.
  • A known limit: an empty queue is not quite "no more work is coming". If a check lands in the instant between one job finishing and the job it triggers being queued, the bench can stop one step early. Only the server knows what each job queues next, so closing that properly needs a server-side signal (for example an idle event on /kbs/{id}/events sent once a finished job's follow-ups are committed). I'm happy to propose that separately if you think it's worth it.
  • The seconds column is unchanged: it's read from chunks.extracted_at, not from when the bench stopped waiting.
  • README: the The chunks of one document are extracted one at a time #588 paragraph says why the bench waits.

Verified

  • The wait loop, run against a local database with a noop job due in 20 s: it waited, then reported settled after 24 s once the worker finished the job. With a noop due in 5 minutes it settled immediately.
  • --table on round 1's record shows the new row (- for a record from before this change).
  • Not yet run inside a full round, which costs model calls; the loop is the same code the test exercised.

…it scores a round

Signed-off-by: Jeremy Capps <jeremylwcapps@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks @jeremycapps, this closes the gap you found in #1002.

Reviewed at 35a3da8. The query matches the jobs table (status, run_at), the one-minute window covers the debounce, and marking a run settled: false and naming it under the table keeps a snapshot out of the means.

Two notes, neither blocks this:

  • A failed job that is queued again with a backoff longer than a minute will read as settled. Worth a line in the README if you see it happen in a round.
  • The count covers every base in the database, as the extraction wait above it does. That is fine on a bench database that holds one run at a time.

The server-side idle signal is worth proposing. Please open an issue for it first so we can agree on the shape before any code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants