Skip to content

XERK-427: give the raw-cursor loop's entry lookups their own budget term - #615

Merged
xerhab merged 1 commit into
mainfrom
XERK-427
Sep 1, 2026
Merged

XERK-427: give the raw-cursor loop's entry lookups their own budget term#615
xerhab merged 1 commit into
mainfrom
XERK-427

Conversation

@xerhab

@xerhab xerhab commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

What

rawCursors in turma/archive.js computes per-file byte cursors for the raw archive layer each heartbeat. Its per-beat loop charges a budget for two kinds of work — one unit per manifest entry (the SELECT filePath row lookup) and one per file offered — but the budget was a single ARCHIVE_RAW_CURSOR_MAX (2000), which mirrors only the agent's file cap ARCHIVE_RAW_MANIFEST_FILES_MAX (2000).

So an in-cap agent offering F files across N transcripts cost the hub N + F work and overran by exactly N. The last N offers got no cursor; the agent read that as 0 and re-pushed from the start, and ingestRaw refused each on its offset check — a wasted small POST per affected file per beat. The hub's own comment claimed an in-cap agent "never reaches this"; it was not arithmetically true.

The interaction that made it worth fixing now (from the ticket): XERK-424's reserved backlog slice exists so older, unarchived sessions get their sidecars shipped — and because the manifest is newest-first, this budget truncated exactly that slice.

Why this shape (ticket's recommended option 2)

Keep charging the lookup — it is real work, and charging only files reopens the QA F4 walk-around, where unknown / no-filePath ids cost a free SELECT and move the stall rather than bound it. Instead, size the budget to the true worst case an in-cap agent presents.

Added ARCHIVE_RAW_CURSOR_LOOKUP_MAX (200, mirroring the agent's ARCHIVE_MANIFEST_MAX=200 manifest-entry cap) and set budget = ARCHIVE_RAW_CURSOR_MAX + ARCHIVE_RAW_CURSOR_LOOKUP_MAX. Both terms now mean what they read as; a well-behaved agent (≤200 entries, ≤2000 files → ≤2200 work) is never truncated, while total per-beat work stays bounded at 2200 ops. Corrected the false comment and the warning message; updated .claude/rules/turma-archive.md.

How verified

  • Tests: reworked the existing two-term cap test; added an XERK-427 regression proving files spread across transcripts keep every cursor. node --test turma/tests/*.test.js → 1690/1690 pass (archive 41, archive-budget 29, server 700, …). The regression fails against the old single-budget line and passes with the fix (confirmed by reverting only that line).
  • QA agent (PASS, no defects): exercised the real module at production defaults (200×10 boundary → all cursors, no truncation warning) and under a hostile 25k-offer manifest (bounded to 2200 ops, ~8ms, 19k dropped with one throttled warning); confirmed the agent-side caps in hub-agent.py bound the producer to ≤200 entries / ≤2000 files; caught 3/3 mutations (old single budget; dropped lookup charge; dropped stat charge); checked positiveEnvInt degenerate inputs (0/-5/abc → 200, can't disable the budget); confirmed no dependents on the old value outside archive.js.

rawCursors charged both the per-file stat AND the per-manifest-entry row
lookup against a single budget (ARCHIVE_RAW_CURSOR_MAX, 2000) sized only to
the agent's file cap (ARCHIVE_RAW_MANIFEST_FILES_MAX, 2000). So an agent
inside its own caps offering F files across N transcripts cost the hub N + F
work and overran by exactly N: the last N offers got no cursor, the agent
read that as 0 and re-pushed from the start, and ingestRaw refused each on
its offset check — a wasted small POST per affected file per beat. The hub's
own comment claimed an in-cap agent never reached it; it was not true.

The interaction XERK-424 warned about: the reserved backlog slice exists so
older, unarchived sessions get their sidecars shipped, and this budget
truncated exactly that slice (the manifest is newest-first).

Fix (ticket option 2, the recommended one): keep charging the lookup — it is
real work, and charging only files reopens QA F4 (unknown/no-filePath ids
walk the budget for free) — and size the budget to the true worst case an
in-cap agent presents. Add ARCHIVE_RAW_CURSOR_LOOKUP_MAX (200, mirrors the
agent's ARCHIVE_MANIFEST_MAX) and set the budget to the SUM of the two terms.
Both numbers now mean what they read as. Correct the false comment + warning.

Tests: reworked the existing cap test to the two-term budget; added a
regression proving files spread across transcripts keep every cursor (fails
against the old single-budget line). Updated .claude/rules/turma-archive.md.
@xerhab
xerhab merged commit 5190250 into main Sep 1, 2026
5 checks passed
@xerhab
xerhab deleted the XERK-427 branch September 1, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant