Conversation
rawCursors charged both the per-file stat AND the per-manifest-entry row lookup against a single budget (ARCHIVE_RAW_CURSOR_MAX, 2000) sized only to the agent's file cap (ARCHIVE_RAW_MANIFEST_FILES_MAX, 2000). So an agent inside its own caps offering F files across N transcripts cost the hub N + F work and overran by exactly N: the last N offers got no cursor, the agent read that as 0 and re-pushed from the start, and ingestRaw refused each on its offset check — a wasted small POST per affected file per beat. The hub's own comment claimed an in-cap agent never reached it; it was not true. The interaction XERK-424 warned about: the reserved backlog slice exists so older, unarchived sessions get their sidecars shipped, and this budget truncated exactly that slice (the manifest is newest-first). Fix (ticket option 2, the recommended one): keep charging the lookup — it is real work, and charging only files reopens QA F4 (unknown/no-filePath ids walk the budget for free) — and size the budget to the true worst case an in-cap agent presents. Add ARCHIVE_RAW_CURSOR_LOOKUP_MAX (200, mirrors the agent's ARCHIVE_MANIFEST_MAX) and set the budget to the SUM of the two terms. Both numbers now mean what they read as. Correct the false comment + warning. Tests: reworked the existing cap test to the two-term budget; added a regression proving files spread across transcripts keep every cursor (fails against the old single-budget line). Updated .claude/rules/turma-archive.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
rawCursorsinturma/archive.jscomputes per-file byte cursors for the raw archive layer each heartbeat. Its per-beat loop charges a budget for two kinds of work — one unit per manifest entry (theSELECT filePathrow lookup) and one per file offered — but the budget was a singleARCHIVE_RAW_CURSOR_MAX(2000), which mirrors only the agent's file capARCHIVE_RAW_MANIFEST_FILES_MAX(2000).So an in-cap agent offering F files across N transcripts cost the hub
N + Fwork and overran by exactly N. The last N offers got no cursor; the agent read that as 0 and re-pushed from the start, andingestRawrefused each on its offset check — a wasted small POST per affected file per beat. The hub's own comment claimed an in-cap agent "never reaches this"; it was not arithmetically true.The interaction that made it worth fixing now (from the ticket): XERK-424's reserved backlog slice exists so older, unarchived sessions get their sidecars shipped — and because the manifest is newest-first, this budget truncated exactly that slice.
Why this shape (ticket's recommended option 2)
Keep charging the lookup — it is real work, and charging only files reopens the QA F4 walk-around, where unknown / no-
filePathids cost a freeSELECTand move the stall rather than bound it. Instead, size the budget to the true worst case an in-cap agent presents.Added
ARCHIVE_RAW_CURSOR_LOOKUP_MAX(200, mirroring the agent'sARCHIVE_MANIFEST_MAX=200 manifest-entry cap) and setbudget = ARCHIVE_RAW_CURSOR_MAX + ARCHIVE_RAW_CURSOR_LOOKUP_MAX. Both terms now mean what they read as; a well-behaved agent (≤200 entries, ≤2000 files → ≤2200 work) is never truncated, while total per-beat work stays bounded at 2200 ops. Corrected the false comment and the warning message; updated.claude/rules/turma-archive.md.How verified
XERK-427regression proving files spread across transcripts keep every cursor.node --test turma/tests/*.test.js→ 1690/1690 pass (archive 41, archive-budget 29, server 700, …). The regression fails against the old single-budget line and passes with the fix (confirmed by reverting only that line).hub-agent.pybound the producer to ≤200 entries / ≤2000 files; caught 3/3 mutations (old single budget; dropped lookup charge; dropped stat charge); checkedpositiveEnvIntdegenerate inputs (0/-5/abc→ 200, can't disable the budget); confirmed no dependents on the old value outsidearchive.js.