From 9cf48b92fa0648133d8153e0fc952f671542915a Mon Sep 17 00:00:00 2001 From: Jean-Baptiste Date: Sat, 21 Nov 2026 04:11:00 +0100 Subject: [PATCH 1/2] fix(session-index): bound writer command deadlines by the work a folder write carries Effects, certificate and name-derivation commands get a deadline that scales with their serialized payload and entry count, so a large folder finishes on a busy machine, and a retry after an unknown outcome reuses rows that are already durably committed. --- .ai/contexts/session-cache.md | 6 +- .ai/contexts/session-index-writer.md | 44 +++++- CHANGELOG.md | 1 + session-index-writer-client.js | 17 ++- session-index-writer.js | 30 +++- test/session-index-large-folder-1135.test.js | 144 +++++++++++++++++++ 6 files changed, 236 insertions(+), 6 deletions(-) create mode 100644 test/session-index-large-folder-1135.test.js diff --git a/.ai/contexts/session-cache.md b/.ai/contexts/session-cache.md index f45499a40..971844315 100644 --- a/.ai/contexts/session-cache.md +++ b/.ai/contexts/session-cache.md @@ -123,7 +123,11 @@ population, with at most two fresh folder attempts; unrelated folders are not rescanned. Retry snapshot failures remain scoped to their folder so later folders can publish. Shutdown prevents new retry workers and terminates admitted retry workers through the same registered scan-worker set. Population has no -wall-clock deadline. Explicit remote/subset scans +wall-clock deadline. Folder writer commands use payload-scaled deadlines; +a retry preserves durably certified matching rows from an interrupted +publication in the same writer lifetime (see session-index-writer.md, #1135). +The retry still re-enumerates and validates sources; it does not claim to skip +transcript parsing on a forced/cold retry. Explicit remote/subset scans retain the ten-minute ceiling. Population returns `{ok,...}`; initial completion, Projects reconciliation time and continuation-choice sweeping require success. diff --git a/.ai/contexts/session-index-writer.md b/.ai/contexts/session-index-writer.md index c4a50c737..3881582bf 100644 --- a/.ai/contexts/session-index-writer.md +++ b/.ai/contexts/session-index-writer.md @@ -718,7 +718,7 @@ receipt continuity without refusing token/witness-valid certificates. The commands bracket producer operations at the facade. They do not alter the generic chunk scheduler, producer-unit packing, transaction budgets, -admission or deadlines (#1135). Certificate commands participate in existing +admission. The payload deadlines below apply to folder commands (#1135). Certificate commands participate in existing project-deletion ownership fencing. Repair invalidates its folder seal before scanning and joins startup population; its marker clears only after successful sealed publication. @@ -766,3 +766,45 @@ The obsolete single-row worker `certifyFullRead` command is removed; certificate units invoke the store method directly inside `certifyFullReads`. The real worker facade test uses 80 certificates and an intervening setting write to prove `continuous:false` returns the body ACK. + +## Large-folder deadlines and interrupted publication (#1135) + +The transport scales the started-command deadline for effects, full-read +certification and batched name derivation. The multiplier is the larger of +one, the serialized payload's 32 KiB blocks, and its 32 array-entry blocks +(including nested effect/row arrays). Each block receives the existing +30-second allowance; the timer is capped at Node's maximum positive timer +delay. Small/control commands, bootstrap, heartbeat and admission retain +their existing budgets. This bounds a command by the work carried in its +payload without changing the worker's transaction packing, yields, fences +or scheduling. The conservative structural count also covers many small +rows; bytes cover large text/metrics/witness payloads. This is a finite +work allowance, not a guarantee under arbitrary CPU starvation. + +After a startup folder's body ACK, the facade retains its existing pending +token, configuration and proposed per-row certificates for a retry in this +writer lifetime. It does not treat this memory as committed progress: on +retry, it rereads the durable pending token and each row certificate/witness, +and compares them with the newly scanned source candidate and configuration. +Only matching, durably certified rows bypass replacement. A full retry uses +the existing partial writer to preserve those rows, deletes vanished inventory +rows, and still seals as a full scan only after the remaining writes and +certificates. Multiple failed retries retain the preserved candidates. +A superseded token, changed configuration/source/row or absent certificate +requires republication. An unknown body outcome has no new certified progress +and cannot authorize skipping its rows. No receipt table or parallel durable +progress mechanism is introduced. After process restart the provenance hint +is gone and A2 conservatively performs full validation again. + +The scan coordinator still allows only three delayed retries (30 s, 120 s, +600 s), then retains its error with the retry time at Infinity until an +explicit forced scan or a later startup. Unknown certificate outcomes leave +the folder pending/unsealed and initial completion false on a cold failure. +Seal remains last; an unknown seal outcome is logged without compensation. +A2's successfully published deferred sources still count as complete for +population while their folder remains pending. + +The regression uses real disposable SQLite, the production scan worker, +facade and writer, with a virtual slow transport to exercise command expiry. +The interrupted certificate test commits a prefix of single-row chunks, +then loses the next outcome and verifies that retry preserves certified rows. diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f5d340b3..1a556c966 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ What changes for you in each release of Switchboard. How to write an entry: [doc - Codex sessions run non-interactively by a program (`codex exec`: scripted reviews and fixes) are now left out of the sidebar and grid like Claude Agent SDK sessions, under the same **Hide SDK-launched Sessions** setting. They still count in the heatmap and are found by search; the ones you have open stay listed. Codex sessions already indexed are re-checked once at the next start. (#1145) ### Fixed +- Indexing a very large project now finishes on a busy machine. (#1135) - After restarting the app or Windows, the sidebar shows the projects as they were at exit, even if the last project load finished just before you closed the app or signed out. (#1136) - On Windows, a slow start no longer leaves tabs without process cleanup: tabs already open, restored ones included, are covered once cleanup starts, and programs they launch afterwards end when the tab closes. Programs launched before that can keep running. (#1140, #1156) - Claude and Codex sessions in `.work-files/worktrees/`, and Codex sessions in `.worktrees`, `.claude-worktrees` and `.claude/worktrees`, appear under their existing repository, including nested worktrees and sessions already indexed. Repository settings, archive and hidden state follow the group, and project deletion removes the session metadata it previews for both runtimes. Archiving a repository covers its worktrees' sessions; a new session in a worktree reopens the repository. Resume and fork use the recorded worktree directory and refuse to launch if it no longer exists; Claude sessions in a deleted `.claude/worktrees` directory are refused instead of starting in the repository. (#1137) diff --git a/session-index-writer-client.js b/session-index-writer-client.js index e40567f88..6b71d7ffa 100644 --- a/session-index-writer-client.js +++ b/session-index-writer-client.js @@ -1,6 +1,21 @@ 'use strict'; +const { Buffer } = require('node:buffer'); + // see .ai/contexts/session-index-writer.md +function commandTimeoutMs(command, args, timeoutMs) { + if (!['effects', 'certifyFullReads', 'deriveCachedNames'].includes(command)) return timeoutMs; + let rows = 0; + const pending = [args]; + while (pending.length) { + const value = pending.pop(); + if (Array.isArray(value)) rows += value.length; + if (value && typeof value === 'object') for (const child of Object.values(value)) pending.push(child); + } + const batches = Math.max(1, Math.ceil(rows / 32), Math.ceil(Buffer.byteLength(JSON.stringify(args)) / 32768)); + return Math.min(2147483647, timeoutMs * batches); +} + function createWriterWorkerClient({ workerFactory, onSettings = () => {}, onBusyCounts = () => {}, onBootstrap = () => {}, timeoutMs = 30000, maxRestarts = 3, now = () => Date.now(), setTimeout = global.setTimeout, clearTimeout = global.clearTimeout, heartbeatMs = 30000, log = console }) { let worker, closing, stopped = false, terminated = false, nextId = 0, restartTimer; @@ -130,7 +145,7 @@ function createWriterWorkerClient({ workerFactory, onSettings = () => {}, onBusy const job = active.get(message.id); if (!job) return; if (message.started) { - job.timer ||= setTimeout(() => settle(job, Object.assign(new Error(`writer command timed out for request ${job.id}; outcome unknown`), { code: 'WRITER_TIMEOUT', requestId: job.id })), timeoutMs); + job.timer ||= setTimeout(() => settle(job, Object.assign(new Error(`writer command timed out for request ${job.id}; outcome unknown`), { code: 'WRITER_TIMEOUT', requestId: job.id })), commandTimeoutMs(job.command, job.args, timeoutMs)); return; } active.delete(message.id); diff --git a/session-index-writer.js b/session-index-writer.js index 248cda5e5..3a4a5b47d 100644 --- a/session-index-writer.js +++ b/session-index-writer.js @@ -72,6 +72,7 @@ function createSessionIndexWriter(options = {}, cache = require('./session-cache }; let backend, recorder, projectsDir, cacheTail = Promise.resolve(), closing, accepting = true, terminated = false; const pending = new Set(); + const scanResumes = new Map(); const commandScope = new AsyncLocalStorage(); const owner = {}; const admitted = () => !terminated && (accepting || commandScope.getStore() === owner); @@ -119,7 +120,24 @@ function createSessionIndexWriter(options = {}, cache = require('./session-cache let startupFence = scanResult?.startup ? { folder: scanFolder, sessionIds, witness: folderReadWitness(backend, scanFolder, sessionIds) } : null; if (name === 'applyProjectsWorkerEffects' && backend.getProjectsRevision() !== sourceVersion) throw Object.assign(new Error('Projects effect snapshot changed while queued'), { code: 'WRITER_STALE' }); const continuationCollector = { deferred: new Set(), accounted: new Set() }; - let computeArgs = args; + let computeArgs = args, computeName = name; + const resumed = new Map(); + const previous = scanResult?.startup && !scanResult.unverified && scanResumes.get(scanFolder); + if (previous && previous.config === scanResult.fullIndexConfig && previous.config === currentConfig() + && backend.getFolderMeta(scanFolder)?.fullIndexPending === previous.token) { + for (const row of scanResult.sessions) { + const candidate = previous.rows.get(row.sessionId); + const durable = backend.getCachedSession(row.sessionId); + if (candidate && row.fullReadStamp === candidate.stamp && rowWitness(row) === candidate.witness + && durable?.fullReadStamp === candidate.stamp && rowWitness(durable) === candidate.witness) resumed.set(row.sessionId, candidate); + } + if (resumed.size) { + const present = new Set(scanResult.sessions.map(row => row.sessionId)); + computeName = 'writeScannedFolderPartial'; + computeArgs = [{ ...scanResult, sessions: scanResult.sessions.filter(row => !resumed.has(row.sessionId)), + toDelete: scanResult.partial ? scanResult.toDelete : scanResult.inventory.rows.filter(row => !present.has(row.sessionId)).map(row => row.sessionId) }]; + } + } if (folder !== undefined && cache.prepareContinuationRefresh) { const opts = name === 'refreshFolder' ? args[1] : {}; const preparedContinuations = await cache.prepareContinuationRefresh(folder, opts); @@ -129,7 +147,8 @@ function createSessionIndexWriter(options = {}, cache = require('./session-cache recorder.begin(); let result, revision, effects; try { - result = cache[name](...computeArgs); + result = cache[computeName](...computeArgs); + if (resumed.size) result += resumed.size; revision = recorder.revision(); } finally { effects = recorder.finish(); @@ -200,6 +219,8 @@ function createSessionIndexWriter(options = {}, cache = require('./session-cache for (const [key, publication] of publications) { if (!publication?.began || publication.failed) continue; const rows = certificates.filter(row => row.folder === key); + if (scanResult?.startup) scanResumes.set(key, { token: publication.token, config: publication.config, + rows: new Map([...resumed, ...rows.map(row => [row.sessionId, row])]) }); if (!rows.length) continue; try { const certificateAck = await send('certifyFullReads', key, publication.token, rows, { receipt: forwarded }); @@ -229,7 +250,10 @@ function createSessionIndexWriter(options = {}, cache = require('./session-cache if (forwarded) { if (acknowledgment?.committedRevision !== undefined && sealed.beforeRevision !== acknowledgment.committedRevision) continuous = false; acknowledgment = sealed; - } else if (scanResult) scanResult.fullIndexOutcome = sealed; + } else if (scanResult) { + scanResult.fullIndexOutcome = sealed; + if (sealed) scanResumes.delete(key); + } } } catch (error) { (options.log || console).warn('[session-index-writer] full index seal outcome unknown: ' + error.message); } } diff --git a/test/session-index-large-folder-1135.test.js b/test/session-index-large-folder-1135.test.js new file mode 100644 index 000000000..822721b29 --- /dev/null +++ b/test/session-index-large-folder-1135.test.js @@ -0,0 +1,144 @@ +'use strict'; +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const { EventEmitter } = require('node:events'); +const { fixture, supported } = require('./fixtures/warm-start-1144.cjs'); +const { createWriterWorkerClient } = require('../session-index-writer-client'); +const { createWorker } = require('./fixtures/project-deletion-worker.cjs'); +const sqlite = { skip: !supported && 'SQLite requires native binding or Node 22' }; + +async function loadFixture(t) { + const f = await fixture(t, { separateReader: true }); + const client = createWriterWorkerClient({ workerFactory: () => createWorker(f.db), timeoutMs: 300000, heartbeatMs: 300000 }); + await client.ready; + f.worker.client.request = client.request; + t.after(() => client.terminate()); + return f; +} + +function slowTransport(t, f, costOf = args => Math.max(1, Math.ceil(Buffer.byteLength(JSON.stringify(args)) / 4096))) { + const worker = new EventEmitter(), commands = []; + const request = f.worker.client.request; + let latest; + const client = createWriterWorkerClient({ + timeoutMs: 10, heartbeatMs: 100000, + workerFactory: () => worker, + setTimeout(fn, ms) { latest = { fn, ms }; return latest; }, + clearTimeout() {}, + }); + worker.terminate = async () => {}; + worker.postMessage = message => { + worker.emit('message', { id: message.id, started: true }); + const deadline = latest; + const cost = costOf(message.args); + commands.push({ command: message.command, cost, deadline: deadline.ms }); + request(message.command, ...message.args).then(result => { + if (cost > deadline.ms) deadline.fn(); + worker.emit('message', { id: message.id, result }); + }, error => worker.emit('message', { id: message.id, error: { message: error.message, code: error.code } })); + }; + worker.emit('message', { ready: true, settings: [] }); + f.worker.client.request = client.request; + t.after(() => client.terminate()); + return commands; +} + +function observeWrites(t, f) { + const writes = new Map(), upsert = f.db.upsertCachedSessions; + t.mock.method(f.db, 'upsertCachedSessions', rows => { + for (const row of rows) writes.set(row.sessionId, (writes.get(row.sessionId) || 0) + 1); + return upsert(rows); + }); + return writes; +} + +function loseCertificateBatch(t, f) { + const certify = f.db.certifyFullRead; + const committed = new Set(); + let armed = true; + t.mock.method(f.db, 'nextChunkSize', () => 1); + t.mock.method(f.db, 'certifyFullRead', (...args) => { + if (armed && committed.size === 2) { + armed = false; + throw Object.assign(new Error('writer command timed out; outcome unknown'), { code: 'WRITER_TIMEOUT' }); + } + const result = certify(...args); + if (armed && result) committed.add(args[2]); + return result; + }); + return committed; +} + +test('large cold folder finishes when slow writer commands exceed a fixed deadline', sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 12; i++) f.write('large-' + i, 'body '.repeat(1800)); + const commands = slowTransport(t, f); + const result = await f.cache.populateCacheViaWorker(); + assert.equal(result.ok, true, 'large folder must finish under simulated writer load'); + assert.ok(commands.some(command => command.cost > 10), 'the fixture exceeds the old per-command deadline'); + assert.equal(f.db.isInitialScanComplete(), true); + assert.equal(f.db.getCachedByFolder(f.folder).length, 12); + assert.equal(f.db.getFolderMeta(f.folder).fullIndexPending, null); + assert.ok(f.db.getCachedByFolder(f.folder).every(row => row.fullReadStamp)); +}); + +test('retry after an unknown certificate batch preserves durably certified rows', sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 8; i++) f.write('resume-' + i); + const writes = observeWrites(t, f), committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + assert.equal(committed.size, 2); + assert.equal(f.db.isInitialScanComplete(), false); + assert.ok(f.db.getFolderMeta(f.folder).fullIndexPending); + const retried = await f.restart(); + assert.equal(retried.ok, true, retried.error); + for (const id of committed) assert.equal(writes.get(id), 1, 'certified row must not be rewritten: ' + id); + for (const row of f.db.getCachedByFolder(f.folder)) { + assert.ok(row.fullReadStamp); + if (!committed.has(row.sessionId)) assert.equal(writes.get(row.sessionId), 2, 'uncertified row must be republished'); + } + assert.equal(f.db.isInitialScanComplete(), true); + assert.equal(f.db.getFolderMeta(f.folder).fullIndexPending, null); +}); + +test('resume revalidates changed sources while preserving stable neighbors', sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 8; i++) f.write('changed-' + i); + const writes = observeWrites(t, f), committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + const changed = [...committed][0]; + fs.appendFileSync(f.write(changed), JSON.stringify({ type: 'user', cwd: f.root, uuid: 'appended', message: { content: 'new bytes' } }) + '\n'); + assert.equal((await f.restart()).ok, true); + assert.equal(writes.get(changed), 2); + assert.equal(f.db.getCachedSession(changed).messageCount, 2); + for (const id of [...committed].slice(1)) assert.equal(writes.get(id), 1, 'unchanged certified neighbor survives'); +}); + +test('many small rows receive a work budget even without a large byte payload', async t => { + const f = { worker: { client: { request: async () => true } } }; + slowTransport(t, f, args => args[0].length / 4); + await assert.doesNotReject(f.worker.client.request('deriveCachedNames', Array.from({ length: 128 }, () => ({})))); +}); + +for (const changed of ['token', 'certificate', 'configuration']) { + test('resume refuses a changed ' + changed, sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 4; i++) f.write('guard-' + i); + const writes = observeWrites(t, f), committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + if (changed === 'token') f.db.beginFullIndex(f.folder, 'superseded'); + if (changed === 'certificate') for (const id of committed) f.db.invalidateFullRead(id); + if (changed === 'configuration') f.db.setSetting('global', { disabledHarnesses: ['codex'] }); + const retried = await f.restart(); + assert.equal(retried.ok, true, retried.error); + for (const id of committed) assert.equal(writes.get(id), 2, 'changed ' + changed + ' forbids reuse'); + }); +} + +test('a single large text payload receives a byte budget independently of row count', async t => { + const f = { worker: { client: { request: async () => true } } }; + slowTransport(t, f); + await assert.doesNotReject(f.worker.client.request('effects', [{ name: 'upsertSearchEntries', + args: [[{ id: 'large', type: 'session', body: 'x'.repeat(131072) }]] }], 0)); +}); From 98ab07ee9116a4816fd91be3bb9c1d89cef7004f Mon Sep 17 00:00:00 2001 From: Jean-Baptiste Date: Sat, 21 Nov 2026 06:00:00 +0100 Subject: [PATCH 2/2] fix(session-index): estimate writer payloads in one walk and cap the scaled deadline The deadline estimate no longer serializes the payload on the main thread, the scaled deadline is capped at ten minutes, and the resume guards (deletion on a full retry, durable witness, source stamp) are tested. --- .ai/contexts/session-index-writer.md | 33 +++-- session-index-writer-client.js | 39 ++++-- test/session-index-large-folder-1135.test.js | 128 ++++++++++++++++++- 3 files changed, 180 insertions(+), 20 deletions(-) diff --git a/.ai/contexts/session-index-writer.md b/.ai/contexts/session-index-writer.md index 3881582bf..8a598103f 100644 --- a/.ai/contexts/session-index-writer.md +++ b/.ai/contexts/session-index-writer.md @@ -717,8 +717,9 @@ continuity across chunks. An unrelated write between chunks breaks Projects receipt continuity without refusing token/witness-valid certificates. The commands bracket producer operations at the facade. They do not alter -the generic chunk scheduler, producer-unit packing, transaction budgets, -admission. The payload deadlines below apply to folder commands (#1135). Certificate commands participate in existing +the generic chunk scheduler, producer-unit packing, transaction budgets or +admission. The payload deadlines below apply to every `effects` command, +including refresh, project-worker and remote producers (#1135). Certificate commands participate in existing project-deletion ownership fencing. Repair invalidates its folder seal before scanning and joins startup population; its marker clears only after successful sealed publication. @@ -769,16 +770,24 @@ prove `continuous:false` returns the body ACK. ## Large-folder deadlines and interrupted publication (#1135) -The transport scales the started-command deadline for effects, full-read -certification and batched name derivation. The multiplier is the larger of -one, the serialized payload's 32 KiB blocks, and its 32 array-entry blocks -(including nested effect/row arrays). Each block receives the existing -30-second allowance; the timer is capped at Node's maximum positive timer -delay. Small/control commands, bootstrap, heartbeat and admission retain +The transport scales the started-command deadline for every `effects` command, +full-read certification and batched name derivation. The multiplier is the larger +of one, the estimated payload's 32 KiB blocks, and its 32 work-row blocks. +Bulk session, search, entrypoint and title effects count their immediate rows; +other effects count once. Certification and name derivation count their candidates. +Nested metrics are not counted again as work rows in both session upserts and +metric replacement; their payload size still contributes to the byte estimate. +A single iterative walk estimates bytes from string/key lengths (two bytes per +UTF-16 code unit), structural delimiters and fixed-size scalar allowances, +without serializing a copy. A WeakSet visits shared/cyclic objects once; BigInt +uses the scalar allowance. An unreadable payload falls back to the maximum. +Each block receives the existing 30-second allowance, capped by +`MAX_WORK_COMMAND_TIMEOUT_MS` at ten minutes, matching the subset scan ceiling. +Small/control commands, bootstrap, heartbeat and admission retain their existing budgets. This bounds a command by the work carried in its payload without changing the worker's transaction packing, yields, fences -or scheduling. The conservative structural count also covers many small -rows; bytes cover large text/metrics/witness payloads. This is a finite +or scheduling. The work-row count also covers many small rows; estimated bytes +cover large text/metrics/witness payloads. This is a finite work allowance, not a guarantee under arbitrary CPU starvation. After a startup folder's body ACK, the facade retains its existing pending @@ -808,3 +817,7 @@ The regression uses real disposable SQLite, the production scan worker, facade and writer, with a virtual slow transport to exercise command expiry. The interrupted certificate test commits a prefix of single-row chunks, then loses the next outcome and verifies that retry preserves certified rows. +Retry regressions also remove a transcript, edit a durable title without +clearing its certificate, and change only source ctime while retaining identical +parsed row witnesses. Each compares the resulting index with a separate full +rebuild; the source-only case isolates the source-stamp reuse guard. diff --git a/session-index-writer-client.js b/session-index-writer-client.js index 6b71d7ffa..a0e0c9af0 100644 --- a/session-index-writer-client.js +++ b/session-index-writer-client.js @@ -1,19 +1,40 @@ 'use strict'; -const { Buffer } = require('node:buffer'); +const MAX_WORK_COMMAND_TIMEOUT_MS = 10 * 60 * 1000; // see .ai/contexts/session-index-writer.md function commandTimeoutMs(command, args, timeoutMs) { if (!['effects', 'certifyFullReads', 'deriveCachedNames'].includes(command)) return timeoutMs; - let rows = 0; - const pending = [args]; - while (pending.length) { - const value = pending.pop(); - if (Array.isArray(value)) rows += value.length; - if (value && typeof value === 'object') for (const child of Object.values(value)) pending.push(child); + try { + let rows = 0, bytes = 0; + if (command === 'effects') { + for (const effect of args[0]) { + rows += ['upsertCachedSessions', 'upsertSearchEntries', 'setCachedEntrypoints', 'setCachedCustomTitles'].includes(effect.name) + ? effect.args[0].length : 1; + } + } else rows = args[command === 'certifyFullReads' ? 2 : 0].length; + const pending = [args], seen = new WeakSet(); + while (pending.length) { + const value = pending.pop(); + if (typeof value === 'string') bytes += value.length * 2 + 2; + else if (value && typeof value === 'object') { + if (seen.has(value)) continue; + seen.add(value); + bytes += 2; + if (Array.isArray(value)) { + bytes += value.length; + for (const child of value) pending.push(child); + } else for (const key of Object.keys(value)) { + bytes += key.length * 2 + 3; + pending.push(value[key]); + } + } else bytes += 8; + } + const batches = Math.max(1, Math.ceil(rows / 32), Math.ceil(bytes / 32768)); + return Math.min(MAX_WORK_COMMAND_TIMEOUT_MS, timeoutMs * batches); + } catch { + return MAX_WORK_COMMAND_TIMEOUT_MS; } - const batches = Math.max(1, Math.ceil(rows / 32), Math.ceil(Buffer.byteLength(JSON.stringify(args)) / 32768)); - return Math.min(2147483647, timeoutMs * batches); } function createWriterWorkerClient({ workerFactory, onSettings = () => {}, onBusyCounts = () => {}, onBootstrap = () => {}, timeoutMs = 30000, maxRestarts = 3, diff --git a/test/session-index-large-folder-1135.test.js b/test/session-index-large-folder-1135.test.js index 822721b29..df08be449 100644 --- a/test/session-index-large-folder-1135.test.js +++ b/test/session-index-large-folder-1135.test.js @@ -2,8 +2,10 @@ const test = require('node:test'); const assert = require('node:assert/strict'); const fs = require('node:fs'); +const path = require('node:path'); const { EventEmitter } = require('node:events'); -const { fixture, supported } = require('./fixtures/warm-start-1144.cjs'); +const { fixture, assertRebuildOracle, supported } = require('./fixtures/warm-start-1144.cjs'); +const { rowWitness } = require('../src/main/session-index/full-read-state'); const { createWriterWorkerClient } = require('../session-index-writer-client'); const { createWorker } = require('./fixtures/project-deletion-worker.cjs'); const sqlite = { skip: !supported && 'SQLite requires native binding or Node 22' }; @@ -142,3 +144,127 @@ test('a single large text payload receives a byte budget independently of row co await assert.doesNotReject(f.worker.client.request('effects', [{ name: 'upsertSearchEntries', args: [[{ id: 'large', type: 'session', body: 'x'.repeat(131072) }]] }], 0)); }); + +function deadlineTransport(t, timeoutMs = 30000) { + const worker = new EventEmitter(); + const timers = new Set(); + let sent; + worker.postMessage = message => { sent = message; }; + worker.terminate = async () => {}; + const client = createWriterWorkerClient({ + workerFactory: () => worker, timeoutMs, heartbeatMs: 1234567, + setTimeout(fn, ms) { const timer = { fn, ms }; timers.add(timer); return timer; }, + clearTimeout(timer) { timers.delete(timer); }, + }); + worker.emit('message', { ready: true }); + t.after(() => client.terminate()); + return async (command, ...args) => { + const result = client.request(command, ...args); + result.catch(() => {}); + assert.doesNotThrow(() => worker.emit('message', { id: sent.id, started: true }), 'payload estimation must not escape the message listener'); + const deadline = [...timers].find(timer => timer.ms !== 1234567); + assert.ok(deadline, 'started work has a deadline'); + worker.emit('message', { id: sent.id, result: true }); + await result; + return deadline.ms; + }; +} + +test('deadline byte estimate avoids serialization and tracks representative payload size', async t => { + const deadline = deadlineTransport(t, 1); + const args = [[{ name: 'upsertSearchEntries', args: [[{ + id: 'large', body: 'ASCII café 漢字 😀 \"\n'.repeat(4000), count: 42, enabled: true, + }]] }], 0]; + const realBlocks = Math.ceil(Buffer.byteLength(JSON.stringify(args)) / 32768); + const stringify = t.mock.method(JSON, 'stringify'); + const estimatedBlocks = await deadline('effects', ...args); + assert.equal(stringify.mock.callCount(), 0, 'deadline estimation must not serialize the payload'); + assert.ok(estimatedBlocks >= realBlocks / 3 && estimatedBlocks <= realBlocks * 3, + 'estimated byte blocks stay within a factor of three of serialized size'); +}); + +test('all scaled command deadlines are capped at ten minutes', async t => { + const deadline = deadlineTransport(t); + const rows = Array.from({ length: 1000 }, () => ({})); + for (const [command, args] of [ + ['effects', [[{ name: 'upsertCachedSessions', args: [rows] }], 0]], + ['deriveCachedNames', [rows]], + ['certifyFullReads', ['folder', 'token', rows]], + ]) assert.equal(await deadline(command, ...args), 10 * 60 * 1000, command + ' has a ten-minute ceiling'); +}); + +test('nested metrics do not count as duplicate work rows', async t => { + const deadline = deadlineTransport(t, 1); + const metrics = Array.from({ length: 160 }, () => ({})); + const effects = [{ name: 'upsertCachedSessions', args: [[{ sessionId: 'one', dailyMetrics: metrics }]] }, + { name: 'replaceSessionMetrics', args: ['one', structuredClone(metrics)] }]; + assert.equal(await deadline('effects', effects, 0), 1, 'two operations with small nested metrics fit one work block'); +}); + +test('BigInt payloads still receive a bounded command timer', async t => { + const deadline = deadlineTransport(t); + const ms = await deadline('deriveCachedNames', [{ value: 1n }]); + assert.ok(ms >= 30000 && ms <= 600000); +}); + +test('cyclic payloads still receive a bounded command timer', async t => { + const deadline = deadlineTransport(t); + const row = { sessionId: 'cycle' }; + row.self = row; + const ms = await deadline('deriveCachedNames', [row]); + assert.ok(ms >= 30000 && ms <= 600000); +}); + +test('resumed full retry removes a deleted transcript and its search entry', sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 6; i++) f.write('r' + i); + const committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + const victim = f.db.getCachedByFolder(f.folder).find(row => !committed.has(row.sessionId)).sessionId; + assert.ok(f.db.connection.prepare('SELECT id FROM search_map WHERE id = ?').get(victim), 'victim has a search entry before deletion'); + fs.unlinkSync(path.join(f.dir, victim + '.jsonl')); + const retried = await f.restart(); + assert.equal(retried.ok, true, retried.error); + assert.equal(f.db.getCachedSession(victim), null, 'vanished transcript deleted on resumed full retry'); + assert.equal(f.db.connection.prepare('SELECT count(*) AS n FROM search_map WHERE id = ?').get(victim).n, 0, 'vanished transcript search entry is deleted'); + assert.equal(f.db.getFolderMeta(f.folder).fullIndexPending, null); + await assertRebuildOracle(t, f); +}); + +test('resume rejects a durable witness change even when its certificate survives', sqlite, async t => { + const f = await loadFixture(t); + for (let i = 0; i < 6; i++) f.write('r' + i); + const writes = observeWrites(t, f), committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + const changed = [...committed][0], before = f.db.getCachedSession(changed); + f.db.connection.prepare('UPDATE session_cache SET customTitle = ? WHERE sessionId = ?').run('external', changed); + const external = f.db.getCachedSession(changed); + assert.equal(external.fullReadStamp, before.fullReadStamp, 'external edit leaves certificate intact'); + assert.notEqual(rowWitness(external), rowWitness(before), 'only the durable witness is invalidated'); + const retried = await f.restart(); + assert.equal(retried.ok, true, retried.error); + assert.equal(writes.get(changed), 2, 'changed durable witness forbids reuse'); + assert.equal(f.db.getCachedSession(changed).customTitle, before.customTitle); + for (const id of [...committed].slice(1)) assert.equal(writes.get(id), 1, 'stable certified neighbor is reused'); + await assertRebuildOracle(t, f); +}); + +test('resume rejects a changed source observation with identical bytes and row witness', sqlite, async t => { + const f = await loadFixture(t); + const timestamp = new Date('2024-01-02T00:00:00.000Z'); + for (let i = 0; i < 6; i++) fs.utimesSync(f.write('r' + i), timestamp, timestamp); + const writes = observeWrites(t, f), committed = loseCertificateBatch(t, f); + assert.equal((await f.cache.populateCacheViaWorker()).ok, false); + const changed = [...committed][0], before = f.db.getCachedSession(changed); + const file = path.join(f.dir, changed + '.jsonl'); + fs.utimesSync(file, timestamp, timestamp); + const fresh = require('../read-session-file').readSessionFile(file, f.folder, f.root); + assert.equal(rowWitness(fresh), rowWitness(before), 'ctime change keeps the parsed row witness identical'); + assert.notEqual(fresh.fullReadStamp, before.fullReadStamp, 'ctime changes only source observation'); + const retried = await f.restart(); + assert.equal(retried.ok, true, retried.error); + assert.equal(writes.get(changed), 2, 'changed source stamp forbids reuse'); + assert.equal(f.db.getCachedSession(changed).fullReadStamp, fresh.fullReadStamp); + for (const id of [...committed].slice(1)) assert.equal(writes.get(id), 1, 'stable certified neighbor is reused'); + await assertRebuildOracle(t, f); +});