diff --git a/CLAUDE.md b/CLAUDE.md index 7c10d88..e12c768 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -139,10 +139,12 @@ These are tracked gaps that the project slash commands help address: installed copy), macOS arm64 (unsigned, Metal) and Linux x64 (xvfb, CPU). Each reached coordinated startup, loaded the native AI engine (Windows x64 on a Hyper-V GPU falls back from Vulkan to CPU, which the check reports but allows) and ran the packaged OCR worker on a synthetic - image. Still not verified: Defender real-time-protection timing (Sandbox disables Defender), the - interactive UAC/decline path, the zip's **Install now** click-through, the vision model (timed - out in the memory-limited sandbox), the Atlas, log-export, settings-import and large-library - restore workers, and a signed and notarized macOS build. The checklist lives in + image. Run 42702179765 added a seeded 5 MB LZ4 search index: all four restored it through + `oramaRestoreWorker` (`parsedIn: 'worker'`, 800/800 documents). Still not verified: Defender + real-time-protection timing (Sandbox disables Defender), the interactive UAC/decline path, the + zip's **Install now** click-through, the vision model (timed out in the memory-limited sandbox), + the Atlas, log-export and settings-import workers, and a signed and notarized macOS build. The + checklist lives in [`docs/RELEASING.md` ยง Packaged-integrity checks](docs/RELEASING.md#packaged-integrity-checks) โ€” keep that the single source and do not re-fork it here. diff --git a/docs/RELEASING.md b/docs/RELEASING.md index cb3d737..16997cb 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -332,7 +332,8 @@ image-preprocess and embedding workers crashed in v5.1.2 already has a library and check that the `[OramaVectorService] Restored database` log line carries `parsedIn: 'worker'`. Restore only uses the worker for saved files of 4 MB or more (smaller ones always log `parsedIn: 'main'`), so use a library that large; `'main'` on a - bigger file means the worker failed to load and restore fell back to the main thread. + bigger file means the worker failed to load and restore fell back to the main thread. The + packaged smoke workflow below checks this on every platform with a seeded 5 MB index. - [ ] **Worker dependency trees.** `verify:packaged-artifacts` (pre-flight step 6) fails when a worker bundle's externals, with their full dependency closure, do not resolve inside `app.asar.unpacked`. On the running app, confirm no log line shows a worker failing with @@ -428,16 +429,20 @@ branch before tagging (Actions โ†’ **Packaged Smoke** โ†’ **Run workflow**, or with `--no-sandbox`. - **Every leg** launches the app with a throwaway profile through `scripts/packaged-launch-smoke.js` (no model downloads, updater off). It passes only if coordinated startup completes, the native AI - engine loads, and the packaged OCR worker reads a synthetic image, with no `Cannot find module`, - worker error, `ERR_DLOPEN_FAILED`, integrity failure or failed startup in the log. A GPU backend - that fails and falls back to the CPU (Vulkan on the runner's Hyper-V GPU) is recorded under - `gpuFallbacks` and allowed; `NoBinaryFoundError` on the CPU backend fails the run. Logs, the - verdict and the Windows signature table are uploaded as - `packaged-smoke---`. + engine loads, the packaged OCR worker reads a synthetic image, and a seeded search index is + restored through the restore worker, with no `Cannot find module`, worker error, + `ERR_DLOPEN_FAILED`, integrity failure or failed startup in the log. A GPU backend that fails and + falls back to the CPU (Vulkan on the runner's Hyper-V GPU) is recorded under `gpuFallbacks` and + allowed; `NoBinaryFoundError` on the CPU backend fails the run. Logs, the verdict and the Windows + signature table are uploaded as `packaged-smoke---`. +- **Restore worker.** Before launch the script writes a 5 MB LZ4-compressed `fileChunks` collection + of 800 synthetic passages, in the app's own saved format, into the profile's `vector-db` folder. + That is past the 4 MB threshold, so the run fails unless the `Restored fileChunks database` line + shows `parsedIn: 'worker'` with all 800 documents inserted. A missing worker bundle or LZ4 binding + would otherwise fall back to the main thread silently. - **Still manual.** The runners have no models and click nothing, so the vision model, Atlas, log - export, settings import, a large-library restore, the UAC and **Install now** paths, Defender - timing, the tamper test and the updater live smoke stay with the checklist and the Windows Sandbox - run above. + export, settings import, the UAC and **Install now** paths, Defender timing, the tamper test and + the updater live smoke stay with the checklist and the Windows Sandbox run above. ### Windows zip (no-install build) diff --git a/scripts/packaged-launch-smoke.js b/scripts/packaged-launch-smoke.js index 302b2b7..d41796a 100644 --- a/scripts/packaged-launch-smoke.js +++ b/scripts/packaged-launch-smoke.js @@ -2,7 +2,8 @@ /** * Launch a packaged StratoSort build with a throwaway profile and check that it * really works: startup completes, the local AI engine's native binaries load, - * and a packaged worker (OCR) runs on a dropped image. Used by + * a packaged worker (OCR) runs on a dropped image, and a seeded search index is + * restored through the packaged restore worker. Used by * .github/workflows/packaged-smoke.yml on Windows, macOS and Linux runners. * * node scripts/packaged-launch-smoke.js --exe @@ -58,6 +59,103 @@ const FORBIDDEN = [ /Coordinated startup failed/ ]; +// The search index is read back at startup by the oramaRestoreWorker bundle, +// which decodes LZ4 with a native binding that must sit in app.asar.unpacked. +// It only runs for saved files of at least 4 MB (RESTORE_WORKER_MIN_BYTES in +// OramaVectorService), and when it cannot load, restore quietly falls back to +// the main thread. So the profile gets a real, compressed collection above +// that size, and the run passes only if the app restores it in the worker. +const RESTORE_WORKER_MIN_BYTES = 4 * 1024 * 1024; +const SEED_COLLECTION = 'fileChunks'; +// A fresh profile's embedding dimension (all-MiniLM-L6-v2, the default model). +const SEED_DIMENSION = 384; + +/** + * The text of a saved collection, in the app's own compact form + * (`{"docs":{"docs":{...},"count":N}}`, see serializeCollectionForDisk). + * Embeddings are pseudo-random but deterministic, so the file barely + * compresses and every run writes the same bytes. + * @param {{ docCount: number, dimension?: number }} options + * @returns {string} + */ +function buildSeedCollection({ docCount, dimension = SEED_DIMENSION }) { + let state = 0x2545f491; + const next = () => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0; + return state / 0x100000000 - 0.5; + }; + const docs = {}; + for (let i = 0; i < docCount; i += 1) { + const content = `Synthetic smoke-test passage ${i}. No real data.`; + docs[String(i + 1)] = { + id: `smoke-chunk-${i}`, + embedding: Array.from({ length: dimension }, next), + fileId: `smoke-file-${i % 50}`, + chunkIndex: i, + content, + startOffset: 0, + endOffset: content.length + }; + } + return JSON.stringify({ docs: { docs, count: docCount } }); +} + +/** + * Write a compressed collection of at least RESTORE_WORKER_MIN_BYTES (with + * margin) into the profile's vector-db folder, using the app's own LZ4 codec. + * @param {string} userData + * @param {{ lz4?: any }} [deps] + * @returns {{ collection: string, docCount: number, bytes: number, file: string }} + */ +function seedVectorIndex(userData, deps = {}) { + const lz4 = deps.lz4 || require('../src/shared/lz4Codec'); + if (!lz4.isLz4BindingAvailable) { + throw new Error('lz4-napi does not load in this runner, so the search index cannot be seeded'); + } + const dir = path.join(userData, 'vector-db'); + fs.mkdirSync(dir, { recursive: true }); + const target = Math.ceil(RESTORE_WORKER_MIN_BYTES * 1.25); + let docCount = 800; + for (let attempt = 0; attempt < 6; attempt += 1) { + const compressed = lz4.compressSync(Buffer.from(buildSeedCollection({ docCount }), 'utf8')); + if (compressed.length >= target) { + const file = path.join(dir, `${SEED_COLLECTION}.json.lz4`); + fs.writeFileSync(file, compressed); + return { collection: SEED_COLLECTION, docCount, bytes: compressed.length, file }; + } + docCount = Math.ceil(docCount * (target / Math.max(compressed.length, 1)) * 1.1); + } + throw new Error('Could not build a seeded collection large enough for the restore worker'); +} + +/** + * Whether the seeded collection came back through the restore worker, read + * from the app's structured "Restored database" line. + * @param {string} logs + * @param {{ collection: string, docCount: number }} seed + * @returns {{ ok: boolean, reason: string | null, line: any }} + */ +function restoreWorkerCheck(logs, seed) { + const want = `[OramaVectorService] Restored ${seed.collection} database`; + const entry = parseJsonLines(logs) + .map((p) => p.entry) + .find((e) => e.msg === want); + if (!entry) return { ok: false, reason: `no "${want}" line`, line: null }; + const line = { + parsedIn: entry.parsedIn, + total: entry.total, + inserted: entry.inserted, + failed: entry.failed + }; + if (entry.parsedIn !== 'worker') { + return { ok: false, reason: `parsed in ${entry.parsedIn}, not the worker`, line }; + } + if (entry.total !== seed.docCount || entry.inserted !== seed.docCount || entry.failed !== 0) { + return { ok: false, reason: 'not every seeded document was restored', line }; + } + return { ok: true, reason: null, line }; +} + const GPU_BACKENDS = new Set(['vulkan', 'cuda', 'metal']); function parseJsonLines(logs) { @@ -101,9 +199,15 @@ function gpuFallbackOrFailure(logs) { return { fallbacks, failures, selectedBackend: selected ? selected.entry.actual : null }; } -/** The verdict on one run's logs. Pure, so it is unit-tested on recorded lines. */ -function evaluateLogs(logs) { +/** + * The verdict on one run's logs. Pure, so it is unit-tested on recorded lines. + * @param {string} logs + * @param {{ seed?: { collection: string, docCount: number } | null }} [options] + */ +function evaluateLogs(logs, { seed = null } = {}) { const missing = REQUIRED.filter((r) => !r.pattern.test(logs)).map((r) => r.id); + const restore = seed ? restoreWorkerCheck(logs, seed) : null; + if (restore && !restore.ok) missing.push(`restore-worker (${restore.reason})`); const gpu = gpuFallbackOrFailure(logs); const failures = [ ...gpu.failures, @@ -112,7 +216,13 @@ function evaluateLogs(logs) { return { signature: String(p), line: line.slice(0, 400) }; }) ]; - return { missing, failures, aiBackend: gpu.selectedBackend, gpuFallbacks: gpu.fallbacks }; + return { + missing, + failures, + aiBackend: gpu.selectedBackend, + gpuFallbacks: gpu.fallbacks, + restore: restore ? restore.line : null + }; } function readLogs(userData) { @@ -218,6 +328,10 @@ async function main() { const inbox = path.join(root, 'inbox'); const uncategorized = path.join(root, 'Uncategorized'); seedProfile(userData, inbox, uncategorized); + const seed = seedVectorIndex(userData); + console.log( + `[smoke] seeded ${seed.collection} with ${seed.docCount} documents (${seed.bytes} bytes compressed)` + ); const started = Date.now(); const child = spawn(opts.exe, opts.args, { @@ -235,26 +349,24 @@ async function main() { }); let dropped = []; - let logs = ''; while (Date.now() - started < opts.timeoutMs && !exitInfo) { - logs = readLogs(userData); - if (!dropped.length && /Coordinated startup completed successfully/.test(logs)) { + const current = readLogs(userData); + if (!dropped.length && /Coordinated startup completed successfully/.test(current)) { dropped = writeSamples(inbox); console.log( `[smoke] startup complete after ${Date.now() - started} ms; dropped ${dropped.join(', ')}` ); } - const missing = REQUIRED.filter((r) => !r.pattern.test(logs)); - if (dropped.length && missing.length === 0) { + if (dropped.length && evaluateLogs(current, { seed }).missing.length === 0) { await sleep(5000); // let in-flight work log any late failure break; } await sleep(1000); } - logs = readLogs(userData); + const logs = readLogs(userData); killTree(child); - const { missing, failures, aiBackend, gpuFallbacks } = evaluateLogs(logs); + const { missing, failures, aiBackend, gpuFallbacks, restore } = evaluateLogs(logs, { seed }); const summary = { platform: process.platform, arch: process.arch, @@ -264,6 +376,8 @@ async function main() { droppedSamples: dropped, aiBackend, gpuFallbacks, + seededIndex: { collection: seed.collection, docCount: seed.docCount, bytes: seed.bytes }, + restore, missing, failures, ok: missing.length === 0 && failures.length === 0 && !exitInfo @@ -290,4 +404,11 @@ if (require.main === module) { }); } -module.exports = { evaluateLogs, gpuFallbackOrFailure }; +module.exports = { + evaluateLogs, + gpuFallbackOrFailure, + buildSeedCollection, + seedVectorIndex, + restoreWorkerCheck, + RESTORE_WORKER_MIN_BYTES +}; diff --git a/test/packagedLaunchSmoke.test.js b/test/packagedLaunchSmoke.test.js index 88d1327..afddcb9 100644 --- a/test/packagedLaunchSmoke.test.js +++ b/test/packagedLaunchSmoke.test.js @@ -5,7 +5,15 @@ * binary: the engine must be allowed to fall back to the CPU, while the CPU * floor failing (the clean-PC break of 5.1.2) must still fail the check. */ -const { evaluateLogs } = require('../scripts/packaged-launch-smoke'); +const path = require('path'); +const fs = require('fs'); +const { + evaluateLogs, + buildSeedCollection, + seedVectorIndex, + RESTORE_WORKER_MIN_BYTES +} = require('../scripts/packaged-launch-smoke'); +const { loadPersistedCollection } = require('../src/main/workers/oramaRestoreLoader'); const line = (entry) => JSON.stringify({ level: 'info', context: 'LlamaService', ...entry }); @@ -41,7 +49,8 @@ describe('packaged launch smoke verdict', () => { missing: [], failures: [], aiBackend: 'cpu', - gpuFallbacks: [{ backend: 'vulkan', loaded: 'cpu' }] + gpuFallbacks: [{ backend: 'vulkan', loaded: 'cpu' }], + restore: null }); }); @@ -85,3 +94,76 @@ describe('packaged launch smoke verdict', () => { ]); }); }); + +describe('seeded search index for the restore worker', () => { + const restored = (fields) => + line({ msg: '[OramaVectorService] Restored fileChunks database', ...fields }); + const seed = { collection: 'fileChunks', docCount: 900 }; + const base = [VULKAN_FAILED, CPU_SELECTED, INITIALIZED]; + + test('the seed is the app compact form, deterministic, with 384-dim vectors', () => { + const text = buildSeedCollection({ docCount: 3 }); + expect(buildSeedCollection({ docCount: 3 })).toBe(text); + const parsed = JSON.parse(text); + expect(parsed.docs.count).toBe(3); + const docs = Object.values(parsed.docs.docs); + expect(docs.map((d) => d.id)).toEqual(['smoke-chunk-0', 'smoke-chunk-1', 'smoke-chunk-2']); + expect(docs.every((d) => d.embedding.length === 384)).toBe(true); + }); + + test('the written file is past the worker threshold and the app loader reads every doc', async () => { + // The suite's fs is the in-memory mock; the seed and the app loader both use it. + fs.mkdirSync(require('os').tmpdir(), { recursive: true }); + const dir = fs.mkdtempSync(path.join(require('os').tmpdir(), 'strato-seed-')); + try { + const written = seedVectorIndex(dir); + expect(written.file).toBe(path.join(dir, 'vector-db', 'fileChunks.json.lz4')); + expect(fs.statSync(written.file).size).toBe(written.bytes); + expect(written.bytes).toBeGreaterThanOrEqual(RESTORE_WORKER_MIN_BYTES); + const load = await loadPersistedCollection(written.file, { + decompress: true, + keepText: false + }); + expect(load.status).toBe('ok'); + expect(load.docList).toHaveLength(written.docCount); + expect(load.docList[0].embedding).toHaveLength(384); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + + test('refuses to seed when the LZ4 binding does not load', () => { + expect(() => seedVectorIndex('unused', { lz4: { isLz4BindingAvailable: false } })).toThrow( + /lz4-napi does not load/ + ); + }); + + test('a restore in the worker with every document passes', () => { + const result = evaluateLogs( + logsOf(...base, restored({ parsedIn: 'worker', total: 900, inserted: 900, failed: 0 })), + { seed } + ); + expect(result.missing).toEqual([]); + expect(result.restore).toEqual({ parsedIn: 'worker', total: 900, inserted: 900, failed: 0 }); + }); + + test('a restore that fell back to the main thread fails', () => { + const result = evaluateLogs( + logsOf(...base, restored({ parsedIn: 'main', total: 900, inserted: 900, failed: 0 })), + { seed } + ); + expect(result.missing).toEqual(['restore-worker (parsed in main, not the worker)']); + }); + + test('a missing or partial restore fails', () => { + expect(evaluateLogs(logsOf(...base), { seed }).missing).toEqual([ + 'restore-worker (no "[OramaVectorService] Restored fileChunks database" line)' + ]); + expect( + evaluateLogs( + logsOf(...base, restored({ parsedIn: 'worker', total: 900, inserted: 899, failed: 1 })), + { seed } + ).missing + ).toEqual(['restore-worker (not every seeded document was restored)']); + }); +});