fix(v-memory): sub-batch embeddings so large corpora don't trip the 300s timeout - #9
Open
khymerao wants to merge 1 commit into
Open
fix(v-memory): sub-batch embeddings so large corpora don't trip the 300s timeout#9khymerao wants to merge 1 commit into
khymerao wants to merge 1 commit into
Conversation
…00s timeout refresh --with-embeddings flattened ALL chunks into one embedder subprocess call capped at timeout=300. On a sizable corpus (~1340 chunks here) that call exceeds 300s, returns None, and silently degrades the WHOLE refresh to FTS5-only (0 vectors) with no error. Split the embed into bounded EMBED_BATCH (256) sub-batches so every call stays well under the timeout; degrade stays all-or-nothing. Adds _embed_batched selftests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
/v:memory-refresh --with-embeddingssilently populates 0 vectors on any sizable corpus — the dense (semantic) recall lane never activates, and there's no error to explain why.Root cause
reindex_batchflattens every chunk into a singleembedder(flat)call, andembed_textscaps that subprocess attimeout=300. On a real corpus the one call exceeds 300s:embed_textsis degrade-safe by design — a timeout returnsNone, which persists all chunks withNULLembeddings (FTS5-only). So the whole refresh degrades quietly:doctorshows1340 chunks (0 with vectors)even though bootstrap succeeded and the embedder works fine on small inputs.Fix
Split the embed into bounded
EMBED_BATCH(256) sub-batches (_embed_batched), so no single embedder call approaches the timeout. The ONNX model reloads per sub-batch (~3s) — cheap versus losing the dense lane entirely. Degrade stays all-or-nothing: any failed sub-batch → whole resultNone→ FTS5-only (unchanged contract, never a partial vector set).Verified
refresh --rebuild --with-embeddings→1340 chunks (0 with vectors)1340 chunks (1340 with vectors), full rebuild ~2:38 (6 sub-batches), no timeoutdoctor: dense engaged (scale gate ≥ 80)--selftestgreen, incl. 3 new_embed_batchedchecks (order/count preserved, splits by size[256,256,88], all-or-nothing on failure)Scope
One file —
scripts/compound-v-memory.py. No new deps, no API change, degrade-safe semantics unchanged.