Skip to content

fix(v-memory): sub-batch embeddings so large corpora don't trip the 300s timeout - #9

Open
khymerao wants to merge 1 commit into
procoders:mainfrom
khymerao:fix/v-memory-embed-batch-timeout
Open

fix(v-memory): sub-batch embeddings so large corpora don't trip the 300s timeout#9
khymerao wants to merge 1 commit into
procoders:mainfrom
khymerao:fix/v-memory-embed-batch-timeout

Conversation

@khymerao

Copy link
Copy Markdown

What

/v:memory-refresh --with-embeddings silently populates 0 vectors on any sizable corpus — the dense (semantic) recall lane never activates, and there's no error to explain why.

Root cause

reindex_batch flattens every chunk into a single embedder(flat) call, and embed_texts caps that subprocess at timeout=300. On a real corpus the one call exceeds 300s:

full 1340 chunks -> None in 301.4s

embed_texts is degrade-safe by design — a timeout returns None, which persists all chunks with NULL embeddings (FTS5-only). So the whole refresh degrades quietly: doctor shows 1340 chunks (0 with vectors) even though bootstrap succeeded and the embedder works fine on small inputs.

Fix

Split the embed into bounded EMBED_BATCH (256) sub-batches (_embed_batched), so no single embedder call approaches the timeout. The ONNX model reloads per sub-batch (~3s) — cheap versus losing the dense lane entirely. Degrade stays all-or-nothing: any failed sub-batch → whole result None → FTS5-only (unchanged contract, never a partial vector set).

Verified

  • Before: refresh --rebuild --with-embeddings1340 chunks (0 with vectors)
  • After: 1340 chunks (1340 with vectors), full rebuild ~2:38 (6 sub-batches), no timeout
  • doctor: dense engaged (scale gate ≥ 80)
  • --selftest green, incl. 3 new _embed_batched checks (order/count preserved, splits by size [256,256,88], all-or-nothing on failure)

Scope

One file — scripts/compound-v-memory.py. No new deps, no API change, degrade-safe semantics unchanged.

…00s timeout

refresh --with-embeddings flattened ALL chunks into one embedder subprocess
call capped at timeout=300. On a sizable corpus (~1340 chunks here) that call
exceeds 300s, returns None, and silently degrades the WHOLE refresh to
FTS5-only (0 vectors) with no error. Split the embed into bounded EMBED_BATCH
(256) sub-batches so every call stays well under the timeout; degrade stays
all-or-nothing. Adds _embed_batched selftests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant