Skip to content

Improve Coconut top-k seeding and leaf reads - #78

Closed
sophisid wants to merge 2 commits into
mainfrom
fix/coconut-query-refinement
Closed

sophisid wants to merge 2 commits into
mainfrom
fix/coconut-query-refinement

Conversation

@sophisid

Copy link
Copy Markdown
Collaborator

Coconut exact top-k previously started with an empty heap, ran queries serially, and reopened leaf files for each candidate vector. This change seeds refinement from distinct records near the query's sortable-SAX key, honors the requested query workers, and reads complete candidate leaves into private query buffers. Exact refinement still considers every unseeded SAX summary and the live insertion buffer.

Top-k and range now share checked leaf reads, including partial final leaves; missing or truncated reads raise an error instead of producing incomplete results. Contiguous build inputs are borrowed only while writing the index. Regression tests cover FAISS ground truth, streaming inserts, worker counts, ties, directory endpoints, oversized k, input lifetime, invalid configuration, and failed reads.

The README and pinned upstream audit explain the remaining storage differences: the key directory is resident, and inserted records remain in an unmerged buffer. This does not implement Berkeley DB storage or Coconut-LSM flush/merge. /scripts/experiments/ is added to .gitignore; experiment scripts, datasets, and unrelated MESSI changes are excluded.

Validation:

  • Release build of the C++ library and Python bindings with BUILD_COCONUT=ON.
  • CTest: test_coconut_topk_py and test_streaming_py passed (five Coconut checks and four existing streaming checks).
  • Release library build with BUILD_COCONUT=OFF, with MPI, CUDA, FFTW-backed algorithms, and C++ FAISS disabled.
  • git diff --check passed.

@sophisid sophisid closed this Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant