[TamimEhsan]

Summary

Give each IVF cluster its own storage, so IVF::add() only touches the clusters that receive new points instead of copying the whole index on every call.

Before this change, all clusters shared four combined buffers (batch data, extra-bit codes, raw vectors, IDs). Any add() rebuilt all four, so its cost and peak memory grew with the whole index, not with the points added. Now:

• Per-cluster storage. IVF holds one private ClusterStorage per cluster. Cluster stays a non-owning view and the search path is unchanged.
• add() touches only the clusters it adds to. Untouched clusters are neither reallocated nor copied; their storage pointers are unchanged after add(). A touched cluster is rebuilt at exactly its new size next to its old storage and swapped in, unless it has reserved room (below).
• New API: IVF::reserve(cluster_id, num_points) and Python IvfIndex.reserve(cluster_id, num_points). These preallocate room so that later add() calls grow the cluster in place, without copying it. Reserved room is not saved, so a loaded index has none. In Python, reserve validates its arguments and takes the same exclusive-access guard as add. The change is additive; no existing signature changes.
• Strong exception guarantee kept. A cluster growing in place repacks its partial tail batch into scratch memory, which is copied in only after every touched cluster has succeeded. On failure, clusters grown in place shrink back to their old size, so the index and its saved bytes are unchanged.
• Staging is sized by the number of touched clusters, not by the total cluster count.

Saved-index format

Unchanged. save() still writes all batch data, then all extra-bit or raw data, then all IDs, in cluster order, which is byte-for-byte the old shared-buffer layout. load() reads that layout into each cluster's own storage.

To pin this, tests/fixtures/persistence/pre-ivf-per-cluster-storage/ adds frozen IVF files written by main at 64fc943. They cover 4-bit and raw (nbits=32) storage under L2 and IP, with four uneven clusters of 0, 1, 32 and 97 points. Unlike the single-cluster v0.5.2 fixtures, these catch a change in how multi-cluster data is ordered on disk. The test requires that re-saving each file reproduces it byte-for-byte. The generator, its CMake project and the SHA-256 manifest are committed alongside the fixtures.

How other IVF libraries do it

Checked against current source of FAISS, cuVS, Knowhere (Milvus), ScaNN, pgvector, Lance and Elasticsearch (DiskBBQ). SPFresh is described from its paper.

Storing clusters in memory. Every index that can be updated in memory gives each cluster its own storage:

• FAISS: ArrayInvertedLists (code and ID vectors per list), BlockInvertedLists (32-vector FastScan blocks per list, including IndexIVFRaBitQFastScan), and GPU IVF (a device vector per list).
• cuVS IVF-Flat, PQ and SQ: one list object per cluster.
• Knowhere IVF_FLAT_CC: a chain of fixed-size segments per cluster. Knowhere's IVF_RABITQ wraps FAISS.
• ScaNN tree-AH: a packed code buffer and ID list per partition.
• pgvector ivfflat: a linked chain of 8 KB pages per list.

The only shared-buffer design found is cuVS IVF-RaBitQ: flat arrays with a start offset per cluster, like main. It has no extend API; you can only build it.

Adding points and growing a cluster.

Approach	Used by
Geometric growth (std::vector)	FAISS ArrayInvertedLists, ScaNN
Power-of-two capacity	FAISS BlockInvertedLists and OnDiskInvertedLists; cuVS up to 1024 rows, then multiples of 1024
Exact fit	cuVS with conservative_memory_allocation; this branch by default
Room reserved by the caller	This branch (reserve)
Fixed-size segments, so existing data never moves	Knowhere IVF_FLAT_CC (48 vectors by default), pgvector (8 KB pages)
No in-place add: new rows stay unindexed until an optimize or merge builds a new segment	Lance, Elasticsearch
No add at all	cuVS IVF-RaBitQ

When adding to a partial FastScan block, FAISS BlockInvertedLists repacks the new vectors into the tail block one at a time, in place, as this branch does when a cluster has room. FAISS doesn't promise that a failed add leaves the index unchanged; this branch does, by repacking the tail in scratch first.

Searching while adding. Only Knowhere's IVF_FLAT_CC is designed for this: its segments never move, and each list keeps an atomic size. The others, including this branch, may move a cluster's memory while it grows, so writes need exclusive access.

Deleting points. FAISS remove_ids swaps the last entry into each removed slot and then shrinks the list. pgvector marks entries dead and reclaims them during VACUUM. Lance and Elasticsearch use deletion markers and drop the rows at the next rebuild or merge.

Updating centroids as data changes. This branch, like main, keeps centroids fixed after construction. Others adapt:

• cuVS: adaptive_centers updates centroids on extend.
• ScaNN: a small gradient step toward each added point, plus reassigning and splitting partitions.
• SPFresh: splits large posting lists, merges small ones and reassigns nearby points.
• Elasticsearch: re-clusters on merge, starting from the existing centroids.
• Lance: retrains when it optimizes.

Saving to disk. Disk-oriented formats use one file with clusters stored back-to-back and an offset per cluster: Lance (one file per index segment) and Elasticsearch (.clivf posting lists plus .mivf offsets, one set per Lucene segment). FAISS OnDiskInvertedLists also uses a single memory-mapped file, with a power-of-two slot per list and a free-slot list. This branch keeps the existing format, a single file with clusters in order.

Spilling points to a second cluster. Elasticsearch DiskBBQ also stores each vector in a second, nearby cluster (SOAR) to improve recall, and ScaNN has a spilling option. That changes cluster sizes and the file layout, and is out of scope here.

Possible follow-ups, not in this PR:

• Growth mode: an optional power-of-two growth mode like cuVS, so one-point adds amortize without calling reserve.
• Concurrent search: a segmented layout like Knowhere's, if search must run during add.

Known tradeoff: search speed

The below comparision is only true when hugepage is enabled. For hugepage turned off the qps and recall stays almost same, the difference due to run to run noise.

Search is up to about 3–4% slower than main on GIST1M, with identical results. Most per-cluster buffers are under 2 MiB, so they no longer get the huge-page advice (MADV_HUGEPAGE) the single shared buffer received. Whether to address this before merging is still open.

Setup: GIST1M (1M × 960, 1000 queries), 9-bit IVF, L2, recall@10 against the official ground truth. main is 64fc943. Intel Xeon Gold 6418H, g++ 14.3.0, Release, RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF with runtime dispatch, transparent huge pages set to madvise. Single-thread search pinned to one core. main and this branch alternate for five rounds, and each value is the median QPS.

Clusters	nprobe	recall@10	main QPS	branch QPS	change
256	16	0.929	894	860	−3.8%
256	32	0.980	508	491	−3.3%
1024	32	0.906	1406	1420	+1.0%
1024	64	0.964	833	826	−0.8%
4096	64	0.879	1097	1065	−2.9%
4096	128	0.948	779	772	−0.9%

• Quality: in every case both builds loaded the same index file and returned identical results (matching result hashes), so recall is unchanged.
• Cause: at 256 clusters, perf stat shows 2–3× more dTLB load misses on the branch (1.2–1.7M against 0.4–0.7M) and 57k page faults against 3.6k.
• Noise: a first run at 256 clusters and nprobe 32 showed −16% with a wide branch range (349–475 QPS). It did not reproduce, and the table shows the re-run.
• Indexes: the 256- and 1024-cluster indexes are pre-existing files. The 4096-cluster index was built by main from the same centroid files.

A possible fix: after construct and load, keep all clusters in one huge-page buffer as main does, and give a cluster its own buffer only once it grows. That should recover main's search speed for indexes that are never added to, at the cost of more complex storage code.

Docs

• The IVF::add header comment and the Python add docstring describe the per-cluster cost and reserve.
• docs/docs/index/ivf.md: the "Cost of add" section is rewritten, and reserve is documented.
• docs/docs/index/usage.md: reserve is listed as requiring exclusive access.

Benchmark

benchmarks/hotpaths/ivf_add.cpp adds a hot-path benchmark for add(), in single (one target cluster) and routed (nearest-centroid) modes, registered as the CTest smoke tests IvfAddHotpathSingle and IvfAddHotpathRouted.

Why this matters for LlamaIndex and Haystack integrations

Vector stores in frameworks such as LlamaIndex and Haystack receive data incrementally. Documents are chunked and embedded in small batches, often a few dozen vectors at a time, and written to the store batch by batch, and users keep inserting, updating and deleting documents in a live store. An adapter would turn each of these writes into an IvfIndex.add() call.

On main, each add() copies the whole index. Ingesting N vectors in batches of size b therefore costs about N²/b, and each call briefly needs about twice the index's memory. At about 98 ms per one-point add, document-by-document ingestion is impractical. With this change:

Workload	main	this branch	branch + reserve
One-point add (200 adds)	98 ms	74 µs	7 µs
Routed batch add, 64 / 128 / 1024 clusters	1×	1.7× / 1.9× / 5.9× faster	—
Peak extra memory during add	whole index	touched clusters only	none until the reserved room is used

• Streaming ingestion: an adapter can forward each write batch straight to add(), with no client-side buffering or periodic full rebuilds.
• Upserts: an overwrite or document refresh (delete, then re-add) costs about the size of the affected clusters, not of the index.
• Bulk loads: before a large ingest, reserve() can pre-size clusters, for example in proportion to their current sizes, so later batches grow in place without copying.
• Memory: writes no longer double peak memory, which matters for a store running inside a server process.
• Compatibility: search results and the saved-file format are unchanged, so existing indexes keep working.

An integration still has to provide:

• ID mapping: a map from string document or node IDs to PIDs below kSearchBufferMaxPointCount.
• Text and metadata: a side store for both, and metadata filtering. IVF stores only vectors and IDs.
• Concurrency: a lock or writer queue. The Python wrapper rejects an add that overlaps a search instead of waiting.
• Saving: save() still rewrites the whole file, and reserved room is not saved.
• Recall over time: centroids are fixed at construction. If new data drifts away from the training set, recall drops, so a rebuild or re-cluster option is needed.

Testing

Built with RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF, Release, g++ 14.3.0, Linux x86-64.

• C++ CTest: 414/414 passed, including the IvfAddHotpath* smoke tests. One test is skipped (ThreadCount.BoundParallelForUsesRuntimePlacementAndPropagatesErrors), as on main.
• Python: rebuilt and reinstalled into the project environment, confirmed the installed extension exposes IvfIndex.reserve, then python -m pytest tests/python: 703 passed, 2 skipped (the wheel tests).
• ./scripts/check-format.sh (clang-format 15.0.7), ./scripts/check-python.sh, python -m mkdocs build --strict --config-file docs/mkdocs.yml and git diff --check all pass.
• Fixture provenance: built 64fc943 separately. It loads all four new fixtures (cluster sizes 0, 1, 32 and 97), re-saves them byte-for-byte, and its search matches the .expected files. All eight manifest hashes match.

New and updated tests:

• EveryWayOfAddingTheSamePointsGivesByteIdenticalIndexes gains a reserved case: half the clusters grow in place and half are rebuilt, for bits {1, 4, 9, 32} under L2 and IP, and the result must be byte-identical to a single add.
• AddingToOneClusterLeavesOtherClustersUntouched: untouched clusters keep their storage pointers.
• UnreservedClustersGrowToExactlyTheirNewSize: a rebuilt cluster has no spare capacity.
• IvfReserveTest: growing in place until the reserved room runs out, exact error types and messages, and confirmation that spare capacity changes neither the saved file nor a loaded index.
• FailedAddLeavesClustersGrowingInPlaceUnchanged: an injected failure in one cluster, after another has grown in place, leaves the saved bytes unchanged, and a retry matches a clean index. With the old in-place tail write restored, only this test fails.
• Python: test_reserved_adds_match_a_single_add (bits 1, 4 and 32) and test_reserve_rejects_invalid_arguments.

Not run: a full ./scripts/check-tidy.sh build-tidy (required before merge), and a search benchmark with recorded numbers (see the TODO above).