[pma1999]

Summary

Adds add and remove to IVF and IvfIndex, so a built or loaded index can gain and lose points without the original vectors. This follows the design discussion in #123.

The motivation is txtai (neuml/txtai#1251). Its backend has to keep a float32 copy of every vector because a built IvfIndex can't be extended or hide a point. For 20k vectors with 384 dimensions that copy is 30.88 MB of a 33.59 MB file, while the native 1-bit index is 2.70 MB.

API:

void IVF::add(const float* data, size_t n, const PID* cluster_ids = nullptr,
              bool faster = false, size_t num_threads = max);
size_t IVF::remove(const PID* ids_to_remove, size_t n);  // returns how many were newly removed

new_ids = index.add(vectors)                       # or add(vectors, cluster_ids=labels)
removed = index.remove(new_ids[:10])

• add quantizes the new vectors against the rotator and centroids the index already stores. New points get the PIDs max_elements() + i. cluster_ids is optional: without it each vector goes to its nearest centroid, chosen the way a query is routed. The partial last batch of each cluster is unpacked with the new fastscan::unpack_codes (the exact inverse of pack_codes) and repacked with the new points. The grown index is built next to the old one and swapped in, so the index is unchanged if add throws.
• remove sets f_add = +inf for each point. The estimated and lower-bound distances of that point are then +inf, so it never enters the result buffer and never triggers reranking.

Saved-index format and compatibility: the format does not change, so no version discriminator is needed. No file written before this change has an infinite f_add, so nothing in an old file is reinterpreted. A file with removed points loads in the published 0.5.0 wheel and never returns them (checked with nbits 1, 4 and 32, L2). The convention is documented in docs/docs/index/ivf.md and next to the persistence recipe in DEVELOPMENT.md.

Costs and limits, documented in docs/docs/index/ivf.md:

• Every add copies the whole storage, so it needs memory for both copies and each call has a fixed cost that grows with the index. Adding many vectors per call is recommended. There is no spare capacity.
• The centroids never move. Recall can drop when the added vectors come from a different distribution than the first ones.
• remove keeps the storage of removed points and cannot be undone.
• HNSW is not covered. Inserting a node needs the raw vectors of the existing nodes, which the index doesn't keep.

Not in this change, as discussed in #123: spare capacity so small adds don't move data, and an explicit deleted bitmap instead of +inf (which would need a format version).

Testing

Commands run on Linux x86-64 (i7-12700KF, GCC 13), following CONTRIBUTING.md and AGENTS.md:

Check	Result
C++ suite, portable Release (RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF)	256/256 passed
C++ suite, native tuning	256/256 passed
C++ suite, ASan and UBSan with the CI flags	251 passed, 5 skipped (AVX-512 only, this CPU has no AVX-512), no reports
python -m pytest tests/python -ra -q after reinstalling the extension	602 passed, 21 skipped (optional dependencies)
./scripts/check-format.sh (clang-format 15.0.7), ./scripts/check-python.sh, git diff --check	clean
./scripts/check-tidy.sh build-tidy (clang-tidy 15, full)	clean
mkdocs build --strict	clean
.github/scripts/test_ci_changes.py	12 passed

The new tests cover L2 and inner product, bit widths 1 to 9 and 32, and high_accuracy on and off:

• Adding the same points in one call, in chunks, with routing, with explicit cluster_ids and with several threads gives byte-identical files.
• Removed points never appear in the results, survivors keep their order and distances, and unfilled slots hold kPidMax with an infinite distance.
• After save and load, search matches the original index. A further add on the loaded index keeps the removed points removed and gives the same results as the original index.
• Invalid arguments leave the index unchanged.

I also checked that the tests can fail. Dropping the tombstone for 5 to 7 bit indexes only (an earlier version of the tests did not cover those widths) passes that earlier version and fails the extended tests. Each of these mutations makes at least one IVF or FastScan packing test fail: no tombstone, a finite tombstone (FLT_MAX), the wrong nibble order in unpack_codes, new PIDs without the offset, new points quantized against the wrong centroid, and dropping the old tail vectors when repacking.

Where the new C++ tests ran. CI runs are on my fork, with the same workflows: pma1999#1 (runs 36687459981 Test, 36687460043 Python Wheel, 36687460062 Release wheels, 36687459335 Documentation), all jobs green.

Backend	Where it ran
AVX2, native	local; fork CI Ubuntu job
AVX2 only, emulated	Intel SDE -hsw (local and the installed-consumer job)
AVX-512 with VPOPCNTDQ, emulated	Intel SDE -icl (local and the installed-consumer job)
ARM NEON	fork CI Native Linux ARM64 and Native Apple Silicon jobs (full ctest)
MSVC	fork CI Windows x86-64 (VS 2026) job (full ctest)
ASan and UBSan	local and fork CI
Python on Linux x86-64, Linux ARM64, macOS ARM64, Windows	wheel jobs run the full tests/python (597 to 600 passed per platform for CPython 3.11)

.github/workflows/test.yaml now adds IvfAddTest.* and IvfRemoveTest.* to the SDE filter, so the tombstone, add and save/load paths run with real AVX2 and AVX-512 dispatch. Locally that adds about 100 s to the -icl run. The installed-consumer job took 347 s on the fork, against a 15 minute limit.

Not run:

• Real AVX-512 hardware, only the SDE emulation.
• NEON, MSVC and macOS were only run in the fork CI, not on my machine.
• There is no scalar fallback for standard FastScan (it throws without AVX2, AVX-512 or NEON), so IVF search needs one of those. The generic HACC fallback isn't reached through IVF search in any job.

Measurements (1,000,000 vectors, 128 dimensions, 1,024 clusters, 1 bit, 8 threads, portable build, i7-12700KF; the benchmark scripts are not part of this PR, and I can share them):

• add of 1,000 vectors in one call: about 5 ms. The same vectors one call each: about 3 s (2.8 s and 3.7 s in two runs).
• add of 100,000 vectors: 0.16 s. remove of 100,000 ids: about 3 ms.
• Memory: a process that had loaded the 33 MB file held about 65 MB and peaked at about 100 MB during one add of 1,000 vectors.
• Recall after growing an index (100k vectors, 384 dimensions, inner product, nbits 4, nprobe = clusters / 16): 0.897 recall@10 for an index built on 10% or 50% of the data and grown with add, against 0.904 for a full rebuild. Starting from 1% gives 0.844. If the later vectors come from topics the first ones did not cover, recall falls to 0.52 to 0.74.

Refs #123.