Add resumable offline and online vector storage migration

[ajroetker]

Existing standalone tables retain primary-LSM vector ownership after #728. This adds explicit, resumable offline and online migration to shared vector ownership while preserving artifact identities, models, configurations, and queries.

Online migration co-commits a candidate reference map with ordinary writes, validates its complete coverage, publishes a durable ownership decision, drains inline values without reappending payloads, consolidates ANN serving generations, and waits for bounded primary overlap rewrites to remove superseded inline values. The catalog and compiled storage owner reconcile interrupted publication. Updates, deletes, enrichment and old snapshots retain their normal version/lease protection; cancellation is limited to prepublication jobs.

antfly storage migrate --table NAME --to vector-store --job ID supports online execution with --url and offline execution with --catalog plus --replica-root. The API creates jobs at POST /tables/{table}/storage/migrations, observes them with GET /tables/{table}/storage/migrations/{job}, and advances/publishes/cancels them with POST on the job. GET never starts work or reconciles catalog publication. The standalone server and offline command are shipped in the same binary.

The offline command streams the whole physical root into a resumable shadow, replays committed derived work, uses the same conversion/verification and native ANN lifecycle, and publishes through recoverable generation exchange. Standalone and the command share exclusive catalog admission. The offline candidate uses a bounded 64 MiB block cache under the normal memory budget when no caller cache is supplied. Descriptor-cache hits now avoid filename allocation; a repeated Debug microbenchmark measures 6.5× faster cached acquisitions, with no end-to-end speedup claim yet.

Scope is local, single-shard, single-replica standalone tables. Configuration/topology changes and overlapping backup/restore are fenced. Reverse conversion, HA, replication and broader backup support remain separate work. Operator instructions and recovery details are in zig/VECTOR_STORE.md.

Validation so far:

• 23 focused migration checks, including both legacy ANN conversion modes, preparation/publication faults, physical-copy faults, repeated restart, cancellation, concurrent models/chunks, old readers, resource rejection and last-index rebuild.
• 68 source payload tests.
• DocStore/dependency suite: 887 passed, 21 skipped, no unexpected logged errors or leaks.
• OpenAPI consistency, changed-file Python formatting/lint, four completion tests and 12 packaging tests.
• Job observation/recovery and table-admin route authorization checks.
• The final production migration/vector-store suite passes all 13 checks, including online restart, offline lock/resume/catalog recovery, multi-model reads, last-index rebuild, cancellation, and abrupt process death after a WAL-only page receipt or primary reclamation request. Both online and offline conversion preserve all 32 ordered top-10 neighbor lists for a 4,096-vector native ANN before and after restart.
• The page-commit regression and focused DB migration suite pass (12 tests, no leaks), including crashes between the reclamation manifest request and job receipt. The Linux fixed-step fixture failures are addressed with a deadline-based completion driver and an explicitly injected timed GC retry; all 12 focused tests pass.
• Primary reclamation regression and LSM suite: 493 passed, 23 skipped. The new case covers zero tombstones, bounded input I/O, old readers, restart after partial progress and obsolete-value removal.
• HA mutation inventory/classifier checks and focused C-ABI owner test pass. The final packaged Linux Antfly and inference E2E jobs pass; the dense checkpoint fixture now disables optional workers while manually installing certificates.
• Generated Python/TypeScript/Go clients are updated; Python 133 tests plus lint/typechecking, TypeScript 283 tests plus typechecking, and Go SDK tests pass.

The final 1M fresh/online/offline screen completed with exactly 999,000 retained source payloads in every arm. Total allocated disk is 3.980/3.977/4.006 GB, with migrated primary SSTables down to 349/379 MB. Offline conversion plus churn fell from 29.9 to 8.85 minutes across the diagnostic screens, including explicit reclamation; online takes 7.49 minutes. Sampled peak RSS is 6.84/11.67/7.14 GiB. Reopened semantic QPS differs by up to about 6% online and 12% offline from fresh, and the offline arm has lower recall. These single sequential runs do not establish equivalent query quality or performance. Final 50K migration reclaims primary SSTables to 17–19 MB and total disk to within about 1% of fresh storage; all three arms retain the expected 49,000 live payloads. Online conversion plus churn takes 18.49 seconds (including primary reclamation), offline 11.52 seconds. The 50K screen exposed forced primary flushes after every page; pages now sync the WAL without forcing an SSTable, with publication retaining the full barrier. The first 1M online migration recovered correctly but retained old inline values below normal compaction thresholds; the new reclamation phase makes that work explicit and durable. Sustained queries cannot indefinitely defer this requested work.

Fresh and migrated 50K tables show a shared post-churn semantic-query slowdown, also measured after restart. A no-churn control retains high throughput after restart, so the follow-up target is deleted-identity lookup work, not migration alone. Results and limitations are recorded in zig/VECTOR_STORAGE_MIGRATION_FINDINGS.md. No performance equivalence or migration-duration guarantee is claimed.

The repository now requires a human /ci run e26ab3a1dd7b28e1ede78c4a5b41f9551aba5ff4 comment before starting CI for a PR revision. The two Linux unit step-count failures are fixed and pass locally; the final CI rerun awaits that approval.

[ajroetker]

/ci run 9da8209

[ajroetker]

/ci run 5f84999

[ajroetker]

/ci run 3f38836