[aaltshuler]

What & why

Upgrade the complete Lance family from 11.0.0 to 13.0.0, preserving graph format 16, explicit V2_2 files, and the existing full-text analyzer generation. Format-16 graphs need no storage conversion. Genuine predecessor/candidate tests cover reads, writes, search, Blob contents, history, maintenance, rollback, and whole-root backup restoration.

The adapters preserve pagination faults/accounting, opaque object-store context, strict manifest-cache identity, and typed completion evidence. Retire three obsolete compatibility paths:

• The merge-insert writer uses strict native SourceDedupeBehavior::Fail; typed source uniqueness checks remain. The indexed-update regression keeps the same BTREE UUID/coverage while the rewritten row moves to an uncovered fragment, then updates it again. Lance's processed_row_ids still exists; the upstream fix is the index-coverage prefilter correction in lance#6965.
• Remove the Legacy-only nearest partition fence. Supported graph writers and supported upgrade sources use V2_2; native parallel ordering and the complete public nearest result remain pinned.
• Remove clone index/base/fragment-reuse preflight and its fixtures. Production branch creation clones only __manifest, which has no fragment-reuse index. Keep exact source-version refusal, scoped main-source manifest-location reuse, local ETag normalization, and configured commit-handler/session/failure semantics. A native nested-clone guard requires a cold indexed query. Remove the now-unused core prost and dev lance-linalg dependency edges; package versions and transitive feature selections remain unchanged.

The cleanup guard now puts the attached manifest's embedded timestamp at −10 days and a detached-only data file at −8 days, on opposite sides of Lance 11's retained floor and Lance 13's seven-day listing cutoff. It preserves the inline transaction when changing the fixture timestamp. Cleanup documentation distinguishes file age from snapshot age: an old detached-only file can make a recent descendant or the current graph unreadable; files shared with retained attached versions remain protected.

Consolidate two remaining adapters: use Lance's native ChainedWrappingObjectStore in place of StackedWrapper, preserving memo-outside-probe order and pagination refusal; remove the test-only generic stage_merge_insert adapter, trait forwarding and duplicate key validator. The four staged behavior owners now exercise production stage_keyed_write using keyed fixtures, with update-only rebase semantics and all substantive assertions retained. Refresh the affected comments, source-guard delimiter and vocabulary inventory. These APIs predate Lance 13; this follow-up removes duplication without changing the storage contract (282 fewer lines).

Backing issue / RFC

• Implements the dependency compatibility audit required by accepted RFC 0043.

Checklist

• One focused dependency upgrade; no new graph or CLI capability
• Existing regression owners extended and genuine Lance 11/13 CI journey added
• Compatibility docs and changelog updated
• Publication, snapshot, cleanup, bounded-I/O and memory safeguards retained

Validation

Adapter consolidation (91ecceef)

• Clean pre-edit core/engine unit baseline: 654 passed. Final affected qualification: 890 passed, zero failed, three intentionally ignored — 651 core/engine unit tests, 63 write tests, 69 native Lance guards, 35 source/API guards, 19 write-cost tests, one deterministic storage-cost golden and 52 vocabulary-guard unit tests.
• The four migrated staged-write owners preserve superseded-fragment visibility, committed upsert values, disjoint-delete rebasing and the limitation of independently staged overlapping upserts. Native wrapper tests retain all four pagination allow/refuse combinations and explicitly pin ordinary wrapper order. Existing CSV endpoint-read checks confirm caching and low-memory fallback across every load mode.
• The deterministic storage-cost golden passes unchanged. Final-tree all-target Clippy for core/catalog/engine passes with warnings denied. Formatting, docs, AGENTS, dependency-source, spelling and required-test inventory checks pass. Independent diff review found no remaining issues.
• The complete Rust-string vocabulary audit remains blocked by drift already present at 0a36f664: 1,693 unclassified and 529 stale occurrences. CI explicitly disables that audit. Exact baseline-source and final-source scans prove this follow-up introduces no mismatches and removes two existing unclassified occurrences; the obsolete helper's inventory entries are removed.
• Fresh-head GitHub checks are running. The complete CI and release-profile results below are historical qualification of their explicitly named heads.

Retirement follow-up (0a36f664)

• Local affected debug owners: 1,739 passed, zero failed, 12 intentionally ignored across core/catalog/engine units, integration, native Lance guards, schema, search, writes and source guards. This includes the stale-BTREE repeated update, strict duplicate rejection, cold nested-clone index lookup and corrected cleanup chronology.
• The corrected cleanup fixture passes on genuine Lance 13 and intentionally fails the deletion-count assertion on the genuine Lance 11 predecessor: (old_versions, data_files_removed) is (0, 0) on 11 versus (0, 1) on 13. The equally old attached file remains readable on 13; the fresh descendant of the swept detached file does not. The temporary predecessor test edit was restored.
• Deterministic storage-cost golden passes unchanged. No retirement changes a golden row.
• Final committed-tree all-target Clippy for core/catalog/engine passes with warnings denied and the locked dependency graph. Formatting, docs, AGENTS, spelling, dependency-source and CI-owner inventory checks pass; the source guards were also rerun after dropping the two unused dependency edges.
• Final committed-tree release-profile qualification passes: 1,703 passed, zero failed, 12 intentionally ignored across core/catalog/engine units, integration, native Lance guards, schema, search and writes, with failpoints enabled. The four strengthened guards pass in release. This uses the macOS archive flags (RUSTFLAGS=-C code-model=large, RUST_MIN_STACK=16777216); the linker emitted the previously observed large unwind-table warning, without build or runtime failures.
• Fresh-head CI, GQT, and DST are green at 0a36f664, including CI Gate and GQ Logic Tests. Workspace: 4,717 passed, 39 intentionally skipped, plus two doctests. Storage compatibility: 147 passed (8 cross-version, 100 engine, 4 Lance-column and 35 protocol guards). The full configured GQT corpus passes all 399 cases; DST passes 108 tests with 31 intentionally ignored, including its storage-cost golden. RustFS passes 121 default and 3 failpoint tests; Azure passes all 15 required tests and the shared-container inventory check.
• Earlier upgrade qualification below records its stated historical heads.

Prior upgrade qualification

• Genuine Lance 11 baseline: 930 core/catalog/engine/guard tests and 3 benchmark-counter tests passed; the exact predecessor's DST golden passes both locally and in main CI.
• Local debug workspace at 0881e10b: 4,734 passed, zero failed, 39 intentionally ignored.
• Configured cloud qualification: all 16 RustFS/S3 and 15 Azurite/Azure required cells passed, with exact inventories and skip rejection. Tests run serially as CI requires. An earlier local parallel S3 attempt had transport errors; the emulator stayed healthy and the complete serial run passed.
• Strengthened genuine Lance 11 ↔ 13 journey passed, including uppercase analyzer queries, nullable vectors, managed Blob values, trained/untrained/uncovered FTS, candidate rebuilds, history, compaction, cleanup, rollback reads, and byte-preserving backup restoration.
• Workspace Clippy, formatting, cargo-deny, dependency-source, documentation, AGENTS, workflow/CI inventory, storage-upgrade checker and spelling checks passed.
• Full DST suite: 108 passed, zero failed, 31 intentionally ignored; the audited cost golden passes without diagnostic instrumentation.
• Uninstrumented macOS release binaries at 0881e10b built with the archive workflow flags. The genuine Lance 11 ↔ 13 journey passed against the release CLI; three existing process journeys passed against the release CLI/server (query and mutation, branch merge, full cluster lifecycle). Version/commit/channel and all three binary startup checks passed. The release CLI also passed real init/load/read journeys against both isolated S3 and Azure emulators, with no skips. The macOS linker reported its large unwind-table warning; builds and runtime checks succeeded.
• Previously qualified-head CI, GQT, and DST are green at 4faecee9: 4,720 workspace tests plus 2 doctests; all 399 GQT corpus cases and the served/DST-backed groups; all 16 S3 and 15 Azure required cells; 108 DST tests and shipped-engine Clippy. All three corrected overlay tests explicitly pass in debug. Storage compatibility ran 8 storage-upgrade cases (including 6 genuine predecessor journeys), 100 engine upgrade/recovery tests, 4 Lance-column tests and 35 protocol guards without failures or runtime skips.
• Combined release-profile mechanism qualification (with failpoints enabled) is complete: 1,707 passed, zero failed, 12 intentionally ignored across core/catalog/engine units, integration, native Lance guards, schema, search and writes. The first run reproduced three existing fixtures that wrongly expected experimental overlays to be readable in release (both Lance 11 and 13 gate them). Commit 4faecee9 keeps all debug positives and asserts typed release refusal, real cleanup with zero deletions and intact original/overlay inventories, plus continued modern/legacy planning. All 577 corrected engine unit tests pass in release, as do core/catalog units; the other optimized targets are unchanged by this test-only correction. Formatting, docs, 35 source guards and the environment-access guard also pass. Other-platform archive packaging remains release-workflow-owned.

Compatibility boundaries

• Experimental data overlays, lineage spilling and stable-row-ID fragment reuse remain disabled in production builds. The removed malformed external-FRI fixtures exercised an unsupported raw row-address shape. Native clone creation preserves origins but need not read or reject malformed external payloads before branch publication; the adapter does not promise that validation.
• Existing compatible FTS certificates keep generation lance11.0.0-frostem1.20260821.3-v1; package pins and the unchanged analyzer dependency graph are separately audited.
• Release and cross-version qualification use unmodified crates.io packages. The predecessor is built from exact commit 0a62bb7b2aa5b99b012f413807c216277f7f9596.

Audited DST cost-baseline changes

The genuine Lance 11 source at 0a62bb7b2aa5b99b012f413807c216277f7f9596 passes the existing golden. The Lance 13 candidate reproduces the same 17 read-count changes seen in CI. Temporary object-store tracing independently accounts for every successful physical get_opts call in both deterministic replays (26 ledgers total); its totals exactly equal every l.get row. All other golden counters, including writes, deletes, lists and adapter calls, are unchanged. Failed-read counts are unchanged. Tracing has been restored before final qualification.

The reductions come from Lance 13 batching adjacent page/column requests in the catalog's __manifest/.../data/*.lance files. This includes page-initialization metadata inside data files, not fewer native _versions/*.manifest reads. Lance 13's RequestBatch gathers ranges before the I/O scheduler coalesces them, and primitive-page initialization combines cache-miss ranges. The same data-file inventory is accessed for every changed label. For example, one merge reads three catalog data files in 9+14+9 requests under 11 and 4+8+4 under 13. A diff reads the same file in 16 requests under 11 and 8 under 13; distinct 4-byte headers at 0..4 and 768..772 become one 0..772 request, and the nearby column payloads become a shared range. Request sizes/offsets are diagnostic evidence, not new byte-count goldens.

Two labels also expose a separate native index-metadata cost: scalar-index opening under existing v0 fragment reuse now first loads the raw index listing to choose the remapping generation, then the legacy v0 opener looks up that listing again. Where that lookup is not cached, it costs a native table-manifest HEAD plus payload GET. AddFriend has two additional pairs and _audit has three additional pairs in aggregate. These are included below, rather than attributed to batching or hidden by a broader budget.

Golden label	l.get 11 → 13	Catalog data-file GETs 11 → 13	Native table-manifest offset
AddFriend	117 → 111	18 → 8	+2 HEAD, +2 payload
BranchDelete	31 → 26	9 → 4	unchanged
Cleanup	187 → 163	56 → 32	unchanged
_audit	749 → 693	131 → 69	+3 HEAD, +3 payload
_setup	50 → 40	17 → 7	unchanged
_verify	1620 → 1520	229 → 129	unchanged
MergeTargetHistory2	116 → 100	32 → 16	unchanged
MergeTargetHistory32	116 → 100	32 → 16	unchanged
MergeBackAfterThreeWay	134 → 118	32 → 16	unchanged
DiffPageIdle1	47 → 39	16 → 8	unchanged
DiffContinuationIdle1	40 → 32	16 → 8	unchanged
DiffPageIdle9	47 → 39	16 → 8	unchanged
DiffContinuationIdle9	40 → 32	16 → 8	unchanged
BranchDiffPageIdle1	47 → 39	16 → 8	unchanged
BranchDiffContinuationIdle1	25 → 22	7 → 4	unchanged
BranchDiffPageIdle9	47 → 39	16 → 8	unchanged
BranchDiffContinuationIdle9	25 → 22	7 → 4	unchanged

No unexplained counter movement remains. The small/large history pairs remain equal. This approves deliberate regeneration of the exact deterministic memory-store golden; it is not a wall-clock or cloud-performance claim.

Exact upstream mechanisms

• Lance 13 RequestBatch, wired into SchedulerContext at lines 3089–3109.
• Lance 13 primitive-page initialization combines initialization ranges before issuing reads.
• Lance 13 scalar-index opener enters the remapping dispatcher: raw index listing at 312, then existing v0 opener at 325.
• That v0 opener invokes find_frag_reuse_index_meta, which loads the listing again. Lance 11 called the legacy opener directly.

[ragnorc]

Alignment note from #1016 (branch create at a commit), for this bump's API audit. #1016 adds two dependencies on Lance behavior:

1. Branch creation at a non-head version. ManifestCoordinator::create_branch_at checks the __manifest dataset out at the version that published a commit (checkout_version(V)), then runs create_branch_recoverably(ds, native, V) through the clone adapter. That relies on Dataset::create_branch(name, V) resolving a bare version number on the checked-out branch, and on the adapter's exact source-version check accepting V when the dataset is checked out at V. This PR keeps that refusal, so it should pass. feat(branch): start a branch at a graph commit #1016's branching.rs start tests (a main commit and a named-branch commit) exercise it.
2. The collector's path inventory. The branch-start retention probe reuses collector::capture_dataset_paths on each pinned table version and probes every object it lists. A Lance change to what a manifest references (overlay files here, fragment trees at a later bump) needs capture_dataset_paths to follow, as cleanup already does. The probe then follows with no change.

[ragnorc]

Recommendation: merge. I found no blocking defect at 91ecceef.

This pull request moves the Lance family from 11.0.0 to 13.0.0. Graph storage format stays 16. Data files stay V2_2. The full-text analyzer generation stays lance11.0.0-frostem1.20260821.3-v1. A format-16 graph does not need a storage conversion.

What changes for OmniGraph

Lance 13 changes three behaviors that this engine relies on.

Stock cleanup_old_versions still ignores detached manifests. It can delete a data file that only a detached version names, once that file is older than seven days. The manifest remains. The version opens. A scan of that version fails. OmniGraph commits stay detached until one graph publication, so this can make the current graph unreadable. omnigraph cleanup is the only supported cleanup. The changelog and the maintenance guide say this. stock_cleanup_sweeps_a_week_old_detached_data_file_lance_13 pins both sides of the seven-day cutoff.

Flat nearest search no longer forces one scan partition. Lance 13 keeps the KNN sort through V2 payload hydration. WorkMemory::stream merges on that order when the plan has more than one partition. Indexed nearest still sorts by distance and row id in Scanner::ann before that hydration.

Merge-insert now fails on duplicate source keys (SourceDedupeBehavior::Fail). OmniGraph still folds pending upserts last-wins in dedupe_merge_batches_by_id before stage_keyed_write. Issue #925 is unchanged. That check is OmniGraph's typed uniqueness, not this Lance flag.

Contracts that still hold

I checked these in the Lance 13.0.0 tag 0923349 and in this tree.

A catalog overwrite with CommitBuilder::with_max_retries(0) is still a strict overwrite. Lance does not rebase that attempt. Engine commits still set skip_auto_cleanup.

Compaction of a graph table still commits frag_reuse_index: None. Lance sets the stable-row-id fragment-reuse flag only when a fragment-reuse index is present. This build does not create one. Debug builds can read that flag. Release builds cannot. A debug write of that index would make a release reader refuse the dataset.

Spilled row lineage is opt-in (lance.row_lineage.spill) and debug-gated. OmniGraph does not set that key. A spilled sequence is not an inline change-feed proof. The bounded small-file reader refuses it.

Scanner::batch_size on a V2 read still beats LANCE_DEFAULT_BATCH_SIZE. get_batch_size prefers the environment variable. That function serves the legacy scan and strict_batch_size. Pipelined reads and compaction use the V2 path. explicit_scanner_batch_size_beats_lance_default_batch_size_env passed here.

Session cache defaults stay 6 GiB for indexes and 1 GiB for metadata. Issue #702 stays open.

DataFusion stays 54.0.0. Arrow stays 58.3.0. The toolchain stays 1.97.1. Lance 13 requires Rust 1.91. Two object_store crates remain: 0.13.2 for DataFusion and 0.14.2 for Lance. The quick-xml advisories that applied to the 0.14 line are no longer ignored. paste and rsa remain.

Local manifest discovery still returns an unquoted inode-mtime-size tag. object_store 0.14 quotes the same token. The adapter quotes only a bare three-part hex tag on a direct local path. A quoted tag is left unchanged. A mismatch is a cache miss.

Dataset::create_branch still refuses a source whose version moved. capture_dataset_paths still lists referenced_lance_files, and that list includes overlay data files. Lance 13 destructures the fragment, so a new file field fails to compile. Fragment metadata trees are not a field in 13.0.0. This matches the alignment note from #1016. capture_dataset_paths does not need a change for this bump.

Scalar index delta segments can still return a stale value for an updated row. The guard that expects that bug passed. A whole-index rebuild stays required.

Production vector builds stay ivf_flat. The 5-bit ivf_rq default does not apply.

Liability

The clone adapter no longer repairs inherited index bases and no longer rejects a malformed external fragment-reuse index before publication. Production branch creation clones the __manifest dataset only. stock_lance_nested_clone_preserves_inherited_index_base_issue_7840 passed. A later clone of an indexed graph table would not get the old preflight. That path is not used today.

The new operator hazard is external cleanup. The code path is unchanged. The Lance rule is new, and the docs and the guard carry it.

The DST golden changes 17 l.get rows. Every one is lower. I did not re-run the DST suite. The pinned DST job on this commit is green. This is a request-count change. It is not a wall-clock measurement.

Findings

Blocking: none.

Optional: the upgrade guide says a graph the new fleet has written is not qualified for the old reader. See the inline note. Keep the mixed-fleet ban.

Issues this does not close

#970, #971, #895, #702, #925, #841, #958, and #1094 stay open. #1094 is the order of OmniGraph's own cleanup. It is separate from the Lance stock-cleanup rule above.

Validation

GitHub checks on 91ecceef are green, including CI Gate, GQ Logic Tests, Test Workspace, Storage Upgrade Compatibility, RustFS, Azurite, Azure contract and deployment, DST, and cargo deny. Devin Review is a skipped trial. It is not a review of this diff. There was no earlier review.

I ran these locally on that commit. All passed.

• flat_knn_v2_late_payload_preserves_order
• stock_cleanup_sweeps_a_week_old_detached_data_file_lance_13
• explicit_scanner_batch_size_beats_lance_default_batch_size_env
• stock_lance_nested_clone_preserves_inherited_index_base_issue_7840
• index_delta_segment_serves_stale_entries_of_an_updated_row
• rfc_detached_only_detached_commit_never_triggers_auto_cleanup
• changes::write_receipts_identify_exact_commits_and_mutation_noops
• merge_truth_table::add_edge_add_edge_keyed_twin_converges
• export::export_jsonl_round_trips_branch_snapshot
• nearest_large_k_preserves_global_order_through_payload_hydration
• GQT runner: binary_dispatches_both_runner_modes, replay_uses_frozen_case_and_rejects_changed_evidence, contention_action_records_the_retryable_effect_and_keeps_legacy_fail
• GQT cases: nearest_corpus_smaller_than_limit, nearest_sees_rows_inserted_after_index_build, second_merge_of_merged_branch, filtered_nearest_returns_matching_rows_not_postfiltered_topk

I did not re-run the full GQT corpus, the DST suite, or the S3 and Azure jobs. Those checks are green on this commit. The rollback binary in CI is the PR base 0a62bb7b, not current main 9e2425d2. The three commits on main after that base do not change storage. I did not add a GQT case. The changed contracts are not row results. The Rust guards above are the owners.

Inline comment on docs/user/operations/upgrade.md line 122: Storage Upgrade Compatibility on this commit builds the Lance 11 CLI from 0a62bb7b and reads the candidate writes with it. That job is green. This sentence says those writes are not qualified for the old reader. Please keep the ban on mixed fleets. Please also say that the journey qualifies that one predecessor, and that it does not qualify any other old binary or a live mixed fleet.