Repository navigation
Conversation
Index maintenance rebuilds every lagging index from the whole table and never refreshes full-text coverage. The RFC adds segments over fragments whose row IDs no segment of the index has indexed, which keeps answers equal to a full build's under Lance 13's stable-row-ID prefilter, merges segments under a bounded policy, and folds moved rows by a filtered BTREE merge or a whole rebuild. Full text is maintained like the other kinds; the index build contract does not change.
…l-index-reconciliation
WhyA routine append should not scan every row of a table to refresh an index. On current WhatThe RFC adds one rule. An index gains a segment only over fragments that hold row ids no segment of that index has indexed. I checked the rule against Lance 13.0.0. It matches the pinned prefilter. The decision record still has three gaps. Close them before acceptance. How the rule holds on the pinLance 13.0.0 builds the scalar and full-text mask from the union of the segment bitmaps (
The BTREE merge calls The row-id lineage page still says secondary indexes reference row addresses and that stable row ids in indexes are in progress. The 13.0.0 prefilter cited above works in stable row ids. The guards pin the code. That citation is right.
Blocking
Optional
Tradeoff and liabilityThe append path removes a cost that grows with table size. It adds a fresh test, a segment budget, and a second way to certify full text. Query work grows with the segment count until a merge. The report has to name Five more index kinds would each add a merge row or a rebuild row. One decision function and one rule keep that growth small. The 13.0.0 fence has to stay easy to delete. A permanent fresh test, after Lance search already applies segment ownership, would be a second copy of a fixed upstream rule. Correctness stays ahead of cost. The mixed fresh-and-moved row is a correctness gap in the table. The upstream fixes do not weaken the 13.0.0 rule. They change how long that rule should live. Validation
Recommendation: revise the three blocking points, then accept the draft. This is a comment review. Inline comments (also part of this review):
"Refused by maintenance" (line 96) is a different outcome from the code. A search calls
When any uncovered fragment is moved, publish one BTREE segment from a filtered merge whose new input covers every uncovered fragment. Full text and vector already rebuild, which covers both. Include the identity and endpoint BTREEs in this function.
Keep the append extension as the durable design. Mark the fresh test and the full-text whole rebuild as the 13.0.0 fence. Remove that fence in the bump that contains #9785 and #9872. That is the same bump that makes
|
…l-index-reconciliation
State the full-text contract change, add the mixed fresh-and-moved BTREE case, name the identity and endpoint BTREEs, record lance#9785 and lance#9872 and mark the moved-row half as the Lance 13.0.0 fence, state the moved-row costs, and block on PR ModernRelay#1150.
WhyA routine append should not read every row to refresh an index. This RFC adds a segment only over fragments whose row ids no segment of that index has indexed. Under the Lance 13.0.0 stable-row-id prefilter, that rule keeps each answer equal to a full build. WhatCommit HowI read the RFC at ImpactThe append rule can stand. The new compaction sentence cannot. An implementation that follows it rebuilds every index after a compaction whose bitmaps already cover the new fragments. Prior findings
The six optional notes are resolved in the text. Blocking
On a stable-row-id table, Lance applies After that apply, an index that covered the rewritten fragments still covers the current fragments. The missing-fragment arm remains the right fallback when a bitmap still names a fragment the manifest has dropped, or when a sequence cannot be read. OmniGraph compaction rewrites the bitmap, so that arm does not run. State the covered result: index work after compaction is only the fragments the rewritten bitmaps still omit. Today's optimize pays no index rebuild for that covered result. The new sentence calls the moved-row action the cost today's optimize already pays. An implementation that follows the sentence scans every current row and reads every index entry after a compaction that now does no index read. The owner of this fact is Lance's Rewrite apply. Optional
PR #1171 discovers change-feed rows by fragment identity. It does not share this decision. Its compaction note, that row ids and content are preserved, matches the bitmap rewrite above. Tradeoff and liabilityCorrectness stays ahead of cost. The fresh-segment rule is the durable part. It matches the 13.0.0 prefilter ( The compaction sentence adds liability. Five more maintenance drafts that each restate compaction coverage will drift. #1168 F14 and this paragraph already disagree. One sentence that points at the Rewrite apply removes that drift. The fresh test, the segment budget, and the second full-text certificate remain the necessary new state. Validation
No RecommendationRevise before acceptance. Correct the compaction coverage sentence and the test bullet that depends on it. The four earlier blocking findings can stand. Inline on Blocking. On a stable-row-id table this Rewrite does the opposite of this sentence.
After that apply, a fully covered index still covers the current fragments. Keep the missing-fragment arm for a bitmap that still names a dropped fragment. Say that OmniGraph compaction rewrites the bitmap, so index work after it is only the fragments those bitmaps still omit. Today's optimize pays no index rebuild for the covered result. This sentence calls the moved-row action that cost. |
Lance's stable-row-ID Rewrite apply moves each row-ID index's bitmap to the rewritten fragments, so index work after a compaction is only the fragments the bitmaps still omit; the previous text had every compaction take the moved-row action. State the rule's assumption that coverage leaves a fragment only with the fragment, the two Lance applies that break it, and why graph writers do neither. Name lance#9481 and lance#10041 in the fence, own the segment policy, and correct the full-text merge line reference.
WhyA routine append should not read every row to refresh an index. This RFC adds a segment only over fragments whose row ids no segment of that index has indexed. Commit WhatThe interdiff against HowI read the RFC at ImpactThe 13.0.0 rule can stand. An implementer who follows the fence will wait for lance#9785 to handle address-domain BTREE compaction. Lance 14.0.0 already changed that compaction, and current Prior findings
Blocking
On that tag,
The published btree page now says the same split. The Put the remap, or an explicit decision to keep the The same builder is today's Optional
Tradeoff and liabilityCorrectness stays ahead of cost. The fresh-segment rule matches the 13.0.0 prefilter. The corrected compaction paragraph follows the bitmap Lance writes, which is the coverage The fence adds liability. It joins two releases that Lance already shipped apart. The next bump to 14.0.0 fails compaction once any BTREE segment is version 1, while the text still describes a later uncovered-fragment repair. Five more maintenance drafts that copy "the #9785 bump also remaps BTREE" will miss that refusal. One sentence per release keeps the obligation where the code changes. Query fan-out still grows with the segment count until a merge. The report has to name Validation
No review and no review comment on this head at post time. The two earlier issue comments are the reviews on RecommendationRevise before acceptance. Split the Lance 14.0.0 compaction obligation from the later stale-entry bump, and name the Inline on Blocking. Lance 14.0.0 already contains this change. The bump that contains lance#9785 does not have to be the one that introduces it.
On
State the remap, or the decision to keep the refusal, on the bump to Lance 14.0.0. Keep the moved-row fence for the later bump that contains #9785 and #9872. The new guard should expect a newly built BTREE to lose coverage, and full text and vector to keep it. Inline on Optional. |
docs/rfcs/2026-10-25-incremental-index-reconciliation.md, the second step of #840. Index maintenance rebuilds every lagging index from the whole table, so a few appended rows cost a pass over every row on each run, and full-text coverage is never refreshed by maintenance at all; PR #1150 putsindex buildon that same mechanism.The RFC makes the work proportional to what changed under one rule: an index gains a segment only over fragments whose row IDs no segment of that index has indexed. Lance 13's prefilter on a stable-row-ID dataset admits a row only through its current live copy, so under the rule every answer equals a full build's, for any number of segments and any merge of them. Fresh fragments get a segment of their own; rows moved by updates are folded in by Lance's filtered BTREE merge or a whole rebuild (full text, vector); a bounded merge policy caps segments per index; full text is maintained like the other kinds, with certificates on subset builds and on merges of certified segments. Compaction needs no index work of its own: Lance's stable-row-ID
Rewriteapply moves each index's bitmap to the rewritten fragments. For BTREE and vector indexes theindex buildandoptimizecontract does not change; a certified full-text index is maintained instead of reported pending. The storage format does not change.Every Lance fact the design rests on is cited to the 13.0.0 source (prefilter, subset builder, segment merges,
CreateIndexapply, row-ID sequences, the global BM25 scorer), and the evidence section lists the surface guards that pin each one.For review: the decision to challenge is the soundness argument for the rule and its conservative fresh test (smallest row ID of a fragment above the largest of every covered fragment, else treated as moved); the argument assumes coverage leaves a fragment only when the fragment leaves the manifest. Lance 13 breaks that on in-place column rewrites and on compaction of fragments with newer data overlays; graph writers produce neither, and the RFC says what a writer that does must change first. The row-ID lineage page says indexes reference row addresses and that stable row IDs in indexes are in progress, while the cited 13.0.0 code works in stable row IDs, so the guards pin the code. Lance's
CreateIndexapply checks no coverage overlap, so the reconciler must. Open: the merge policy's constants, and whether optimize may defer full-text and vector rebuilds for a small moved tail.