bp-153 — the rebuild, the frequency gauges, the probe re-home, compaction (D6/D7/§3/§6) - #53
Open
ascalva wants to merge 9 commits into
Open
bp-153 — the rebuild, the frequency gauges, the probe re-home, compaction (D6/D7/§3/§6)#53ascalva wants to merge 9 commits into
ascalva wants to merge 9 commits into
Conversation
Item 5's whole target is `ops/lifecycle/launcher.py` (`_code_backfill_incomplete` at :374-393, call site :551) and Item 7 needs its Launcher method and handler registration there, but the file was never in write_scope. Item 7's verb must ENQUEUE, so the job kind + checkpointing handler go beside their two siblings in `scheduler/code_sync.py` rather than into a new module that splits one lane across two homes. Both added as bare globs, and the omission recorded in §5 for the reviewer. Third instance of this shape this wave — the note in §5 suggests deriving write_scope from the items' cited file:lines rather than restating it. No status field touched.
…ction path §3 makes physical maintenance part of the store's semantics, and no compaction path existed in `vectorstore.py` (the note verified it). Four additions, each arriving with the call that needs it: * `project(columns, where=)` — a projected, predicate-pushed read. `vector` is 2560 floats per row, so a scan that does not name it costs a fraction of one that does; the rebuild's baseline reads id/layer/text over the whole code lane and must never pay for geometry it does not read. * `atom_row_count()` — |V| as a server-side `count_rows`, so the standing gauge can run on a cadence instead of materializing every vector. * `supersede_legacy_code_rows()` — keep-and-link (D2) pointed at the retired ROW MODEL: the rebuild lands the atom plane into the same table as the rows it replaces, and without this both answer the default current-view search. Nothing is deleted, |V| cannot fall, D5's "purge is the ONE removal" untouched. * `compact()` + `CompactionReport` — compact fragments, drop old versions, and carry BOTH the before and after row counts so "semantically invisible" can be asserted without asserting it vacuously. §3 Q5 resolves to its in-scope case, not the finding: verified against the INSTALLED lancedb 0.33.0 (the bp-103 rule — read the package, not the docs), the capability is there. The shim declares `optimize` rather than the `compact_files` + `cleanup_old_versions` pair the API also carries, found by running it: both of those are deprecated as of 0.21.0 and route through `Table.to_lance()`, which raises ImportError without the optional `pylance` package. `optimize` does both halves on the supported path with no new dependency — measured 7 -> 1 versions, row count unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-avoided D6's standing gauges (Item 4). `n_doc`/`n_occ` already existed per atom; what was missing is the batch and aggregate side that makes them observable forever rather than investigable occasionally: * `n_doc_counts` / `rank_frequency` — every atom's document frequency in ONE GROUP BY, and the rank-frequency histogram of LIFETIME n_doc as plain data. Zipf conformance is a falsifiable corpus property and must be CHECKED, not assumed (T2/T4), so the caller gets the numbers rather than a plot. * `occupied_atoms` / `occupancy_count` / `lane_gauges` — the per-lane counts, taken at one cut from one scan so a |M| from after a landing can never be divided by a |V| from before it. * `frequency_gauges(vectors, memberships)` -> `FrequencyGauges` — three aggregate queries plus one server-side row count. No vector crosses into Python, which is what makes it registrable on a cadence. `dedup_factor` IS the D7 falsifier kept observable (the S5 amendment): it fails its keep by sitting at ~1.0 after a rebuild, which would mean the model bought nothing and D7's economics are false. Tests, each with its precondition asserted first: * n_doc vs n_occ must DIFFER, and the only input that can tell them apart is a duplicate L0b window pair — so the pair is asserted to exist as TWO rows with distinct chunk_index inside ONE path before the two readings are compared, and a non-repeated atom is used as the control that makes the inequality mean something (2 vs 1 lifetime 6 vs 1). * the dedup factor gets an explicit CONTROL store where every landing is a fresh atom and the gauge reads ~1.0 — otherwise "dedup > 1" establishes nothing. * the histogram asserts the fixture's distribution is not flat before reading a ranking off it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one deliberate migration into the atom+membership model, as four phases in strict blast-radius order — each is the next one's precondition. `measure_baseline` (Item 1, READ-ONLY). The economics were measured on a July cut and the corpus grows, so the constant is not portable and the RATIO is. Re-derived against the live ledger + live store: 1,663 versions, 52,200 Sigma per-version chunks vs 22,897 distinct atoms = 2.280x, -2.6% from the design's 2.34x — inside the ~10% band, so Item 3's economics stand and no finding is owed. Per lane L0a 2.55x / L0b 2.06x / L1 1.96x. Carry-forward seed 15,186 atoms (66.3%). Nothing was written: store 33,861 rows before and after. `capture_slice` (Item 2, step 0). The shipped `capture_commit_diffs` has never successfully run — the one attempt died in TimeoutError (job 300240). This wraps it in a clock without changing what it does. R6's falsifier is "a slice exceeding its budget WITHOUT leaving a checkpoint": there is no such window here, because the durable progress IS the `_commit_diffs_captured` marker table and the budget is only ever checked between commits. `seed_carry_forward` (Item 3). Bulk embed-reuse by canonical re-hash: a pre-D1 row whose header-free body hashes to a wanted atom becomes an atom-keyed row carrying the SAME vector. Zero embedder calls. The header is verified against the row's own `source_path` before anything is stripped — a wrong strip is silent identity corruption, and this repo has files whose first line is a comment. L1 is excluded on purpose: its windows were cut over header-bearing prose and do not survive D0's pin, so L1 re-embeds. The embedder pin is a REFUSAL, not a warning: a contradicting dimension seeds nothing at all, because a partial seed is the geometry-mixing corruption the pin exists to prevent. `rebuild_slice` + `rebuild_step` (Item 3). Every ledger version becomes a fiber, landed through the D2 write path with its HEAD blob passed explicitly. Resumability is a property of the WALK, not of the token: landing a version twice writes no row and embeds nothing, so the token only saves re-derivation. The tests demonstrate exactly that — a run is killed mid-slice, resumed, and compared against an independent single-shot rebuild for identical |V|, |M| and fibers chunk-for-chunk; then re-run from scratch with the token thrown away, and with an unreadable token. `retire_legacy_rows` — NOT enumerated by the plan and reported separately for that reason. The rebuild lands the atom plane beside the duplicated rows it replaces and both would answer the default search. It is a SUPERSESSION (keep-and-link, D2), never a removal: nothing deleted, |V| cannot fall, D5 untouched, reversible by the same update in the other direction. §8(g) is asserted as an EQUALITY rather than an inequality: the standing |M|/|V| gauge must reproduce the baseline's ratio exactly, because |M| IS Sigma per-version chunks and |V| IS the distinct atom count — the same number reached from two independent directions, derived-and-counted before the run and stored-and-queried after it. The fixture's sharing (across files AND across versions) is asserted first, since at zero sharing the whole claim is vacuous. One defect found by the acceptance test and fixed: `pending_commits` called `_ensure_schema` before reading, which is a CREATE TABLE — it raises on a `mode=ro` connection and silently mutates the file on a read-write one. The dry-run reads the LIVE ledger, where those tables have never existed, so this was the ordinary path. It now answers the absent-table case instead of creating it, with a ratchet that opens the ledger read-only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`supersession_chains` documented a chain as "the ordered DISTINCT sequence of its blobs". The code (`:155`) collapses only ADJACENT repeats, so the sentence described a different function — and it is the exact distinction the design's F4 dispute rests on, which means the prose was contradicting the argument that cites it as evidence. A revert threads [A, B, A] — three runs, two edges — where a distinct collapse would thread [A, B] and erase the revert entirely. Verified against the real ledger rather than asserted: the first successful `commit_diffs` capture in the project's history (1,377 commits, bp-153 Item 2) shows `ops/lifecycle/launcher.py` re-occupying blob e09f038 at NON-adjacent run positions 10 and 12 — a re-occupancy a distinct collapse would have lost. Two tests, because a revert is the ONLY input that tells the two readings apart and the existing `repo` fixture's history is strictly increasing (which is why this drifted unnoticed): a dedicated revert fixture that computes BOTH readings side by side and asserts they differ, and a ratchet on the docstring that asserts the contradicting phrase is ABSENT — gaining the word "adjacent" while keeping the wrong claim would otherwise still pass. Issue #28's defect class, caught where it was load-bearing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…probe Two changes, bundled because both live in `ops/lifecycle/launcher.py`. THE PROBE RE-HOME (Item 5, the note's §6 re-home (1)). Closes #39. `_code_backfill_incomplete` counted distinct `(source_path, digest)` pairs over the code lane. bp-152 shed both columns from atom rows, so against a rebuilt store every atom row collapses to the single tuple ('', '') and the probe reads 1 < 1,663 on every daemon start — enqueueing a backfill forever. That is finding-0166's named falsifier returning through a different door, and bp-152 shipped the shed knowing this re-home was bp-153's. The store side now reads `memberships.fibers()`: a version IS its fiber, so this is the same number at a sturdier home, and the honest form of the F6 re-home (an endpoint is resolvable iff its fiber is non-empty). ⚑ Only the DATA SOURCE moved — cadence, call site, `ingestion.code.enabled` gate and enqueued kind all stand, and a second test pins that, because a trigger-level change is out of design. The test builds a genuinely REBUILT store and computes the OLD reading beside the new one, asserting the old one WOULD have looped. Without that counterfactual the assertion is just "the probe says complete", which the un-re-homed code could also produce on some other input. It also asserts the probe still says INCOMPLETE when a version really is missing — becoming a constant False is the other way to stop the loop, and it is useless. THE VERB (Item 7). `palace code-rebuild` — listed in USAGE, in the module docstring, and dispatched in `main()`; `CODE_REBUILD_KIND` + a checkpointing handler registered unconditionally in `build_components` beside its two siblings. Wiring is the deliverable: a method nothing routes to is not a verb, so the tests read the dispatch out of the SOURCE and assert the enqueued kind has a handler. This is the first handler to use the queue's checkpoint/resume protocol, and the one it was built for: `checkpoint` clears the lease, so a yielded row reads as waiting rather than orphaned and the next claim stamps a fresh PER-BATCH deadline — the shape §2.10 requires. It ENQUEUES; it never stops the daemon, and a test reads the method's source for the calls it must not make. `deploy` remains the separate owner-in-loop gate, untouched. `--dry-run` runs Item 1's read-only pass in-process and writes nothing, asserted against a real seeded ledger with an empty queue afterwards. It passes `repo=self.repo_root` rather than taking `build_code_corpus_sync`'s default, which resolves the repo from the CWD's git toplevel — the measurement would otherwise describe whichever checkout the command was typed in. References #18 (the code_sync TimeoutError wedge): the slicing is what bounds that failure mode. Left OPEN — it closes when the live capture succeeds post-deploy, not when the machinery lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l maintenance The firewall is a row PREFILTER (`provenance IN (...)` with prefilter=True), so the one thing that could quietly break it is a physical rewrite that drops or rewrites the column it filters on. Compaction and legacy-row retirement are exactly that kind of rewrite, so the claim is checked rather than assumed. Degenerate input named and defended against: an empty result satisfies "no code leaked" without testing anything. So the notes are asserted RETRIEVABLE before and after (identical hit ids, not merely "still no code"), the code lane is asserted still reachable through its own explicit provenance set, the dataset version count is asserted to have actually fallen, and every row is asserted to still carry a non-empty provenance — a firewall that held by deleting the corpus is not a firewall. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two errors this branch introduced in its own test file: the `ledger` fixture is a generator and needs `Iterator[...]` rather than the yielded type, and the resume token is `str | None` but was inferred `str` from the `is not None` assert above its loop. Baseline verified back at 69 errors in 20 files (570 checked) — unmoved from the pin, so this branch contributes zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The journal carries the measured Item 1 numbers (2.280x over 1,663 versions at the live cut, -2.6% from the design's 2.34x — inside the band, so the falsifier did not fire), the first successful commit_diffs capture in the project's history (1,377 commits, 4 budgeted slices, re-run captured 0), the gate verbatim with both diff-innocence checks, and Item 3's live re-entry condition stated explicitly rather than left as an incomplete item. The deskcheck goes into `docs/tracks/code-ingest.md`'s `backlog_deskcheck`, NOT into `docs/DESKCHECK-QUEUE.md`: that file is generated by `scripts/board.py` and says so in its first line. Running the generator was tried and reverted — it produced 34 rows of unrelated board catch-up (bp-128..bp-150, bp-154, the erratum-relation track) that belong to a /triage sweep, not to this PR. The drift is recorded in the journal's open questions so it is not lost. Scoped deliberately to the ARC (bp-151 -> bp-155 -> bp-152 -> bp-153) rather than flipping the track's phase: the rest of code-ingest (bp-095, the seed run, integrator densification) is still work-owed, and a track-level flip would over-claim. The arc is READY TO DESKCHECK, not done — its closing act is the live rebuild, which is the owner's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last plan of the vector-membership family (bp-151 → bp-155 → bp-152 → bp-153). It builds the
one deliberate migration into the atom+membership model — sliced, checkpointed, resumable — and
leaves standing gauges that keep the dedup factor observable forever.
closes #39
What
measure_baseline— the read-only pass that re-derives D7's economics at the CURRENT cut. Run for real against the live store + ledger.capture_slice— the first successfulcommit_diffscapture in the project's history, proven on a copy.seed_carry_forward+rebuild_slice+rebuild_step— the four-phase walk (capture → seed → land → compact). Machinery complete; live run owner-gated.n_doc/n_occin batch, the rank-frequency histogram, `:139docstring correction. This is #39.VectorStore.compact— the §3 path that did not exist, plus the widened shim Protocol.palace code-rebuild(+--dry-run), the kind, and the checkpointing handler.Why
bp-152 shed
source_path/digestfrom code atom rows and left two live consequences this planowes: the corpus is still in the duplicated row model, and the daemon's incompleteness probe reads a
column that no longer exists. Item 5 is the urgent half — against a rebuilt store the old probe
collapses every atom row to the single tuple
('',''), reads1 < 1,663, and enqueues a backfillon every daemon start, forever. That is finding-0166's named falsifier returning through a
different door, which is why #39 has been holding the deploy.
Verification
Item 1 — measured against the live store and ledger, read-only
−2.6% is inside the ~10% band, so the §8(g) falsifier did NOT fire and Item 3 proceeded on the
blessed economics. Per lane: L0a 2.55× (2.54×), L0b 2.06× (2.05×), L1 1.96× (2.42× — filed as
#51). The seed is 66.3% of the atom set.
Embedder identity: MATCHES.
stored_dim=2560= the liveEmbeddingConfig.dim;ledger_mismatched=0. Themodelis not recorded on a pre-D1 row — structurally, the Arrow schemais shared with the prose lane and has no embedder column (the gap bp-152's
atomsledger closesgoing forward) — so the report says
unrecorded_rows=15186rather than assuming. Closed by evidenceoutside the store:
config/defaults.toml'smodel = "qwen3-embedding:4b"has exactly one commitin its whole history (
7502109, 2026-06-25), predating the code lane. No model change can havehappened under those rows.
Read-only verified, not asserted: stub embedder that raises if called; live store 33,861 rows
before and after;
|M|stayed 0; nomemberships.sqlitecreated beside the live vault catalog.Item 2 — the first successful capture
Confirmed at HEAD first: the live snapshots db has zero
commit_diffs/_commit_diffs_capturedtables (§3 Q1 / panel S2 hold — shipped since bp-099, never run). Proven on a copy carrying the
snapshotstable verbatim in live rowid order; the live 5 GB db was openedmode=roonly, becausethe daemon owns that file and single-writer is the invariant.
commit_diffsrows; re-run captured 0 (idempotence)progress is
_commit_diffs_captured, and the budget is only checked between commits, each itsown transaction. The 300240
TimeoutErrormode is bounded, not hoped away.ops/lifecycle/launcher.pyre-occupies blobe09f038eat non-adjacent run positions 10 and 12. Adjacent-collapse kept it; adistinct-collapse would have erased it — the receipt the F4 dispute wanted.
Item 3 — resumability, proven not to depend on the token
A run is killed mid-slice (zero budget forces a yield with work already landed, asserted, so it
is a real resume), resumed from its token to completion, and compared against an independent
single-shot rebuild of the same ledger: identical
|V|, identical|M|, identical fiberschunk-for-chunk. Then the token is discarded entirely and the walk re-run from scratch — 0 new
occupancies, 0 embeds — and once more with an unreadable token. Fiber equality holds because
derivation is pure, so the token buys time and nothing else.
§8(g) is asserted as an EQUALITY, not the inequality the design warns about.
|atoms| ≤ Σ chunksholds vacuously at zero savings; instead the standing
|M|/|V|gauge must reproduce the baseline'sratio exactly, because
|M|is Σ per-version chunks and|V|is the distinct atom count —the same number from two independent directions. The fixture's sharing (across files and across
versions) is asserted before any ratio is read.
Gate (final tree)
uv run ruff check .uv run mypy core agents eval ops scheduler scriptsuv run mypy(argless)uv run python -m ops.type_gateuv run pytest -qThe 6 = the five known-red (
test_dream_v2_live;test_worktree_enforcement×3 —test_a_deny_cross_worktree,test_c_unsafe_direction_narrow_not_loosened,test_d_no_pointer_is_no_plan_not_main_fallback;test_core_imports_nothing_outside_core) plusthe known
test_scheduler_liveflake. No new failures.Diff-innocence, twice, because counts drift both ways:
and a
str | Noneassignment) and are fixed in23e1693→ back to 69;test_scheduler_livebuilds its ownSupervisorwithhandlers={"ping": handler}and never callsbuild_components, so the handler I registered cannot reach it.Things the reviewer should look at deliberately
retire_legacy_rowsis not enumerated by the plan. The rebuild lands the atom plane into thesame table as the duplicated rows it replaces, so both would answer the default current-view search —
the dedup would be bought and then not served. It is a supersession (keep-and-link, D2), never a
removal: nothing deleted,
|V|cannot fall, D5's "purge is the ONE removal" untouched, andreversible by the same update in the other direction. Reported separately for exactly this reason.
§3 Q5 resolved to its in-scope case, but the obvious path was wrong. Widening the
VectorTableProtocol was ordinary work — the capability exists in the installed lancedb 0.33.0 (verified
against the package, not the docs, per bp-103). But
compact_files+cleanup_old_versionsaredeprecated as of 0.21.0 and route through
Table.to_lance(), which raisesImportErrorwithoutthe optional
pylancepackage. Found by running it. The shim declaresoptimizeinstead: onecall, both halves, supported path, no new dependency. Measured 7 → 1 versions, rows unchanged.
A read that was writing.
pending_commitscalled_ensure_schemabefore reading — aCREATE TABLEthat raises on amode=roconnection and silently mutates the file otherwise. Thedry-run reads the live ledger, where those tables have never existed, so this was the ordinary
path rather than a corner. Caught by the acceptance test, fixed, and ratcheted.
Write-scope amendment (
dbf6ae2). Addedops/lifecycle/launcher.py— Item 5's entire target(
_code_backfill_incompleteat:374-393, call site:551) plus Item 7's method and handlerregistration — and
scheduler/code_sync.py, where the job kind belongs beside its two siblings.Bare globs, no inline comments, status field untouched. Third instance this wave → #52.
Scope addition: the deskcheck filing.
docs/DESKCHECK-QUEUE.mdis generated byscripts/board.pyand says so in its first line, so the statement went intodocs/tracks/code-ingest.md'sbacklog_deskcheck— the view's actual source. Running the generatorwas tried and reverted: it produced 34 rows of unrelated board catch-up (bp-128…bp-150, bp-154,
the erratum-relation track) that belong to a
/triagesweep, not to this PR. That drift is recordedin the journal so it is not lost. Scoped to the arc, not the track: the rest of code-ingest
(bp-095, the seed run, integrator densification) is still work-owed and a track-level flip would
over-claim.
Not done here, on purpose
The live rebuild has not run, and could not have. The daemon runs pre-bp-152 code, so the
rebuild physically cannot execute until the deploy this PR unblocks. Re-entry, recorded in the
journal: wedge drained (observed depth 0 at build); the live rebuild runs post-deploy via
palace code-rebuild— an owner op. Order: merge →deploy→code-rebuild --dry-runtore-confirm the ratio at the then-current cut →
code-rebuildto enqueue. Expected at today's cut:~22,897 atoms landed, ~15,186 free by carry-forward, so ~7,700 real embeds against the
duplicated model's 52,200.
References #18 (the
code_syncTimeoutError wedge) — the slicing addresses its failure mode andItem 2 demonstrates the bound. Left open: it closes when the live capture succeeds, not when
the machinery lands.
Issues filed
real ledger: 1,663 versions, 1,663 chain members, an empty difference both ways. The reasoning
assumed chains follow HEAD's first-parent line; they do not.
the aggregate holds. Exactly the shape Amendment A1.4 warns about.
write_scopeomits files the plan's own items cite byfile:line; derive it instead ofrestating it.
Non-goals held: no notes-lane migration (PD-2/R5), no IDF ranking (PD-4), no materialized edge table
(PD-3), no L1 re-slotting (PD-5), no embedder change, no ANN tuning, no design-note edits, no
deploy, and the old duplicated backfill was never run against live data.🤖 Generated with Claude Code
https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U