Skip to content

feat(build): #90 — build the GAUL lookup arrow-native, and refuse an empty one - #220

Merged
Polichinel merged 1 commit into
developmentfrom
feat/90-pyarrow-native-lookup-build
Aug 4, 2026
Merged

feat(build): #90 — build the GAUL lookup arrow-native, and refuse an empty one#220
Polichinel merged 1 commit into
developmentfrom
feat/90-pyarrow-native-lookup-build

Conversation

@Polichinel

Copy link
Copy Markdown
Collaborator

B3 of the sequenced plan. Closes #90. The builder was the last runtime pandas import in the repository — the sources are parquet, the output is parquet, and pandas was only ever the thing in the middle.

Proven against the real producer, not asserted

Rebuilt land_gaul from the views-datafactory checkout and compared to the committed artifact:

rows 64,742 → 64,742
column order identical
schema types identical, including dictionary<string> on the four name columns
all 10 columns value-identical
adr / region / n_cells / n_dropped_incomplete / lookup_version / source_provenance byte-identical

That is #90's re-baselined acceptance criterion met exactly: contents identical, and the flat declared lookup_version key reproduced byte-for-byte (C-60).

One deliberate difference: the written file no longer carries a pandas schema-metadata blob. It described an index arrow does not have, the only consumer reads the file as arrow (contract.gaul_lookup.load), and reproducing it would have meant keeping knowledge of pandas' metadata format in the one script this change exists to remove pandas from. Said in the docstring rather than left to be discovered.

The committed artifact is not regenerated here. The builder is what #90 asks for; rewriting a delivered artifact is a separate act with no delivery benefit.

C-76 closed — and the guard failed its own first test

build() now refuses a zero-row result instead of writing one and printing cells=0 as though that were an outcome.

The first draft did not survive its mutation test. On an empty table pa.array([True] * 0) infers NULL type, so pc.and_ raised ArrowNotImplementedError in the completeness filter before the zero-row check could speak — the confusing-late-failure C-76 exists to prevent, relocated by one function. The mask is now explicitly pa.bool_(), and the test asserts the refusal a human can act on.

C-75 decided: retire — and deliberately not here

The entry's Owner field required whoever took #90 to make the keep-or-retire call on GaulLookupEnricher, "not to defer a third time."

Retire. Zero production callers (only three test files import it); its "build/verification path" justification is spent now that this path is arrow-native; it holds the package's last pandas reference; and its _gather duplicates the shipping gather in contract/historical.py, which four test files cover independently. C-45 is the precedent and was resolved by deleting.

Not executed in this PR. The retirement touches ten files — the module, its 39 tests, references in two other test files, its CIC, ADR-012, gaul_lookup.py's docstring, the machinery list in test_clone_readiness.py, and the pandas-importer assertion in test_doc_accuracy.py. Mixing that with a builder rewrite is exactly what epic #148's S5 refused to do. It is the next change.

Epic #85 and tracking #93 stay open until it lands — their claim, pandas pushed to the seams, only becomes true when that module is gone.

Verification

373 passed / 40 xfailed / 0 failed
ruff check .  ->  All checks passed!

The fidelity suite's synthetic source is now an arrow table (_load_source returns a table with a gid column; arrow has no index) — same column vocabulary, so the builder's contract with that seam is unchanged.

Register 83 / 15 open / 68 resolved.

…empty one

The lookup builder was the last runtime pandas import in the repository. The sources
are parquet and the output is parquet; pandas was only ever the thing in the middle.

Proven against the real producer, not asserted
Rebuilt land_gaul from the views-datafactory checkout and compared to the committed
artifact:

  rows                      64,742 -> 64,742
  column order              identical
  schema types              identical (including dictionary<string> on the four names)
  ALL 10 COLUMNS            VALUE-IDENTICAL
  adr / region / n_cells / n_dropped_incomplete / lookup_version / source_provenance
                            byte-identical

That is #90's re-baselined acceptance criterion met exactly: contents identical, and
the flat declared `lookup_version` key reproduced byte-for-byte (C-60).

One deliberate difference: the written file no longer carries a `pandas` schema-metadata
blob. It described an index arrow does not have, the only consumer reads the file as
arrow (`contract.gaul_lookup.load`), and reproducing it would have meant keeping
knowledge of pandas' metadata format in the one script this change exists to remove
pandas from. Stated in the docstring rather than left to be discovered.

The committed artifact is NOT regenerated here. The builder is what #90 asks for;
rewriting a delivered artifact is a separate act with no delivery benefit.

C-76 closed, and the guard failed its own first test
`build()` now refuses a zero-row result instead of writing one and printing `cells=0`
as though that were an outcome. The first draft did not survive its mutation test: on
an empty table `pa.array([True] * 0)` infers NULL type, so `pc.and_` raised
ArrowNotImplementedError in the completeness filter BEFORE the zero-row check could
speak -- the confusing-late-failure C-76 exists to prevent, relocated by one function.
The mask is now explicitly typed.

C-75 decided: RETIRE, and not in this PR
The entry's Owner field required whoever took #90 to make the keep-or-retire call on
GaulLookupEnricher "not to defer a third time". Retire: zero production callers, its
"build/verification path" justification is spent now that this path is arrow-native, it
holds the package's last pandas reference, and its _gather duplicates the shipping
gather in contract/historical.py -- which four test files cover independently. C-45 is
the precedent and was resolved by deleting.

Not executed here: the retirement touches ten files including a CIC, ADR-012 and four
test files. Mixing that with a builder rewrite is what epic #148's S5 explicitly refused
to do. It is the next change.

Epic #85 and tracking #93 stay OPEN until it lands -- their claim only becomes true when
that module is gone.

373 passed / 40 xfailed / 0 failed. ruff clean. Register 83/15/68.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Polichinel
Polichinel merged commit 1013787 into development Aug 4, 2026
4 checks passed
@Polichinel
Polichinel deleted the feat/90-pyarrow-native-lookup-build branch August 4, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant