S5 — build_gaul_lookup.py: pyarrow-native build

[Polichinel]

Epic: #85 · S5 · [UNILATERAL — lowest priority]

Background

scripts/build_gaul_lookup.py builds the GAUL lookup parquet from views-datafactory's source parquets using pandas + pyarrow. It runs offline, outside the delivery path, so it touches nothing interconnected and has zero samples relevance. Pandas-free for completeness only.

Work

• Make the builder pandas-free (pure pyarrow): the join/filter/dtype/sort steps currently done with pandas (build_gaul_lookup.py ~:71-166) reimplemented on arrow tables; keep the embedded provenance metadata that _read_version reads.

Acceptance criteria

• The builder produces a lookup parquet identical to the current one from the same datafactory inputs (table contents + embedded provenance metadata).
• No pandas import in the script.

Parity / validation

Build the lookup both ways from the same inputs; assert identical arrow table and identical provenance metadata. (Downstream tests/test_enrichment.py lookup-integrity assertions must still pass against the rebuilt file.)

Dependencies

Independent; lowest urgency. Keep in sync with S4 and gaul_schema.py (the 9-column contract + rename map).

Files

scripts/build_gaul_lookup.py.

[Polichinel]

Collision with epic #181's S5 (#186) — read before starting this.

Both stories rewrite scripts/build_gaul_lookup.py, and #186 lands first.

This issue's acceptance criterion is "produces a lookup parquet identical to the current one from the same datafactory inputs (table contents + embedded provenance metadata)". #186 deliberately changes the embedded metadata — it adds a flat declared lookup_version key so the provenance stamp can no longer silently degrade to the string "unknown" (register C-60), and rebuilds the committed artifact. Table contents stay identical; metadata does not.

Also stale here: this issue says "keep the embedded provenance metadata that _read_version reads". _read_version moved out of enrichment.py to contract/gaul_lookup.version() in #152, and #186 rewrites it to read one declared key and raise when it is absent.

Re-baseline this issue's parity target against the tree after #186 merges.

[Polichinel]

#186 has now merged (#201) — this issue's parity target needs re-baselining.

The builder's embedded metadata changed as predicted: it now writes a flat, declared lookup_version key, and refuses a build whose ledger yields no digest for the region's declared stamp dataset (stamp_dataset(region) — area-majority for all, <region>_region otherwise, no fallback).

So this issue's "produces a lookup parquet identical to the current one … + embedded provenance metadata" should now read: table contents identical, metadata including the lookup_version key the pandas-free builder must reproduce byte-for-byte.

Also: _provenance now takes a required datasets keyword, and _read_version no longer exists anywhere — the reader is contract/gaul_lookup.version(), which reads one key and raises.