[BUG] Discover validation registry through importlib.resources

[Seth Fitzsimmons (sethfitz)]

Fixes #661

Note

A third alternative for #661, alongside #662 (teach the filesystem walk to read inside a zip) and #663 (replace discovery with a codegen-emitted index). Merging any one resolves the issue.

What

overture.schema.pyspark._registry.REGISTRY comes back empty when overture-schema-pyspark is loaded from a wheel placed on sys.path rather than installed to a real directory, which is what AWS Glue does with --extra-py-files. Every feature type then reports as unregistered.

Why

_registry walked the generated tree with pathlib.Path.rglob. Under zipimport the namespace portion's __path__ entry points inside the archive, pathlib cannot traverse there, and rglob returns no matches without raising — so the empty registry is indistinguishable from the documented "generated tree absent" case.

Fix

Read the tree through importlib.resources.files, which resolves a namespace portion whether that portion is a directory on disk or a member of an archive. This is the API the stdlib provides for exactly this problem, so discovery no longer has to know how the package was loaded: any loader exposing a resource reader works, rather than only the ones the walk was taught about.

files() raises NotADirectoryError on a namespace package with any non-directory portion through Python 3.12, which is why it isn't usable as-is (as #661 notes). CPython fixed it in 3.13 and the importlib_resources backport carries the same fix from 6.2, so the backport is a dependency below 3.13 and evaporates when the floor rises. Glue 4.0 runs Python 3.10 and Glue 5.0 runs Python 3.11, so on Glue the backport is always the code path taken.

Two things this preserves that are worth naming. files() multiplexes every portion of the namespace, so a generated tree assembled from more than one installed distribution is still walked whole — the plural-__path__ behaviour the previous implementation had. And a missing expressions/generated/ subtree still yields an empty registry rather than raising.

Discovery now takes the root package name instead of a list of portion paths. That lets _walk drop its own import_module of the root (files() performs it) and lets the tests point the walk at a synthetic package on sys.path.

Verification

End to end, against a real wheel that is never unpacked — the deployment shape, not a simulation of it:

$ PYTHONPATH=/tmp/wheeltest/overture_schema_pyspark-0.1.1-py3-none-any.whl python -c "..."
registry module loaded from: .../overture_schema_pyspark-0.1.1-py3-none-any.whl/overture/schema/pyspark/_registry.py
registry entries: 15

Control arm, same harness and same wheel shape built from main: registry entries (UNFIXED code): 0. The check distinguishes fixed from unfixed rather than merely reporting a number.

Also confirmed the wheel carries the PEP 420 generated tree (15 modules, zero __init__.py), and make check passes — full suite, mypy, lint, docformat.

On the tests

Three tests are added; one of them is the gate. test_iter_generated_module_names_reads_a_zipimported_tree builds a wheel-shaped zip, puts it on sys.path, and walks the resulting namespace portion — so the portion string under test comes from real import machinery rather than being hand-written, which is the coupling the bug turned on. Verified by isolation: with the new signature but a pathlib walk behind it, that test fails and the other two pass.

The other two — a directory-backed tree and an absent tree — pass under both implementations. They are behaviour-preservation guards for the rewrite, not proofs of the fix.

The zip fixture writes explicit directory entries, because zipimport recognises a namespace portion inside an archive only when the archive carries one, and that is what uv_build emits. Omitting them would test a shape no real wheel has.

Not addressed

The issue reports that the task "erroneously succeeds". overture-validate already exits 1 on an unresolvable feature type, so the silent success is on the consumer side — a caller treating feature_type in REGISTRY as "is this one of ours". Worth a separate issue against data-platform. Switching those jobs from --extra-py-files to --additional-python-modules would also sidestep zipimport entirely and let pip resolve click / overture-schema-system instead of shipping them by hand.

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/666/schema/index.html
🕐 Updated	Aug 17, 2026 21:13 UTC
📝 Commit	f7cc054
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.