[BUG] Discover validation registry via a generated index module

[Adam Lastowka (Rachmanin0xFF)]

Fixes #661

Note

This is one of two alternative fixes for #661. The other, #662, keeps discovery on the filesystem and teaches the walk to read inside a zip. Merging either one resolves the issue.

What

Same bug as #662: REGISTRY comes back empty on AWS Glue, so every feature type silently reports as unregistered and the task erroneously succeeds 🫠. This happens when overture-schema-pyspark is loaded as a raw .whl on sys.path (Glue's --extra-py-files) instead of installed to a real directory.

Why

_registry.py discovered generated modules by walking expressions/generated/ with pathlib.Path(root_path).rglob("*.py"), which works... only when a real filesystem directory exists! When the package is a raw wheel on sys.path, those modules live inside the zip, and pathlib can't traverse into one.

Fix

We stop discovering by filesystem. make generate-pyspark now emits an _index.py alongside the validation modules that imports each one and exposes them as MODULES, and the registry imports that index. An ordinary import resolves whether the package is a directory or a wheel on sys.path. The generated modules were always importable under zipimport; only enumerating them by walking the tree was broken.

The index is generated from the same specs as the modules. The registry-building policy stays in _registry.py; codegen only decides which modules exist. There's also a test asserts that the index covers exactly the modules on disk, guarding the codegen step.

Approach vs #662

• [BUG] Fix validation registry discovery when loaded via zipimport #662 keeps automatic filesystem discovery and adds pathlib-plus-zipfile walking that has to know every way the package might be loaded.
• This PR moves discovery to an ordinary import of a generated index. The runtime shrinks to "import the index, read MODULES", zipimport-safe for free, at the cost of a small addition to the codegen (one extra emitted module).

An earlier revision of this PR used a hand-maintained entry-point table; that was replaced with the codegen-emitted index so nothing has to be kept in sync by hand.

Verification

• Built a wheel, added it to sys.path without extracting it (matching Glue's --extra-py-files): all 15 generated modules resolve; previously len(REGISTRY) == 0. Confirmed _index.py ships in the wheel.
• make check passes (mypy, lint, docformat, full suite across the codegen and pyspark packages).
• A present-but-broken module raises instead of being silently skipped; an absent generated tree yields a clean empty registry.

Yet-untested on Glue!

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/663/schema/index.html
🕐 Updated	Aug 14, 2026 07:15 UTC
📝 Commit	ad77f79
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.

[Copilot]

Pull request overview

Updates the PySpark validation registry discovery mechanism so it works when overture-schema-pyspark is loaded from a wheel on sys.path (zipimport, e.g., AWS Glue), by switching from filesystem walking to importlib.metadata entry-point discovery.

Changes:

• Replace filesystem-based generated-module discovery in _registry.py with discovery via the overture.pyspark_validations entry-point group.
• Add/adjust tests to (a) assert the registry populates when the generated tree exists and (b) ensure the declared entry-point table matches the generated modules on disk.
• Add overture.pyspark_validations entry points to pyproject.toml and document the behavior change in the changelog.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
packages/overture-schema-pyspark/src/overture/schema/pyspark/_registry.py	Switch registry population from filesystem walking to entry-point-driven discovery.
packages/overture-schema-pyspark/tests/test_registry.py	Update/extend tests to validate entry-point-based discovery and prevent drift vs generated tree.
packages/overture-schema-pyspark/pyproject.toml	Declare one entry point per generated validation module (overture.pyspark_validations).
packages/overture-schema-pyspark/changelog.d/661.bugfix.md	Document the fix for zipimport / AWS Glue registry discovery.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

packages/overture-schema-pyspark/src/overture/schema/pyspark/_registry.py:60

• Modules that import successfully but are missing ENTRY_POINT or MODEL_VALIDATION are currently skipped silently. That can cause validations to be dropped without any signal, which is especially hard to diagnose in production (and contradicts the module docstring’s “never dropped without notice” claim).

        if entry_point is None or validation is None:
            continue

packages/overture-schema-pyspark/tests/test_registry.py:51

• This PR’s core regression risk is zipimport behavior (raw wheel on sys.path), but the updated tests only exercise the on-disk case. Consider adding a regression test that loads the generated expressions/generated tree from a zip on sys.path (without an extracted directory) and asserts REGISTRY populates, so the original Glue failure mode is covered automatically.

def test_registry_discovers_generated_models() -> None:
    """The registry finds generated modules through the index."""
    if not _generated_module_names():
        pytest.skip("generated tree not present; run `make generate-pyspark`")

    generated_entries = [
        key for key in REGISTRY if ":" in key and key.startswith("overture.schema.")
    ]
    assert generated_entries, "registry found no generated feature modules"