Per-format install extras: nlr-gat[sienna|plexos|reeds|plots|...] - #29
Merged
Merged
Conversation
Closes #23. Also closes #11 (removes EGRET from code and docs, not just the docs page as originally scoped — the handler had zero test coverage or other consumers, confirmed before deletion). ## The packaging half Trims base dependencies to the truly format-agnostic floor (numpy, pandas, pydantic, click, loguru, pyyaml, orjson) and moves everything format-specific into extras: - `sienna` — h5py, polars, geopandas - `plexos` — h5py + duckdb + plexos2duckdb, bundled together "for now" per the H5PLEXOS.jl deprecation notice (README); `plexos-duckdb` kept as a backward-compat alias for the plexos2duckdb-only slice, since it's already published under that name on PyPI - `reeds` — declared empty; ReEDsScenario needs nothing beyond base - `plots` — matplotlib + PySide6 (gat.quickplots), independent of source format - `server`/`client` — gained duckdb/pyarrow, their actual real deps, previously hidden inside the old blanket base dependencies - `all` — convenience bundle of everything above - `dev` — now pulls in every format + plots extra, fixing the root cause of #23 (the dual-backend PLEXOS suite silently never ran because `pip install -e ".[dev,reports]"` never installed plexos-duckdb) - removed `tables` entirely — confirmed dead, never imported anywhere in src/gat ## The lazy-import half (required for the extras to mean anything) Declaring extras is meaningless if the import graph doesn't respect them — `from gat.scenariohandlers import PlexosScenario` previously eagerly imported every handler (Sienna, PLEXOS, EGRET, ReEDS, ...) regardless of which one was requested, so a `[reeds]`-only install would ImportError the moment anything touched gat.scenariohandlers at all. Fixed at every layer that had the same eager-`import *`/blanket- top-level-import shape: - `gat/scenariohandlers/__init__.py`, `gat/simulations/__init__.py`, `gat/systems/__init__.py` — converted to PEP 562 module-level `__getattr__` (mirroring the pattern gat/__init__.py already used), with `__all__` so `from package import *` and `from package import Name` both still resolve lazily and correctly. - `gat/datahelpers/__init__.py` — gutted its blanket `from .X import *` re-exports entirely. Every real caller already imports specific submodules directly (gat.datahelpers.sienna_system, .parsers, .h5Parsers, ...); only one example script relied on the flat re-export, fixed to the fully-qualified path. - `gat/datahelpers/parsers.py` — h5py import moved from module level into the three functions that actually touch h5py.File(...). This module sits on BaseScenario's mandatory import path (`from gat.datahelpers.parsers import *`), so every handler — including ReEDS and EGRET, which never touch h5 — was paying for h5py. - `gat/colors.py` — random_color() and standard_color_dict relocated here from gat.quickplots (dependency-free; the old locations re-export them for backward compatibility). Both were on the mandatory path too: standard_color_dict/random_color are used by TechnologyMapping in gat/models/scenario.py, which BaseScenario imports unconditionally — so constructing ANY scenario handler transitively imported matplotlib + triggered quickplots' macOS Qt-backend-detection, just to pick a fallback hex color. - Two dead, shadowed `_find_solution_files` method definitions found and removed along the way (base.py, reeds.py each defined the method twice in the same class — Python silently keeps only the later one; harmless but confusing, cleaned up while touching this code for the h5py fix). ## Verification - sys.modules probing confirms real isolation: touching PlexosScenario only imports h5py; SiennaScenario only h5py+geopandas+polars; neither touches the other's deps or matplotlib/duckdb inappropriately. - Built the wheel and installed bare/sienna/plexos/reeds/plots/all/dev into six independent, fresh venvs. Confirmed each extra's handler imports cleanly, unrelated handlers fail with a clear ModuleNotFoundError (not a confusing crash), and dev's self-referential extra resolves every format + plots correctly. - Full local suite: 236 passed, 58 skipped, 19 failed — the failures are exactly the #27 class (legacy h5 regression baselines vs. an unavailable original fixture), already triaged and dismissed by the maintainer as expected given the H5PLEXOS.jl deprecation timeline. Zero new failures from this change. ## EGRET removal (closes #11, expanded from docs-only) Deleted gat/scenariohandlers/egret.py and its docs page. Confirmed before deletion: egret_map_simple (its only real dependency, in config_maps.py) had zero other consumers, EGRETScenario was referenced nowhere outside its own file and the __init__ import, and no test exercised it at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Autodoc imports every scenario handler module directly (gat.scenariohandlers.plexos, .reeds, .sienna, ...) to pull docstrings, and the gallery examples construct SiennaScenario and call gat.quickplots. Neither is covered by doc/reports alone now that format-specific deps moved into extras (see #23) — the docs build would otherwise ImportError on the first gallery example. Verified locally: pip install -e '.[doc,reports,all]' into a fresh venv, confirmed SiennaScenario/PlexosScenario/MultiScenario/ gat.quickplots all import cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI caught this: gat.backends.duckdb_backend._write_parquet goes through pandas.to_parquet(), which needs a parquet engine — pyarrow was only declared under server/client, but the core v1 engine (gat.scenario.Scenario / GATDatabase) needs it too, and it's reached via the plexos extra (duckdb-backed PlexosScenario, from_plexos_duckdb, etc.), not just server/client. Verified in a fresh isolated venv (pip install -e '.[dev,reports]'): tests/test_duckdb_backend.py 22/22 passed, including the exact TestScenario/TestGroupedQueries tests that errored in CI before this fix (ImportError resolving pandas' parquet engine). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
micahpw
added a commit
that referenced
this pull request
Aug 3, 2026
Follow-up to the earlier 'Sienna fixture refresh' run on main (run 30832525538) — the cache was refreshed but the baselines were never regenerated to match, leaving main's own tests.yml red since then and causing spurious sienna-regression failures on unrelated PRs (e.g. #29, a packaging-only change that doesn't touch Sienna at all). Regenerated against that run's exact artifact. Suite: 6/6 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oolkit into feat/split-packaging-extras-lazy-imports
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23. Closes #11 (expanded from docs-only to full removal — see below).
Packaging
Base deps trimmed to the format-agnostic floor (numpy, pandas, pydantic, click, loguru, pyyaml, orjson). Everything format-specific moves into extras:
siennaSiennaScenarioplexosPlexosScenario(both backends, for now)reedsReEDsScenarioplotsgat.quickplots/.plotpropertyserver/clientalldevnow pulls in every format + plots — fixes #23's root cause directly (pip install -e ".[dev,reports]"never installedplexos-duckdb, so the dual-backend PLEXOS suite silently never ran, anywhere, ever).plexos-duckdbkept as a backward-compat alias since it's already published on PyPI under that name.tablesremoved outright — confirmed dead, never imported anywhere insrc/gat.Lazy imports (the part that makes the extras real)
Declaring extras means nothing if the import graph ignores them.
from gat.scenariohandlers import PlexosScenariopreviously eagerly imported every handler regardless of which one was wanted — a[reeds]-only install wouldImportErrorthe instant anything touchedgat.scenariohandlersat all. Fixed everywhere with the same shape:scenariohandlers/__init__.py,simulations/__init__.py,systems/__init__.py→ PEP 562__getattr__(mirroringgat/__init__.py's existing pattern), with__all__so bothimport *and named imports stay lazy.datahelpers/__init__.py→ gutted its blanketfrom .X import *. Every real caller already used fully-qualified submodule imports; the one exception (an example script) is fixed.datahelpers/parsers.py→ h5py import moved into the 3 functions that actually touchh5py.File. This module sits onBaseScenario's mandatory path, so ReEDS/EGRET were paying for h5py despite never using it.gat/colors.py→random_color/standard_color_dictrelocated here (dependency-free; old locations re-export for compat). Both were reachable frommodels/scenario.py'sTechnologyMapping, whichBaseScenarioimports unconditionally — so constructing any handler transitively pulled in matplotlib + triggered quickplots' macOS Qt-backend detection just to pick a fallback color._find_solution_filesdefinitions removed (base.py,reeds.pyeach defined it twice in one class — Python keeps only the later copy silently).Verification
sys.modulesprobing: touchingPlexosScenarioonly imports h5py;SiennaScenarioonly h5py+geopandas+polars. Neither leaks into the other or into matplotlib/duckdb.ModuleNotFoundError(not a confusing crash) where a format's deps are absent.EGRET removal (#11, expanded scope)
egret.pyand its docs page deleted outright rather than just de-listed. Confirmed before deletion: its only real dependency (egret_map_simple) had zero other consumers,EGRETScenariowas referenced nowhere but its own file and the__init__import, and nothing tested it.🤖 Generated with Claude Code