perf(bfabric): import polars lazily in the eagerly-loaded entities - #585
Conversation
polars is by far the heaviest dependency (~200 MB installed, ~70 ms to import), but three modules on the `import bfabric` path pulled it in at module scope: entities/core/has_many.py the .polars convenience property entities/dataset.py to_polars() and its write_* helpers entities/multiplexkit.py the .ids property (uses pl.col) None of them need polars unless the caller actually asks for a DataFrame, so each import moves into the function body, with the type-only names under TYPE_CHECKING. This is the pattern ResultContainer.to_polars already used. `import bfabric` drops from ~296 ms to ~184 ms and no longer loads polars at all. Behaviour is unchanged -- polars remains a hard dependency, just not an eagerly imported one. The remaining module-level imports (utils/polars_utils, utils/table_lint, operations/dataset/*) are in modules that *are* the polars-based features and are not on the import path.
The perf win is invisible to the existing suite: every test process imports polars for other reasons, so nothing notices if a module-scope `import polars` comes back. A probe runs in a fresh interpreter, blocks polars at sys.meta_path so an eager import raises instead of quietly populating sys.modules, and then imports every module in the package in turn. Blocking rather than inspecting sys.modules is what keeps each offender attributable, since the real module would be cached after the first import. It reports the offending import statement, not just that polars got loaded: bfabric/entities/dataset.py:6 (reached by `import bfabric`) Coverage is package-wide rather than just `import bfabric`, which reaches only 53 of the 118 modules -- an eager import landing in e.g. bfabric.transfer would otherwise slip through. The modules that *are* the polars features (operations/dataset, utils/polars_utils, utils/table_lint) are allowlisted, and a module whose optional extra is not installed (e.g. zeep) is skipped.
leoschwarz
left a comment
leoschwarz
left a comment
There was a problem hiding this comment.
Thank you. I think that if we want to really optimise the imports, we will have to do it more systematically. Many of the imports are not needed for all use cases. Also, polars could be made optional behind a feature gate for instance. I do use it heavily but e.g. if your command line tool does not depend on it, then it could be gated behind a feature flag.
For me, it's ok to go as a quick fix. I'm pushing a test case to spot the regression though, is it ok with you?
yes ofc this was mainly to decrease the compiled size of program which would only upload workunits. |
2 participants
Three modules on the
import bfabricpath importedpolarsat module scope, none of which need it unless the caller actually asks for aDataFrame:entities/core/has_many.py.polarsconvenience propertyentities/dataset.pyto_polars()and itswrite_*helpersentities/multiplexkit.py.idsproperty (usespl.col)Each import moves into the function body, with type-only names under
TYPE_CHECKING— the patternResultContainer.to_polarsalready used.Effect
polars is by far the heaviest dependency — ~200 MB of a ~261 MB install — so this matters most for short-lived processes. A cron-driven uploader was paying ~70 ms per tick, plus the install weight, for a DataFrame library it never touches.
Scope
Behaviour is unchanged. polars remains a hard dependency and every API works exactly as before; it is simply not loaded until a
DataFrameis requested..polars,to_polars()and.idswere all exercised after the change and still return real DataFrames.The remaining module-level imports (
utils/polars_utils,utils/table_lint,operations/dataset/*) are untouched: those modules are the polars-based features and are not on theimport bfabricpath.Testing
Full
tests/bfabricsuite passes (822 on this base); ruff and basedpyright clean.Split out of #581, where it was noticed while packaging an instrument uploader — the change touches core entity modules that every consumer imports, so it is worth reviewing on its own rather than riding along with unrelated upload work.