perf(preprocessor): slice transform blocks from output_indices_ - #29
Open
ChrisW09 wants to merge 1 commit into
Open
perf(preprocessor): slice transform blocks from output_indices_#29ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
The dict form of ``transform`` -- the default -- measured each block's width by
calling ``transformer.transform(X[columns])`` again, so every per-feature
pipeline ran twice per call. On 20k rows x 20 features with PLE that was 43.5 ms
against 21.9 ms for ``return_array=True``, for identical numbers.
``column_transformer_.output_indices_`` already records the exact output slice
for each transformer, and ``output_dims_`` on this same class already reads it.
Use it here too. Also more precise than what it replaces: the ``width = 1``
fallback for a non-estimator entry was a guess.
return_array=True : 21.9 ms -> 21.9 ms
return_array=False: 43.5 ms -> 21.7 ms
Closes #20
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #20.
Problem
transform(..., return_array=False)— the default — transformed the data once through theColumnTransformer, then re-ran each per-feature pipeline a second time purely to read.shape[1]:So every pipeline ran twice per call, for identical numbers.
Fix
column_transformer_.output_indices_already records the exact output slice for eachtransformer — and
output_dims_, on this same class, already reads it. Use it here too.It is also strictly more precise than what it replaces: the
width = 1fallback for anon-estimator entry was a guess, whereas the indices are authoritative.
Result
The dict path now costs the same as the array path, which is what it should always have
cost — the split is pure bookkeeping.
Returned values are unchanged: same keys, same widths, same numbers.
Tests
Four added to
tests/test_preprocessor.py:test_dict_transform_calls_each_pipeline_once— wraps each fitted pipeline'stransformin a counter and asserts every count is exactly 1. This is the direct regression guard and
fails on the old code with counts of 2.
test_dict_blocks_reassemble_the_stacked_array— hstacking the blocks reproducesreturn_array=Truetest_dict_block_widths_match_output_dims— widths agree withoutput_dims_and sum tolen(get_feature_names_out())test_dict_keys_cover_every_input_featureFull suite: 484 passed, 9 xfailed. pyright back to its pre-existing 71 errors — my first pass
added 12, all the familiar "
transformreturnsdict | ndarray, pyright can't narrow it"noise, resolved with the
cast(...)pattern the repo already uses intest_adaptive_output_dim.py.One incidental cleanup:
tests/test_preprocessor.pyhad a pre-existing ruffI001(unsorted imports) on the exact block this change edits, so I sorted it rather than leave a
lint error attached to my diff. That file is now ruff-clean; the unrelated
RUF100inpretab/__init__.pyis untouched and still present onmain.🤖 Generated with Claude Code