Skip to content

perf(preprocessor): slice transform blocks from output_indices_ - #29

Open
ChrisW09 wants to merge 1 commit into
mainfrom
perf/transform-uses-output-indices
Open

perf(preprocessor): slice transform blocks from output_indices_#29
ChrisW09 wants to merge 1 commit into
mainfrom
perf/transform-uses-output-indices

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #20.

Problem

transform(..., return_array=False) — the default — transformed the data once through the
ColumnTransformer, then re-ran each per-feature pipeline a second time purely to read
.shape[1]:

width = transformer.transform(X[columns]).shape[1]

So every pipeline ran twice per call, for identical numbers.

Fix

column_transformer_.output_indices_ already records the exact output slice for each
transformer — and output_dims_, on this same class, already reads it. Use it here too.

It is also strictly more precise than what it replaces: the width = 1 fallback for a
non-estimator entry was a guess, whereas the indices are authoritative.

Result

return_array=True :  21.9 ms -> 21.9 ms
return_array=False:  43.5 ms -> 21.7 ms      (20k rows x 20 features, PLE)

The dict path now costs the same as the array path, which is what it should always have
cost — the split is pure bookkeeping.

Returned values are unchanged: same keys, same widths, same numbers.

Tests

Four added to tests/test_preprocessor.py:

  • test_dict_transform_calls_each_pipeline_once — wraps each fitted pipeline's transform
    in a counter and asserts every count is exactly 1. This is the direct regression guard and
    fails on the old code with counts of 2.
  • test_dict_blocks_reassemble_the_stacked_array — hstacking the blocks reproduces
    return_array=True
  • test_dict_block_widths_match_output_dims — widths agree with output_dims_ and sum to
    len(get_feature_names_out())
  • test_dict_keys_cover_every_input_feature

Full suite: 484 passed, 9 xfailed. pyright back to its pre-existing 71 errors — my first pass
added 12, all the familiar "transform returns dict | ndarray, pyright can't narrow it"
noise, resolved with the cast(...) pattern the repo already uses in
test_adaptive_output_dim.py.

One incidental cleanup: tests/test_preprocessor.py had a pre-existing ruff I001
(unsorted imports) on the exact block this change edits, so I sorted it rather than leave a
lint error attached to my diff. That file is now ruff-clean; the unrelated RUF100 in
pretab/__init__.py is untouched and still present on main.

🤖 Generated with Claude Code

The dict form of ``transform`` -- the default -- measured each block's width by
calling ``transformer.transform(X[columns])`` again, so every per-feature
pipeline ran twice per call. On 20k rows x 20 features with PLE that was 43.5 ms
against 21.9 ms for ``return_array=True``, for identical numbers.

``column_transformer_.output_indices_`` already records the exact output slice
for each transformer, and ``output_dims_`` on this same class already reads it.
Use it here too. Also more precise than what it replaces: the ``width = 1``
fallback for a non-estimator entry was a guess.

    return_array=True :  21.9 ms -> 21.9 ms
    return_array=False:  43.5 ms -> 21.7 ms

Closes #20

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(preprocessor): transform runs every pipeline twice by default

1 participant