fix(preprocessor): make handle_missing="error" actually reject NaN - #31
Open
ChrisW09 wants to merge 1 commit into
Open
fix(preprocessor): make handle_missing="error" actually reject NaN#31ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
The docstring says "error" lets missing values "reach the transformers, which
then raise on NaN". Only PLE did. ``handle_missing`` merely dropped the
SimpleImputer and was forwarded as a constructor argument to exactly one
transformer, so the guarantee depended on whether the chosen method happened to
notice NaN -- and the scikit-learn scalers ignore missing values by design while
the PreTab expansion families declare ``allow_nan``:
hm=error minmax no error, output all finite = False
hm=error rbf no error, output all finite = False
hm=error ple ValueError: Input contains NaN.
The feature-map case was worse than pass-through: unsupervised center placement
runs ``np.percentile`` over the column, so one NaN makes *every* center NaN and
the whole feature block NaN, not just the affected rows.
Enforce the policy in one place instead. ``get_numerical_transformer_steps``
now puts a ``RaiseOnNaNTransformer`` in the slot the imputer would occupy when
``add_imputer`` is False, so every numerical method raises a ``PretabDataError``
that names the option and its remedy. All 22 registered methods now reject NaN;
the "median" path is untouched.
Closes #16
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #16.
Problem
The
Preprocessordocstring promises that"error"lets missing values "reach thetransformers, which then raise on NaN". Only PLE did.
handle_missingdid two things: dropped theSimpleImputer, and got forwarded as aconstructor argument to exactly one transformer (
ple, viaNUMERICAL_METHODS). Everythingelse was left to notice NaN on its own — and doesn't. The scikit-learn scalers ignore missing
values by design, and the PreTab expansion families declare
allow_nan:The feature-map case is worse than plain pass-through: unsupervised center placement runs
np.percentileover the column, so a single NaN makes every center NaN and the entirefeature block NaN — not just the ten affected rows.
Fix
Enforce the policy in one place rather than per transformer.
get_numerical_transformer_stepsnow puts a
RaiseOnNaNTransformerin the slot the imputer would have occupied whenadd_imputeris False:This is the option the issue recommended, and it makes the documented contract literally
true. The error names the setting and its remedy:
Result
All 22 registered numerical methods now reject NaN under
handle_missing="error"— verifiedby iterating
NUMERICAL_METHODS, with zero methods failing to raise. The"median"path isuntouched and still imputes.
Tests
Ten added:
tests/test_reproducibility.py(next to the existinghandle_missingtests): a12-method parametrised rejection test, a 3-method check that
"median"still imputes, acheck that the error is a
PretabDataErrornaming the option, a pipeline-shape assertionthat
nan_checkandimputerare mutually exclusive, and a check that clean data stilltransforms normally under
"error".tests/test_encoder_feature_counts.py: four unit tests forRaiseOnNaNTransformeritself.Full suite: 504 passed, 9 xfailed. pyright unchanged at 71. Ruff on
pretab/clean apartfrom the pre-existing
RUF100in__init__.py— my first pass added aRUF022by insertingthe new export out of alphabetical order in
__all__, now fixed.🤖 Generated with Claude Code