Skip to content

fix(preprocessor): make handle_missing="error" actually reject NaN - #31

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/handle-missing-error-rejects-nan
Open

fix(preprocessor): make handle_missing="error" actually reject NaN#31
ChrisW09 wants to merge 1 commit into
mainfrom
fix/handle-missing-error-rejects-nan

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #16.

Problem

The Preprocessor docstring promises that "error" lets missing values "reach the
transformers, which then raise on NaN"
. Only PLE did.

handle_missing did two things: dropped the SimpleImputer, and got forwarded as a
constructor argument to exactly one transformer (ple, via NUMERICAL_METHODS). Everything
else was left to notice NaN on its own — and doesn't. The scikit-learn scalers ignore missing
values by design, and the PreTab expansion families declare allow_nan:

hm=error minmax : no error; output all finite = False
hm=error rbf    : no error; output all finite = False
hm=error ple    : raised ValueError - Input contains NaN.

The feature-map case is worse than plain pass-through: unsupervised center placement runs
np.percentile over the column, so a single NaN makes every center NaN and the entire
feature block NaN — not just the ten affected rows.

Fix

Enforce the policy in one place rather than per transformer. get_numerical_transformer_steps
now puts a RaiseOnNaNTransformer in the slot the imputer would have occupied when
add_imputer is False:

if add_imputer:
    steps.append(("imputer", SimpleImputer(...)))
else:
    steps.append(("nan_check", RaiseOnNaNTransformer()))

This is the option the issue recommended, and it makes the documented contract literally
true. The error names the setting and its remedy:

PretabDataError: Input contains NaN, but handle_missing='error' was requested.
Fix: pass handle_missing='median' to impute missing values before the numerical
method, or remove them from the input.

Result

All 22 registered numerical methods now reject NaN under handle_missing="error" — verified
by iterating NUMERICAL_METHODS, with zero methods failing to raise. The "median" path is
untouched and still imputes.

Tests

Ten added:

  • tests/test_reproducibility.py (next to the existing handle_missing tests): a
    12-method parametrised rejection test, a 3-method check that "median" still imputes, a
    check that the error is a PretabDataError naming the option, a pipeline-shape assertion
    that nan_check and imputer are mutually exclusive, and a check that clean data still
    transforms normally under "error".
  • tests/test_encoder_feature_counts.py: four unit tests for RaiseOnNaNTransformer itself.

Full suite: 504 passed, 9 xfailed. pyright unchanged at 71. Ruff on pretab/ clean apart
from the pre-existing RUF100 in __init__.py — my first pass added a RUF022 by inserting
the new export out of alphabetical order in __all__, now fixed.

🤖 Generated with Claude Code

The docstring says "error" lets missing values "reach the transformers, which
then raise on NaN". Only PLE did. ``handle_missing`` merely dropped the
SimpleImputer and was forwarded as a constructor argument to exactly one
transformer, so the guarantee depended on whether the chosen method happened to
notice NaN -- and the scikit-learn scalers ignore missing values by design while
the PreTab expansion families declare ``allow_nan``:

    hm=error minmax  no error, output all finite = False
    hm=error rbf     no error, output all finite = False
    hm=error ple     ValueError: Input contains NaN.

The feature-map case was worse than pass-through: unsupervised center placement
runs ``np.percentile`` over the column, so one NaN makes *every* center NaN and
the whole feature block NaN, not just the affected rows.

Enforce the policy in one place instead. ``get_numerical_transformer_steps``
now puts a ``RaiseOnNaNTransformer`` in the slot the imputer would occupy when
``add_imputer`` is False, so every numerical method raises a ``PretabDataError``
that names the option and its remedy. All 22 registered methods now reject NaN;
the "median" path is untouched.

Closes #16

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(preprocessor): handle_missing="error" lets NaN into the output

1 participant