Skip to content

fix(preprocessor): raise a clear error on duplicate column names - #43

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/duplicate-column-names
Open

fix(preprocessor): raise a clear error on duplicate column names#43
ChrisW09 wants to merge 1 commit into
mainfrom
fix/duplicate-column-names

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #37.

Problem

dup = pd.DataFrame(np.column_stack([rng.normal(size=50)] * 2), columns=["a", "a"])
Preprocessor(numerical_method="minmax").fit(dup, rng.normal(size=50))
AttributeError: 'DataFrame' object has no attribute 'dtype'
  ... pretab/preprocessor.py, line 307, in _detect_column_types

X[col] returns a DataFrame rather than a Series when col is duplicated, so the dtype
inspection has nothing to read. (nunique() on the preceding line happens to work on a
frame, which is why the failure surfaces one line later and looks unrelated.)

Fix

Detect duplicates up front and raise a typed error naming them:

PretabDataError: Duplicate column names are not supported: ['a'].
Fix: rename the columns so every name is unique.

Rejecting rather than accommodating is the honest outcome here: the ColumnTransformer this
builds keys its transformers by column name (f"num_{feature}"), so duplicate labels could
not be routed unambiguously even if type detection coped with them.

Tests

Three added to tests/test_exceptions.py: the reported single-duplicate case, a frame with
two distinct duplicated labels (asserting both are listed), and a unique-column frame to
confirm the check is not over-eager.

Full suite: 483 passed, 9 xfailed. ruff check clean on changed files; pyright unchanged at
its pre-existing 71 errors.

🤖 Generated with Claude Code

Fitting on a DataFrame with a repeated column label failed with

    AttributeError: 'DataFrame' object has no attribute 'dtype'

from inside ``_detect_column_types``, with nothing pointing at the cause:
``X[col]`` returns a DataFrame rather than a Series for a duplicated label, so
the dtype inspection has nothing to read. (``nunique()`` on the line above
happens to work on a frame, which is why it surfaced one line later.)

Check for duplicates up front and raise ``PretabDataError`` naming them. The
ColumnTransformer this builds keys its transformers by column name, so
duplicates could not be routed unambiguously even if detection coped with them
-- rejecting them early is the honest outcome.

Closes #37

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChrisW09
ChrisW09 force-pushed the fix/duplicate-column-names branch from a3c4e02 to 49863eb Compare July 27, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(preprocessor): raise a clear error on duplicate column names

1 participant