fix(preprocessor): validate embeddings against the fitted dimensions - #40
Open
ChrisW09 wants to merge 1 commit into
Open
fix(preprocessor): validate embeddings against the fitted dimensions#40ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
``fit`` recorded ``embedding_dimensions_`` for every embedding array it was
given, but nothing ever read it back, so ``transform`` accepted anything:
fitted {'embedding_1': 8}
transform(embeddings=(100, 3)) -> accepted
transform(embeddings=(7, 8)) -> {'num_a': (100, 1), 'embedding_1': (7, 8)}
The row-count case is the damaging one -- the returned dict has blocks of
different heights, which surfaces either as a shape error far from the cause or,
if the caller indexes rather than stacks, as silently misaligned rows.
Validate in ``transform`` against what ``fit`` saw: the number of arrays, each
array's width, that each is 2D, and that its row count matches ``X``. Raise
``PretabDataError`` naming the offending array and both shapes.
``fit`` and ``transform`` now share one normalization helper, so a single array
and a one-element list are treated identically on both sides.
Closes #34
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ChrisW09
force-pushed
the
fix/validate-embeddings-at-transform
branch
from
July 27, 2026 20:48
48cff5e to
3fc1d11
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #34.
Problem
fitpopulatesembedding_dimensions_for every embedding array it receives — and nothingever reads it back.
transformchecked only "were embeddings expected at all", so everymismatch was accepted silently:
That last one is the damaging case: the returned dict has blocks of different heights. A
caller who stacks them gets a shape error a long way from the cause; a caller who indexes
them gets silently misaligned rows.
Two more went unchecked: passing fewer arrays than were fitted (the missing block just
vanished from the dict), and a 1-D array.
Fix
transformnow validates against whatfitrecorded — array count, per-array width, 2-Dshape, and row count against
X— raisingPretabDataErrorthat names the array and bothshapes:
fitandtransformshare one normalization helper, so a bare array and a one-element listare treated identically on both sides — previously each branch had its own
isinstanceladder.
Left as is
Fitting with embeddings and transforming without them still returns the dict without the
embedding blocks, rather than raising. The issue flagged the asymmetry with the opposite
direction (which does raise). I left it alone because tightening it could break a caller who
deliberately transforms without embeddings, and it is a design call rather than a defect —
happy to change it if you want the symmetry.
Impact
This is the documented integration point for an embedding host such as DeepTab — the caller
most likely to wire up the wrong array and least likely to spot it.
Tests
Six added to
tests/test_preprocessor.py: matching embeddings still pass through (includingthe
float32cast), wrong width, wrong row count, wrong array count, a two-array round trip,and a 1-D array.
Full suite: 486 passed, 9 xfailed.
ruff checkclean on changed files; pyright unchanged at71.
tests/test_preprocessor.pyalso picks up the import sort this file has needed sincebefore these changes.
🤖 Generated with Claude Code