Skip to content

fix(architectures): initialize Trompt's init_rec prompt parameter - #440

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/trompt-init-rec
Open

fix(architectures): initialize Trompt's init_rec prompt parameter#440
ChrisW09 wants to merge 1 commit into
mainfrom
fix/trompt-init-rec

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Part of #423 (item 6). This one is currently making CI intermittently red on main.

Problem

Trompt.__init__ creates its initial prompt representation as

self.init_rec = nn.Parameter(torch.empty(config.P, config.d_model))

and never initializes it, so the parameter keeps whatever was in the allocation. When that memory holds garbage the model emits NaN logits.

This is not theoretical — it is the cause of the flaky CI failures observed on unrelated PRs:

  • tests/test_models.py::test_experimental_lss_fit_predict_evaluate[TromptLSS] on Tests (Python 3.11, windows-latest):
    ValueError: Expected parameter loc (Tensor of shape (32,)) of distribution Normal(...) to satisfy the constraint Real(), but found invalid values: tensor([nan, nan, ...])
  • tests/test_models.py::test_experimental_classifier_fit_predict_evaluate[TromptClassifier] on Tests (Python 3.12, ubuntu-latest)

Both appeared on a PR touching only deeptab/metrics/classification.py — the failures are unrelated to that change and reproduce from this uninitialized parameter.

The sibling prompt embeddings in ImportanceGetter (deeptab/core/inspection.py:32-35) use the identical torch.empty pattern followed by torch.nn.init.normal_(..., std=0.01). init_rec simply missed the call.

Fix

Apply the same torch.nn.init.normal_(self.init_rec, std=0.01). As a bonus this makes Trompt seed-reproducible, which it could not be while a parameter came from uninitialized memory.

Tests

tests/test_trompt_init.py:

  • Monkeypatches torch.empty to return NaN-filled tensors — simulating a garbage allocation deterministically — and asserts init_rec is finite after construction. Verified this fails on main and passes with the fix.
  • A smoke test that Trompt predictions are finite after a fit.

🤖 Generated with Claude Code

nn.Parameter(torch.empty(P, d_model)) was never initialized, so the
initial prompt representation kept whatever happened to be in the
allocation. When that memory held garbage the model produced NaN logits:
this is the cause of the intermittent CI failures in
test_experimental_classifier_fit_predict_evaluate[TromptClassifier] and
test_experimental_lss_fit_predict_evaluate[TromptLSS], where the LSS run
died with 'Expected parameter loc ... to satisfy the constraint Real(),
but found invalid values: tensor([nan, ...])'.

The sibling prompt embeddings in ImportanceGetter use exactly this
pattern with a following torch.nn.init.normal_(std=0.01); init_rec just
missed the call. Matching that convention also makes the model
seed-reproducible, which it could not be before.

Part of #423

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant