fix(architectures): initialize Trompt's init_rec prompt parameter - #440
Open
ChrisW09 wants to merge 1 commit into
Open
fix(architectures): initialize Trompt's init_rec prompt parameter#440ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
nn.Parameter(torch.empty(P, d_model)) was never initialized, so the initial prompt representation kept whatever happened to be in the allocation. When that memory held garbage the model produced NaN logits: this is the cause of the intermittent CI failures in test_experimental_classifier_fit_predict_evaluate[TromptClassifier] and test_experimental_lss_fit_predict_evaluate[TromptLSS], where the LSS run died with 'Expected parameter loc ... to satisfy the constraint Real(), but found invalid values: tensor([nan, ...])'. The sibling prompt embeddings in ImportanceGetter use exactly this pattern with a following torch.nn.init.normal_(std=0.01); init_rec just missed the call. Matching that convention also makes the model seed-reproducible, which it could not be before. Part of #423 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #423 (item 6). This one is currently making CI intermittently red on
main.Problem
Trompt.__init__creates its initial prompt representation asand never initializes it, so the parameter keeps whatever was in the allocation. When that memory holds garbage the model emits NaN logits.
This is not theoretical — it is the cause of the flaky CI failures observed on unrelated PRs:
tests/test_models.py::test_experimental_lss_fit_predict_evaluate[TromptLSS]on Tests (Python 3.11, windows-latest):ValueError: Expected parameter loc (Tensor of shape (32,)) of distribution Normal(...) to satisfy the constraint Real(), but found invalid values: tensor([nan, nan, ...])tests/test_models.py::test_experimental_classifier_fit_predict_evaluate[TromptClassifier]on Tests (Python 3.12, ubuntu-latest)Both appeared on a PR touching only
deeptab/metrics/classification.py— the failures are unrelated to that change and reproduce from this uninitialized parameter.The sibling prompt embeddings in
ImportanceGetter(deeptab/core/inspection.py:32-35) use the identicaltorch.emptypattern followed bytorch.nn.init.normal_(..., std=0.01).init_recsimply missed the call.Fix
Apply the same
torch.nn.init.normal_(self.init_rec, std=0.01). As a bonus this makes Trompt seed-reproducible, which it could not be while a parameter came from uninitialized memory.Tests
tests/test_trompt_init.py:torch.emptyto return NaN-filled tensors — simulating a garbage allocation deterministically — and assertsinit_recis finite after construction. Verified this fails onmainand passes with the fix.🤖 Generated with Claude Code