fix(metrics): correct inverted log-term sign in GammaDeviance - #427
Open
ChrisW09 wants to merge 1 commit into
Open
fix(metrics): correct inverted log-term sign in GammaDeviance#427ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
The deviance computed 2*mean(log(y/mu) + (y-mu)/mu) instead of 2*mean(log(mu/y) + y/mu - 1). Both forms are 0 at mu == y, so the existing perfect-prediction test passed, but for any imperfect prediction the metric disagreed with sklearn's mean_gamma_deviance and was unbounded below: predicting mu = 1000*y scored -14.6, i.e. the metric rewarded unbounded over-prediction. It is the primary metric for lss:gamma, so HPO and model selection were driven toward degenerate models. Adds a test comparing against sklearn on imperfect predictions (a perfect-prediction test cannot detect a sign error) and a non-negativity test for extreme over-prediction. Fixes #417 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #417
Problem
GammaDeviancecomputed2*mean(log(y/mu) + (y-mu)/mu)instead of the gamma deviance2*mean(log(mu/y) + y/mu - 1).Both forms are 0 at
mu == y, so the existing perfect-prediction test passed. For any imperfect prediction it disagrees withsklearn.metrics.mean_gamma_deviance(0.341 vs 0.135 on a small example), and the metric is unbounded below: predictingmu = 1000*yscores -14.6, so it rewards unbounded over-prediction. It is the primary metric forlss:gammainMETRIC_REGISTRY, so HPO and model selection minimizing it were driven toward degenerate over-predicting models.Fix
One-line sign correction in
deeptab/metrics/distributional.py.Tests
test_gamma_deviance_matches_sklearn— compares against sklearn on imperfect predictions (a perfect-prediction test cannot detect a sign error).test_gamma_deviance_nonnegative_for_overprediction— deviance must stay positive for extreme over-prediction.Full
tests/test_metrics.pypasses (202 tests).🤖 Generated with Claude Code