Skip to content

fix(metrics): count confidence 1.0 in ECE and stop thresholding 1-D integer labels - #428

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/classification-metric-edge-cases
Open

fix(metrics): count confidence 1.0 in ECE and stop thresholding 1-D integer labels#428
ChrisW09 wants to merge 1 commit into
mainfrom
fix/classification-metric-edge-cases

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #421

Problems

  1. ExpectedCalibrationError ignores confidence == 1.0. Every bin was half-open [lo, hi), including the last [0.9, 1.0), so predictions at exactly 1.0 fell into no bin and contributed 0. A model predicting the wrong class with probability 1.0 on every sample — the worst possible calibration — scored ECE = 0.0.
  2. Accuracy/F1Score binarize 1-D input at 0.5. The docstrings promise 1-D integer-label support, but any 1-D input was thresholded, mangling multiclass labels (label 2 → 1). Accuracy()([0,1,2,2], [0,1,2,2]) returned 0.5 for perfect predictions. This bites the natural composition Accuracy()(y, model.predict(X)), since predict returns 1-D labels.

Fix

  • Make the final ECE bin right-inclusive.
  • Add a shared _labels_from_predictions helper: 2-D input → argmax, 1-D integer-valued input → labels as-is, 1-D fractional input → thresholded at 0.5 (so binary probability scores keep working).

Tests

test_ece_counts_full_confidence, test_accuracy_1d_multiclass_labels, test_f1_1d_multiclass_labels. Full tests/test_metrics.py passes.

🤖 Generated with Claude Code

…nteger labels

ExpectedCalibrationError used half-open bins everywhere, including the
last one, so samples at confidence exactly 1.0 (saturated softmax, or
hard labels passed as probabilities) fell into no bin: a model that is
always wrong at 100% confidence scored a perfect ECE of 0. The final
bin is now right-inclusive.

Accuracy and F1Score thresholded any 1-D input at 0.5 despite
documenting 1-D integer-label support, mangling multiclass labels
(label 2 -> 1). Accuracy()([0,1,2,2], [0,1,2,2]) -- perfect predictions
-- returned 0.5. 1-D integer-valued input is now treated as labels;
only fractional 1-D scores are thresholded, so binary probability
inputs keep working.

Fixes #421

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ChrisW09

Copy link
Copy Markdown
Collaborator Author

CI note: the two red test jobs on this PR (test_experimental_lss_fit_predict_evaluate[TromptLSS] on py3.11/windows and test_experimental_classifier_fit_predict_evaluate[TromptClassifier] on py3.12/ubuntu) are unrelated to this change — this PR only touches deeptab/metrics/classification.py and its tests.

They are a pre-existing flake: Trompt.init_rec is created with nn.Parameter(torch.empty(...)) and never initialized, so it can start as NaN depending on what is in the allocation (issue #423, item 6). Fixed in #440, which reproduces it deterministically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Classification metrics: ECE ignores confidence == 1.0; Accuracy/F1 threshold 1-D multiclass labels at 0.5

1 participant