Fix four scoring defects surfaced by the DAT-3459 SIG rescore (spotting, KIE syntax, grounding bbox, boxed JSON) - #9
Open
FHLiang221 wants to merge 1 commit into
Open
Conversation
- text spotting: replace the 'simplified to VQA' scorer (string-matching the
box list against the flattened polygon answer scored every model 0) with
the official RRC-style single-image metric: axis-aligned GT polygons,
### ignore regions, greedy IoU>0.5 matching, recognition h-mean. Adds the
official box extractor with \boxed{} unwrapping.
- KIE F1: unwrap single-element list values on the prediction side (mirror
of _generate_combinations on GT) so answers written in the GT's own
{'key': ['value']} syntax stop scoring 0 - the GT fed back as a
prediction used to score 0.0
- VQA-with-position/grounding: rebuild the GT box from the four numeric
answer strings when the sample bbox field is null (exported eval sets),
so the IoU half of the score stays live
- CC-OCR KIE _extract_json: restore braces when a model merges the boxed
braces with the JSON braces, leaving a brace-less body
Canary tests in tests/test_sig_scoring_fixes.py (14 tests), including
GT-as-prediction == 1.0 for spotting and KIE.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four scoring defects in
datbench/datasets_scoring/that graded answer format instead of content — found by rescoring the DAT-3459 SIG-scaling arms from saved predictions. This package scores db-document (the DatBench composite), so these defects shaped every db-document number.###ignore regions, greedy IoU>0.5 match, recognition precision/recall h-mean) plus the official box extractor with\boxed{}unwrapping._generate_combinationsunpacks GT list values to scalars but predictions were stringified as-is, so{'key': ['value']}compared"['value']"vs"value". The ground truth fed back as a prediction scored 0.0. Fix: mirror the unwrap on the prediction side (single-element lists only).bboxfield is null (db-document) — the GT box exists only as four numeric answer strings. Fix: rebuild it from the answers whenbboxis absent; an explicitbboxstill wins.{}for\boxed{"k": "v"}— the brace-matching extractor handsjson.loadsa brace-less body when the boxed braces swallow the JSON braces. Fix: restore the braces.Impact (DAT-3459 arms, rescored locally from saved predictions)
None of the repairs can turn a wrong value into a right one. Canary tests included (
tests/test_sig_scoring_fixes.py, 14 tests).Companion
datnanovlm #489 applies the same fixes to
datvlmeval's standalone-eval copies of these scorers; after this merges, datnanovlm needs a submodule pointer bump.🤖 Generated with Claude Code