Skip to content

Fix four scoring defects surfaced by the DAT-3459 SIG rescore (spotting, KIE syntax, grounding bbox, boxed JSON) - #9

Open
FHLiang221 wants to merge 1 commit into
mainfrom
freddie-sig-scoring-fixes
Open

Fix four scoring defects surfaced by the DAT-3459 SIG rescore (spotting, KIE syntax, grounding bbox, boxed JSON)#9
FHLiang221 wants to merge 1 commit into
mainfrom
freddie-sig-scoring-fixes

Conversation

@FHLiang221

Copy link
Copy Markdown

What

Four scoring defects in datbench/datasets_scoring/ that graded answer format instead of content — found by rescoring the DAT-3459 SIG-scaling arms from saved predictions. This package scores db-document (the DatBench composite), so these defects shaped every db-document number.

  1. Text spotting was 'simplified to VQA' — the model's box list was string-matched against the flattened polygon answer, which can essentially never match, so every run scored a hard 0. Replaced with the official RRC-style single-image metric (axis-aligned GT polygons, ### ignore regions, greedy IoU>0.5 match, recognition precision/recall h-mean) plus the official box extractor with \boxed{} unwrapping.
  2. KIE F1 zeroed the GT's own answer syntax_generate_combinations unpacks GT list values to scalars but predictions were stringified as-is, so {'key': ['value']} compared "['value']" vs "value". The ground truth fed back as a prediction scored 0.0. Fix: mirror the unwrap on the prediction side (single-element lists only).
  3. Grounding/VQA-with-position rows never earned their IoU half where the exported row's bbox field is null (db-document) — the GT box exists only as four numeric answer strings. Fix: rebuild it from the answers when bbox is absent; an explicit bbox still wins.
  4. CC-OCR KIE returned {} for \boxed{"k": "v"} — the brace-matching extractor hands json.loads a brace-less body when the boxed braces swallow the JSON braces. Fix: restore the braces.

Impact (DAT-3459 arms, rescored locally from saved predictions)

  • Spotting: 8-way tie at 0 → real scores (0.8–3.5%); the all-synthetic arm stays ~0 legitimately (fabricated coordinates)
  • OCRBench Extraction: SIG-trained arms 0.9–54.7 → 61.5–72.0; pure-mined arms unchanged (the repair is a no-op where there was no artifact)
  • Referring: all real-data arms roughly double (e.g. 5.8 → 13.7); CC-OCR KIE: the all-SIG arm 41.8 → 49.2, others unchanged

None of the repairs can turn a wrong value into a right one. Canary tests included (tests/test_sig_scoring_fixes.py, 14 tests).

Companion

datnanovlm #489 applies the same fixes to datvlmeval's standalone-eval copies of these scorers; after this merges, datnanovlm needs a submodule pointer bump.

🤖 Generated with Claude Code

- text spotting: replace the 'simplified to VQA' scorer (string-matching the
  box list against the flattened polygon answer scored every model 0) with
  the official RRC-style single-image metric: axis-aligned GT polygons,
  ### ignore regions, greedy IoU>0.5 matching, recognition h-mean. Adds the
  official box extractor with \boxed{} unwrapping.
- KIE F1: unwrap single-element list values on the prediction side (mirror
  of _generate_combinations on GT) so answers written in the GT's own
  {'key': ['value']} syntax stop scoring 0 - the GT fed back as a
  prediction used to score 0.0
- VQA-with-position/grounding: rebuild the GT box from the four numeric
  answer strings when the sample bbox field is null (exported eval sets),
  so the IoU half of the score stays live
- CC-OCR KIE _extract_json: restore braces when a model merges the boxed
  braces with the JSON braces, leaving a brace-less body

Canary tests in tests/test_sig_scoring_fixes.py (14 tests), including
GT-as-prediction == 1.0 for spotting and KIE.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant