ara-diac-small: full-set correction — 8.259 DER-CE; subset 3.658 unrepresentative - #54
Merged
Conversation
…58 was unrepresentative Full 1,200-paragraph measurement (teacher reproduces 2.5815 vs documented 2.5793, confirming protocol consistency): student scores 8.259 vs 3.658 on the first-300 subset. The subset sits in the training-domain neighborhood; the remaining 900 paragraphs expose a domain-generalization gap the subset hid. Catalog number is now the full-set figure; the miss against the strict gate is +5.68pp, disclosed. Standing rule: student-tier numbers publish from full sets only, subset figures labeled at first publication. Also corrects paper finding 3 to match the results log: the Arabic 33M collapse ran on corrupted labels (retracted there), so the clean-label pretrained-or-collapse evidence is Thai. Metrics chain: new ara-diac-small-1.0-fullset provenance entry (der_teacher_fullset 2.5815 / der_student_fullset 8.259), metadata and index updated for fp32 and int8 variants.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
imf metricspasses for all 10 modelsTest plan
imf cli metrics— all models trace to RESULTS.md (validated against the pushed branch ref; flip ref to main post-merge)