From a95e56e3424899dd2cd6a7e4e3cff372d37cc6ae Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 29 Aug 2026 06:41:40 +0800 Subject: [PATCH 1/2] docs: run-004 reproduced the retracted verdict on the mojibake snapshot Identical-to-4-decimals (83.0797 vs 83.08) + live mojibake decode of best/ = same poisoned data, not a capacity result. run-005 is the real clean-label test. --- docs/PUBLICATION-NOTES.md | 13 +++++++++++++ docs/RESULTS.md | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index 767f6e8..4edb439 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -103,3 +103,16 @@ stack. The frontier law (pretrained-or-collapse) and the controlled capacity/aux/optimizer experiments give it empirical heft. Package as one paper (current paper.adoc); the margin/head finding alone is also a strong short workshop paper if a split is ever wanted. + +## ara-diac-tiny: the "clean-label re-run" wasn't (2026-08-29) + +Narrative caution for the paper: the Aug-24 retraction flagged that the +83.08 tiny-collapse number was measured on mojibake labels. The +attempted clean re-run (run-004) silently reused the same Aug-23 +snapshot via a baked-in `labels_file` path — and reproduced 83.0797 +exactly, which is itself the tell: identical-to-4-decimals means same +data, not same capacity law. Two lessons for the experiments section: +(1) label-provenance must be content-hashed into the run record, not +inferred from filenames; (2) a reproduced number is only evidence of +reproducibility when the input pipeline is versioned. run-005 +(regenerating labels live from the r6 teacher) is the honest test. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index c64dae9..501e1a3 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -177,6 +177,24 @@ conclusion for Arabic is UNPROVEN pending a clean-label re-run. The Thai tiny verdict is unaffected (umt5/sentencepiece labels were byte-exact); the pretrained-backbone law rests on Thai evidence. +## ara-diac-tiny run-004 — the retracted verdict reproduced, on poisoned data (2026-08-29) + +The spec intended to rerun the tiny tier on clean labels silently +consumed the Aug-23 label snapshot (pre `decode_joined` fix). Evidence +chain: + +- run-004/best decodes `كتاب` as `ÙÙاتÙب` — UTF-8-as-Latin1 mojibake, + the exact label corruption the retraction describes +- the 300-paragraph windowed gate (n=300, teacher reproduces its + documented 1.3205) scores the student **83.0797** vs the retracted + **83.08** — identical to four decimals: the poisoned-data constant, + not a capacity result +- `final_eval.json` in the run dir is the durable provenance + +Verdict: run-004 says nothing about from-scratch capacity; the +retraction stands. **run-005** (fresh teacher labels, no snapshot, +2026-08-29) is the actual clean-label falsification test — in flight. + ## ara-diac-small-1.0 — Arabic client tier (2026-08-24) Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph, From ed061cc6bd6f202dee70c7e96018387a107dfa23 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 29 Aug 2026 06:42:22 +0800 Subject: [PATCH 2/2] chore: ara-diac-small metrics refs flipped to main (post-merge) The full-set tables live on main's docs/RESULTS.md now; the docs/ara-fullset branch ref was the pre-merge stopgap. Validation clean: all models trace to RESULTS.md. --- models/metrics-sources.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index ac36611..80da65c 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -79,7 +79,7 @@ tha-g2p-small-1.0: - {row: "greedy (runtime protocol)", column: Student PER, as: per_student} ara-diac-small-1.0: repo: interscript/interscript-ml - ref: docs/ara-fullset + ref: main path: docs/RESULTS.md anchor: ara-diac-small-10-arabic-client-tier-2026-08-24 protocol: "windowed DER-CE (1400-byte windows, greedy, haraqat-projected, Misraj evaluator); 300-para subset and full 1,200-paragraph SadeedDiac-25" @@ -90,7 +90,7 @@ ara-diac-small-1.0: - {row: "Student (ByT5-small, full-set)", column: "DER-CE (full 1,200)", as: der_student_fullset} ara-diac-small-1.0-int8: repo: interscript/interscript-ml - ref: docs/ara-fullset + ref: main path: docs/RESULTS.md anchor: ara-diac-small-10-arabic-client-tier-2026-08-24 protocol: "windowed DER-CE (1400-byte windows, greedy, haraqat-projected, Misraj evaluator); 300-para subset and full 1,200-paragraph SadeedDiac-25"