From a95e56e3424899dd2cd6a7e4e3cff372d37cc6ae Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 29 Aug 2026 06:41:40 +0800 Subject: [PATCH] docs: run-004 reproduced the retracted verdict on the mojibake snapshot Identical-to-4-decimals (83.0797 vs 83.08) + live mojibake decode of best/ = same poisoned data, not a capacity result. run-005 is the real clean-label test. --- docs/PUBLICATION-NOTES.md | 13 +++++++++++++ docs/RESULTS.md | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index 767f6e8..4edb439 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -103,3 +103,16 @@ stack. The frontier law (pretrained-or-collapse) and the controlled capacity/aux/optimizer experiments give it empirical heft. Package as one paper (current paper.adoc); the margin/head finding alone is also a strong short workshop paper if a split is ever wanted. + +## ara-diac-tiny: the "clean-label re-run" wasn't (2026-08-29) + +Narrative caution for the paper: the Aug-24 retraction flagged that the +83.08 tiny-collapse number was measured on mojibake labels. The +attempted clean re-run (run-004) silently reused the same Aug-23 +snapshot via a baked-in `labels_file` path — and reproduced 83.0797 +exactly, which is itself the tell: identical-to-4-decimals means same +data, not same capacity law. Two lessons for the experiments section: +(1) label-provenance must be content-hashed into the run record, not +inferred from filenames; (2) a reproduced number is only evidence of +reproducibility when the input pipeline is versioned. run-005 +(regenerating labels live from the r6 teacher) is the honest test. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index c64dae9..501e1a3 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -177,6 +177,24 @@ conclusion for Arabic is UNPROVEN pending a clean-label re-run. The Thai tiny verdict is unaffected (umt5/sentencepiece labels were byte-exact); the pretrained-backbone law rests on Thai evidence. +## ara-diac-tiny run-004 — the retracted verdict reproduced, on poisoned data (2026-08-29) + +The spec intended to rerun the tiny tier on clean labels silently +consumed the Aug-23 label snapshot (pre `decode_joined` fix). Evidence +chain: + +- run-004/best decodes `كتاب` as `ÙÙاتÙب` — UTF-8-as-Latin1 mojibake, + the exact label corruption the retraction describes +- the 300-paragraph windowed gate (n=300, teacher reproduces its + documented 1.3205) scores the student **83.0797** vs the retracted + **83.08** — identical to four decimals: the poisoned-data constant, + not a capacity result +- `final_eval.json` in the run dir is the durable provenance + +Verdict: run-004 says nothing about from-scratch capacity; the +retraction stands. **run-005** (fresh teacher labels, no snapshot, +2026-08-29) is the actual clean-label falsification test — in flight. + ## ara-diac-small-1.0 — Arabic client tier (2026-08-24) Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph,