From e04841575a252b19bba2d0c55c5411542cde72a7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 23 Aug 2026 18:07:45 +0800 Subject: [PATCH] =?UTF-8?q?docs:=20ara-diac-tiny=20verdict=20=E2=80=94=203?= =?UTF-8?q?3M=20student=20collapsed=20(83.08%=20DER-CE=20vs=20teacher=201.?= =?UTF-8?q?32%);=20eval=20=C3=97100=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit evaluate_der multiplied the Misraj evaluator's already-percent DER by 100 again; the harness was validated by the teacher reproducing its documented tier (1.32% on the 300-paragraph replication). The tiny tier is closed as a negative result across both task families. --- docs/RESULTS.md | 32 ++++++++++++++++++++++++++++---- src/gpu/modal_distill.py | 2 +- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index a576b4c..897a532 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -101,7 +101,31 @@ parameter count — depth-pruning yields no useful intermediate rung (263M). (3) The pretrained rung is the whole quality cliff: 300M at 12.06% (run-003, full labels; 12.63% on the 23K subset) vs 70M at 78.5%. -Conclusion: G2P client tier ships at the ByT5-small rung (~300MB int8) -today; a 30–70MB G2P tier requires byte-level pretraining of the small -model first (future work). Copy-task languages (Arabic/Hebrew -diacritization) have a different viability and are evaluated separately. +Conclusion: G2P client tier ships at the ByT5-small rung — 246MB at +int8, 202MB at int4 (parity 0.0734pp, quality cost ~0.17pp CER; PRs +#30/#31) — today; a 30–70MB G2P tier requires byte-level pretraining +of the small model first (future work). Copy-task languages are +evaluated separately below. + +## ara-diac-tiny verdict — 33MB from-scratch student collapsed (2026-08-23) + +The Arabic copy-task hypothesis test: a 33M-parameter custom byte-level +student (d384, 8+8) trained CE on 11,792 r6-teacher labels for 3 +epochs (train CE converged to 0.46). Gate harness: windowed DER-CE at +the 1400-byte r5 window, greedy, haraqat-projected, Misraj evaluator — +identical to rababa's eval_sadeed_windowed; validated by the teacher +reproducing its documented tier on this replication. + +| Model | DER-CE (300 Sadeed paragraphs) | +|---|---| +| Teacher (r6, run-006-morph) | 1.32% | +| **Student (33M from-scratch)** | **83.08%** — REJECTED | + +Gate ≤ teacher + 0.5pp: the student misses by two orders of magnitude +with the same collapse signature as the Thai tiny tier (train loss +converges, test generalization absent). Verdict: sub-100M from-scratch +byte students do not generalize for diacritization any more than for +G2P — a pretrained backbone is non-negotiable. The Arabic client tier +therefore ships at the ByT5-small rung (ara-diac-small) or parks until +byte-level pretraining exists. The 30MB tier is closed as a negative +result across both task families. diff --git a/src/gpu/modal_distill.py b/src/gpu/modal_distill.py index 0a7c6a8..c753c5c 100644 --- a/src/gpu/modal_distill.py +++ b/src/gpu/modal_distill.py @@ -1162,7 +1162,7 @@ def der_ce(model) -> dict: _, _, total_der, _, _ = E.caculate_errors_on_sentences( paragraphs, gts, gt_missing_diacritic_is_error=False ) - return {"der_ce": round(100 * total_der, 4), "n": len(inputs)} + return {"der_ce": round(total_der, 4), "n": len(inputs)} # evaluator already returns % result = {"teacher": der_ce(teacher), "student": der_ce(student)} result["gate_delta"] = round(result["student"]["der_ce"] - result["teacher"]["der_ce"], 4)