Skip to content

docs: ara-diac-tiny verdict (collapsed); eval DER x100 fix #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 23, 2026
Merged

docs: ara-diac-tiny verdict (collapsed); eval DER x100 fix #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 28 additions & 4 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,31 @@ parameter count — depth-pruning yields no useful intermediate rung
(263M). (3) The pretrained rung is the whole quality cliff: 300M at
12.06% (run-003, full labels; 12.63% on the 23K subset) vs 70M at 78.5%.

Conclusion: G2P client tier ships at the ByT5-small rung (~300MB int8)
today; a 30–70MB G2P tier requires byte-level pretraining of the small
model first (future work). Copy-task languages (Arabic/Hebrew
diacritization) have a different viability and are evaluated separately.
Conclusion: G2P client tier ships at the ByT5-small rung — 246MB at
int8, 202MB at int4 (parity 0.0734pp, quality cost ~0.17pp CER; PRs
#30/#31) — today; a 30–70MB G2P tier requires byte-level pretraining
of the small model first (future work). Copy-task languages are
evaluated separately below.

## ara-diac-tiny verdict — 33MB from-scratch student collapsed (2026-08-23)

The Arabic copy-task hypothesis test: a 33M-parameter custom byte-level
student (d384, 8+8) trained CE on 11,792 r6-teacher labels for 3
epochs (train CE converged to 0.46). Gate harness: windowed DER-CE at
the 1400-byte r5 window, greedy, haraqat-projected, Misraj evaluator —
identical to rababa's eval_sadeed_windowed; validated by the teacher
reproducing its documented tier on this replication.

| Model | DER-CE (300 Sadeed paragraphs) |
|---|---|
| Teacher (r6, run-006-morph) | 1.32% |
| **Student (33M from-scratch)** | **83.08%** — REJECTED |

Gate ≤ teacher + 0.5pp: the student misses by two orders of magnitude
with the same collapse signature as the Thai tiny tier (train loss
converges, test generalization absent). Verdict: sub-100M from-scratch
byte students do not generalize for diacritization any more than for
G2P — a pretrained backbone is non-negotiable. The Arabic client tier
therefore ships at the ByT5-small rung (ara-diac-small) or parks until
byte-level pretraining exists. The 30MB tier is closed as a negative
result across both task families.
2 changes: 1 addition & 1 deletion src/gpu/modal_distill.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1162,7 +1162,7 @@ def der_ce(model) -> dict:
_, _, total_der, _, _ = E.caculate_errors_on_sentences(
paragraphs, gts, gt_missing_diacritic_is_error=False
)
return {"der_ce": round(100 * total_der, 4), "n": len(inputs)}
return {"der_ce": round(total_der, 4), "n": len(inputs)} # evaluator already returns %

result = {"teacher": der_ce(teacher), "student": der_ce(student)}
result["gate_delta"] = round(result["student"]["der_ce"] - result["teacher"]["der_ce"], 4)
Expand Down
Loading