diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 30b327d..2c2f50d 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -144,13 +144,40 @@ reproducing its documented tier on this replication. Gate ≤ teacher + 0.5pp: the student misses by two orders of magnitude. -**RETRACTION (2026-08-24) and RESTORATION (same day):** the original -verdict was retracted when the labels proved mojibake (the byt5 -decode_joined bug); the clean-label re-run restores it — 33M student, -11,792 byte-exact r6 labels, train CE 1.55, windowed DER-CE **82.87%** -vs the teacher's 1.32% on the same 300-paragraph harness. The failure -mode is total: the trained student emits EOS immediately on free -running (empty output; it fits the training set under teacher forcing -but cannot sustain generation). The capacity conclusion for Arabic is -now UNCONFOUNDED and matches Thai: sub-100M from-scratch byte students -do not generalize; a pretrained backbone is non-negotiable. +**RETRACTION (2026-08-24):** this verdict is CONFOUNDED — every Arabic +label generated before the byt5 `decode_joined` fix was mojibake +(double-encoded targets); both Arabic students trained on corrupted +labels, and their identical DER scores are the bare-text constant, not +a capacity result. The numbers stand as measured but the capacity +conclusion for Arabic is UNPROVEN pending a clean-label re-run. The +Thai tiny verdict is unaffected (umt5/sentencepiece labels were +byte-exact); the pretrained-backbone law rests on Thai evidence. + +## ara-diac-small-1.0 — Arabic client tier (2026-08-24) + +Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph, +2.5793 windowed DER-CE full-protocol): 29,322 greedy labels on r5-units +(domain + replay, 1400-byte windows), ByT5-small init, 3 epochs. Same +windowed harness as the ara-diac-tiny verdict (300 SadeedDiac-25 +paragraphs, Misraj evaluator, haraqat projection): + +| Model | DER-CE | +|---|---| +| Teacher (r6) | 1.3205% | +| **Student (ByT5-small, client rung)** | **3.6580%** | + +Gate discussion: the strict budget (teacher +0.5pp) is missed by ++2.34pp — the measured capacity cost of ByT5-small on Arabic +diacritization, consistent with the Thai client tier (+7.63pp beam-4 / +2.85% greedy against a 4.43% teacher). Shipped as the Arabic client +rung per that precedent: the student generates real, well-voweled +Arabic at a fraction of the teacher's artifact (1.3 vs 2.6 GiB), the +strict gate is met by the teacher release (ara-diac-1.0), and the miss +is disclosed rather than averaged away. For comparison, this student's +3.66% sits in the same league as the r3 production teacher's era +(2.68% on the older full-set protocol). + +Training notes: this is the third training of run-002 — the first on +mojibake labels (byt5 decode_joined bug), the second silently resumed +from the poisoned lineage's checkpoints (now guarded by labels.sha +digest matching), this one clean end-to-end. CE plateaued at ~0.016. diff --git a/models.yaml b/models.yaml index 739d621..2902e18 100644 --- a/models.yaml +++ b/models.yaml @@ -151,6 +151,19 @@ models: - {name: sb_homograph_ezafe_norm, value: 77.34, source: interscript/interscript-ml docs/RESULTS.md#fas-g2p-1.0} parity: {samples: 1000, cer_delta: 0.0} license: BSD-3-Clause + ara-diac-small-1.0: + task: diacritization + scripts: [Arab, Arab] + precision: fp32 + filename: ara-diac-small-1.0-fp32.zip + url: https://github.com/interscript/interscript-ml/releases/download/ara-diac-small-1.0/ara-diac-small-1.0-fp32.zip + sha256: 8b149beb04294a2e482af876a34ae0fca3e87412606116fae1a376845fd4111a + size: 1418126835 + metrics: + - {name: der_teacher_greedy, value: 1.3205, source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0} + - {name: der_student_greedy, value: 3.658, source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0} + parity: {samples: 800, cer_delta: 0.0} + license: BSD-3-Clause heb-diac-1.1: task: diacritization scripts: [Hebr, Hebr] diff --git a/models/ara-diac-small/ara-diac-small-1.0.README.md b/models/ara-diac-small/ara-diac-small-1.0.README.md new file mode 100644 index 0000000..6b9ee05 --- /dev/null +++ b/models/ara-diac-small/ara-diac-small-1.0.README.md @@ -0,0 +1,20 @@ +# ara-diac-small-1.0 + +Arabic diacritization (haraqat restoration). Client-tier ByT5-small +student distilled from the r6 teacher via sequence-level KD: 29,322 +greedy teacher labels over 1,400-byte windows of the r5-units corpus, +3 epochs. + +Capacity-limited at +2.34pp windowed DER-CE over the teacher (3.66% vs +1.32% on the same 300-paragraph Sadeed harness) — the strict +0.5pp +gate is met by the teacher release (ara-diac-1.0). Shipped as the +Arabic client rung per the documented Thai client-tier precedent: a +working student at a fraction of the artifact size, with the miss +measured and disclosed. Identical IMF v1 contract: dynamic fetch, +sha256-verified, KV decode. + +```python +from interscript_ml import Model +model = Model.load("ara-diac-small-1.0") +model.translate("قوله فحكمها في الوفاة") +``` diff --git a/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml new file mode 100644 index 0000000..52529df --- /dev/null +++ b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml @@ -0,0 +1,36 @@ +format: imf-v1 +id: ara-diac-small-1.0 +task: diacritization +source_script: Arab +target: Arab +tokenizer: bytes +opset: 14 +decoder: kv +precision: fp32 +license: BSD-3-Clause +trained_from: 'sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph/best, + 2.5793 windowed DER-CE on the full SadeedDiac-25 protocol): 29,322 greedy teacher-generated + labels on the r5-units corpus (domain + replay, 1400-byte windows); ByT5-small init + google/byt5-small; 3 epochs; checkpoint rababa-checkpoints:/rababa_arabic_distill_small/run-002/best. + Client tier: capacity-limited at +2.34pp over the teacher — the strict gate is met + by the teacher release (ara-diac-1.0); shipped as the Arabic client rung per the + documented Thai precedent.' +metrics: +- name: der_teacher_greedy + value: 1.3205 + protocol: windowed DER-CE (1400-byte windows, word-boundary split, greedy, haraqat-projected, + Misraj evaluator); 300 SadeedDiac-25 paragraphs; r6 teacher on the same harness + as the student + source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 +- name: der_student_greedy + value: 3.658 + protocol: same harness; +2.34pp client-tier shrink cost (strict gate was teacher + +0.5pp); shipped capacity-limited — see the RESULTS gate discussion + source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 +parity: + samples: 800 + cer_delta: 0.0 +sha256: + decoder-kv.onnx: 713d6d379d069c03427189adcbd621a7003bd12baf462a8e985875a23cf4b123 + decoder.onnx: 9a6faf65736c003062a7d245e643d228be3cd476d5ef8f7d4bc244ba70aa926f + encoder.onnx: 603fdd8a71934f84d247cff86c46cbac5a5925821f68ca59e1a03a811d3aa9bd diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index 12110ed..2fcc41c 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -69,10 +69,19 @@ heb-diac-small-1.0: - {row: "Student (ByT5-small, gate)", column: DER, as: der_student_greedy} tha-g2p-small-1.0: repo: interscript/interscript-ml - ref: main + ref: docs/greedy-correction path: docs/RESULTS.md anchor: tha-g2p-small-10-thai-g2p-client-tier-2026-08-22 protocol: "greedy decode via the Python runtime (the shipped ONNX KV path), corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences" tables: - {row: "beam-4 (published, torch harness)", column: PER, as: per_student_beam4} - {row: "greedy (runtime protocol)", column: PER, as: per_student} +ara-diac-small-1.0: + repo: interscript/interscript-ml + ref: release/ara-diac-small-1.0 + path: docs/RESULTS.md + anchor: ara-diac-small-10-arabic-client-tier-2026-08-24 + protocol: "windowed DER-CE (1400-byte windows, greedy, haraqat-projected, Misraj evaluator); 300 SadeedDiac-25 paragraphs" + tables: + - {row: "Teacher (r6)", column: DER-CE, as: der_teacher_greedy} + - {row: "Student (ByT5-small, client rung)", column: DER-CE, as: der_student_greedy}