From f758d4bc5d6f534da7bd01a1cf1b958d6cb2febc Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 24 Aug 2026 21:37:20 +0800 Subject: [PATCH 1/4] release: ara-diac-small-1.0 registration (capacity-limited client rung, +2.34pp disclosed) --- docs/RESULTS.md | 47 +++++++++++++++---- .../ara-diac-small-1.0.README.md | 20 ++++++++ .../ara-diac-small-1.0.metadata.yaml | 30 ++++++++++++ models/metrics-sources.yaml | 11 ++++- 4 files changed, 97 insertions(+), 11 deletions(-) create mode 100644 models/ara-diac-small/ara-diac-small-1.0.README.md create mode 100644 models/ara-diac-small/ara-diac-small-1.0.metadata.yaml diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 30b327d..b73f1b6 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -144,13 +144,40 @@ reproducing its documented tier on this replication. Gate ≤ teacher + 0.5pp: the student misses by two orders of magnitude. -**RETRACTION (2026-08-24) and RESTORATION (same day):** the original -verdict was retracted when the labels proved mojibake (the byt5 -decode_joined bug); the clean-label re-run restores it — 33M student, -11,792 byte-exact r6 labels, train CE 1.55, windowed DER-CE **82.87%** -vs the teacher's 1.32% on the same 300-paragraph harness. The failure -mode is total: the trained student emits EOS immediately on free -running (empty output; it fits the training set under teacher forcing -but cannot sustain generation). The capacity conclusion for Arabic is -now UNCONFOUNDED and matches Thai: sub-100M from-scratch byte students -do not generalize; a pretrained backbone is non-negotiable. +**RETRACTION (2026-08-24):** this verdict is CONFOUNDED — every Arabic +label generated before the byt5 `decode_joined` fix was mojibake +(double-encoded targets); both Arabic students trained on corrupted +labels, and their identical DER scores are the bare-text constant, not +a capacity result. The numbers stand as measured but the capacity +conclusion for Arabic is UNPROVEN pending a clean-label re-run. The +Thai tiny verdict is unaffected (umt5/sentencepiece labels were +byte-exact); the pretrained-backbone law rests on Thai evidence. + +## ara-diac-small-1.0 — Arabic client tier (2026-08-24) + +Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph, +2.5793 windowed DER-CE full-protocol): 29,322 greedy labels on r5-units +(domain + replay, 1400-byte windows), ByT5-small init, 3 epochs. Same +windowed harness as the ara-diac-tiny verdict (300 SadeedDiac-25 +paragraphs, Misraj evaluator, haraqat projection): + +| Model | DER-CE | +|---|---| +| Teacher (r6) | 1.32% | +| **Student (ByT5-small, client rung)** | **3.66%** | + +Gate discussion: the strict budget (teacher +0.5pp) is missed by ++2.34pp — the measured capacity cost of ByT5-small on Arabic +diacritization, consistent with the Thai client tier (+7.63pp beam-4 / +2.85% greedy against a 4.43% teacher). Shipped as the Arabic client +rung per that precedent: the student generates real, well-voweled +Arabic at a fraction of the teacher's artifact (1.3 vs 2.6 GiB), the +strict gate is met by the teacher release (ara-diac-1.0), and the miss +is disclosed rather than averaged away. For comparison, this student's +3.66% sits in the same league as the r3 production teacher's era +(2.68% on the older full-set protocol). + +Training notes: this is the third training of run-002 — the first on +mojibake labels (byt5 decode_joined bug), the second silently resumed +from the poisoned lineage's checkpoints (now guarded by labels.sha +digest matching), this one clean end-to-end. CE plateaued at ~0.016. diff --git a/models/ara-diac-small/ara-diac-small-1.0.README.md b/models/ara-diac-small/ara-diac-small-1.0.README.md new file mode 100644 index 0000000..6b9ee05 --- /dev/null +++ b/models/ara-diac-small/ara-diac-small-1.0.README.md @@ -0,0 +1,20 @@ +# ara-diac-small-1.0 + +Arabic diacritization (haraqat restoration). Client-tier ByT5-small +student distilled from the r6 teacher via sequence-level KD: 29,322 +greedy teacher labels over 1,400-byte windows of the r5-units corpus, +3 epochs. + +Capacity-limited at +2.34pp windowed DER-CE over the teacher (3.66% vs +1.32% on the same 300-paragraph Sadeed harness) — the strict +0.5pp +gate is met by the teacher release (ara-diac-1.0). Shipped as the +Arabic client rung per the documented Thai client-tier precedent: a +working student at a fraction of the artifact size, with the miss +measured and disclosed. Identical IMF v1 contract: dynamic fetch, +sha256-verified, KV decode. + +```python +from interscript_ml import Model +model = Model.load("ara-diac-small-1.0") +model.translate("قوله فحكمها في الوفاة") +``` diff --git a/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml new file mode 100644 index 0000000..0f726bd --- /dev/null +++ b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml @@ -0,0 +1,30 @@ +format: imf-v1 +id: ara-diac-small-1.0 +task: diacritization +source_script: Arab +target: Arab +tokenizer: bytes +opset: 14 +decoder: kv +precision: fp32 +license: BSD-3-Clause +trained_from: 'sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph/best, + 2.5793 windowed DER-CE on the full SadeedDiac-25 protocol): 29,322 greedy + teacher-generated labels on the r5-units corpus (domain + replay, 1400-byte + windows); ByT5-small init google/byt5-small; 3 epochs; checkpoint + rababa-checkpoints:/rababa_arabic_distill_small/run-002/best. Client tier: + capacity-limited at +2.34pp over the teacher — the strict gate is met by the + teacher release (ara-diac-1.0); shipped as the Arabic client rung per the + documented Thai precedent.' +metrics: +- name: der_teacher_greedy + value: 1.3205 + protocol: windowed DER-CE (1400-byte windows, word-boundary split, greedy, + haraqat-projected, Misraj evaluator); 300 SadeedDiac-25 paragraphs; r6 teacher + on the same harness as the student + source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 +- name: der_student_greedy + value: 3.658 + protocol: same harness; +2.34pp client-tier shrink cost (strict gate was teacher + +0.5pp); shipped capacity-limited — see the RESULTS gate discussion + source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index 12110ed..1f11a8d 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -69,10 +69,19 @@ heb-diac-small-1.0: - {row: "Student (ByT5-small, gate)", column: DER, as: der_student_greedy} tha-g2p-small-1.0: repo: interscript/interscript-ml - ref: main + ref: docs/greedy-correction path: docs/RESULTS.md anchor: tha-g2p-small-10-thai-g2p-client-tier-2026-08-22 protocol: "greedy decode via the Python runtime (the shipped ONNX KV path), corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences" tables: - {row: "beam-4 (published, torch harness)", column: PER, as: per_student_beam4} - {row: "greedy (runtime protocol)", column: PER, as: per_student} +ara-diac-small-1.0: + repo: interscript/interscript-ml + ref: main + path: docs/RESULTS.md + anchor: ara-diac-small-10-arabic-client-tier-2026-08-24 + protocol: "windowed DER-CE (1400-byte windows, greedy, haraqat-projected, Misraj evaluator); 300 SadeedDiac-25 paragraphs" + tables: + - {row: "Teacher (r6)", column: DER-CE, as: der_teacher_greedy} + - {row: "Student (ByT5-small, client rung)", column: DER-CE, as: der_student_greedy} From e6a44dc40649f475c34de58e243e54e527178e20 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 25 Aug 2026 06:27:08 +0800 Subject: [PATCH 2/4] fix(metrics): ara-diac-small provenance reads the release branch --- models/metrics-sources.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/models/metrics-sources.yaml b/models/metrics-sources.yaml index 1f11a8d..2fcc41c 100644 --- a/models/metrics-sources.yaml +++ b/models/metrics-sources.yaml @@ -78,7 +78,7 @@ tha-g2p-small-1.0: - {row: "greedy (runtime protocol)", column: PER, as: per_student} ara-diac-small-1.0: repo: interscript/interscript-ml - ref: main + ref: release/ara-diac-small-1.0 path: docs/RESULTS.md anchor: ara-diac-small-10-arabic-client-tier-2026-08-24 protocol: "windowed DER-CE (1400-byte windows, greedy, haraqat-projected, Misraj evaluator); 300 SadeedDiac-25 paragraphs" From f2bd73cac6f22e013f0dcc8d237ebb4f56432890 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 25 Aug 2026 06:33:00 +0800 Subject: [PATCH 3/4] docs(results): ara-diac-small cells at full metric precision --- docs/RESULTS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b73f1b6..2c2f50d 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -163,8 +163,8 @@ paragraphs, Misraj evaluator, haraqat projection): | Model | DER-CE | |---|---| -| Teacher (r6) | 1.32% | -| **Student (ByT5-small, client rung)** | **3.66%** | +| Teacher (r6) | 1.3205% | +| **Student (ByT5-small, client rung)** | **3.6580%** | Gate discussion: the strict budget (teacher +0.5pp) is missed by +2.34pp — the measured capacity cost of ByT5-small on Arabic From e88432d26869a16b156dcbf29d4d8df8bedb7382 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 25 Aug 2026 09:43:43 +0800 Subject: [PATCH 4/4] release: ara-diac-small-1.0 (fp32, parity cer_delta 0.0pp on 800 samples) --- models.yaml | 13 ++++++++++ .../ara-diac-small-1.0.metadata.yaml | 24 ++++++++++++------- 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/models.yaml b/models.yaml index a8f5f83..f6862d5 100644 --- a/models.yaml +++ b/models.yaml @@ -151,3 +151,16 @@ models: - {name: sb_homograph_ezafe_norm, value: 77.34, source: interscript/interscript-ml docs/RESULTS.md#fas-g2p-1.0} parity: {samples: 1000, cer_delta: 0.0} license: BSD-3-Clause + ara-diac-small-1.0: + task: diacritization + scripts: [Arab, Arab] + precision: fp32 + filename: ara-diac-small-1.0-fp32.zip + url: https://github.com/interscript/interscript-ml/releases/download/ara-diac-small-1.0/ara-diac-small-1.0-fp32.zip + sha256: 8b149beb04294a2e482af876a34ae0fca3e87412606116fae1a376845fd4111a + size: 1418126835 + metrics: + - {name: der_teacher_greedy, value: 1.3205, source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0} + - {name: der_student_greedy, value: 3.658, source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0} + parity: {samples: 800, cer_delta: 0.0} + license: BSD-3-Clause diff --git a/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml index 0f726bd..52529df 100644 --- a/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml +++ b/models/ara-diac-small/ara-diac-small-1.0.metadata.yaml @@ -9,22 +9,28 @@ decoder: kv precision: fp32 license: BSD-3-Clause trained_from: 'sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph/best, - 2.5793 windowed DER-CE on the full SadeedDiac-25 protocol): 29,322 greedy - teacher-generated labels on the r5-units corpus (domain + replay, 1400-byte - windows); ByT5-small init google/byt5-small; 3 epochs; checkpoint - rababa-checkpoints:/rababa_arabic_distill_small/run-002/best. Client tier: - capacity-limited at +2.34pp over the teacher — the strict gate is met by the - teacher release (ara-diac-1.0); shipped as the Arabic client rung per the + 2.5793 windowed DER-CE on the full SadeedDiac-25 protocol): 29,322 greedy teacher-generated + labels on the r5-units corpus (domain + replay, 1400-byte windows); ByT5-small init + google/byt5-small; 3 epochs; checkpoint rababa-checkpoints:/rababa_arabic_distill_small/run-002/best. + Client tier: capacity-limited at +2.34pp over the teacher — the strict gate is met + by the teacher release (ara-diac-1.0); shipped as the Arabic client rung per the documented Thai precedent.' metrics: - name: der_teacher_greedy value: 1.3205 - protocol: windowed DER-CE (1400-byte windows, word-boundary split, greedy, - haraqat-projected, Misraj evaluator); 300 SadeedDiac-25 paragraphs; r6 teacher - on the same harness as the student + protocol: windowed DER-CE (1400-byte windows, word-boundary split, greedy, haraqat-projected, + Misraj evaluator); 300 SadeedDiac-25 paragraphs; r6 teacher on the same harness + as the student source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 - name: der_student_greedy value: 3.658 protocol: same harness; +2.34pp client-tier shrink cost (strict gate was teacher +0.5pp); shipped capacity-limited — see the RESULTS gate discussion source: interscript/interscript-ml docs/RESULTS.md#ara-diac-small-1.0 +parity: + samples: 800 + cer_delta: 0.0 +sha256: + decoder-kv.onnx: 713d6d379d069c03427189adcbd621a7003bd12baf462a8e985875a23cf4b123 + decoder.onnx: 9a6faf65736c003062a7d245e643d228be3cd476d5ef8f7d4bc244ba70aa926f + encoder.onnx: 603fdd8a71934f84d247cff86c46cbac5a5925821f68ca59e1a03a811d3aa9bd