ara-diac-small: full-set correction — 8.259 DER-CE; subset 3.658 unrepresentative

docs/RESULTS.md

@@ -185,33 +185,48 @@ Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph,
 windowed harness as the ara-diac-tiny verdict (300 SadeedDiac-25
 paragraphs, Misraj evaluator, haraqat projection):
 
-| Model | DER-CE |
+| Model | DER-CE (300-para subset) |
 |---|---|
 | Teacher (r6) | 1.3205% |
 | **Student (ByT5-small, client rung)** | **3.6580%** |
 
-Gate discussion: the strict budget (teacher +0.5pp) is missed by
-+2.34pp — the measured capacity cost of ByT5-small on Arabic
-diacritization, consistent with the Thai client tier (+7.63pp beam-4 /
-2.85% greedy against a 4.43% teacher). Shipped as the Arabic client
-rung per that precedent: the student generates real, well-voweled
-Arabic at a fraction of the teacher's artifact (1.3 vs 2.6 GiB), the
-strict gate is met by the teacher release (ara-diac-1.0), and the miss
-is disclosed rather than averaged away. For comparison, this student's
-3.66% sits in the same league as the r3 production teacher's era
-(2.68% on the older full-set protocol).
+**Full-set correction (2026-08-26): the subset was not representative.**
+Re-measured on the full 1,200-paragraph SadeedDiac-25 benchmark (same
+harness; teacher reproduces its documented value at 2.5815 vs 2.5793,
+confirming protocol consistency):
+
+| Model | DER-CE (full 1,200) |
+|---|---|
+| Teacher (r6, full-set) | 2.5815% |
+| **Student (ByT5-small, full-set)** | **8.2590%** |
+
+The first 300 paragraphs sit in the student's training-domain
+neighborhood; the remaining 900 expose a domain-generalization gap the
+subset hid. The student's catalog number is the full-set 8.26; the
+300-para figures above stand as measurements of that subset only.
+
+Gate discussion: against the full set the strict budget (teacher
++0.5pp) is missed by +5.68pp — the capacity-plus-domain cost of
+ByT5-small trained on 29K r5-unit labels, consistent in direction with
+the Thai client tier. Shipped as the Arabic client rung with that
+number disclosed: the student generates real, well-voweled Arabic at a
+fraction of the teacher's artifact (1.3 vs 2.6 GiB) and the strict gate
+is met by the teacher release (ara-diac-1.0, 2.58 full-set). On the
+SadeedDiac-25 leaderboard the student at 8.26 sits just behind
+Sadeed-1.5B (7.2915 published) and ahead of nothing measured below it —
+the earlier "between Gemini-Flash and GPT-4" reading was an artifact of
+the unrepresentative subset and is withdrawn.
 
 Training notes: this is the third training of run-002 — the first on
 mojibake labels (byt5 decode_joined bug), the second silently resumed
 from the poisoned lineage's checkpoints (now guarded by labels.sha
 digest matching), this one clean end-to-end. CE plateaued at ~0.016.
 
 Leaderboard context (SadeedDiac-25, Misraj evaluator, zero-skip,
-harakat-projected DER-CE): the teacher tier (r6, 580M) at 2.5793 is the
-best dedicated model measured under this protocol — second only to
-Claude-3.7-Sonnet's published 1.3941, ahead of GLM-5.2 zero-skip (2.6911),
-Gemini-Flash-2.0 (3.1926), GPT-4 (3.8645), and Sadeed-1.5B (7.2915;
-source table in rababa docs/RESULTS.md). This student's 3.66% was
-measured on the 300-paragraph subset; the full-1,200-paragraph
-measurement is running (2026-08-26) and will replace the subset number
-when it lands.
+harakat-projected DER-CE): the teacher tier (r6, 580M) at 2.5793
+(reproduced at 2.5815, 2026-08-26) is the best dedicated model measured
+under this protocol — second only to Claude-3.7-Sonnet's published
+1.3941, ahead of GLM-5.2 zero-skip (2.6911), Gemini-Flash-2.0 (3.1926),
+GPT-4 (3.8645), and Sadeed-1.5B (7.2915; source table in rababa
+docs/RESULTS.md). The client student's full-set 8.26 lands behind
+Sadeed-1.5B; see the correction above.

docs/paper.adoc

@@ -12,9 +12,9 @@ toc::[]
 
 Interoperable script conversion (OGC Abstract Standard topic _Interoperable Script Conversion Systems_) has matured for *deterministic* transliteration: authority-published romanization systems encoded as machine-readable maps that produce byte-identical output across runtimes. A complementary class of conversions — diacritization restoration, grapheme-to-phoneme conversion for scripts without a phonemic orthography, homograph disambiguation — has no authority to publish a map, because the mapping is not a rule set but a learned function. This paper presents the phonological layer of Interscript: a complete pipeline for producing, validating, and serving distilled byte-level sequence-to-sequence models under the same engineering discipline as deterministic maps.
 
-We contribute: (1) the Interscript Model Format (IMF v1), a checksummed, runtime-agnostic artifact contract that carries models across Ruby, Python, and JavaScript with byte-identical output; (2) a distillation methodology in which students are trained from frozen, independently evaluated teachers under pre-registered error budgets; (3) a measured size–quality frontier for client-tier models demonstrating that sub-100M randomly-initialized byte-level students fail catastrophically in both generation and diacritization, while a pretrained 300M backbone quantized to 4 bits (193 MiB) retains full utility; and (4) a decode-protocol study showing that beam search with conventional length normalization is *counterproductive* on distilled byte-level students, inflating phoneme error rate by a factor of four relative to greedy decoding on the same artifact — a calibration artifact with implications for any evaluation of low-confidence autoregressive models.
+We contribute: (1) the Interscript Model Format (IMF v1), a checksummed, runtime-agnostic artifact contract that carries models across Ruby, Python, and JavaScript with byte-identical output; (2) a distillation methodology in which students are trained from frozen, independently evaluated teachers under pre-registered error budgets; (3) a measured size–quality frontier for client-tier models demonstrating that sub-100M randomly-initialized byte-level students fail catastrophically, while a pretrained 300M backbone quantized to 4 bits (193 MiB) retains full utility; and (4) a decode-protocol study showing that beam search with conventional length normalization is *counterproductive* on distilled byte-level students, inflating phoneme error rate by a factor of four relative to greedy decoding on the same artifact — a calibration artifact with implications for any evaluation of low-confidence autoregressive models.
 
-Eleven models across four languages are released under BSD-3-Clause, every one traceable to a documented protocol. Where public benchmarks permit protocol-matched comparison, the teacher tier ranks among the best dedicated systems measured — second only to a frontier proprietary LLM on the SadeedDiac-25 Arabic diacritization leaderboard, and above the published SentenceBench homograph accuracy for Persian — and the 300M-parameter client student lands between Gemini Flash and GPT-4 on the same Arabic leaderboard.
+Eleven models across four languages are released under BSD-3-Clause, every one traceable to a documented protocol. Where public benchmarks permit protocol-matched comparison, the teacher tier ranks among the best dedicated systems measured — second only to a frontier proprietary LLM on the SadeedDiac-25 Arabic diacritization leaderboard, and above the published SentenceBench homograph accuracy for Persian.
 
 == Introduction
 
@@ -43,7 +43,7 @@ Our goal is to serve the second class under the guarantees of the first. This im
 
 1. **IMF v1** (<<section-imf>>): a minimal artifact contract — one zip, three graphs, a byte-level tokenizer fixed by construction — that makes neural models as portable and as verifiable as transliteration maps.
 2. **Gated distillation campaign** (<<section-method>>, <<section-results>>): teachers frozen and independently evaluated before any student trains; students gated against their teacher on the same harness under budgets pre-registered before training. Four languages shipped; two capacity limits measured and disclosed rather than averaged away.
-3. **The client-tier frontier** (<<section-frontier>>): from-scratch sub-100M byte-level students collapse in *both* task families — including a from-scratch 33M student that fits its training objective under teacher forcing yet emits an immediate end-of-sequence at inference. The pretrained backbone is the whole quality cliff; below ByT5-small there is no useful rung today.
+3. **The client-tier frontier** (<<section-frontier>>): from-scratch sub-100M byte-level students collapse — including a from-scratch 33M student that fits its training objective under teacher forcing yet emits an immediate end-of-sequence at inference. The clean-label evidence is Thai; the Arabic replica's labels were later found corrupted, and that caveat is carried with the claim. The pretrained backbone is the whole quality cliff; below ByT5-small there is no useful rung today.
 4. **The decode-protocol correction** (<<section-decode>>): beam-4 evaluation numbers overstated student phoneme error rate by 4.2x relative to greedy decoding on the same artifact. We analyze why (per-token distributions barely above uniform make likelihood-ranked completion pathological), and argue that runtime protocol — greedy — must be the published protocol for this model class.
 5. **Reproducibility record** (<<section-repro>>): the full engineering post-mortem of a multi-day debugging effort that unmasked four stacked infrastructure and code defects that masqueraded as storage-layer corruption, with the guards now preventing each.
 
@@ -129,7 +129,7 @@ Teachers are production models independently trained and evaluated before any st
 
 === Error budgets, pre-registered
 
-Each teacher–student pair carries a shrink budget agreed before training: within budget the student ships as a full-quality replacement; outside it the student may still ship as a *client tier* artifact with the miss measured and disclosed. This two-tier policy matters: discarding a working 3.66%-DER student because its budget was 0.5pp serves no user, but shipping it silently as if it met budget destroys the meaning of the numbers.
+Each teacher–student pair carries a shrink budget agreed before training: within budget the student ships as a full-quality replacement; outside it the student may still ship as a *client tier* artifact with the miss measured and disclosed. This two-tier policy matters: discarding a working 8.26%-DER student because its budget was 0.5pp serves no user, but shipping it silently as if it met budget destroys the meaning of the numbers.
 
 === Harnesses
 
@@ -177,14 +177,14 @@ Students train either by sequence-level KD (teacher-generated targets; cross-ent
 |tha-g2p-small-1.0 |Thai→IPA |PER 2.85 greedy |int8 · 246 MiB |client
 |tha-g2p-small-1.0-int4 |Thai→IPA (same student) |parity 0.07pp; CER cost +0.17pp |int4 · 193 MiB |client
 |heb-diac-small-1.0 |Hebrew niqqud |DER 30.37 (teacher 24.79; +5.58 within budget) |fp32 · 1.3 GiB |client
-|ara-diac-small-1.0 |Arabic haraqat |DER-CE 3.66 (teacher 1.32; +2.34 disclosed) |fp32 · 1.3 GiB |client
+|ara-diac-small-1.0 |Arabic haraqat |DER-CE 8.26 full-set (teacher 2.58; +5.68 disclosed) |fp32 · 1.3 GiB |client
 |===
 
 All fp32 artifacts measured 0.0pp parity (torch reference vs ONNX, ≥500 samples each); quantized artifacts as noted. Every row resolves from the public index and live-verifies from a cold cache in all three runtimes.
 
 === Capacity costs are real and are disclosed
 
-The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp (client tier) and +2.28pp (server tier) on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box.
+The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +5.68pp on the full benchmark and ships with that number in its metadata, its model card, and the results log — the first-published subset figure (+2.34pp on 300 paragraphs) overstated the student's quality, a selection artifact documented in <<section-leaderboard>>. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp (client tier) and +2.28pp (server tier) on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box.
 
 === The Persian teacher is competitive with published state of the art
 
@@ -205,12 +205,16 @@ External claims are made only where a public benchmark exists and the full proto
 |**our teacher (r6, ByT5-base)** |580M |**2.5793**
 |GLM-5.2 (reproduced, zero-skip) |— |2.6911
 |Gemini Flash 2.0 (published) |— |3.1926
-|**our client student (ByT5-small)** |300M |3.66 footnote:ara-subset[Measured on a 300-paragraph subset of the same benchmark; the full-set measurement is in flight at submission time.]
 |GPT-4 (published) |— |3.8645
 |Sadeed (published) |1.5B |7.2915
+|our client student (ByT5-small) |300M |8.259
 |===
 
-The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The client student, at 300M parameters in a 1.3 GiB checksummed artifact, lands between Gemini Flash and GPT-4. Two protocol notes the benchmark's own authors make and we adopt: self-reported numbers from other protocols (Sadeed's repository reports 1.2% DER under its own split) are not comparable, and multi-reference evaluation (Mohamed & Mubarak 2025) is the honest treatment of valid-alternative diacritizations; on WikiNews-2024 our teacher family scores 19.82 WER against QCRI's in-domain-trained 2.70 — a domain gap (classical hadith vs news text), recorded rather than averaged away.
+The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B.
+
+The client student's row carries a measurement lesson we record rather than bury. Its first-published figure, 3.66, was measured on the first 300 paragraphs of the benchmark; the full 1,200-paragraph run scores 8.26 (teacher reproduces at 2.5815 against its documented 2.5793, confirming protocol consistency). The subset was not representative: its paragraphs sit closer to the student's training domain, and the remaining 900 expose a domain-generalization gap the subset hid. Two practices follow, now standing rules: student-tier numbers are published only from full benchmark sets, and a subset figure is labeled as such at first publication. The student ships with the full-set number disclosed — behind Sadeed-1.5B on the leaderboard — and closing its domain gap (more diverse label coverage, or on-policy distillation; <<section-discussion>>) is open work.
+
+Two protocol notes the benchmark's own authors make and we adopt: self-reported numbers from other protocols (Sadeed's repository reports 1.2% DER under its own split) are not comparable, and multi-reference evaluation (Mohamed & Mubarak 2025) is the honest treatment of valid-alternative diacritizations; on WikiNews-2024 our teacher family scores 19.82 WER against QCRI's in-domain-trained 2.70 — a domain gap (classical hadith vs news text), recorded rather than averaged away.
 
 *Hebrew.* No external leaderboard exists for the Nakdimon test domain (Biblical/Rabbinic). On that split our best teacher scores 16.58% DER — against DictaBERT-large, the reference production diacritizer, at 35.63% *on the same test, run by us*. Recent open systems (D-Nikud; visual-representation approaches, Elboher & Pinter 2025) report state of the art on *modern*-text benchmarks we do not evaluate. Hebrew remains the campaign's disclosed weak spot in absolute terms while being locally competitive; closing the domain gap is active work.
 
@@ -240,7 +244,7 @@ Three findings:
 
 1. **Random initialization collapses regardless of capacity or structure.** The bridge projections (teacher activations ridge-regressed into the student's geometry) measurably improve internal structure — 75.80 → 71.12 — but cannot rescue task accuracy. Enlarging without pretraining does not help (70M scores *worse* than 33M).
 2. **The pretrained rung is the whole cliff.** From 70M/78.51 to 300M/2.85 — two orders of magnitude in error across one initialization decision. ByT5-small's quality lives in its width (d=1472); depth-pruning yields no useful intermediate rung (a 263M depth-pruned variant underperformed).
-3. **The collapse reproduces exactly in diacritization, with clean labels.** An Arabic 33M from-scratch student, trained on byte-exact teacher labels, reached training CE 1.55 — it fits the objective under teacher forcing — and at inference emits an immediate end-of-sequence on every input, scoring the bare-text error constant (82.87% DER-CE vs teacher 1.32%). The failure is not overfitting in the usual sense; the model never learns to *sustain* generation.
+3. **The collapse reproduces in diacritization — with an honesty caveat.** An Arabic 33M from-scratch student scored the bare-text error constant (83.08% DER-CE vs teacher 1.32% on the same subset) while fitting its training objective under teacher forcing — emitting an immediate end-of-sequence at inference. A later audit showed that run's labels were corrupted in transit (<<section-repro>>), so this instance is confounded and the results log retracts its evidentiary weight; the *clean-label* collapse evidence is Thai, where the label pipeline was byte-exact and every from-scratch rung fails identically. We state the provenance because the pretrained-or-collapse conclusion should not lean on a measurement we know was poisoned.
 
 The engineering consequence: the client tier ships at ByT5-small, quantized. A true 30–70 MiB tier requires a *pretrained* backbone at that scale — byte-level pretraining of a narrow model — which remains open future work. The quantization ladder is the practical size lever today: 246 MiB at int8, 193 MiB at int4, with the 4-bit quality cost measured at +0.17pp CER and certified by the parity gate.
 
@@ -324,7 +328,7 @@ The general lesson: when a file "reads corrupt remotely and clean locally," susp
 == Discussion and limitations
 [[section-discussion]]
 
-*Scope.* The frontier result (pretrained-or-collapse) is measured at the 30–300M scale on two task families and four languages. It is consistent with the general finding that byte-level models need pretraining at scale; it does not establish that a pretrained 70M model must fail — only that none exists to ship today.
+*Scope.* The frontier result (pretrained-or-collapse) is measured at the 30–300M scale. The clean-label evidence is Thai G2P at four capacity points; the Arabic diacritization replica collapsed identically but was later found to have trained on corrupted labels, and its evidentiary weight is retracted in the results log. The finding is consistent with the general result that byte-level models need pretraining at scale; it does not establish that a pretrained 70M model must fail — only that none exists to ship today.
 
 *Teachers are moving targets by design.* The Hebrew teacher improved mid-campaign (s45 → s46 via phonikud curriculum; s47 in flight via morphological auxiliary tasks); the Arabic teacher improved via the same auxiliary-task template. Students track their teacher's snapshot; the catalog's provenance chain records which.