diff --git a/docs/paper.adoc b/docs/paper.adoc index 81f0003..fc7f059 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -207,6 +207,7 @@ External claims are made only where a public benchmark exists and the full proto |our previous teacher (r6) |580M |2.5793 |GLM-5.2 (reproduced, zero-skip) |— |2.6911 |GLM-5.3-Flash (reproduced, zero-skip, reasoning_effort=low) |— |8.7978 +|GLM-5.3 (reproduced, zero-skip, reasoning_effort=low) |— |9.8971 |Gemini Flash 2.0 (published) |— |3.1926 |GPT-4 (published) |— |3.8645 |Sadeed (published) |1.5B |7.2915 @@ -214,7 +215,7 @@ External claims are made only where a public benchmark exists and the full proto |our client student 1.0 (ByT5-small) |300M |8.259 |=== -A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. +A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp. The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.