From 21e138c1d1c0d5c5b3d652a9e45f8237c91aa82f Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 1 Sep 2026 06:32:32 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20GLM-5.3-Flash=20attribution=20correctio?= =?UTF-8?q?n=20=E2=80=94=20wrong=20haraqat=2010.05%,=20convention=200.125p?= =?UTF-8?q?p?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first reading (orthography-shaped) was wrong; the per-position decomposition with convention normalization and controls localizes the regression to wrong haraqat at ~4x GLM-5.2's rate — which matches our r7 teacher's to within 0.02pp. Missing 1.01%, extra 0.20%, the whole U+0670 effect 0.125pp (310 marks, zero in GT; controls <=0.009pp). Method note: apply the decomposition to any future LLM row before interpreting its DER. --- docs/PUBLICATION-NOTES.md | 19 +++++++++++++------ docs/paper.adoc | 2 +- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index ff4db83..8eff961 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -177,12 +177,19 @@ raw, 8.7978/6.6368 zero-skip** at reasoning_effort=low — ~3.4x worse DER than GLM-5.2 (2.5060/2.6911), behind Sadeed-1.5B (7.2915). Two protocol facts discovered live: the API rejects disabled thinking outright (400 code 1210; valid efforts exactly low/high/max), so the -plain-completion protocol is inexpressible for this model; and the gap -is orthography-shaped (Quranic dagger-alif conventions vs the GT's -MSA; 9.8% not-fully-diacritized). Paper: leaderboard row + protocol -note (PR #100). This strengthens the dedicated-model thesis: the -frontier's newest generalist regressed on this classical-knowledge -task while the dedicated 580M teacher improved. +plain-completion protocol is inexpressible for this model. ATTRIBUTION +(2026-09-01, corrects the first orthography-shaped reading): the gap +is wrong haraqat — 10.05% of positions vs GLM-5.2's 2.64% (matching +our r7 teacher's 2.62% to 0.02pp); missing 1.01%, extra 0.20%, and +the whole dagger-alif U+0670 convention effect is 0.125pp (rules +derived from aligned positions; controls <=0.009pp). Paper: leaderboard +row + protocol note (PR #100, attribution corrected in PR after #69 +in rababa). This strengthens the dedicated-model thesis: the +frontier's newest generalist lost classical-Arabic mark knowledge its +predecessor had, while the dedicated 580M teacher improved. The +per-position decomposition method (missing/wrong/extra + +convention-normalization with controls) is reusable — apply to any +future LLM row before interpreting its DER. Sibling measurement finding (same day): resumable LLM-eval checkpoints silently resume error sentinels — the first pass read diff --git a/docs/paper.adoc b/docs/paper.adoc index 833f3f8..81f0003 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -214,7 +214,7 @@ External claims are made only where a public benchmark exists and the full proto |our client student 1.0 (ByT5-small) |300M |8.259 |=== -A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B — driven by Quranic-convention orthography (dagger-alif forms absent from the ground truth; 9.8% not-fully-diacritized words) and the reasoning floor. The frontier's newest generalist regressed on this classical-knowledge task while our dedicated 580M teacher improved; we record it rather than average it away. +A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.