From 02cd8d4a6b0a815365b820d8fa42dc9945ca61c3 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 2 Sep 2026 17:59:49 +0200 Subject: [PATCH] =?UTF-8?q?docs(paper):=20glm-4.7-flash=20leaderboard=20ro?= =?UTF-8?q?w=20=E2=80=94=2013.2256=20zero-skip;=20axis=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourth row of the GLM family regression on SadeedDiac-25; the only family member regressing on both axes (missing 6.67%, wrong 9.01%); bootstrap vs GLM-5.2 +7.945pp CI [+7.516, +8.370]. Full record: rababa#75. --- docs/paper.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/paper.adoc b/docs/paper.adoc index fc7f059..ddbf02d 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -208,6 +208,7 @@ External claims are made only where a public benchmark exists and the full proto |GLM-5.2 (reproduced, zero-skip) |— |2.6911 |GLM-5.3-Flash (reproduced, zero-skip, reasoning_effort=low) |— |8.7978 |GLM-5.3 (reproduced, zero-skip, reasoning_effort=low) |— |9.8971 +|glm-4.7-flash (reproduced, zero-skip, thinking-disabled) |— |13.2256 |Gemini Flash 2.0 (published) |— |3.1926 |GPT-4 (published) |— |3.8645 |Sadeed (published) |1.5B |7.2915 @@ -215,7 +216,7 @@ External claims are made only where a public benchmark exists and the full proto |our client student 1.0 (ByT5-small) |300M |8.259 |=== -A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp. +A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp. The axis completes with glm-4.7-flash, the last plain-completion GLM row (thinking-disabled still accepted on 4.x): 13.2256 zero-skip, the worst frontier DER, and the only family member regressing on both axes at once (missing 6.67% — worst of the family — plus wrong 9.01%); +7.945pp behind GLM-5.2 by paired bootstrap (CI [+7.516, +8.370]). Every measurable GLM successor lost classical-Arabic mark knowledge its predecessor had. The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.