Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/paper.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -208,14 +208,15 @@ External claims are made only where a public benchmark exists and the full proto
|GLM-5.2 (reproduced, zero-skip) |— |2.6911
|GLM-5.3-Flash (reproduced, zero-skip, reasoning_effort=low) |— |8.7978
|GLM-5.3 (reproduced, zero-skip, reasoning_effort=low) |— |9.8971
|glm-4.7-flash (reproduced, zero-skip, thinking-disabled) |— |13.2256
|Gemini Flash 2.0 (published) |— |3.1926
|GPT-4 (published) |— |3.8645
|Sadeed (published) |1.5B |7.2915
|**our client student 2.0 (ByT5-small)** |300M |**4.822**
|our client student 1.0 (ByT5-small) |300M |8.259
|===

A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp.
A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp. The axis completes with glm-4.7-flash, the last plain-completion GLM row (thinking-disabled still accepted on 4.x): 13.2256 zero-skip, the worst frontier DER, and the only family member regressing on both axes at once (missing 6.67% — worst of the family — plus wrong 9.01%); +7.945pp behind GLM-5.2 by paired bootstrap (CI [+7.516, +8.370]). Every measurable GLM successor lost classical-Arabic mark knowledge its predecessor had.

The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.

Expand Down
Loading