Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions docs/PUBLICATION-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,12 +177,19 @@ raw, 8.7978/6.6368 zero-skip** at reasoning_effort=low — ~3.4x worse
DER than GLM-5.2 (2.5060/2.6911), behind Sadeed-1.5B (7.2915). Two
protocol facts discovered live: the API rejects disabled thinking
outright (400 code 1210; valid efforts exactly low/high/max), so the
plain-completion protocol is inexpressible for this model; and the gap
is orthography-shaped (Quranic dagger-alif conventions vs the GT's
MSA; 9.8% not-fully-diacritized). Paper: leaderboard row + protocol
note (PR #100). This strengthens the dedicated-model thesis: the
frontier's newest generalist regressed on this classical-knowledge
task while the dedicated 580M teacher improved.
plain-completion protocol is inexpressible for this model. ATTRIBUTION
(2026-09-01, corrects the first orthography-shaped reading): the gap
is wrong haraqat — 10.05% of positions vs GLM-5.2's 2.64% (matching
our r7 teacher's 2.62% to 0.02pp); missing 1.01%, extra 0.20%, and
the whole dagger-alif U+0670 convention effect is 0.125pp (rules
derived from aligned positions; controls <=0.009pp). Paper: leaderboard
row + protocol note (PR #100, attribution corrected in PR after #69
in rababa). This strengthens the dedicated-model thesis: the
frontier's newest generalist lost classical-Arabic mark knowledge its
predecessor had, while the dedicated 580M teacher improved. The
per-position decomposition method (missing/wrong/extra +
convention-normalization with controls) is reusable — apply to any
future LLM row before interpreting its DER.

Sibling measurement finding (same day): resumable LLM-eval
checkpoints silently resume error sentinels — the first pass read
Expand Down
2 changes: 1 addition & 1 deletion docs/paper.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ External claims are made only where a public benchmark exists and the full proto
|our client student 1.0 (ByT5-small) |300M |8.259
|===

A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B — driven by Quranic-convention orthography (dagger-alif forms absent from the ground truth; 9.8% not-fully-diacritized words) and the reasoning floor. The frontier's newest generalist regressed on this classical-knowledge task while our dedicated 580M teacher improved; we record it rather than average it away.
A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away.

The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.

Expand Down
Loading