diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index c9ce241..ff4db83 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -169,3 +169,24 @@ overstatement now has three quantified instances - it is a finding, not a nuisance: domain-neighborhood evaluation subsets inflate student quality by 2-4x in this regime. Frame it in the measurement section as a generalizable warning with the three pairs as evidence. + +## GLM-5.3-Flash on SadeedDiac-25 (2026-08-31): frontier generalist regression + +First measurement of GLM-5.3-Flash (320B/18B active): **8.5721/6.5335 +raw, 8.7978/6.6368 zero-skip** at reasoning_effort=low — ~3.4x worse +DER than GLM-5.2 (2.5060/2.6911), behind Sadeed-1.5B (7.2915). Two +protocol facts discovered live: the API rejects disabled thinking +outright (400 code 1210; valid efforts exactly low/high/max), so the +plain-completion protocol is inexpressible for this model; and the gap +is orthography-shaped (Quranic dagger-alif conventions vs the GT's +MSA; 9.8% not-fully-diacritized). Paper: leaderboard row + protocol +note (PR #100). This strengthens the dedicated-model thesis: the +frontier's newest generalist regressed on this classical-knowledge +task while the dedicated 580M teacher improved. + +Sibling measurement finding (same day): resumable LLM-eval +checkpoints silently resume error sentinels — the first pass read +15.96 DER because 140 exhausted-retry empties were resumed as done. +Belongs in the paper's measurement-discipline paragraph with the +subset-inflation instances; the resume path now drops empty rows +(rababa PR #65). diff --git a/docs/paper.adoc b/docs/paper.adoc index f5f18f6..833f3f8 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -220,7 +220,7 @@ The teacher is the best dedicated (task-trained, runnable-locally) model measure The r6 teacher's auxiliary-task design is validated by a controlled ablation: an identical run differing only in the auxiliary stream's output representation — broad-phonemic IPA of the same training units (deterministic converter) in place of morphological analysis — scores 2.6588 (vs 2.6775 for no auxiliary task and 2.5793 for the morphological one), while a held-out probe confirms the phonemic projection itself was learned (2.3% CER on the IPA stream). The improvement is therefore attributable to lexical-morphological knowledge injection, not to phonemic supervision per se: the strong form of the "phonemes help diacritization" hypothesis fails where its weak form (any structured auxiliary projection beats none) barely holds. -The client student's row carries a measurement lesson we record rather than bury. Its first-published figure, 3.66, was measured on the first 300 paragraphs of the benchmark; the full 1,200-paragraph run scores 8.26 (teacher reproduces at 2.5815 against its documented 2.5793, confirming protocol consistency). The subset was not representative: its paragraphs sit closer to the student's training domain, and the remaining 900 expose a domain-generalization gap the subset hid. Two practices follow, now standing rules: student-tier numbers are published only from full benchmark sets, and a subset figure is labeled as such at first publication. The student ships with the full-set number disclosed — behind Sadeed-1.5B on the leaderboard — and closing its domain gap (more diverse label coverage, or on-policy distillation; <>) is open work. The measured levers have since moved it: the 2.0 rung (r7-teacher labels + the Muon optimizer, pre-registered as E4) reaches **4.822** — a 42% error reduction at identical architecture and artifact size — with the optimizer contributing −2.96pp and the teacher's fresher labels −0.47pp of the −3.44pp total; the factorial decomposition of the remaining gap is in <>. +The client student's row carries a measurement lesson we record rather than bury. Its first-published figure, 3.66, was measured on the first 300 paragraphs of the benchmark; the full 1,200-paragraph run scores 8.26 (teacher reproduces at 2.5815 against its documented 2.5793, confirming protocol consistency). The subset was not representative: its paragraphs sit closer to the student's training domain, and the remaining 900 expose a domain-generalization gap the subset hid. Two practices follow, now standing rules: student-tier numbers are published only from full benchmark sets, and a subset figure is labeled as such at first publication. A second, sibling lesson came from the newest external row: resumable API-evaluation checkpoints must be validated, not just resumed — exhausted retries were checkpointed as empty predictions, and a first GLM-5.3-Flash pass read 15.96 DER until the 140 sentinel rows were detected and re-fetched (the recorded number is 8.5721 raw / 8.7978 zero-skip, with the resume path now dropping empty rows by default). The student ships with the full-set number disclosed — behind Sadeed-1.5B on the leaderboard — and closing its domain gap (more diverse label coverage, or on-policy distillation; <>) is open work. The measured levers have since moved it: the 2.0 rung (r7-teacher labels + the Muon optimizer, pre-registered as E4) reaches **4.822** — a 42% error reduction at identical architecture and artifact size — with the optimizer contributing −2.96pp and the teacher's fresher labels −0.47pp of the −3.44pp total; the factorial decomposition of the remaining gap is in <>. Two protocol notes the benchmark's own authors make and we adopt: self-reported numbers from other protocols (Sadeed's repository reports 1.2% DER under its own split) are not comparable, and multi-reference evaluation (Mohamed & Mubarak 2025) is the honest treatment of valid-alternative diacritizations; on WikiNews-2024 our teacher scores 17.38 WER against QCRI's in-domain-trained 2.70 — a domain gap (classical hadith vs news text), recorded rather than averaged away.