results: GLM-5.3-Flash on SadeedDiac-25 — 8.57/6.53 raw, 8.80/6.64 zero-skip - #63
Merged
Conversation
….6368 zero-skip First measurement of GLM-5.3-Flash (320B/18B active) on the benchmark; the 2026-08-17 403 cleared. reasoning_effort=low (the API rejects disabled thinking outright — HTTP 400 code 1210 — so the plain-completion protocol is not expressible for this model). ~3.4x worse DER than GLM-5.2 and behind Sadeed-1.5B: Quranic- convention orthography (dagger alif, 9.8% not-fully-diacritized words) against the benchmark's MSA conventions. Recorded as a dedicated-model data point. Measurement hygiene: an initial pass resumed 140 checkpoint rows that the pre-fix both-knobs payload had retried into empty strings — Total DER read 15.96 with 11.7% catastrophic empties. Rows purged and re-fetched; final numbers are 1,200/1,200 with zero empty responses. Also tracks sadeed_evaluator.py: eval_sadeed_glm.py imports it, but it lived only as an untracked local file — the eval was not reproducible from a fresh clone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First protocol-matched GLM-5.3-Flash row on SadeedDiac-25 (the
2026-08-17 attempt was blocked by HTTP 403; access cleared).
vs GLM-5.2 (thinking disabled): raw 2.5060/1.5537, zero-skip
2.6911/1.7179 — the newer model is ~3.4x worse on DER and lands
behind Sadeed-1.5B (7.2915).
Protocol delta the API forces: the model rejects
thinking: {"type": "disabled"}outright (HTTP 400 code 1210 —"always engages in thinking"); valid
reasoning_effortvalues areexactly low/high/max. Run at
low, the nearest analog to plaincompletion. Driver of the gap: Quranic-convention orthography
(dagger-alif forms the GT lacks; 9.8% not-fully-diacritized words)
plus the reasoning floor. Recorded as a dedicated-model data point —
the frontier's newest generalist regressed on this classical-knowledge
task relative to its predecessor.
Measurement hygiene: an initial pass resumed 140 checkpoint rows
that the pre-#62 both-knobs payload had retried into empty strings —
Total DER read 15.96 contaminated by 11.7% empties. Purged and
re-fetched: final numbers are 1,200/1,200, zero empty responses.
Also tracks
sadeed_evaluator.py(imports byeval_sadeed_glm.pybut previously lived only as an untracked local file — the eval was
not reproducible from a fresh clone).
Follow-ups in #59/#62: the eval script now refuses glm-5.3* without
an explicit effort and warns when responses carry reasoning_content.