Skip to content

results: GLM-5.3 (full) on SadeedDiac-25 — 9.8971 zero-skip; 5.x regression is family-wide #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Sep 1, 2026
Merged

results: GLM-5.3 (full) on SadeedDiac-25 — 9.8971 zero-skip; 5.x regression is family-wide #71

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,21 @@ canonical Arabic teacher. Script: train_arabic_r8.py; artifacts:
rababa_arabic_byt5/run-008-ipa (EVAL_DONE, ipa_probe.json,
sadeed_preds_windowed.csv).

## GLM-5.3 (full) on SadeedDiac-25 (2026-09-01)

**9.8971/7.8219 zero-skip, 9.9760/7.9285 raw** (Misraj;
`results/sadeed-glm-5-3/`; 1,200/1,200, zero empties — 41 retry
sentinels caught by the #65 guard). reasoning_effort=low: glm-5.3
rejects disabled thinking with the same 400/1210 as Flash. The 5.x
generation regression is family-wide: GLM-5.2 2.6911 -> Flash 8.7978
(wrong haraqat 10.05%) -> full 5.3 9.8971 (missing 3.75%, wrong
8.59%) — while GLM-5.2's wrong-rate (2.64%) matches our dedicated
580M teacher (2.62%) to within 0.02pp. Flash significantly better
than the full model (bootstrap CI excludes zero); both decisively
behind 5.2 (CI [-18,028, -14,685] error positions). The dedicated-
model thesis gains its cleanest form: the newest generalist
generation lost classical-Arabic mark knowledge its predecessor had.

## Comparison protocol ledger (2026-09-01)

Every external comparison we cite, rowed by what was actually run.
Expand All @@ -433,6 +448,7 @@ row here does not belong in the paper.
| our client students (4.822 / 8.259) | same | same | same; full-set-only publication rule | yes |
| GLM-5.2 (2.5060 raw / 2.6911 zero-skip) | same evaluator | temp 0, `thinking.type=disabled` (plain completion) | raw: word-structure skips; zero-skip: projected | yes — script + CSVs (results/sadeed-glm-5-2/) |
| GLM-5.3-Flash (8.5721 raw / 8.7978 zero-skip) | same evaluator | temp 0, `reasoning_effort=low` (thinking CANNOT be disabled — API 400 code 1210) | same; dagger-alif convention skips; 0 empty responses after sentinel purge | yes — script + CSVs (results/sadeed-glm-5-3-flash/) |
| GLM-5.3 (9.9760 raw / 9.8971 zero-skip) | same evaluator | temp 0, `reasoning_effort=low` (same 400/1210 rejection) | same; 41 sentinels caught + refetched, 0 empties final | yes — script + CSVs (results/sadeed-glm-5-3/) |
| Claude-3.7-Sonnet (1.3941) | published number | their protocol, undisclosed to us | unknown | no — vendor-published |
| Gemini-Flash-2.0 (3.1926) / GPT-4 (3.8645) | published numbers | theirs | unknown | no — vendor-published |
| Sadeed-1.5B (7.2915) | published, same benchmark | theirs (their repo reports 1.2 under its own split — not comparable) | unknown | partially — paper + code, split differs |
Expand Down
39 changes: 39 additions & 0 deletions results/sadeed-glm-5-3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# GLM-5.3 (full) on SadeedDiac-25 (2026-09-01)

Same harness as every LLM row: neutral prompt, `temperature 0`,
structure-preserving cleanup, Misraj evaluator, 1,200/1,200 responses,
zero empty responses (41 exhausted-retry sentinels were caught by the
resume guard and re-fetched). Decode protocol: `reasoning_effort=low`
— glm-5.3 rejects `thinking: {"type": "disabled"}` outright (HTTP 400
code 1210), identical to 5.3-Flash; plain completion is inexpressible
for the 5.x generation.

## Results (Misraj evaluator, percentages)

| Protocol | DER (CE) | DER (w/o CE) | WER (CE) | WER (w/o CE) | NFDW |
|---|---|---|---|---|---|
| raw | **9.9760** | 7.9285 | 31.2581 | 25.0347 | 12.86 |
| projected zero-skip | **9.8971** | 7.8219 | 31.0748 | 24.6340 | 12.49 |

## The 5.x generation regression (attribution, same 1,200 paragraphs, convention-normalized)

| model | missing | **wrong haraqat** | extra | protocol |
|---|---|---|---|---|
| our r7 teacher | 0.11% | **2.62%** | 0.15% | greedy argmax |
| GLM-5.2 | 0.61% | **2.64%** | 0.17% | thinking disabled |
| GLM-5.3-Flash | 1.01% | **10.05%** | 0.20% | reasoning_effort=low |
| GLM-5.3 (full) | **3.75%** | **8.59%** | 0.11% | reasoning_effort=low |

Paired bootstrap (error-position deltas, 95% CI): Flash significantly
better than the full 5.3 (−2,152 positions, CI [−3,863, −546]); both
catastrophically behind GLM-5.2 (−16,251, CI [−18,028, −14,685]).
The two 5.x variants regress on different axes — Flash emits wrong
marks (10.05%), the full model leaves marks missing (3.75%) — but the
family verdict is one line: the 5.x generation lost the
classical-Arabic mark knowledge GLM-5.2 had (2.64%, matching our
dedicated 580M teacher's 2.62% to within 0.02pp). The convention
effect (U+0670) is 0.015pp here — negligible again.

## Files

- `sadeed_preds_raw.csv`, `sadeed_preds_projected.csv`
Loading