Skip to content

docs: comparison protocol ledger — every row we cite, by what was actually run #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 31, 2026
Merged

docs: comparison protocol ledger — every row we cite, by what was actually run #67

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,3 +410,33 @@ none) and fails in its strong form (phonemic specifically). r6 stays the
canonical Arabic teacher. Script: train_arabic_r8.py; artifacts:
rababa_arabic_byt5/run-008-ipa (EVAL_DONE, ipa_probe.json,
sadeed_preds_windowed.csv).

## Comparison protocol ledger (2026-09-01)

Every external comparison we cite, rowed by what was actually run.
Ours are reproducible by construction; vendor rows are not, and the
difference is load-bearing — the same disclosure discipline that
exposed GLM-5.3-Flash's orthography regression. A number without its
row here does not belong in the paper.

| Row | Benchmark / evaluator | Decode protocol | Skip policy | Reproducible |
|---|---|---|---|---|
| our teacher r7 (2.2864/1.3343) | SadeedDiac-25, Misraj, full 1,200 | greedy, windowed zero-skip 1400B, temp-free argmax | zero skipped; haraqat projected | yes — harness + ckpt public |
| our teacher r6 (2.5793/1.5317) | same | same | same | yes |
| our client students (4.822 / 8.259) | same | same | same; full-set-only publication rule | yes |
| GLM-5.2 (2.5060 raw / 2.6911 zero-skip) | same evaluator | temp 0, `thinking.type=disabled` (plain completion) | raw: word-structure skips; zero-skip: projected | yes — script + CSVs (results/sadeed-glm-5-2/) |
| GLM-5.3-Flash (8.5721 raw / 8.7978 zero-skip) | same evaluator | temp 0, `reasoning_effort=low` (thinking CANNOT be disabled — API 400 code 1210) | same; dagger-alif convention skips; 0 empty responses after sentinel purge | yes — script + CSVs (results/sadeed-glm-5-3-flash/) |
| Claude-3.7-Sonnet (1.3941) | published number | their protocol, undisclosed to us | unknown | no — vendor-published |
| Gemini-Flash-2.0 (3.1926) / GPT-4 (3.8645) | published numbers | theirs | unknown | no — vendor-published |
| Sadeed-1.5B (7.2915) | published, same benchmark | theirs (their repo reports 1.2 under its own split — not comparable) | unknown | partially — paper + code, split differs |
| Hebrew rows (s46 16.43 / DictaBERT 35.63) | Nakdimon Biblical test, run by us | beam-4 (ours) / vendor default | per-evaluator | ours yes; Dicta run-by-us on their artifact |
| Tencent Hy4-preview (win/tie/loss vs GLM 5.3 / Kimi K3) | 203 tasks, 163 internal experts, blind pairwise | `reasoning_effort` default high (`no_think` via template kwargs) | unstated | no — internal blind pairwise |

Standing rules this ledger enforces: full-set-only student numbers;
subset figures labeled at first publication; every LLM row discloses
temperature, reasoning knob, and max tokens; resumed checkpoints are
validated for error sentinels, not just presence (rababa #65); new
frontier rows follow the reproduction checklist (knob semantics
first). Paired bootstrap CIs land with the deltas they qualify
(ml-models f93ac41 onward).