diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 63bdf99..eb24de0 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -410,3 +410,33 @@ none) and fails in its strong form (phonemic specifically). r6 stays the canonical Arabic teacher. Script: train_arabic_r8.py; artifacts: rababa_arabic_byt5/run-008-ipa (EVAL_DONE, ipa_probe.json, sadeed_preds_windowed.csv). + +## Comparison protocol ledger (2026-09-01) + +Every external comparison we cite, rowed by what was actually run. +Ours are reproducible by construction; vendor rows are not, and the +difference is load-bearing — the same disclosure discipline that +exposed GLM-5.3-Flash's orthography regression. A number without its +row here does not belong in the paper. + +| Row | Benchmark / evaluator | Decode protocol | Skip policy | Reproducible | +|---|---|---|---|---| +| our teacher r7 (2.2864/1.3343) | SadeedDiac-25, Misraj, full 1,200 | greedy, windowed zero-skip 1400B, temp-free argmax | zero skipped; haraqat projected | yes — harness + ckpt public | +| our teacher r6 (2.5793/1.5317) | same | same | same | yes | +| our client students (4.822 / 8.259) | same | same | same; full-set-only publication rule | yes | +| GLM-5.2 (2.5060 raw / 2.6911 zero-skip) | same evaluator | temp 0, `thinking.type=disabled` (plain completion) | raw: word-structure skips; zero-skip: projected | yes — script + CSVs (results/sadeed-glm-5-2/) | +| GLM-5.3-Flash (8.5721 raw / 8.7978 zero-skip) | same evaluator | temp 0, `reasoning_effort=low` (thinking CANNOT be disabled — API 400 code 1210) | same; dagger-alif convention skips; 0 empty responses after sentinel purge | yes — script + CSVs (results/sadeed-glm-5-3-flash/) | +| Claude-3.7-Sonnet (1.3941) | published number | their protocol, undisclosed to us | unknown | no — vendor-published | +| Gemini-Flash-2.0 (3.1926) / GPT-4 (3.8645) | published numbers | theirs | unknown | no — vendor-published | +| Sadeed-1.5B (7.2915) | published, same benchmark | theirs (their repo reports 1.2 under its own split — not comparable) | unknown | partially — paper + code, split differs | +| Hebrew rows (s46 16.43 / DictaBERT 35.63) | Nakdimon Biblical test, run by us | beam-4 (ours) / vendor default | per-evaluator | ours yes; Dicta run-by-us on their artifact | +| Tencent Hy4-preview (win/tie/loss vs GLM 5.3 / Kimi K3) | 203 tasks, 163 internal experts, blind pairwise | `reasoning_effort` default high (`no_think` via template kwargs) | unstated | no — internal blind pairwise | + +Standing rules this ledger enforces: full-set-only student numbers; +subset figures labeled at first publication; every LLM row discloses +temperature, reasoning knob, and max tokens; resumed checkpoints are +validated for error sentinels, not just presence (rababa #65); new +frontier rows follow the reproduction checklist (knob semantics +first). Paired bootstrap CIs land with the deltas they qualify +(ml-models f93ac41 onward). +