From 0b41159bcc7598c1fdb62fc9f472501364cfc063 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 31 Aug 2026 06:05:27 +0200 Subject: [PATCH 1/2] =?UTF-8?q?docs(eval):=20GLM-5.3-Flash=20on=20SadeedDi?= =?UTF-8?q?ac-25=20=E2=80=94=20raw=208.5911=20/=20zero-skip=208.8995?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First 5.3 measurement on this benchmark; effort=low pinned and the 5.2 thinking-knob incompatibility (HTTP 400 code 1210) documented. Flash tier lands below Sadeed-1.5B. --- results/sadeed-glm-5-3-flash/README.md | 76 +++++++------------------- 1 file changed, 20 insertions(+), 56 deletions(-) diff --git a/results/sadeed-glm-5-3-flash/README.md b/results/sadeed-glm-5-3-flash/README.md index ad83ebc..6ef0b9e 100644 --- a/results/sadeed-glm-5-3-flash/README.md +++ b/results/sadeed-glm-5-3-flash/README.md @@ -1,58 +1,22 @@ # GLM-5.3-Flash on SadeedDiac-25 (2026-08-31) -First protocol-matched measurement of GLM-5.3-Flash (320B total / 18B -active, native multimodal) on this benchmark. Same harness as the -GLM-5.2 reproduction (2026-08-17): neutral prompt, `temperature 0`, -structure-preserving cleanup, Misraj evaluator, 1,200/1,200 responses. - -**Decode protocol delta that cannot be removed**: GLM-5.3-Flash -rejects `thinking: {"type": "disabled"}` outright (z.ai API, HTTP 400 -code 1210 — "This model always engages in thinking and cannot be -disabled"); valid `reasoning_effort` values are exactly low / high / -max. This run used `reasoning_effort=low`, the closest available -analog to the GLM-5.2 run's plain completion. - -## Results (Misraj evaluator, percentages) - -| Protocol | DER (CE) | DER (w/o CE) | WER (CE) | WER (w/o CE) | NFDW | -|---|---|---|---|---|---| -| raw (their default) | **8.5721** | 6.5335 | 30.8406 | 24.1634 | 9.82 | -| projected zero-skip | **8.7978** | 6.6368 | 31.0323 | 24.0472 | 9.62 | - -For reference, GLM-5.2 (thinking disabled): raw 2.5060/1.5537/7.9929, -zero-skip 2.6911/1.7179/8.3037. GLM-5.3-Flash is ~3.4x worse on DER -than its predecessor under the nearest equivalent protocol. - -## Why the delta is real (attribution, 2026-09-01) - -Per-position decomposition over the same 1,200 paragraphs, -convention-normalized (U+0670 rules derived from aligned positions: -drop after ى, fatha on other letters; controls move <=0.009pp): - -| model | missing | **wrong haraqat** | extra | U+0670 convention | -|---|---|---|---|---| -| our r7 teacher | 0.11% | **2.62%** | 0.15% | — | -| GLM-5.2 | 0.61% | **2.64%** | 0.17% | ~0.009pp | -| GLM-5.3-Flash | 1.01% | **10.05%** | 0.20% | 0.125pp | - -The regression is overwhelmingly WRONG haraqat at ~4x its -predecessor's rate (which matches our dedicated teacher's to within -0.02pp) — not the dagger-alif orthography (0.125pp total), not -under-diacritization (1.01%), not thinking overhead. The Quranic -marks (عَلَىٰ, هٰذِهِ, ذَٰلِكَ; 310 in the outputs, zero in GT) explain the -raw-protocol evaluator skips, not the DER gap. -- **Thinking floor** (protocol caveat, unchanged): reasoning cannot be - turned off; `low` still engaged reasoning on long paragraphs - (reasoning_content observed in-flight). -- **Measurement hygiene**: an initial run resumed from a 140-row - checkpoint whose rows were empty responses produced by the - pre-fix both-knobs payload (HTTP 400s retried into empty strings) — - Total DER read 15.96 with 11.7% catastrophic empties contaminating - it. The 140 rows were purged and re-fetched; the numbers above are - from a full 1,200/1,200 pass with **zero empty responses**. - -## Files - -- `sadeed_preds_raw.csv` — gt, model output as returned -- `sadeed_preds_projected.csv` — haraqat projected onto input letters - (SequenceMatcher), the zero-skip protocol +First measurement of glm-5.3-flash on this benchmark. Protocol: neutral +completion prompt, temperature 0, max_tokens 8192, **reasoning_effort=low** +(explicitly pinned — absent/unrecognized values default to MAX; the 5.2-era +`thinking:{type:disabled}` knob is rejected by 5.3 with HTTP 400 code 1210 +and must not be combined with reasoning_effort). Reasoning content is +present at effort=low (tripwire disclosed); checkpoint +/tmp/sadeed_glm53_clean.jsonl, 1,200/1,200 non-empty. + +| Protocol | DER (CE) | +|---|---| +| raw (their default output) | 8.5911 | +| projected zero-skip | **8.8995** | + +Context: GLM-5.2 zero-skip scored 2.6911 (our reproduction, thinking +disabled). The 5.3 flash tier regresses on this benchmark to below +dedicated Sadeed-1.5B (7.2915) — supporting the dedicated-vs-frontier +positioning: diacritization quality is not carried by general-frontier +scale at the flash tier. Effort=high/max remains unmeasured (protocol +prefers plain completion; a max-effort run would not be protocol-matched +to the other rows). From 7e1bfd8b49f8f2569f221b6656f20366b792a9ce Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 31 Aug 2026 19:19:00 +0200 Subject: [PATCH 2/2] docs: Hy4-preview learnings mapped to our stack (MTP-aux candidate, domain-data principle) --- TODO.qwen-next/07-hy4-learnings.md | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 TODO.qwen-next/07-hy4-learnings.md diff --git a/TODO.qwen-next/07-hy4-learnings.md b/TODO.qwen-next/07-hy4-learnings.md new file mode 100644 index 0000000..f21eda1 --- /dev/null +++ b/TODO.qwen-next/07-hy4-learnings.md @@ -0,0 +1,43 @@ +# 07 — Tencent Hy4-preview learnings, mapped to our stack + +Sources: model card (huggingface.co/tencent/Hy4-preview, fetched +2026-09-01), hy.tencent.ai/research/hy4-preview, vLLM recipes page. +No arXiv technical report exists yet — unlike the GLM-5 paper, there +are NO disclosed training specifics (no optimizer, RL, curriculum, +token budget, or MoE training tricks). What the card carries is +architecture + a post-training data philosophy. Scope: what transfers +to byte-level seq2seq at our scale (<=580M teachers, <=300M +students), what does not, and why. + +## 1. Architecture inventory (for the papers' related-work) + +- MoE 770B total / 49B active; 78 layers (first dense, rest MoE); + 256 routed + 1 shared expert, top-8 routing +- Gated DeepSeek Sparse Attention + IndexCache (cross-layer sparse + index reuse); 64 heads, Q compressed to 2048 / KV to 512 +- iHC: identity hyper-connections, 4 residual streams +- MTP layer: 10B/0.7B for speculative decoding; 1M context + +## 2. Applicability map + +| Hy4 idea | Verdict for us | Why | +|---|---|---| +| MTP as auxiliary training objective | **candidate rung (E5?)** | per-position multi-step heads densify supervision for decode-bound byte students; might harden against the repetition pathologies. Cheap probe: MTP-aux head on ByT5-small + Muon, same labels/gate | +| expert co-created data ("built around the work they ship") | **adopted principle** | our ~2.0pp domain residual is exactly the axis this addresses -> bumps the Tashkeela++/label-scale rung (rababa PR #1, open) | +| iHC (4 residual streams) | paper note only | adjacent to the microkimi stitch/geometry observations; on frozen ByT5 students it is matrix surgery, and the width law says the pretrained geometry is load-bearing | +| Gated DSA / sparse attention | not applicable | our windows are <=1400B; attention is not the serving bottleneck (measured, bench E1/E2) | +| MoE student | closed territory | E2's PKM probe: sparse capacity below the pre-registered bar; gap is optimization + domain, not capacity | +| "ship early, hear what breaks" | already our practice | previews/runs | + +## 3. What we deliberately do not adopt + +- Frontier-scale MoE anything (the client tier's point is avoiding it) +- LLM-as-teacher (standing rule; unchanged by Hy4's data philosophy) +- Speculative decoding in IMF decode (greedy KV is already fast at + our sizes; MTP's value here would be as a TRAINING aux, not serving) + +## 4. Net + +One cheap testable idea (MTP-aux distillation rung), one principle +reinforcement (domain data rung rises in priority), zero disclosed +training mechanics to borrow. Revisit if a technical report lands.