diff --git a/TODO.publish/01-glm-4-7-row-completion.md b/TODO.publish/01-glm-4-7-row-completion.md new file mode 100644 index 0000000..eaa95bc --- /dev/null +++ b/TODO.publish/01-glm-4-7-row-completion.md @@ -0,0 +1,36 @@ +# 01 — glm-4.7-flash SadeedDiac-25 row completion + +Status: IN FLIGHT (2026-09-02). Fetch state: 992/1,200 distinct rows +landed, 208 still empty after three passes — the endpoint sits behind +sustained 429s (code 1305; thinking-disabled accepted, plain +completion inexpressible). Clean single fetch pass running +(/tmp/glm47_eval4.log; the #65 guard drops the 208 empties on +resume). Interim tables from contaminated interleaved passes are +VOID — only the final full-fetch tables count. + +## Protocol + +- model glm-4.7-flash, effort: none (display label + "thinking-disabled" = the #62 payload path: thinking.type=disabled) +- same 1,200-paragraph SadeedDiac-25 sweep, greedy contract, raw + + projected zero-skip, Misraj evaluator, full disclosure rows + +## Remaining steps + +- [ ] Fetch to todo=0 (retry passes until provider recovers; each + pass is checkpoint-resumable) +- [ ] Final raw + zero-skip tables (single clean process, no + interleaved writers) +- [ ] results/sadeed-glm-4-7-flash/{README.md, preds CSVs} + + RESULTS.md row + protocol ledger row + paper.adoc row +- [ ] Attribution decomposition (missing/wrong/extra + U+0670 rules; + r7 + GLM-5.2 controls) +- [ ] Paired bootstrap vs GLM-5.2 +- [ ] If provider stays blocked after sustained retries: record as + partial-coverage row with explicit disclosure (owner call, see + TODO.publish/05) + +## Unblock + +Completing this unblocks TODO.publish/04 (site frontier sentence +needs the full regression axis: 5.2 -> 4.7 -> 5.3 -> 5.3-Flash). diff --git a/TODO.publish/02-e6-verdict.md b/TODO.publish/02-e6-verdict.md new file mode 100644 index 0000000..00e4905 --- /dev/null +++ b/TODO.publish/02-e6-verdict.md @@ -0,0 +1,30 @@ +# 02 — E6 verdict (constant-budget register diversification) + +Status: IN FLIGHT (2026-09-02). Training COMPLETE — all 11,073 steps, +final CE ~0.003-0.008 (run-008-tashkeela-mix; /tmp/e6_distill2.log). +`modal_distill.py::main` does not chain the final eval, so +`evaluate_der` was launched separately (/tmp/e6_evalder.log, +resumable, writes final_eval.json). + +## Registered (EXPERIMENTS.md, gate before launch) + +- control: run-006-r7-muon verbatim schedule; delta = 8,000 Tashkeela + classical units replace 8,000 news units (30k total, identical + steps/schedule) +- gate: adopt at <= 4.5218 full-set windowed DER; honest report in + [4.5218, 4.8218); investigate if worse +- prediction: 4.45-4.75 +- data hygiene: 120,000 decontaminated units, 48 contaminated + dropped, 17,036 dups removed (rababa-tashkeela v1.0) + +## Remaining steps + +- [ ] evaluate_der full-set verdict vs gate +- [ ] EXPERIMENTS.md E6 status + RESULTS.md +- [ ] PUBLICATION-NOTES: the E5(-)/E6 rung pair — MTP-aux failed at + +0.26pp (confound disclosed), data-side diversification is the + data-vs-architecture answer either way +- [ ] If adopted: ara-diac-small-2.x material — release is an owner + version decision (see TODO.publish/05) +- [ ] Record training CE curve vs run-006 at matched steps (register + shift visible in convergence speed is itself a finding) diff --git a/TODO.publish/03-d-nikud-vendor-rows.md b/TODO.publish/03-d-nikud-vendor-rows.md new file mode 100644 index 0000000..dda7637 --- /dev/null +++ b/TODO.publish/03-d-nikud-vendor-rows.md @@ -0,0 +1,34 @@ +# 03 — D-Nikud vendor rows (Hebrew Dicta ledger) + +Status: COMPLETE (2026-09-02) — with the premise CORRECTED: D-Nikud +publishes NO numbers on the three Dicta ACL 2020 corpora (verified +against arXiv 2402.00075: their tables are internal 5% splits and +the Nakdimon test pipeline; repo README has no numbers). What was +recorded instead: their Nakdimon test-set table (5 systems, +DEC/CHA/WOR/VOC) as vendor-published ledger rows + the cross-metric +non-comparability disclosure + the angle-bracket matres-lectionis +protocol note. See rababa/docs/RESULTS.md Dicta section. + +## Design + +- D-Nikud (arXiv 2402.00075, NadavShaked/D_Nikud: T5BERT + Bi-LSTM, + ~1.5M-token dataset) published numbers on the same three public- + domain Dicta sets (ACL 2020 test corpora) — add as vendor-published + rows beside ours in the RESULTS.md Hebrew Dicta section. +- Vendor rows are context, not protocol-equal comparison: their + metrics are character-level diacritics accuracy (plus nikud-letter + handling per their paper), not our DER; their decode is theirs. + The ledger row must disclose this asymmetry explicitly (the + standard since the GLM-5.3 attribution correction: never let a + cross-protocol number stand next to ours unqualified). +- License: test corpora public domain; D-Nikud numbers cited from + the paper with arXiv ref. + +## Remaining steps + +- [ ] Pull D-Nikud's paper table numbers for the three Dicta sets + (verify against the paper, not blog/README summaries) +- [ ] Vendor rows + protocol-disclosure note in rababa/docs/ + RESULTS.md Hebrew section +- [ ] Cross-ref in paper.adoc Hebrew section if the numbers appear + there too diff --git a/TODO.publish/04-site-frontier-sentence.md b/TODO.publish/04-site-frontier-sentence.md new file mode 100644 index 0000000..9c8a346 --- /dev/null +++ b/TODO.publish/04-site-frontier-sentence.md @@ -0,0 +1,28 @@ +# 04 — Site frontier sentence (interscript.org ml page) + +Status: BLOCKED on TODO.publish/01 (needs the full regression axis). +The ml.astro page exists on interscript.org (rebrand 2026-08); what +it lacks is the one-line frontier-context claim that the GLM sweep +now substantiates. + +## The claim (draft, number slots to fill from 01) + +Frontier LLMs regress on classical-knowledge diacritization as their +optimization shifts agentic: on SadeedDiac-25, GLM-5.2 {5.2-DER} -> +glm-4.7-flash {4.7-DER} -> GLM-5.3 {9.90} -> GLM-5.3-Flash {8.80} +raw, versus dedicated distilled students at {r6-DER} — the dedicated +model is not nostalgia, it is the only thing on the right side of +the axis. Attribution: the regression is wrong-haraqat, not +convention (U+0670 = 0.125pp total; rababa #69 / ml #116). + +- One sentence + the numbers, linking to the protocol ledger + (rababa docs/RESULTS.md) for the disclosed rows +- Bootstrap CIs quoted from the ledger (paired vs 5.2) + +## Remaining steps + +- [ ] Fill slots from 01's final tables (or record partial-coverage + disclosure if that is the outcome) +- [ ] PR on interscript.org release-branch pattern (never main) +- [ ] Mirror check: the sentence must match ledger numbers exactly — + never-fabricate rule applies to marketing copy too diff --git a/TODO.publish/05-owner-decisions.md b/TODO.publish/05-owner-decisions.md new file mode 100644 index 0000000..ffbf632 --- /dev/null +++ b/TODO.publish/05-owner-decisions.md @@ -0,0 +1,34 @@ +# 05 — Owner-decision register (publish campaign) + +Status: REGISTERED (2026-09-02). Everything in this campaign that is +the owner's call, surfaced in one place. Cross-reference: the fuller +register with rationale lives in TODO.training-work/08; this file is +the publish-facing view. Nothing here is executed without an explicit +instruction naming it. + +## Decisions pending + +1. **GKD ordering** — the last registered training lever (SOTA + strategy 2025: "GKD is next"). Runs after E6's verdict so the + rung ladder stays one-variable-at-a-time. Owner decides: launch + now, after E6, or wait for the E5/E6 writeup. +2. **glm-4.7 partial-coverage policy** — if the 429 wall never + clears: record as partial-coverage row with disclosure, or drop + the row. (Default per protocol honesty: disclose coverage in the + row itself.) +3. **ara-diac-small-2.x release** — iff E6 passes the gate + (<= 4.5218). Version number is always the owner's decision + (rubygems lesson applies to model indices too). +4. **head32 swap-in shape** — in-place + index-v2 vs parallel + `-int8-head32` ids. Five-of-five rebuilds done with flip CIs; + swap-in is a release-side act. +5. **fp16 index entries for ara-diac-2.0** — export-side, low risk, + owner sequencing. +6. **rababa PR backlog** — #51/#52/#53/#48/#49/#26 (from + TODO.training-work/08). + +## Standing constraints (do not re-derive) + +- Version numbers, releases, tags: owner only. +- PRs rebase-merge only on explicit instruction; never main. +- Protocol-matched numbers only; partial coverage disclosed in-row. diff --git a/TODO.qwen-next/01-margin-aware-parity-gates.md b/TODO.qwen-next/01-margin-aware-parity-gates.md new file mode 100644 index 0000000..6ffab47 --- /dev/null +++ b/TODO.qwen-next/01-margin-aware-parity-gates.md @@ -0,0 +1,68 @@ +# 01 — Margin-aware parity gates (adopt now) + +Source: Qwen3.8-Flash-Next quantization practice (Unsloth day-0 analysis) — +the dense backbone tolerates aggressive quantization, but random-access +tensors (embedding tables, output heads, PLE lookup tables) degrade fast +below 4-bit. Precision floor is a property of **how a tensor is read**, +not just its size. They validate with KL-divergence on logits, not +output equality alone. + +## Gap in our stack + +`ml-models/src/imf/parity.py` gates releases on CER-delta between the +torch reference and the ONNX zip (0.2pp fp32 → 3.0pp int4). Our byte +students have **flat top-1 margins** (the greedy-vs-beam lesson: tha-small +published 12.06 PER was really 2.85). On flat distributions, quantization +noise below exact-match detection can still flip near-tie argmaxes — a +golden-set CER gate can pass while decode fragility ships. + +## Deliverables + +1. `run_margin_analysis(model, zip_path, pairs)` in `src/imf/parity.py`: + teacher-forced forward on both sides (torch decoder vs ONNX decoder + session), per-token top1−top2 logit margins, argmax flip rate, KLD. +2. `MarginReport` dataclass + JSON serialization; written next to the zip + as `-margins-.json` (diagnostic, not a release blocker + yet — schema churn avoided). +3. Unit test on the fixture checkpoint (`make_fixture_checkpoint`). +4. Real validation: khm-latn-1.0-fp16.zip (local) + one int8/int4 zip. +5. Policy (documented in docs/EXPERIMENTS.md): embeddings + lm_head + + any memory tables are a separate quantization class — stay fp16 (≥int8 + at minimum) when the body is quantized. + +## Protocol (paper-ready) + +- Probe set: the parity golden inputs (same pairs the CER gate uses). +- Metrics: flip_rate = fraction of teacher-forced positions where + argmax(torch) ≠ argmax(onnx); margin quantiles (p1/p10/p50) of the + reference; KLD mean; per-precision. +- Hypothesis: flip_rate correlates with cer_delta but detects fragility + earlier; int4 artifacts show near-tie flips invisible to the CER gate. + +## Status + +- [x] run_margin_analysis + MarginReport in parity.py +- [x] Wired into modal_export parity gate (every export emits margins) + + read-only `margins` entrypoint for published zips +- [x] Unit test (fixture checkpoint, synthetic near-tie logits) — 8 pass +- [x] **Full-catalog table measured (12 rows, docs/EXPERIMENTS.md E1)**: + fp32 exact everywhere; fp16 benign except khm 2.93% (flattest + margins, p50 0.121); int8 0.26–9.34%. **heb-diac-1.1 int8 = the + outlier: 9.34% flips, only 20% near-tie** — 80% of argmax flips at + confident positions, invisible to the CER gate. Open item: + re-examine the Hebrew int8 artifact (per-channel quantization or + serve fp16) + add a margin threshold to the release policy. + tha-g2p-small int4: 0.26%, all near-tie — flat-but-consistent. +- [x] **Root cause found + export default fixed (fcbe5d3)**: the outlier + was the quantized *head*. Probes on the same pairs: per-channel + alone 8.50% (+25% size, rejected); **head-fp32 body-int8 → 0.26% + flips (36×), KLD 47× lower, 100% near-tie, +0.4% size**. + `export_zips` now excludes `/lm_head/MatMul` from int8 by default. + Shipped int8 zips predate it; re-export = release decision (open). +- [x] Policy registered in docs/EXPERIMENTS.md (E1) +- [x] **Corrected-artifact sweep (2026-08-29, authorized)**: + `rebuild_int8_head32` (PR #76) rebuilds every shipped int8 zip + with the head in fp32 and gates it (parity in-zip + margin + budget). Artifacts land as {mid}-int8-head32.zip; swap-in is a + version decision pending results. Sweep running across khm/urd×2/ + heb/tha; tha int4 left as-is (benign: 0.26%, all near-tie). diff --git a/TODO.qwen-next/02-pkm-memory-layer-student.md b/TODO.qwen-next/02-pkm-memory-layer-student.md new file mode 100644 index 0000000..28bd500 --- /dev/null +++ b/TODO.qwen-next/02-pkm-memory-layer-student.md @@ -0,0 +1,74 @@ +# 02 — PKM memory-layer student (client-tier v2, the big one) + +Sources: +- arXiv 2601.21204 "Scaling Embeddings Outperforms Scaling Experts" + (LongCat-Flash-Lite: 68.5B params, >30B in embedding/lookup tables, + ~3B active — embedding scaling beats expert scaling on the Pareto + frontier in specific regimes). +- Qwen3.8-Flash-Next: first public model on that recipe (51B N-gram/PLE + tables, ~6B active). +- Precedent at our modality: product-key memory was demonstrated on + **character-level** LM (Lample et al., NeurIPS 2019, enwik8/text8). + +## Why this is the right experiment for us + +Wrong turn 1 measured: sub-100M from-scratch byte students collapse; a +pretrained backbone is non-negotiable; the client tier is pinned at +ByT5-small 300M → 246 MiB int8 (Arabic student: 8.26 DER full-set vs +teacher 2.58 — a 5.68pp gap). + +The LongCat result reframes the axis: **parameters and compute are +separable**. Keep compute at ByT5-small, add lookup capacity. Haraqat +restoration is heavily lexical knowledge (word identity + local context) +— table-friendly, not compute-friendly. It is also the architectural +version of our r6/r8 aux-task finding: tagged knowledge injection works; +a memory layer is where that knowledge can physically live. + +## Design (as launched) + +- Backbone: `google/byt5-small` (pretrained, per the non-negotiable rule). + Measured at launch: 300M base, d_model 1472, **4 decoder blocks** + (ByT5 depth lives in the encoder) — memory layers on decoder blocks + [-1, -2, -3]. +- Memory: product-key factorization, C=128 keys/half → 16,384 slots, + top-k=32 sparse reads → **+85.9M params (+29%)** at near-zero FLOPs. + Zero-init output gate preserves the pretrained function at step 0 + (unit-tested: logits bit-identical pre/post injection). +- Training: identical to ara-diac-small run-002 — teacher r6 frozen, + same r5-units corpus (29,322 pairs), same `teacher_labels_v2.jsonl` + (copied into the run dir; labels trusted complete — no relabeling), + same seed → **single-variable comparison**. + +## Protocol + gate (pre-agreed) + +- Spec `ara-diac-small-pkm` in ml-models/src/gpu/modal_distill.py. +- Gate: windowed zero-skip Misraj DER-CE, full 1,200 paragraphs, + ≤ 3.07 (the run-002 gate) — and the comparison target is run-002's + 8.259 full-set student / 2.5815 teacher. +- Verdict rule: PKM wins if it closes ≥1.0pp of the 5.68pp gap at equal + decode-time compute (memory reads are gathers, not matmuls). If no + movement, the capacity story is wrong — the gap is modeling/optimization, + publishable as a negative either way. +- Onnx export (opset-14 TopK/Gather) only if it wins; noted, not built yet. + +## Launch + +- A10G distill slot (never competes with A100 teacher runs) — fits under + the ≤2-big-GPU-apps budget alongside r8. 10,995 steps, ~7h. +- Detached; chain watchdog crash-relaunches, then runs + evaluate_der, then launches the Muon A/B arm (TODO 03), then evals it. + +## Status + +- [x] pkm.py (PKMLayer + byt5 injection helper) + smoke tests (6 pass) +- [x] Spec wiring in modal_distill.py (train + eval load paths) +- [x] Launched detached: rababa_arabic_distill_small/run-003-pkm +- [x] Registered in ml-models docs/EXPERIMENTS.md (E2) +- [x] Engagement probe (`pkm_gates`): step-500 gates 0.0008/0.0028/0.0019 + — all three off zero; memory branch engaged (CE 2.07 → 0.086 @ 900) +- [x] **VERDICT (2026-08-28): 7.5553 full-set DER vs run-002's 8.259 — + 0.704pp of the 5.677pp gap closed (12.4% relative), below the + pre-registered ≥1.0pp win bar.** Positive direction, honestly + reported; capacity is real but not the dominant term. Gate + engagement verified (gates off zero, CE 2.07→0.02). Muon arm (E3) + now tests the optimization half of the remaining gap. diff --git a/TODO.qwen-next/03-muon-optimizer-ab.md b/TODO.qwen-next/03-muon-optimizer-ab.md new file mode 100644 index 0000000..8e7b570 --- /dev/null +++ b/TODO.qwen-next/03-muon-optimizer-ab.md @@ -0,0 +1,46 @@ +# 03 — Muon optimizer A/B (one cheap test, low expected gain) + +Source: Qwen3.8-Flash-Next / LongCat-Flash-Lite trained with Muon +(orthogonalized momentum via Newton–Schulz); "~1/9th training cost" claims +circulate for from-scratch pretraining. + +Our prior: fine-tunes are **knowledge-limited, not optimization-limited** +(RL was a measured negative; data/supervision-side levers won). So the +expected gain on a 3-epoch distill is small — but the test is cheap and +runs in the same A10G slot after the PKM run. + +## Design + +- `muon.py` in ml-models/src/gpu/: single-file Muon — Newton–Schulz + orthogonalization applied to 2D weight matrices (hidden/FFN/attn + projections); embeddings, layer norms, and the lm_head stay on AdamW + (standard split; embeddings are not orthogonalizable objects). +- Flag `optimizer: muon|adamw` in the distill spec (default adamw — + nothing changes for existing specs). +- A/B: same spec as TODO 02 (`ara-diac-small-pkm`), same seed 42, only + the optimizer differs → run-003-pkm (adamw) vs run-004-pkm-muon. + +## Protocol + gate + +- Metric: windowed DER-CE on the val slice during training + full + 1,200-paragraph Misraj at the end; wall-clock per step recorded. +- Adopt if: ≥0.3pp DER improvement at equal steps AND no stability + regressions (loss spikes, grad-norm blowups) AND step overhead <15%. +- Either direction is paper-reportable (training-methods appendix): + "Muon vs AdamW on byte-level distillation fine-tunes" is unmeasured + territory for seq2seq students. + +## Status + +- [x] muon.py (Newton–Schulz, param-group split; shared.weight routing + fixed for transformers 5.x) + unit tests +- [x] Spec `ara-diac-small-pkm-muon` (run-004) wired in modal_distill.py +- [x] **VERDICT (2026-08-28): 4.8287 vs 7.5553 — −2.727pp from the + optimizer alone; adopt gate (≥0.3pp) exceeded 9×. ADOPTED.** + Training CE ~0.007 vs ~0.02 at equal steps; ~1.2s/step vs ~3.4s; + no stability events. Gap decomposition: ≈0.70pp capacity + + 2.73pp optimization + 2.25pp residual. +- [x] Factorial cell 4 (vanilla+Muon, run-005-muon) landed 2026-08-28: + **5.2945** — the 2×2 closes cleanly (optimizer alone −2.96pp, + memory alone −0.70pp / −0.47 under Muon, combined −3.43pp). + Paper carries the full factorial table. diff --git a/TODO.qwen-next/04-speculative-decoding-probe.md b/TODO.qwen-next/04-speculative-decoding-probe.md new file mode 100644 index 0000000..c52fb75 --- /dev/null +++ b/TODO.qwen-next/04-speculative-decoding-probe.md @@ -0,0 +1,22 @@ +# 04 — Speculative decoding probe (PARKED) + +Source: LongCat-Flash-Lite converts embedding sparsity into inference +speed via speculative decoding; Qwen ships day-0 vLLM support. + +Ours: byte-level outputs are long (1400B window → up to 3200 tokens), so +the theory applies — a tiny byte draft model + the student as verifier +could cut server-tier batch decode latency. + +Why parked: +- Teachers run offline; the API serves greedy small students that are + already fast for their workloads. +- We have no latency-budget data showing decode time binds at the API. +- The draft model would itself need training + a parity story — not free. + +Revisit trigger: API latency metrics showing p95 decode > budget, or a +client-tier memory-layer win (TODO 02) that inflates compute enough for +draft/verify to matter. Registered here so the idea isn't lost. + +## Status + +- [x] Parked with explicit revisit trigger (no code, by design) diff --git a/TODO.qwen-next/05-paper-experiment-registration.md b/TODO.qwen-next/05-paper-experiment-registration.md new file mode 100644 index 0000000..b54e29e --- /dev/null +++ b/TODO.qwen-next/05-paper-experiment-registration.md @@ -0,0 +1,36 @@ +# 05 — Paper readiness: experiment registration + +Rule (never fabricate): numbers enter docs/RESULTS.md only when +harness-verified; paper.adoc claims only after a RESULTS.md entry exists. +Before results exist, experiments are **registered** — hypothesis, +protocol, and pre-agreed gate recorded in advance. This is both honest +and what reviewers want to see (pre-registration kills the garden-of- +forking-paths objection on single-run comparisons). + +## Deliverable + +`ml-models/docs/EXPERIMENTS.md` — registry with one entry per experiment: + +- PKM memory-layer student (TODO 02): hypothesis, protocol, 1.0pp verdict + rule, comparison targets (student 8.259 / teacher 2.5815 full-set). +- Margin-aware parity (TODO 01): probe protocol, metrics, policy. +- Muon A/B (TODO 03): adopt-gate, equal-steps protocol. + +Each entry carries `Status: in-flight` until the harness writes the +number; the registry diff itself is the pre-registration evidence. + +## Paper mapping (when results land) + +- PKM result (either direction) → paper.adoc student-frontier subsection: + "parameters ≠ compute: lookup memory on a frozen pretrained backbone" + — extends the pretrained-or-collapse finding with the *constructive* + axis. Negative result → sharpens the collapse story instead. +- Margin gates → the decode-correction/provenance subsection (extends + "our decode was lying" with "our gate was under-reading"). +- Muon → training-methods appendix. + +## Status + +- [x] docs/EXPERIMENTS.md written (3 registered entries) +- [ ] RESULTS.md entries when runs land +- [ ] paper.adoc subsections when RESULTS entries exist diff --git a/TODO.qwen-next/06-glm-5-3-flash-learnings.md b/TODO.qwen-next/06-glm-5-3-flash-learnings.md new file mode 100644 index 0000000..77c1cab --- /dev/null +++ b/TODO.qwen-next/06-glm-5-3-flash-learnings.md @@ -0,0 +1,96 @@ +# 06 — GLM-5.3-Flash learnings, mapped to our stack + +Sources: model card (huggingface.co/zai-org/GLM-5.3-Flash, fetched +2026-08-31), GLM-5 technical report (arXiv 2602.15763), and the +reasoning-effort research PR (merged 2026-08-31). Scope: what applies +to interscript-ml / rababa, what doesn't, and why — recorded so we +don't re-derive it. + +## 1. reasoning_effort defaults to MAX — audit + patch (DONE) + +GLM-5.3-Flash treats an absent **or unrecognized** `reasoning_effort` +as MAX. Our eval's disable mechanism (`thinking: {"type": "disabled"}`, +the GLM-4.x/5.2 knob) is exactly an unrecognized value there. + +- **Audit**: `eval_sadeed_glm.py` is the *only* LLM API call site + across rababa, ml-models, secryst, and api (grep for z.ai / + bigmodel / chat/completions / openai / anthropic, venv excluded). + Everything else is docs/leaderboard references. +- **Our own quantification** (2026-08-17 commit 5cf5cd1): reasoning + mode burned *minutes per long paragraph*; plain completion made the + 1,200-paragraph sweep tractable. On 5.3-Flash a missing parameter + silently re-pays that — ~1h → days, no error to catch. +- **Patch (rababa PR #59)**: `glm-5.3*` refuses to start without an + explicit effort value; both knobs sent when given; any response + with `reasoning_content` prints a WARNING (tripwire proving the + disable didn't take); effort level is in the checkpoint filename + (different effort = different protocol = separate resume state) and + the startup protocol line. +- **Rule (standing)**: every LLM evaluation must disclose the full + decode protocol — temperature, reasoning effort/disabled, max + tokens — next to its numbers. An invalid effort string ALSO falls + back to MAX, so the response tripwire is part of the protocol, not + decoration. + +## 2. Applicability map, learning by learning + +| GLM-5.3-Flash feature | Verdict for us | Why | +|---|---|---| +| reasoning_effort default-max | **adopted** (§1) | silent latency trap at our one call site | +| native multimodal | not applicable | byte-level text seq2seq students; no modality gap to exploit | +| ExtractBench short 96.3 (structured extraction) | low priority | candidate for *non-label* data ops (metadata extraction from corpora, error-triage); the no-LLM-teacher rule for haraqat labels stands | +| hybrid sparse + linear attention | paper note only | our windows are ≤1400 B — attention is not the serving bottleneck; full quadratic attention staying affordable at 300–580 M is itself a client-tier data point | +| mHC hyper-connections | speculative | adjacent to the microkimi bridge/stitch observations (representation geometry), but nothing actionable on frozen ByT5 students | +| 320 B total / 18 B active at 1/10 price | framing, no code | resonates with our dedicated-vs-frontier positioning: the leaderboard compares our 580 M dedicated model against exactly this class of generalist | +| async agent RL infra + algorithms (GLM-5 paper) | **closed — do not revisit** | RL teacher polishing is flat/negative ×3 for us; knowledge-limited at SFT convergence (TODO/RESULTS run-001, run-002) | +| 30 T-token multimodal pretraining | not applicable | avoiding that scale is the point of the client tier | +| MIT license | n/a today | we ship no LLM-generated weights or outputs | + +## 3. Leaderboard refresh (RUNNING 2026-08-31) + +API access unblocked (the 2026-08-17 HTTP 403 is gone). Live probe of +`api.z.ai/api/paas/v4` established the exact contract: + +- valid `reasoning_effort` values are **exactly low, high, max** — no + "minimal"/"medium" +- `thinking: {"type": "disabled"}` is REJECTED outright on glm-5.3-flash: + HTTP 400 code 1210, "This model always engages in thinking and cannot + be disabled" — thinking cannot be turned off at all, only dialed +- `reasoning_effort: "low"` on a short prompt returns zero reasoning + tokens and no reasoning_content — effectively the old plain-completion + protocol; long paragraphs still engage some thinking (the in-run + reasoning_content tripwire fires intermittently, as designed) +- consequence for the script (rababa PR #62): effort set → send + `reasoning_effort` ONLY; the both-knobs form 400s every call + +**COMPLETE 2026-08-31 (rababa PR #63, results/sadeed-glm-5-3-flash/)**: +raw **8.5721/6.5335**, zero-skip **8.7978/6.6368** (Total/Morph DER); +WER 30.84/24.16 raw. ~3.4x worse DER than GLM-5.2 (2.5060 raw / +2.6911 zero-skip) and behind Sadeed-1.5B — the frontier's newest +generalist REGRESSED on this classical-knowledge task while our +dedicated 580M teacher improved (2.2864). dedicated 580M teacher improved (2.2864). CORRECTED DRIVER +(2026-09-01, rababa #69): wrong haraqat 10.05% of positions vs +GLM-5.2's 2.64% (= our r7's 2.62% to 0.02pp); missing 1.01%, extra +0.20%, convention effect 0.125pp total — the regression is genuine +mark errors, not orthography (that reading was wrong; the dagger-alif +marks explain raw-mode evaluator skips only). + the thinking floor. Paper row added +(interscript-ml PR #100) with the protocol footnote. + +Measurement lesson (worth a paragraph in the paper's measurement- +discipline section): an initial pass resumed 140 checkpoint rows that +the pre-#62 payload had retried into empty strings — Total DER read +15.96, contaminated by 11.7% catastrophic empties. Resumed checkpoints +must be validated for empty/error sentinel rows, not just presence. +Also: the user's own retry-loop on the shared /tmp checkpoint will +still carry those 140 empties — flagged, not touched. + +## 4. What we deliberately do not adopt + +- **Async RL anything** — closed negative territory for + diacritization (see map above). +- **LLM-as-teacher for labels** — hallucinated haraqat, standing + rule; multimodality and extraction strength do not change it. +- **Frontier-model distillation into the client tier** — our gap + decomposition (E2/E3 factorial) attributes the remaining student + gap to optimization + domain coverage, not label quality from a + stronger teacher; the r7 teacher already provides fresh labels. diff --git a/TODO.qwen-next/07-tencent-hy4-learnings.md b/TODO.qwen-next/07-tencent-hy4-learnings.md new file mode 100644 index 0000000..6cfc941 --- /dev/null +++ b/TODO.qwen-next/07-tencent-hy4-learnings.md @@ -0,0 +1,92 @@ +# 07 — Tencent Hy4-preview learnings, mapped to our stack + +Source: huggingface.co/tencent/Hy4-preview model card (fetched +2026-09-01; released days earlier — 770B total / 49B active, 78 +layers, 1M context). Companion: Hy4-preview-FP8. The card is candid +that it is a preview and discloses little pre-training recipe — the +substance is architectural + serving-level, with one directly +actionable training direction for us. + +## 1. iHC (identity Hyper-Connections, 4 residual streams) — ACT ON THIS + +Hy4 routes inter-layer information through 4 identity-based residual +streams. GLM-5.3-Flash ships the mHC variant (TODO 06). Two frontier +labs independently adopting hyper-connections is a signal, and it +lands exactly on our open frontier rung: the stitch-down from +ByT5-small pretraining (ridge-fit width bridge, per +PUBLICATION-NOTES). Hyper-connections were designed for the problem +our ridge bridge solves ad hoc — preserving trainable identity paths +across width surgery. Candidate E-item: width-stitch with a +hyper-connection reparameterization vs the ridge bridge, same data +and budget as the existing frontier rungs (controlled, comparable to +the layerdrop rung). This is a paper-B experiment, not a client-tier +change. + +## 2. Native MTP head (10B/0.7B active, 3 speculative tokens) — UN-PARK TRIGGER? + +Hy4 ships multi-token prediction natively for speculative decoding. +Our TODO 04 (speculative decoding probe) is parked on the condition +"API latency data shows p95 decode binding". Check the IMF runtime +benchmarks (RESULTS.md, E1 node tier): if greedy KV decode dominates +wall time for the client tier, an MTP-style extra head on the student +is the cheap version of this — diacritization output is locally +byte-predictable (input letter + haraqat pattern), so a 2-3x step +reduction is plausible at ~0.2% size cost. Decision needed: pull the +benchmark numbers and either open the E-item or re-park with the +measured latency as the recorded reason. + +## 3. Gated DSA + IndexCache (arXiv 2603.12201) — paper note only + +Sparse attention with cross-layer index reuse; same family as +GLM-5.3-Flash's hybrid sparse+linear. Our windows are <=1400B — +attention is not the bottleneck, and quadratic attention staying +cheap at 300-580M is part of the client-tier story. One related-work +line: IndexCache's "compute routing once, reuse across layers" is the +serving-side cousin of what our KV cache does for decode state. + +## 4. What the card does NOT give us + +No pre-training data/curriculum, optimizer, RL algorithm, load +balancing, or long-context recipe — "we scaled model size, context, +and data" plus "a substantially larger post-training run". The only +post-training substance: expert-built task data per domain (SWE, +office, game-dev, science) co-designed with products (CodeBuddy, +WorkBuddy). That is the pattern our r7 news-domain adaptation +followed at our scale — a validation, not a new method. + +## 5. Evaluation methodology contrast (paper-useful) + +Their headline comparison is 163 internal experts, 203 tasks, blind +side-by-side vs GLM 5.3 / Kimi K3 (win/tie/loss) — strong but +non-reproducible. Our external rows are public-protocol, +reproducible-by-construction (the same benchmark+evaluator+decode +disclosure that caught the GLM-5.3-Flash orthography regression). One +sentence for the paper's evaluation section: internal blind pairwise +and public-protocol benchmarks are complementary; only the latter +adversarially constrains vendor-reported gains. + +## 6. reasoning_effort defaults "high", opt-out via chat_template_kwargs no_think + +Third vendor with a non-obvious reasoning knob (GLM-5.3-Flash: +absent/invalid -> MAX, thinking unrejectable; Hy4: default high, +no_think via template kwargs). Standing rule reinforced: every LLM +row must disclose the full decode protocol, and the response-side +reasoning_content tripwire stays. If we ever add Hy4 to the SadeedDiac +LLM rows: read the opt-out semantics BEFORE trusting any "plain" +run. + +## Not applicable (recorded for completeness) + +- 770B/49B active MoE, 256 experts top-8 + shared — more evidence for + the parameters-!=compute axis, no action for us. +- 1M context training recipe — undisclosed; our windows are bounded. +- FP8 companion artifact — our fp16/int8 policy is set by E1. + +## Cross-reference (2026-09-01) + +07-hy4-learnings.md is the canonical Hy4 verdict — its MTP framing +(as a TRAINING auxiliary, not serving-side speculation; decode is +measured non-binding at our sizes per benches E1/E2) overrides the +serving framing in my section 2, and its domain-data prioritization +feeds the 08 plan. My unique content: the evaluation-methodology +contrast and the reasoning_effort knob survey. diff --git a/TODO.qwen-next/08-improvement-comparison-plan.md b/TODO.qwen-next/08-improvement-comparison-plan.md new file mode 100644 index 0000000..46ab296 --- /dev/null +++ b/TODO.qwen-next/08-improvement-comparison-plan.md @@ -0,0 +1,97 @@ +# 08 — Improvement & comparison plan (2026-09-01) + +Two halves: (A) make the models better, (B) make the *comparisons* +better. Every experiment stays pre-registered (EXPERIMENTS.md, gate +before launch); every comparison carries its protocol. Items marked +DECISION are owner calls (versions, spend, priorities). + +## A. Improve + +### A1. Close the student's domain gap (the measured 2.2pp residual) +The E2/E3 factorial attributed the remaining client-tier gap to +domain coverage, not capacity or optimization. Two rungs, in priority +order (07-hy4 raises the first): + +- **A1a. Tashkeela++ / label-scale rung** (rababa PR #1, open): + scale + diversify teacher labels beyond the news mix — classical, + literary, and wiki domains. Gate: student full-set DER beats 4.8218 + by >=0.3pp (the E3-style adopt bar). Cost: one A100 distill run per + label mix; labeling is r7-side and cheap. +- **A1b. On-policy GKD** (registered in the paper's discussion): + student-rollout loss against teacher targets. Highest upside, most + plumbing (needs interleaved teacher scoring of student prefixes). + Gate: >=0.5pp over the A1a rung; ship only with the E4-style + pre-registration note. + +### A2. MTP-aux distillation rung (E5 candidate, from 07-hy4) +Multi-token-prediction heads as a *training* auxiliary on the +ByT5-small student (per-position multi-step targets densify +supervision; NOT serving-side speculation — decode is measured +non-binding). Probe: MTP-aux head + Muon, same labels as 2.0, gate +>=0.3pp over 4.8218 at <1% size delta. Cheap; runs on the existing +distill chain. + +### A3. Teacher-side rungs (paper-facing) +- **r9 = r7 + curriculum on the orthography axis**: the GLM-5.3-Flash + result showed the benchmark punishes Quranic-convention outputs; a + deliberately mixed-orthography training mix (with a held-out + convention-switch probe) is both a teacher improvement and a paper + finding (conventions are learnable, not model-scale-bound). +- **Hebrew modern-text surface**: s46 is Biblical/Rabbinic-strong; + run the D-Nikud/modern benchmarks once to complete the Hebrew + comparison table (eval-only, no training). + +### A4. Artifact tier (release decisions — DECISION) +- head32 swap-in for the 4 shipped int8 zips (options + table in + EXPERIMENTS.md E1): in-place + index-v2 vs parallel -int8-head32 + ids. Verdict is in; only the version call remains. +- tha-g2p-small head32 once its fp32 export lands (in flight). +- fp16 index entries for ara-diac-2.0 (gated at 0.0903, unpublished). + +## B. Compare + +### B1. Statistical footing (extend what just landed) +- Paired bootstrap CI (main as of f93ac41) applies to every + before/after DER; extend it to (a) leaderboard deltas between our + rows (r7 vs r6: is 0.29pp significant at 1,200 paragraphs?), and + (b) head32 vs shipped-int8 margin flips. Standing rule: no delta is + quoted in docs/paper without its CI or an explicit note that the + run predates the policy. + +### B2. Protocol disclosure ledger +One table in RESULTS.md (and paper appendix) rowing every external +comparison by: benchmark, evaluator, decode protocol (greedy/beam, +temp, reasoning knob), skip policy, and reproducibility (public +protocol vs internal blind pairwise — the Hy4 contrast). Includes +our own rows. This makes the GLM-5.3-Flash reasoning-effort caveat a +*structural* feature, not a footnote. + +### B3. Third evaluation surface +ID (SadeedDiac-25) + OOD (WikiNews multi-ref) exist. Add a small +**orthographic-convention probe** (50-100 paragraphs where valid +conventions differ: dagger alif, ʾalif qunyā, hamza carriers) scored +multi-reference-style. Cheap, reuses the multiref evaluator, and +converts the GLM regression into a measurable axis. Also feeds A3. + +### B4. Reproduction cadence for external rows +- Re-run the LLM row of record when a new frontier model lands + (GLM-5.3-Flash took ~2h + ~$ of API). Standing checklist: reasoning + knob semantics FIRST (three vendors, three shapes), empty-sentinel + guard (rababa #65), zero-skip + raw both, RESULTS + paper row same + day. +- Claude/Gemini/GPT-4 rows are published-numbers only; keep them + marked as such in B2's ledger. + +### B5. Client-facing comparison +The site's ml ledger gains the 2.0 rows' numbers + the GLM comparison +context sentence (dedicated 580M vs newest frontier generalist). +Small, after A4 lands. + +## Order of operations (proposal) + +1. A4 decision (unblocks index work) — DECISION +2. A2 MTP-aux probe + A1a label-scale rung in parallel (both ride the + existing chain; A100 budget allows the pair) +3. B1 CI extension + B2 ledger (docs-only, no compute) +4. A3 orthography teacher rung + B3 probe together +5. A1b GKD after A1a's verdict; B5 at the end diff --git a/TODO.qwen-next/09-fable-5-1-learnings.md b/TODO.qwen-next/09-fable-5-1-learnings.md new file mode 100644 index 0000000..c0b79f4 --- /dev/null +++ b/TODO.qwen-next/09-fable-5-1-learnings.md @@ -0,0 +1,57 @@ +# 09 — Claude Fable 5.1 / Mythos 5.1 learnings, mapped to our stack + +Sources: the announcement (anthropic.com/claude-fable-and-mythos-5-1, +fetched 2026-09-01) and the news index (anthropic.com/news). Verified +after the initial web search returned only SEO/rumor noise with +contradictory claims — the guessed `/news/claude-fable-5-1` URL 404'd; +the real page is the combined Fable-and-Mythos announcement dated +2026-09-01. Scope: same as 06/07 — what applies to interscript-ml / +rababa, what doesn't, and why. + +## 1. Reasoning-effort defaults are per-surface (fourth data point) + +"Fable 5.1 defaults to High effort in Claude Code, and to Medium in +Claude Cowork and on Claude.ai." The default for the *same model* +varies by product surface, because the vendor is tuning +cost/latency/quality per context. + +- Extends the 06/07 survey: GLM (absent/invalid → MAX), Hy4 (default + high, `no_think` opt-out), Anthropic (per-surface defaults). +- **Rule reinforced**: never rely on a provider default at a call + site — it is not even stable *within* one vendor's model across + their own surfaces. Our `eval_sadeed_glm.py` pattern (explicit + knob, refusal to start without one on glm-5.3*, effort in the + checkpoint filename + protocol line, `reasoning_content` tripwire) + is the right shape for every future API eval. + +## 2. Applicability map, item by item + +| Fable 5.1 item | Verdict for us | Why | +|---|---|---| +| per-surface effort defaults | **adopted** (§1) | 4th data point for the knob survey; validates explicit-knob rule | +| reward-hacking audits via "natural language autoencoders on internal thinking" | paper-note only | interpretability on *their* reasoning traces; our byte-level seq2seq students have no thinking traces. Worth one line in the paper's interpretability/future-work context, nothing actionable now | +| strengthened distillation defenses (thinking-transcript context-editing restricted) | no impact | we distill our *own* teacher (rababa r6/r7 → students), never a hosted frontier model — and LLM-as-teacher for diacritization is banned outright (hallucinated haraqat) | +| invisible text watermarking + private-preview detection API | no impact | we ship weights as sha-pinned artifacts, not LLM text; our corpora are classical/protected texts, not model output. If we ever ingest web-scale LLM-generated Arabic/Hebrew text into training, watermark taint becomes a provenance question — current hygiene (decontamination scans, dedup) is the mitigation | +| cache reads cut to $0.25/M (25–45% workload savings) | technique noted | provider-specific (Anthropic); our GLM evals run on z.ai. General lesson applies if we ever host on Anthropic: put the static instruction/prompt prefix first and identical across requests so it caches — our sweep template already has that shape | +| safety-FP reductions (bio 85% fewer benign fires, cyber 60% fewer FPs) | n/a | eval-convenience for agentic coding, not a training technique | +| agentic benchmarks at the top (Terminal-Bench-Science 52.6, Terminal-Bench 4.0 55.8, GDPval-AA 1853; 2.5x GPU-kernel speedups, ~50% protein-binder hit rate) | **thesis support** | frontier effort targets agentic/science work, not classical-knowledge tasks — same signal as the GLM-5.3 regression we measured (wrong-haraqat axis up vs 5.2). Dedicated distilled models remain the right call for diacritization | +| protein-binder / GPU-kernel agentic optimization | n/a | no analog in diacritization; no reward-verifiable iteration loop for haraqat (RL already ruled out — knowledge-limited, not exploration-limited) | + +## 3. What is NOT disclosed (and why that matters) + +No architecture, no RL algorithm, no training-data or pretraining- +compute details, no synthetic-data or agent-training method, no +curriculum, no effort-level implementation, no watermark algorithm. +Same wall as 06 (GLM) and 07 (Hy4): frontier labs publish capability +and safety framing, not mechanics. Nothing here is borrowable for our +distillation rungs — the levers stay on our side: data scale/register +diversity (E6), supervision quality, GKD — not architecture copying. + +## 4. Standing conclusions (delta over 06/07) + +- The knob survey is complete enough to freeze: **every** future LLM + evaluation must state the explicit effort/thinking setting next to + its numbers; provider defaults are per-surface and unstable. +- Frontier-vs-dedicated thesis now has three corroborations: our + GLM-5.3 measurement (9.90 raw, worse than 5.2 on the wrong-haraqat + axis), GLM/Hy4 release framing, and Fable 5.1's benchmark choices. diff --git a/TODO.training-work/01-glm-5-3-row-completion.md b/TODO.training-work/01-glm-5-3-row-completion.md new file mode 100644 index 0000000..d07251d --- /dev/null +++ b/TODO.training-work/01-glm-5-3-row-completion.md @@ -0,0 +1,24 @@ +# 01 — GLM-5.3 row completion + +Status: COMPLETE (2026-09-01, rababa #71) — recorded: 9.8971/7.8219 +zero-skip, 9.9760/7.9285 raw; attribution + bootstrap + ledger row +in the PR. Paper row still to add (see below). Previously: 1,200/1,200 collected at +reasoning_effort=low; 41 exhausted-retry empties detected by the #65 +guard and re-fetching now (run log /tmp/glm53_full_eval2.log). + +## Remaining steps + +- [ ] Final tables once todo=0 (raw + projected zero-skip; the + contaminated first-pass tables are void) +- [ ] results/sadeed-glm-5-3/{README.md, preds CSVs} + RESULTS.md row +- [ ] Per-position attribution decomposition (missing/wrong/extra + + U+0670 convention normalization; r7 + GLM-5.2 as controls) — + the standard since rababa #69 +- [ ] Paired bootstrap vs GLM-5.3-Flash and GLM-5.2 +- [ ] Paper leaderboard row + protocol ledger row + PUBLICATION-NOTES + +## Protocol facts (probed 2026-09-01) + +glm-5.3 rejects thinking.type=disabled with HTTP 400 code 1210 (same +as Flash); reasoning_effort=low accepted, 0 reasoning tokens on short +prompts, tripwire fires on long paragraphs. diff --git a/TODO.training-work/02-glm-4-7-flash-row-completion.md b/TODO.training-work/02-glm-4-7-flash-row-completion.md new file mode 100644 index 0000000..f7ede78 --- /dev/null +++ b/TODO.training-work/02-glm-4-7-flash-row-completion.md @@ -0,0 +1,22 @@ +# 02 — GLM-4.7-Flash row completion + +Status: IN FLIGHT (2026-09-01). Old API semantics verified by probe: +thinking.type=disabled ACCEPTED (plain completion expressible), but +the endpoint sits behind sustained 429s (code 1305) — running at +GLM_WORKERS=1, ~2 rows/min aggregate; ETA overnight. Checkpoint +resumes; empties self-heal (#65). + +## Remaining steps + +- [ ] Same recording pipeline as 01: final tables (raw + zero-skip), + results dir + README, RESULTS.md row, attribution + decomposition, paired bootstrap vs GLM-5.2 (its successor) — + this row measures where the pre-5.x Flash line stood on + classical Arabic +- [ ] Paper: cite only if the row adds information beyond GLM-5.2 + (owner call at record time) + +## Note + +The GLM_WORKERS knob (rababa #70) exists because of this endpoint; +if 429s persist tomorrow, resume is idempotent — just relaunch. diff --git a/TODO.training-work/03-e5-mtp-aux-verdict.md b/TODO.training-work/03-e5-mtp-aux-verdict.md new file mode 100644 index 0000000..b80d928 --- /dev/null +++ b/TODO.training-work/03-e5-mtp-aux-verdict.md @@ -0,0 +1,30 @@ +# 03 — E5 (MTP-aux) verdict + +Status: COMPLETE 2026-09-01 (ml #129) — GATE FAILED: 5.0853 (gate +<=4.5218; +0.26pp over the 4.8218 control; prediction missed). NOT +ADOPTED. Confound disclosed in EXPERIMENTS.md (preemption + fresh-head +resume for the final 23% of steps). Nothing shipped. Disclosures for the verdict: run was +preempted at step ~8,650 (Modal-side cancellation), resumed from +step-8,500 whose mtp_head.pt write was interrupted — final ~2,495 +steps trained with a re-initialized aux head (student+optimizer state +intact; resume loss 0.73 = 0.15*ln(384) + CE confirms the diagnosis). +Two launch bugs fixed en route: #115 spec-splat, #117 device. + +## Registered (EXPERIMENTS.md, gate before launch) + +- control: run-006-r7-muon verbatim; delta = MTPHead(3 steps, + beta 0.15, 1.70M params / 0.57%), discarded at inference +- gate: adopt at <= 4.5218 full-set windowed DER; honest report in + [4.5218, 4.8218); investigate if worse +- prediction: 4.5-4.75 + +## Remaining steps + +- [ ] When training completes: evaluate_der full-set (the chain + writes final_eval.json; resumable) +- [ ] Verdict vs gate -> EXPERIMENTS.md status + RESULTS.md +- [ ] If adopted: this becomes ara-diac-small-2.x material — release + is an owner version decision (see 08) +- [ ] Record the training CE curve comparison vs run-006 at matched + steps (the aux head's effect on convergence speed is itself a + finding either way) diff --git a/TODO.training-work/04-label-scale-rung.md b/TODO.training-work/04-label-scale-rung.md new file mode 100644 index 0000000..b6875bc --- /dev/null +++ b/TODO.training-work/04-label-scale-rung.md @@ -0,0 +1,34 @@ +# 04 — Label-scale rung (A1a): diversify the student's domain mix + +Status: TO BUILD. The E2/E3/E4 factorial attributes the remaining +~2.0pp of client-tier gap to domain coverage; the current mix is +news-heavy (r5-units/domain.txt + replay.txt, teacher-labeled by r7). + +## Design + +- Source: Tashkeela (rababa-tashkeela v1.0, the CI already pulls it — + github.com/interscript/rababa-tashkeela) — classical/literary + registers, complementary to the news mix +- Units: split to <=1450B paragraph units like r5-units (same + splitter as the rababa training scripts), dedupe against the + existing domain.txt units, decontaminate against SadeedDiac-25 + (the contamination check from the r3 era: exact + near-dup on + stripped text) +- Labels: r7 teacher greedy (the chain labels automatically from + train file paths — no manual step) +- Spec: ara-diac-small-2-scale, control identical to run-006/E5's + base, unit_limits scaled to keep total steps comparable +- Gate (pre-register in EXPERIMENTS.md before launch): >=0.3pp over + 4.8218 full-set (E3-style adopt bar) +- Prediction: 4.3-4.6 if the domain attribution is right + +## Remaining steps + +- [ ] Pre-register E6 in EXPERIMENTS.md with the gate above +- [ ] Download Tashkeela, build + decontaminate units (CPU, local or + Modal volume put) +- [ ] Add spec to distill_specs.yaml (train_extra with the new unit + file) +- [ ] LAUNCH ONLY WHEN A GPU SLOT FREES (E5 + the owner's layerdrop + run occupy the budget; <=2 big GPU apps) +- [ ] Verdict + record when done diff --git a/TODO.training-work/05-head32-flip-bootstrap.md b/TODO.training-work/05-head32-flip-bootstrap.md new file mode 100644 index 0000000..97a1a1b --- /dev/null +++ b/TODO.training-work/05-head32-flip-bootstrap.md @@ -0,0 +1,33 @@ +# 05 — Head32-vs-shipped flip bootstrap (B1 extension) + +Status: COMPLETE (2026-09-01, ml #120/#121 code + ml #124 record). +heb-diac: 10.985% -> 0.288% flips, delta -10.696pp, 95% CI +[-11.661, -9.726] — the repair is overwhelming. khm-latn: 2.325% -> +2.302%, CI [-0.173, +0.125] — a proper null (nothing was broken +there; head32's role is consistency). Probe-set note disclosed in +EXPERIMENTS.md. Original design below for provenance. +Status: TO BUILD. The paired-bootstrap policy covers DER deltas; the +margin flips (head32 vs shipped int8) still lack CIs because +MarginReport persists aggregates only — per-position flip vectors +are computed then discarded. + +## Design + +- `imf.parity.run_margin_analysis(..., dump_positions=Path)` — + append-only JSONL of {pair_idx, pos, ref_margin, flipped} rows + next to the margins JSON (diagnostic artifact, schema-additive) +- Rerun margin analysis for the two SHIPPED int8 zips and their + head32 twins on the same pairs (khm + heb: one flat-ish, one the + outlier — the informative pair), CPU on Modal, read-only w.r.t. + artifacts +- Paired bootstrap on per-position flip indicators: is head32's + flip-rate reduction significant per model, and is the + confident-flip reduction (the release-relevant number) bounded + away from zero at 95%? + +## Remaining steps + +- [ ] dump_positions parameter + unit test (fixture checkpoint) +- [ ] Modal runs for khm/heb shipped-vs-head32 (4 margin analyses) +- [ ] Bootstrap + record in EXPERIMENTS.md E1 (one CI line per + artifact family) + the ledger's policy note diff --git a/TODO.training-work/06-site-ledger-frontier-context.md b/TODO.training-work/06-site-ledger-frontier-context.md new file mode 100644 index 0000000..b0ca347 --- /dev/null +++ b/TODO.training-work/06-site-ledger-frontier-context.md @@ -0,0 +1,14 @@ +# 06 — Site ledger: 2.0 rows + frontier context (B5) + +Status: BLOCKED on 01/02 landing (needs the final rows), then small. + +## Steps + +- [ ] interscript.org/src/pages/ml.astro: the ara-diac-2.0 / + ara-diac-small-2.0 ledger rows are live from the releases — + add the frontier-comparison sentence (dedicated 580M teacher + at 2.2864 vs the newest frontier generalists measured under + the same protocol: the GLM-5.3 / 5.3-Flash rows from 01/02) + once those numbers are recorded +- [ ] PR to interscript.org -> site (release-branch ref pattern per + convention) diff --git a/TODO.training-work/07-hebrew-modern-text-surface.md b/TODO.training-work/07-hebrew-modern-text-surface.md new file mode 100644 index 0000000..2294692 --- /dev/null +++ b/TODO.training-work/07-hebrew-modern-text-surface.md @@ -0,0 +1,36 @@ +# 07 — Hebrew modern-text surface (A3b) + +Status: COMPLETE (2026-09-01, rababa #72 script + #73 record). +modern_wiki 0.5209 / poetry 0.2377 / rabbinic 0.4523 greedy DER — +dense GT (~0.8 marks/letter), line-level, our-row-only. The Hebrew +weak surface is specifically Nakdimon-style paragraph-level Biblical +text. Research outcome: + +- **The instrument**: Dicta's diacritization test corpora + (github.com/Dicta-Israel-Center-for-Text-Analysis/ + hebrew-diacritization-test-corpora) — ACL 2020 demo, explicitly + **public domain**, 3 sets: Modern (HebrewWiki), Poetry, Rabbinic. + This is the modern-text surface the recent open systems (D-Nikud, + arXiv 2402.00075; visual-representation approaches) report on. + Cloned locally at /tmp/hebrew-diacritization-test-corpora. +- D-Nikud itself: code at github.com/NadavShaked/D_Nikud (TevBERT + + Bi-LSTM); its ~1.5M-token dataset is the training-scale reference — + the test corpora above are the cleaner comparison instrument. + +## Remaining steps + +- [ ] Point the rababa Hebrew eval harness at the three sets + (windowed protocol, same as the Nakdimon row; strip-diacritics + input, Misraj-style DER on haraqat + nikud letters where the + corpus marks them) +- [ ] Run the s46 artifact; record all three DERs beside the + Nakdimon row in the protocol-ledger format (license: public + domain — distributable, no constraint) +- [x] Cross-reference: D-Nikud's published numbers on the same sets + (paper table) as vendor-published rows in the ledger — + CORRECTED 2026-09-02: D-Nikud publishes NO numbers on these + three corpora (paper tables = internal 5% splits + Nakdimon + test pipeline; repo README has none). Their Nakdimon test-set + table is ledgered as vendor rows instead, with the + cross-metric (DEC/CHA/WOR/VOC vs DER) non-comparability + disclosed. See RESULTS.md Dicta section. diff --git a/TODO.training-work/08-owner-decisions.md b/TODO.training-work/08-owner-decisions.md new file mode 100644 index 0000000..36cd298 --- /dev/null +++ b/TODO.training-work/08-owner-decisions.md @@ -0,0 +1,25 @@ +# 08 — Awaiting owner decisions (register — not implementable from here) + +Version numbers, release shapes, and priorities are the owner's. +Everything below has its evidence recorded; none of it is blocked on +me. + +- **head32 swap-in shape**: replace the five shipped int8 zips + in place + cut index-v2 (consumers re-download; sha pins change + deliberately) vs parallel `-int8-head32` ids (no migration). All + five verdicts + the comparison table live in EXPERIMENTS.md E1. +- **fp16 index entries for ara-diac-2.0** — gated at cer_delta + 0.0903, assets built, unpublished. +- **A1b GKD ordering** — after 04's label-scale verdict per the 08 + plan; highest upside, most plumbing. +- **ara-diac-small-2.x release** if E5 (03) passes its <=4.5218 + gate. +- **rababa #51-53** (torch/onnx floor bumps — CI already tests + latest via --upgrade-strategy eager; the pins are contract-only), + stale **#48/#49** (2025, now conflicting with green main), **#26** + (legacy-model rspec expectations). +- **r9 orthography-mix teacher rung**: RE-SCOPED DOWN by evidence — + the attribution correction (rababa #69) measured the entire + dagger-alif convention effect at 0.125pp, and our teacher matches + GT conventions natively. Keep only if a reviewer asks for the + convention axis explicitly. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 8a5a69d..3176c10 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -456,6 +456,30 @@ specifically the Naknimon-style paragraph-level Biblical/Rabbinic test (16.44) — register coverage, not a general modern-text gap. dicta_eval.json beside the checkpoint carries the raw result. +Vendor cross-reference (verified 2026-09-02): D-Nikud (arXiv +2402.00075) publishes NO numbers on these three corpora — the +paper's tables are its internal 5% splits (Table 2, CHA/WOR) and +the Nakdimon test pipeline (Table 3); the repo README carries no +numbers either. TODO 07's "D-Nikud's published numbers on the same +sets" premise was wrong (corrected there). The nearest vendor +surface is the paper's Nakdimon test-set table — DEC/CHA/WOR/VOC: +D-Nikud 98.39/97.15/90.76/93.44, Nakdan (Dicta) 97.95/96.77/94.11/ +94.92, Nakdimon 97.91/96.37/89.75/91.64, Morfix 96.84/94.92/90.38/ +92.39, Snopi 91.29/85.84/76.45/78.91 — ledgered below as +vendor-published. These are token-level macro accuracies under +their decode on their split; not mappable to DER without re-running +their systems, so no cross-metric comparison is drawn. + +Protocol note (angle brackets): the corpora README's intended +protocol removes the matres-lectionis angle brackets from the input +and checks their removal in the output. Our harness keeps brackets +in both input and GT (`seq2seq_der` parses them as units; the model +sees them and can copy them through), so the matres-removal +subtask is not tested and bracket units sit in the denominator. +Numbers stand as recorded, our-row-only; a README-conformant re-run +(bracket letters stripped) is the follow-up if this surface becomes +load-bearing. + ## Comparison protocol ledger (2026-09-01) Every external comparison we cite, rowed by what was actually run. @@ -476,6 +500,7 @@ row here does not belong in the paper. | Gemini-Flash-2.0 (3.1926) / GPT-4 (3.8645) | published numbers | theirs | unknown | no — vendor-published | | Sadeed-1.5B (7.2915) | published, same benchmark | theirs (their repo reports 1.2 under its own split — not comparable) | unknown | partially — paper + code, split differs | | Hebrew rows (s46 16.43 / DictaBERT 35.63) | Nakdimon Biblical test, run by us | beam-4 (ours) / vendor default | per-evaluator | ours yes; Dicta run-by-us on their artifact | +| D-Nikud Nakdimon-test table (DEC/CHA/WOR/VOC, 5 systems incl. Nakdan/Nakdimon) | Nakdimon test set, their evaluation flow | theirs, undisclosed to us | unknown | no — vendor-published (arXiv 2402.00075 Table 3); no D-Nikud numbers exist on the Dicta ACL 2020 corpora | | Tencent Hy4-preview (win/tie/loss vs GLM 5.3 / Kimi K3) | 203 tasks, 163 internal experts, blind pairwise | `reasoning_effort` default high (`no_think` via template kwargs) | unstated | no — internal blind pairwise | ## Statistical footing for the headline deltas (2026-09-01)