Qwen3.8-Flash-Next techniques: margin gates (measured), PKM memory student (in flight), Muon A/B

docs/EXPERIMENTS.md

@@ -0,0 +1,149 @@
+# Experiment registry
+
+Pre-registered experiments: hypothesis, protocol, and gate recorded
+**before** results exist. Numbers graduate to RESULTS.md only when
+harness-verified; paper.adoc claims only after a RESULTS.md entry.
+The registry diff is the pre-registration evidence.
+
+Origin: the Qwen3.8-Flash-Next technique review (2026-08-27; basis paper
+arXiv 2601.21204, LongCat-Flash-Lite — verified). Full working notes:
+`TODO.qwen-next/` in the rababa working tree (uncommitted by design).
+
+---
+
+## E1 — Margin-aware parity gates
+
+- **Status:** implemented + validated across every shipped artifact;
+  policy adopted.
+- **Hypothesis:** byte students have flat top-1 margins, so quantization
+  flips near-tie argmaxes at KLD ~1e-5 — invisible to the CER-delta
+  release gate.
+- **Protocol:** teacher-forced forward on both sides (torch decoder vs
+  ONNX zip graphs) over the parity probe pairs (first 300 test pairs per
+  model); per-position top1−top2 reference margins, argmax flip rate,
+  KL(reference||zip), share of flips below the corpus p10 margin.
+  `imf.parity.run_margin_analysis`; emitted by every `modal_export
+  parity` gate and standalone via `modal_export margins` (read-only for
+  published zips; JSONs on secryst-models:/imf/<model>/).
+- **Measured across the catalog (2026-08-27):**
+
+| Artifact | Precision | Flips | Rate | KLD | ref. margin p50 | near-tie share |
+|---|---|---|---|---|---|---|
+| fas-g2p-1.0 | fp32 | 0/15,103 | 0.00% | 0 | 0.200 | — |
+| tha-g2p-base-1.0 | fp32 | 0/6,932 | 0.00% | 0 | 0.421 | — |
+| heb-diac-1.1 | fp16 | 6/34,178 | 0.02% | ~0 | 0.205 | 1.00 |
+| urd-diac-1.0 | fp16 | 0/7,870 | 0.00% | 0 | 0.213 | — |
+| urd-g2p-1.0 | fp16 | 3/6,281 | 0.05% | ~0 | 0.169 | 1.00 |
+| khm-latn-1.0 | fp16 | 84/2,869 | 2.93% | 6.3e-06 | 0.121 | 0.77 |
+| **heb-diac-1.1** | **int8** | **3,193/34,178** | **9.34%** | 2.4e-05 | 0.205 | **0.20** |
+| urd-g2p-1.0 | int8 | 107/6,281 | 1.70% | 5.1e-06 | 0.169 | 0.97 |
+| khm-latn-1.0 | int8 | 71/2,869 | 2.47% | 1.1e-05 | 0.121 | 0.90 |
+| tha-g2p-small-1.0 | int8 | 63/6,932 | 0.91% | 1.2e-03 | 0.421 | 0.89 |
+| urd-diac-1.0 | int8 | 27/7,870 | 0.34% | 1.6e-05 | 0.213 | 1.00 |
+| tha-g2p-small-1.0 | int4 | 18/6,932 | 0.26% | 4.6e-04 | 0.421 | 1.00 |
+
+All rows passed the CER parity gate at release. Readings:
+- fp32 is exact everywhere (harness sanity); fp16 is benign (≤0.05%)
+  except khm-latn — the flattest margins in the catalog (p50 0.121),
+  2.93% flips, 77% near-tie.
+- **heb-diac-1.1 int8 is the outlier: 9.34% flip rate with only 20% of
+  flips at near-tie positions** — 80% of its argmax flips occur where
+  the reference was confident. That is the dangerous class the CER gate
+  cannot see.
+- **Root cause found and fixed at the export default (2026-08-27,
+  controlled probes on the same 300 pairs):** the culprit is the
+  quantized *head*. Per-channel weights alone: 8.50% flips, 78% still
+  confident-position, +25% artifact size — rejected. Keeping
+  `/lm_head/MatMul` in fp32 (body int8): **0.26% flips (36x fewer),
+  KLD 47x lower, 100% of remaining flips near-tie, +0.4% size**;
+  head-fp32 + per-channel adds nothing. Quantizing the node that
+  computes argmax moves the decision boundary directly.
+  `export_zips` now excludes the head MatMul from int8 by default
+  (`imf.export.head_matmul_names` + `nodes_to_exclude`). Shipped int8
+  zips predate this; re-exporting them is a release decision.
+- The distilled students behave as the decode analysis predicts: flat
+  but consistent — tha-g2p-small's shipped int4 flips 0.26% of
+  positions, all near-tie.
+- **Policy (adopted):** embedding-like tensors — byte embeddings, tied
+  lm_head, relative-attention bias, and any memory-layer lookup tables —
+  are a separate quantization class: fp16 (≥ int8 floor) when the body
+  is quantized. Precision floors are keyed on access pattern, not
+  tensor size (the Qwen/Unsloth lesson).
+
+## E2 — PKM memory-layer student (ara-diac-small run-003-pkm)
+
+- **Status:** COMPLETE (2026-08-28). **Outcome: positive but below the
+  pre-registered win bar.**
+- **Measured (full 1,200 paragraphs, windowed zero-skip, greedy):**
+  PKM student **7.5553** DER-CE vs run-002 vanilla ByT5-small **8.259**
+  (teacher reproduces 2.5815) — 0.704pp of the 5.677pp teacher-student
+  gap closed (12.4% relative) at +85.9M lookup params and near-zero
+  added compute (gathers, not matmuls).
+- **Verdict per the pre-agreed rule (≥1.0pp = win): NOT met.** The
+  capacity axis is real but not the dominant term of the gap; the rest
+  is modeling/optimization (E3 tests the optimization half) and domain
+  coverage (the subset lesson from run-002). Reported exactly as
+  measured — no threshold-moving.
+- Engagement was verified independent of outcome: gates moved off zero
+  by step-500 (0.0008-0.0028) and settled at 0.034-0.053 by step-10,500
+  (15-20x) — the memory branch contributed measurably but modestly,
+  matching the 0.70pp outcome. CE 2.07 → 0.02 over 10,995 steps.
+- Launched 2026-08-27, A10G, labels reused from run-002 (single-variable
+  design). Survived one mid-run eviction: resumed from step-2000.
+- **Hypothesis:** the 5.68pp teacher→student gap (r6 2.5815 → ByT5-small
+  8.259 full-set) is partly a *capacity* gap that lookup memory closes
+  at near-zero compute — parameters and compute are separable (arXiv
+  2601.21204; PKM on character-level LM: Lample et al., NeurIPS 2019).
+- **Design:** google/byt5-small backbone + 3 product-key memory layers
+  on decoder blocks [-2, -4, -6] (128² = 16,384 slots, top-32 reads,
+  ~+25M params), zero-init output gates (pretrained function preserved
+  at step 0 — tested). Teacher r6 frozen; identical corpus, labels,
+  limits, seed, and 3-epoch schedule as run-002.
+- **Protocol:** windowed zero-skip Misraj DER-CE, full 1,200 paragraphs
+  (the published harness; `modal_distill evaluate_der`).
+- **Pre-agreed gate:** ≤ 3.07 windowed DER-CE (the run-002 gate).
+- **Pre-agreed verdict rule:** PKM wins if it closes ≥ 1.0pp of the
+  5.68pp full-set gap at equal decode-time compute. No movement ⇒ the
+  gap is modeling/optimization, not capacity — publishable negative.
+- **Comparison targets:** run-002 student 8.259 / teacher 2.5815
+  (full-set, published in RESULTS.md).
+
+## E3 — Muon optimizer A/B (run-004-pkm-muon)
+
+- **Status:** COMPLETE (2026-08-28). **Outcome: ADOPTED — the gate is
+  exceeded 9x.**
+- **Measured (full 1,200, windowed zero-skip, greedy):** Muon arm
+  **4.8287** DER-CE vs the single-variable AdamW arm (run-003-pkm)
+  7.555 — **−2.727pp from the optimizer alone** (adopt gate ≥0.3pp).
+  Against the shipped vanilla student (8.259): 3.430pp of the 5.677pp
+  canonical gap closed (60.4%). Teacher reproduced at 2.5997 in this
+  container (range across eval containers 2.5793–2.5997, bf16
+  autocast; protocol consistent).
+- Training-side corroboration: CE ~0.007 vs the AdamW arm's ~0.02 at
+  equal steps; step time ~1.2s vs ~3.4s (the <15% overhead gate met
+  with margin — Newton–Schulz is cheap next to 1450-byte windows); no
+  stability events.
+- **Combined with E2, the gap decomposition at the ByT5-small rung:**
+  capacity ~0.70pp (PKM) + optimization ~2.73pp (Muon, on the PKM
+  architecture) + residual ~2.25pp (domain coverage — the subset
+  lesson). Caveat carried: the A/B isolates the optimizer ON the
+  memory architecture; the vanilla+Muon cell (run-005-muon) is queued
+  to complete the 2x2 factorial.
+- **Hypothesis:** orthogonalized-momentum updates (Newton–Schulz; the
+  optimizer Qwen3.8-Flash-Next / LongCat report) help even in
+  knowledge-limited distillation fine-tunes — unmeasured territory for
+  byte-level seq2seq students. Prior expectation: small (RL negative;
+  data-side levers won before).
+- **Design:** identical to E2 except optimizer — Muon on 2D hidden
+  matrices (lr 0.01, momentum 0.95, wd 0.01, cosine), AdamW group for
+  embedding-like params including the memory tables (random-access
+  class per E1 policy). Same seed, data, schedule.
+- **Pre-agreed adopt gate:** ≥ 0.3pp DER improvement at equal steps, no
+  stability regressions, step-time overhead < 15%.
+- **Report:** either direction goes to the paper's training-methods
+  appendix.
+
+## Parked
+
+- **Speculative decoding** (LongCat converts sparsity→speed): revisit
+  only if API latency data shows p95 decode binding. No code, by design.

docs/PUBLICATION-NOTES.md

@@ -0,0 +1,103 @@
+# Publication notes — what is worth publishing, and where it lives
+
+Status: 2026-08-28. Rule: a claim is publishable when it is
+harness-measured and recorded in RESULTS.md (models/protocol numbers)
+or EXPERIMENTS.md (registered experiments). Nothing below cites an
+un-landed number.
+
+## Complete, measured, citable now
+
+### 1. The artifact contract (IMF v1) and cross-runtime parity
+One zip, three ONNX graphs, fixed byte table, per-member SHA-256,
+byte-identical output from Ruby/Python/TypeScript; opset-14 floor for
+old consumer runtimes. Paper: sections 3–4 (section-imf). The framing
+contribution — neural models under the same discipline as
+deterministic transliteration maps.
+
+### 2. The decode-protocol correction
+Beam-4 with length normalization inflates flat byte-student PER 4.2×
+(12.06 published vs 2.85 real, same artifact, greedy). Language-
+dependent at the teacher tier (Hebrew gains ~12 DER points from beam,
+Arabic nothing). Paper: section-decode. This is the paper's most
+quotable calibration result.
+
+### 3. Pretrained-or-collapse (the client-tier frontier law)
+Random-init byte students collapse at every capacity tested (33M
+75.80 PER, 70M 78.51); the pretrained rung is the whole cliff (300M
+→ 2.85). The Arabic replica was label-corrupted in transit —
+disclosed, retracted as evidence, kept as a reproducibility lesson.
+Paper: section-frontier + section-repro.
+
+### 4. Controlled aux-representation ablation (r8, 2026-08-27)
+Identical teacher/corpus/seed/init; only the aux stream's output
+representation varies. Morphology 2.5793 < phonemic IPA 2.6588 < none
+2.6775 (full 1,200-paragraph windowed zero-skip); the IPA projection
+itself was learned (2.3% CER probe). **The strong form of "phonemes
+help diacritization" fails; lexical-morphological knowledge is the
+active ingredient.** Paper: leaderboard section. RESULTS.md (rababa):
+r8 section.
+
+### 5. Margin-aware parity + the head-fp32 quantization fix (E1)
+Teacher-forced margin analysis (flip rate / KLD / near-tie share)
+measured across the entire catalog — invisible to the CER gate.
+Headline diagnosis: the shipped Hebrew int8 artifact flipped 9.34% of
+positions, 80% at confident margins. Controlled probe matrix isolated
+the quantized *head* (not weight granularity: per-channel recovered
+almost nothing at +25% size); keeping the 1.2M-parameter head in fp32
+cut flips 36× (0.26%, all near-tie) at +0.4% size. Now the export
+default + a release gate + a regression test. Paper: IMF section
+"Margin-aware parity" bullet. EXPERIMENTS.md E1.
+
+### 6. The memory-layer capacity experiment (E2, 2026-08-28)
+Single-variable: shipped ByT5-small student vs +3 product-key memory
+layers (+85.9M params, near-zero compute; gates verified engaged).
+8.259 → 7.555 full-set DER — 0.704pp of the 5.677pp teacher-student
+gap (12.4% relative), **below the pre-registered ≥1.0pp bar**. Honest
+conclusion: the distillation gap is dominated by optimization and
+domain coverage, not parameter capacity. First controlled test of the
+LongCat/Qwen "embedding scaling" axis on a byte-level seq2seq student.
+Paper: frontier section (fourth question). RESULTS.md run-003-pkm.
+
+### 7. Measurement-discipline findings
+- Subset-selection artifact: first-300-paragraphs 3.66 vs full-set
+  8.26 (teacher reproduces 2.5815/2.5793 — harness soundness proven).
+  Paper: leaderboard section. Standing rule: full-set-only publication.
+- Leaderboard positioning: r6 2.5793 is the best dedicated model under
+  the protocol (behind Claude-3.7-Sonnet's 1.3941, ahead of GLM-5.2
+  2.6911, Gemini-Flash 3.1926, GPT-4 3.8645, Sadeed-1.5B 7.2915).
+
+### 8. Negative results (honesty assets, all in the log)
+RL teacher polishing flat/negative ×3; microkimi bridges improve
+structure but not accuracy; teacher beam-search unnecessary for
+Arabic; per-channel int8 rejected on measurement; the 30 MiB tier
+closed as infeasible without pretraining.
+
+### 9. Muon optimizer A/B on the memory student (E3) — LANDED 2026-08-28
+**4.8287 vs 7.5553 full-set (−2.727pp from the optimizer alone); adopt
+gate (≥0.3pp) exceeded 9×. ADOPTED.** Training CE ~0.007 vs ~0.02 at
+equal steps, ~1.2s/step vs ~3.4s, no stability events. With E2 this
+completes a controlled decomposition of the distillation gap at the
+ByT5-small rung: ≈0.70pp capacity + 2.73pp optimization + 2.25pp
+residual (domain coverage). The strongest training-methods result in
+the paper — first controlled Muon measurement on byte-level seq2seq
+distillation. Paper: frontier section. EXPERIMENTS.md E3, RESULTS.md
+run-004. Factorial cell 4 (vanilla+Muon, run-005) queued.
+
+### 10. Arabic news-domain adaptation (r7) — ID LANDED 2026-08-28
+**2.2864 / 1.3343 full-set windowed zero-skip — new best dedicated
+model, −0.29pp over r6** (the news mix improved in-domain, not just
+OOD; behind only Claude-3.7-Sonnet's published 1.3941). OOD half
+(WikiNews-2024 multi-ref, gate: beat r6's 19.82/12.46) running via the
+auto-launched actor. → rababa RESULTS.md (recorded) → paper leaderboard
+table + OOD note once OOD confirms; canonical-teacher promotion
+pending that check.
+
+## Venue fit (working notes)
+
+The spine is systems-with-measurements: an artifact contract +
+distillation discipline + three calibration/measurement corrections
+(decode, subset, quantization-margins) that generalize beyond our
+stack. The frontier law (pretrained-or-collapse) and the controlled
+capacity/aux/optimizer experiments give it empirical heft. Package as
+one paper (current paper.adoc); the margin/head finding alone is also
+a strong short workshop paper if a split is ever wanted.

docs/RESULTS.md

@@ -222,6 +222,44 @@ mojibake labels (byt5 decode_joined bug), the second silently resumed
 from the poisoned lineage's checkpoints (now guarded by labels.sha
 digest matching), this one clean end-to-end. CE plateaued at ~0.016.
 
+### run-003-pkm — memory-layer student (2026-08-28, research run)
+
+The qwen-next capacity experiment (EXPERIMENTS.md E2): identical to
+run-002 except three product-key memory layers (+85.9M lookup params,
+zero-init gates) on the ByT5-small decoder — single-variable.
+
+| Model | DER-CE (full 1,200) |
+|---|---|
+| Teacher (r6, full-set) | 2.5815% |
+| **Student + PKM memory (run-003-pkm)** | **7.5553%** |
+| Student vanilla (run-002) | 8.2590% |
+
+0.704pp of the 5.677pp teacher-student gap closed (12.4% relative) at
+near-zero added compute — below the pre-registered ≥1.0pp win bar, so
+the memory axis is real but not the dominant term of the gap. Gates
+verified engaged (0.034-0.053 at completion). Not shipped; the vanilla
+client rung stands.
+
+### run-004-pkm-muon — optimizer A/B on the memory student (2026-08-28)
+
+Identical to run-003-pkm except the optimizer (Muon on 2D hidden
+matrices, AdamW group for embedding-like params; EXPERIMENTS.md E3):
+
+| Model | DER-CE (full 1,200) |
+|---|---|
+| Teacher (r6, this container) | 2.5997% |
+| **ByT5-small + PKM + Muon (run-004)** | **4.8287%** |
+| ByT5-small + PKM + AdamW (run-003) | 7.5553% |
+| ByT5-small vanilla + AdamW (run-002) | 8.2590% |
+
+**−2.727pp from the optimizer alone** — the adopt gate (≥0.3pp)
+exceeded 9x; 3.430pp of the 5.677pp canonical gap closed (60.4%)
+combining memory + optimizer. Training CE ~0.007 vs ~0.02 at equal
+steps; ~1.2s/step vs ~3.4s; no stability events. The teacher-student
+gap at this rung decomposes: ~0.70pp capacity + ~2.73pp optimization
++ ~2.25pp residual (domain coverage). The vanilla+Muon factorial cell
+(run-005-muon) completes the decomposition.
+
 Leaderboard context (SadeedDiac-25, Misraj evaluator, zero-skip,
 harakat-projected DER-CE): the teacher tier (r6, 580M) at 2.5793
 (reproduced at 2.5815, 2026-08-26) is the best dedicated model measured