Qwen3.8-Flash-Next techniques: margin gates (measured), PKM memory student (in flight), Muon A/B - #56
Merged
Merged
Conversation
added 13 commits
August 27, 2026 13:49
…memory students, Muon A/B Margin-aware parity (E1): teacher-forced logit-margin analysis beside the CER gate — flip rate, KLD, near-tie share. Measured on shipped khm-latn: fp16 carries 2.93% silent argmax flips at KLD 6e-06 (CER gate blind to them); median top1-top2 margin 0.12 logits. Policy: embedding-like tensors are a separate quantization class (fp16 floor). PKM memory student (E2): product-key memory layers (Lample et al. 2019; capacity axis per arXiv 2601.21204) injected into byt5-small decoder FFNs behind zero-init gates (+85.9M params). Spec ara-diac-small-pkm: identical teacher/corpus/labels/seed as run-002 → single-variable comparison vs the 8.259 full-set student; gate <= 3.07 windowed DER. Muon (E3): single-file Newton-Schulz orthogonalized momentum with AdamW fallback groups for embedding-like params (incl. memory tables); spec ara-diac-small-pkm-muon for the A/B. docs/EXPERIMENTS.md: pre-registered protocols and verdict rules.
Gate values + memory-table stats from the latest step checkpoint. Step-500 reading: all three decoder gates off zero (0.0008-0.0028) — the memory branch is engaged, not bypassed.
…agged Every shipped artifact measured (300-pair probe, teacher-forced, torch vs ONNX): fp32 exact everywhere; fp16 benign (<=0.05%) except khm (2.93%, 77% near-tie, flattest margins p50 0.121); int8 0.26-9.34%. heb-diac-1.1 int8 is the outlier: 9.34% flips with only 20% at near-tie positions — 80% of argmax flips at confident positions, the class the CER gate cannot see. Open item recorded. margins runner now skips un-exported precisions instead of aborting.
Identical run differing only in the aux stream's output representation (broad-phonemic IPA vs morphology): 2.6588 vs 2.5793 vs 2.6775 baseline, with the IPA projection itself learned (2.3% CER probe). The r6 win is lexical-morphological knowledge injection, not phonemic supervision.
qwen_next_chain: workstation-independent state machine for E2/E3 — driven by volume markers only (best/config.json, final_eval.json, chain_log.jsonl), 20-min stall detection with respawn (distill_sequence resumes from the latest step checkpoint; redundant spawns are benign), idempotent across relaunches. Replaces the local shell chain, which died with the workstation and broke when the checked-out branch changed. evaluate_der now writes final_eval.json into the run dir — the durable verdict record r7-style _init_choice probes can read (run-006-morph lacked one, which is why r7 needed an explicit --init-run).
… flips Controlled probes on heb-diac-1.1 (300 pairs, 34,178 positions): shipped int8 flips 9.34% of teacher-forced positions (80% at confident margins, invisible to the CER gate). Per-channel weights alone barely help (8.50%, +25% size) — rejected. Excluding /lm_head/MatMul from quantization cuts flips to 0.26% (36x), KLD 47x lower, all remaining flips near-tie, at +0.4% artifact size. The head is where argmax is computed; quantizing it moves the decision boundary directly. export_zips now excludes the head from int8 by default. Shipped int8 zips predate the fix; re-export is a release decision. Probes: modal_export int8_pc / int8_head.
…test parity_model now enforces the E1 policy: a release fails if >1% of teacher-forced positions flip argmax at confident margins (near-tie flips are inherent to flat byte models; confident flips mean the decision surface moved). Pre-fix heb-diac int8 measured 7.5% confident flips; every head-fp32 artifact measures <0.7%. paper.adoc: 'Margin-aware parity' bullet in the IMF section — the catalog-wide flip measurement, the head diagnosis (probe matrix), the 36x fix at +0.4% size, and the general lesson: gates must measure the protocol's decision surface, not a downstream aggregate. test_int8_keeps_head_matmul_in_fp32 locks the export default.
…w the 1.0pp bar) Full 1,200-paragraph windowed zero-skip: teacher reproduces 2.5815. Positive but not the pre-registered win; capacity is real (12.4% relative) yet not the dominant term of the teacher-student gap. Engagement verified independent of outcome. E3 (Muon arm) tests the optimization half.
7.5553 full-set vs run-002's 8.2590 (0.704pp closed, 12.4% relative, below the pre-registered bar). Gates settled at 0.034-0.053.
Fourth frontier question: is the teacher-student gap a capacity gap? Single-variable PKM experiment — 8.259 -> 7.555 (0.704pp of 5.677pp, 12.4% relative), below the pre-registered 1.0pp bar; gap is dominated by optimization and domain coverage, not parameters. References added (Lample et al. 2019; Liu et al. 2026). PUBLICATION-NOTES.md: the publication-worthiness inventory — 8 complete measured findings, 2 in-flight slots (E3, r7), negative results, venue notes.
…e); factorial cell 4 queued run-004-pkm-muon vs run-003-pkm differs only in optimizer: 4.8287 vs 7.5553 full-set windowed DER. Adopt gate (≥0.3pp) exceeded 9x. With E2: gap decomposition ≈0.70pp capacity + 2.73pp optimization + 2.25pp residual. Training CE ~0.007 vs ~0.02; ~1.2s/step vs ~3.4s. run-005-muon (vanilla + Muon) queued to close the 2x2 factorial — orchestrator ARMS extended; completed arms skip via markers (idempotent).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three techniques from the Qwen3.8-Flash-Next review (basis paper verified: arXiv 2601.21204, LongCat-Flash-Lite), each landing as a pre-registered experiment in docs/EXPERIMENTS.md:
E1 — Margin-aware parity gates (validated, adopted)
The CER-delta release gate cannot see near-tie argmax flips. imf.parity now also runs a teacher-forced margin analysis (flip rate, KLD, near-tie share) against each precision zip; modal_export emits it on every gate, plus a read-only
marginsentrypoint for already-published zips.Measured on shipped khm-latn-1.0 (300 pairs, 2,869 positions): fp32 0 flips; fp16 84 flips (2.93%), KLD 6.3e-06, 77% of flips at near-tie margins; int8 71 flips (2.47%), KLD 1.1e-05, 90% near-tie — all three passed the CER gate. Median top1−top2 margin is 0.12 logits (the flat-student regime from the decode correction). Policy adopted: embedding-like tensors stay fp16 when the body is quantized.
E2 — PKM memory-layer student (in flight)
Product-key memory (Lample et al., NeurIPS 2019 — demonstrated on character-level LM) injected into byt5-small decoder FFNs behind zero-init gates: +85.9M lookup params at near-zero compute. Spec
ara-diac-small-pkmis identical to run-002 in teacher/corpus/labels/seed — single-variable comparison against the 8.259 full-set student / 2.5815 teacher. Gate ≤ 3.07 windowed DER; pre-agreed verdict rule: ≥1.0pp of the 5.68pp gap closed = capacity axis works; no movement = publishable negative. Unit-tested identity property: gate=0 leaves pretrained logits bit-identical.E3 — Muon optimizer A/B (queued by chain watchdog)
Single-file Newton–Schulz orthogonalized momentum with AdamW fallback groups; embedding-like params (byte embeddings, tied head, memory tables) deliberately stay on AdamW per the E1 access-pattern policy.
ara-diac-small-pkm-muonruns after E2 completes.Notes
rababa_arabic_distill_small/run-003-pkm/run-004-pkm-muon; labels copied from run-002 (no relabeling).