Skip to content

docs: layerdrop full-set verdict 7.44 — frontier complete #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 30, 2026
Merged

docs: layerdrop full-set verdict 7.44 — frontier complete #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/PUBLICATION-NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,15 @@ reproducing 1.3205 across all three measurements as the harness
control. Next rung on the frontier: stitch-down from ByT5-small
pretraining (ridge-fit width bridge), which changes init, not data or
capacity alone.

## layerdrop full-set (2026-08-31): the frontier is complete

7.4413 at 63% of parameters - better than the shipped 1.0, 2.15pp
behind its optimizer-matched full-depth peer. For paper B this closes
the capacity section: a monotone five-point size-quality frontier
(scratch, width-stitch, depth-cut, optimizer-matched, teacher-matched)
with every rung measured full-set under one harness. The subset
overstatement now has three quantified instances - it is a finding,
not a nuisance: domain-neighborhood evaluation subsets inflate student
quality by 2-4x in this regime. Frame it in the measurement section as
a generalizable warning with the three pairs as evidence.
25 changes: 25 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,3 +387,28 @@ representation survives a depth cut that width surgery destroyed.
Full-set gate (1,200 paragraphs) in flight; int8 ~190MB, int4 ~95MB
(the browser-budget artifact). Survived two infra failures en route
(eviction without watchdog; a regressed d_kv derivation) - both fixed.

## ara-diac-small-layerdrop — full-set verdict: 7.44 (2026-08-31)

The 1,200-paragraph gate (teacher reproduces 2.5815):

| rung | params | full-set DER-CE | subset DER-CE |
|---|---|---|---|
| 1.0 (full depth, AdamW, r6) | 300M | 8.259 | 3.658 |
| **layerdrop (enc 6, Muon, r6)** | **~190M (63%)** | **7.4413** | 3.8088 |
| r6 + Muon (full depth) | 300M | 5.2945 | — |
| 2.0 (r7 + Muon, full depth) | 300M | 4.8218 | — |
| scratch d384 | 33M | 74.68 | 83.08 |
| SVD width-stitch d384 | 29M | 82.96 | — |

Reading: halving encoder depth + Muon BEATS full-depth AdamW (7.44 vs
8.26) at 63% of the parameters — but the depth cut costs 2.15pp against
its optimizer-matched peer (5.29). Strict gate (teacher+0.5) failed.
This is the THIRD instance of the first-300 subset overstating quality
(3.66 vs 8.26; 3.81 vs 7.44) — the subset sits in the training-domain
neighborhood; full-set-only stands as the publication rule, now with a
quantified repeat rate. The size-quality frontier is complete and
monotone: 33M/74.7 - 29M/83.0 - 190M/7.4 - 300M/5.3 - 300M/4.8
(teacher 2.28-2.58). Browser-tier decision (user): ship
layerdrop-int4 (~95MB, ~7.5 DER with int4 flip risk ungated) as the
lite rung, or hold the tier at 2.0-int8 (264MB, 4.82).
Loading