Can a ~5M-parameter causal decoder learn multi-digit addition and subtraction from a synthetic corpus, and does it generalize to operand lengths it never saw?
Short answer: it learns the lengths it is trained on, very well, and does not extrapolate to any length outside that range — in either direction.
report.html is the visual write-up. This file covers how to run things and the
throughput work, which is where most of the practical value turned out to be.
Exact-match accuracy, greedy decode, answers accepted in forward or reversed order (see Scoring caveat). Best checkpoint per run.
| corpus | tpp | 1-digit | 2–4d | 2–9d | 5–6d | 10–12d |
|---|---|---|---|---|---|---|
| 2–4 digit (run 3, original) | 12.8 | 0.00% | 78.70% | 9.50% | 0.00% | — |
| 2–9 digit | 12.8 | 0.00% | 29.85% | 47.05% | 34.65% | 0.00% |
| 2–9 digit | 20 | 0.00% | 47.50% | 70.80% | 59.25% | 0.00% |
| 2–9 digit | 100 | 0.00% | 34.85% | 70.20% | 80.85% | 0.00% |
| 1–9 digit | 20 | 11.03% | 61.90% | 86.25% | 81.85% | 0.00% |
| 1–4 digit | 100 | 97.06% | 60.15% | — | — | 0.00% |
| 1–4 digit + SFT | 100 | 100.00% | 99.05% | — | — | 0.00% |
Best model overall is 1–4 digit @ 100 tpp + SFT
(checkpoints/run10_sft/sft_final.pt) — 99.05% on the original 2–4 digit eval.
Best model without SFT, and the widest digit range, is 1–9 digit at 20 tpp
(checkpoints/run7_1d_20tpp/ckpt_final.pt).
Run 9 narrowed to 1–4 digits at 100 tpp and still scored only 60.15% on the original
2–4 digit eval, with 2-digit answers at 1%. The arithmetic was correct; the model
would not stop (18+15= → 3333). Pretraining on a packed stream makes <eos> one
token among millions, never the target of a distinct example.
sft_stop.py applies the recipe from ../llm-modern-arch-experiment/src/modern_lm/sft.py
— whose own docstring says its SFT exists because "packed pretraining text almost never
teaches 'stop after answering'". Loss is masked to -100 on the prompt so only the
answer digits and terminal <eos> are supervised (36.9% of tokens). The SFT set is
balanced by answer length (40k each at 1–5 digits) rather than mirroring pretraining's
~80% four-digit skew, which is what caused the bug.
| answer length | pre-SFT | post-SFT |
|---|---|---|
| 1 digit | 0% | 97% |
| 2 digit | 1% | 94% |
| 3 digit | 33% | 99% |
| 4 digit | 84% | 100% |
| overall (2–4d eval) | 60.15% | 99.05% |
| 5–6 digit (unseen) | 0.00% | 0.00% |
Cost: 40 seconds, ~0.58M supervised tokens — 0.114% of the pretraining budget.
Two things worth noting beyond the headline:
- Strict accuracy went 21.55% → 99.05%, exactly matching lenient. Every earlier run
leaned on a scorer accepting forward or reversed answers, because 50/50 reversed
training left the model emitting a mix. After SFT the two metrics coincide — it emits
forward order every time (
7+5=→21became12). The scoring caveat below no longer applies to the SFT model. - SFT fixes format, not capability. 5–6 digit is 0.00% before and after. Supervising the completion teaches when to stop and which order to emit; it cannot teach arithmetic the model never learned.
Adding mul and div required growing the vocab from 14 to 16 tokens (x, /),
which makes run-8 checkpoints and all earlier checkpoints mutually unloadable (the tied
embedding is V × 256). Same recipe as run 7 otherwise: 1–9 digit, 20 tpp, 287s.
| operation | accuracy | correct length | correct first digit |
|---|---|---|---|
| sub | 83.20% | — | — |
| add | 75.13% | — | — |
| div (exact only) | 60.53% | — | — |
| mul | 5.20% | 54.2% | 12.2% |
Same 511M-token corpus, schedule and vocab at five parameter counts, ffn/dim = 3.0 and
head_dim = 32 held constant.
| 5M | 1M | 0.5M | 0.25M | 0.1M | |
|---|---|---|---|---|---|
| params | 5,129,600 | 1,073,728 | 485,632 | 270,784 | 110,464 |
| throughput | 374k | 870k | 1.25M | 1.58M | 3.89M |
| wall clock | 1367s | 587s | 408s | 325s | 143s |
| eval loss | 0.6831 | 0.6957 | 0.7058 | 0.7262 | 0.7762 |
| div | 100.00% | 100.00% | 98.25% | 82.25% | 0.25% |
| sub | 95.75% | 81.25% | 69.75% | 53.75% | 1.25% |
| add | 88.50% | 70.00% | 63.00% | 45.25% | 0.75% |
| tr_partial | 57.50% | 52.25% | 58.50% | 52.25% | 19.25% |
| tr_reduce chain | 99.75% | 99.25% | 97.75% | 98.25% | 51.75% |
| algebra | 44.25% | 25.25% | 2.75% | 1.50% | 1.00% |
Depth, not parameter count, is the arithmetic floor. 0.25M and 0.1M share the same width (dim 64) and differ only in layers — 5 vs 2. Tripling depth at constant width took division from 0.25% to 82.25% and subtraction from 1.25% to 53.75%. Two layers cannot represent multi-digit arithmetic. Everything at or above 0.25M degrades gracefully.
Traced work is the most robust capability measured. tr_partial holds 52–58% across a
20× parameter range and only breaks at 0.1M; chain fidelity is 99.75% at 5M and still
98.25% at 0.25M. Decomposition turns an arithmetic problem into a format problem, and
format survives capacity reduction far better than computation.
Tokenization dominates small-model runs. Cold-loading the 511M corpus takes 292.5s
— 24× the 12.1s for a 50M slice, because the Python list balloons to ~1.2 GB before the
numpy conversion. On the 0.1M model that was 68% of wall clock. corpus_cache.py reduces
it to 0.0s, keyed on file mtime, SEQL and the vocab list so the seven vocab changes in this
study cannot silently poison a run.
Practical tiers: 1M for real iteration, 0.25M for fast iteration (325s, meaningful signal on everything but algebra), 0.1M for pipeline smoke tests only.
A third point at 485,632 params (9.5% of run 23), same byte-identical corpus, schedule and
vocab, with ffn/dim = 3.0 and head_dim = 32 held constant throughout.
| model | params | tok/s | speedup | wall clock | eval loss |
|---|---|---|---|---|---|
| 5M (run 23) | 5,129,600 | 373,985 | 1.00× | 1367s | 0.6831 |
| 1M (run 24) | 1,073,728 | 870,251 | 2.33× | 587s | 0.6957 |
| 0.5M (run 25) | 485,632 | 1,253,230 | 3.35× | 408s | 0.7058 |
| split | 5M | 1M | 0.5M | Δ vs 5M |
|---|---|---|---|---|
| div direct | 100.00% | 100.00% | 98.25% | −1.75 |
| tr_reduce chain fidelity | 99.75% | 99.25% | 97.75% | −2.00 |
| mul direct | 6.00% | 5.50% | 4.75% | −1.25 |
tr_partial <think> |
57.50% | 52.25% | 58.50% | +1.00 |
tr_sum <think> |
76.00% | 64.25% | 64.25% | −11.75 |
| tr_carry direct | 96.00% | 84.50% | 80.75% | −15.25 |
| add direct | 88.50% | 70.00% | 63.00% | −25.50 |
| sub direct | 95.75% | 81.25% | 69.75% | −26.00 |
| algebra | 44.25% | 25.25% | 2.75% | −41.50 |
Throughput returns diminish sharply below 1M. 20.9% of parameters bought 2.33×; dropping to 9.5% bought only 1.44× more. The embedding, lm_head and data movement are fixed costs that dominate once the body is small. The loss penalty doubled (+0.013 → +0.025) while subtraction lost 26 points. 1M is the efficiency sweet spot.
Capabilities have very different parameter floors. Chain fidelity is nearly free (99.75 → 97.75%) and division almost so (100 → 98.25%), confirming the backward- construction explanation from run 24. Add and sub degrade roughly linearly. Algebra collapses — 44.25 → 25.25 → 2.75% — hitting a hard floor between 0.5M and 1M. Traced multiplication is flat across the whole 10× range, which is what a format ceiling looks like rather than a capacity one.
Note: tr_partial at 0.5M scored above 1M (58.50% vs 52.25%). At n=400 that is within
noise; read it as flat, not as a gain.
A 1.07M-param model on byte-identical corpus, schedule and vocab as run 23. dim 256→128,
layers 6→5, heads 8→4, ffn 768→384, with ffn/dim = 3.0 and head_dim = 32 held constant
so capacity is the only variable. Same 15,600 steps and 511M tokens — 480 tokens/param
rather than 100, so the small model is past its Chinchilla point and any gap is capacity,
not undertraining.
| 5M (run 23) | 1M (run 24) | |
|---|---|---|
| params | 5,129,600 | 1,073,728 (20.9%) |
| throughput | 373,985 tok/s | 870,251 tok/s (2.33×) |
| wall clock | 1367s | 587s |
| eval loss | 0.6831 | 0.6957 |
| SFT held-out | 0.3116 | 0.3293 |
| split | 5M | 1M | Δ |
|---|---|---|---|
| div direct | 100.00% | 100.00% | 0.00 |
| tr_reduce chain fidelity | 99.75% | 99.25% | −0.50 |
| mul direct | 6.00% | 5.50% | −0.50 |
tr_partial <think> |
57.50% | 52.25% | −5.25 |
| tr_carry direct | 96.00% | 84.50% | −11.50 |
tr_sum <think> |
76.00% | 64.25% | −11.75 |
| sub direct | 95.75% | 81.25% | −14.50 |
| add direct | 88.50% | 70.00% | −18.50 |
| algebra | 44.25% | 25.25% | −19.00 |
Format is cheap; arithmetic is not. The 1M model holds reasoning structure almost
perfectly — tr_reduce chain fidelity 99.25% vs 99.75%, tr_partial down only 5.25 points.
What the extra four million parameters buy is accuracy inside the steps: add −18.50,
sub −14.50, algebra −19.00. Emitting a valid chain and computing correctly within it are
separable capabilities with very different parameter costs.
Division was never using the parameters. 100.00% at both scales, every width 4–16
digits. The generator builds it backward (a = q·d, small d), so the quotient is a
pattern-match, not a carry-propagating computation. Multiplication lost only 0.50 points
because at 6% it had nothing to lose.
Practical read: for iterating on corpus design or trace formats — where run 24 shows format learning is cheap — the 1M model gives the same qualitative signal in 10 minutes instead of 23. For final accuracy on anything requiring real computation, the 5M model is meaningfully better.
<think> now declares the step count up front: <think> <3> 847 * 200 = .... Nine new
tokens (<1>–<9>); chains are capped at 9 steps by the carry trace's 8-column limit.
| required partial products | answer accuracy | declared <n> correct |
|---|---|---|
| 1 | 50.8% | 100.0% |
| 2 | 100.0% | 100.0% |
| 3 | 27.3% | 93.2% |
The intervention worked and the outcome got worse. Step-count declaration went from 48.4% (run 22) to 93.2% — the model reliably knows and states how many steps it needs. Answer accuracy at three products fell 40.7% → 27.3%.
So run 22's diagnosis was wrong. I concluded the model dropped terms because it couldn't track chain length. It can: given the ability to declare a count, it declares correctly 93% of the time and still gets the answer wrong. Forced to commit to three steps, it emits three steps with wrong content instead of two steps and a wrong sum.
Dropping a term was never a counting failure — it was what running out of capacity looks like. Two partial products is the ceiling for this model: 2-product problems reached 100.0% (up from 97.5%), and no amount of output scaffolding moved the third.
tr_sum held at 76.00%, tr_carry direct at 96.00%. SFT held-out loss 0.3116, best of
any switch run.
| trace | direct | <think> |
chain exact |
|---|---|---|---|
| sum (new) | 13.20% | 76.60% | 76.60% |
| carry (new) | 99.40% | 69.80% | 77.00% |
| partial | 0.60% | 62.00% | 81.20% |
Both traces were designed from measurements, not intuition:
- sum — 3-term additions collapse (45.5% at 6 digits, 1.8% at 8, 0% at 9) while 2-term sums hold ~90% at any width, so it accumulates pairwise.
- carry — 2-term addition degrades with width (79.4% at 8 digits → 44.4% at 15) but not with carry-chain length (flat ~63% from 1 to 6 consecutive carries), so it emits one column per step. My initial assumption that carry propagation was the problem was wrong; the constraint is holding many columns at once.
Partial-product traces did not improve — 62.00%, identical to run 21. Fixing the closing sum helped 2-product problems (97.5%) but exposed a second bottleneck:
| required partial products | accuracy | emitted right count |
|---|---|---|
| 2 | 97.5% | 100% |
| 3 | 40.7% | 48.4% |
The model drops steps. 50949 * 224 emits the ×200 and ×20 terms, skips ×4, sums what
it has, and returns a consistent but wrong answer. Per-step accuracy is 99.6% (products)
and 99.2% (adds) — the steps it writes are right, it writes too few. The failure is chain-
length control, not arithmetic.
A trace transferred to the direct capability. Column-wise carry traces took direct
wide addition to 99.40%, no <think> needed — run 21 measured 44–80% for the same
operation. First case in the study where a decomposition improved the one-shot skill.
Throughput 383,571 tok/s, fastest of the study.
Three control tokens let one model answer directly or reason first, selected by which token the prompt ends with. Vocab 42 → 45.
direct 9853 * 6 = <ans> 59118 <end>
traced 6864 * 87 = <think> 6864 * 80 = 549120 -> 6864 * 7 = 48048
-> 549120 + 48048 <ans> 597168 <end>
On balanced multiplication — the operation run 19 measured at 0.00%:
| mode | accuracy |
|---|---|
<ans> — answer directly |
1.20% |
<think> — reason first |
62.00% |
| chain exact-match (~90 tokens) | 99.20% |
Same weights, same problems: a 52× difference from one control token. Direct mode is preserved (div 100.00%, add 86.60%), so unlike run 20 there is no trade between reasoning and answering.
Run 20's trace collapse had a mechanical cause. load_sft split the prompt on the
last =, so a trace's whole chain sat inside the prompt and only the final number was
supervised — SFT was training the model to skip its own reasoning. Supervising from the
first control token covers 32 of 44 tokens instead of ~6.
The residual failure moved. On traced problems the model gets 99.0% of partial
products right but only 60.0% of the closing addition, and final accuracy (59.7%)
tracks the sum rather than the products. 6014 * 66 produces 360840 and 36084
correctly, then writes 496924 for 396924. Decomposition solved the multiplication; a
long multi-term addition is the new bottleneck.
<think> is routing, not free capability. Requesting a chain where none applies is
harmful: add falls 86.60% → 0.40%, div 100% → 0.80%, because those operations have no
trace format in training. Use <think> only where decomposition applies.
At max_d=16 the mul answer cap (18) is loose enough that balanced products up to 9×9
appear for the first time. Stratifying multiplication by balance (width of the shorter
operand) separates what every earlier number conflated.
At identical answer lengths, lopsided multiplication works and balanced does not:
| answer length | balance 1–2 | balance 3–4 |
|---|---|---|
| 7 digits | 71.1% | 4.8% |
| 8 digits | 60.7% | 0.0% |
| 9 digits | 60.0% | 3.0% |
| 11 digits | 58.3% | 0.0% |
| 12 digits | 84.2% | 0.0% |
| 14 digits | 34.6% | 0.0% |
Aggregate by balance: 1–2 → 47.59%, 3–4 → 0.63%, 5–6 → 0.00%, 7+ → 0.00%. The cliff is at 3 digits on the shorter operand.
84729 * 7 is a single pass — multiply each digit, propagate one carry. 847 * 293
requires holding three partial products and summing them with alignment: more working
state, independent of output size. The model does the first and cannot do the second.
This corrects the study's central claim. Earlier runs reported a single answer-length ceiling. There are two constraints: answer length caps everything (nothing survives past ~14 digits even lopsided), but balance is the binding constraint for multiplication, and it binds at 3 digits.
It also reframes every earlier mul figure. Run 18's 52.00% vs run 17's 40.25% was not the model improving — run 18's 1–6 digit corpus cannot produce anything past 4×4, so its eval was dominated by lopsided pairs. Reported mul accuracy is only meaningful alongside the balance distribution of its eval set.
| operation | symbolic | word form |
|---|---|---|
| div | 99.79% | 99.79% |
| add | 98.83% | 98.25% |
| sub | 98.67% | 98.42% |
mul (*) |
52.00% | 53.00% |
| pow (in-dist) | 6.83% | 4.58% |
| multi-op | 11.92% | — |
Operator consistency is worth ~30 points. The same symbol scored 22.00% in run 17 —
where build_multi hardcoded * so it appeared only inside compositional expressions —
and 52.00% here, where it also appears as a standalone binary operator. Same model,
budget and digit range. A rendering inconsistency invisible in corpus statistics cost more
than any architecture or curriculum change tried in this study.
But the aggregate overstates it. Matched by answer length, run 18 is worse than run 17 at every width (5d: 86.6% vs 100%, 7d: 46.2% vs 71%, 8d: 20.6% vs 43%). Run 17's eval reached 10-digit answers; run 18's stops at 8. Narrower digits shifted the eval toward easier problems. Run 17's 1–8 digit corpus at 200 tpp still yields the better multiplier.
Balanced wide multiplication was never generated, in any run. build_single caps mul
answers at max_d + 2 and on overflow replaces both operands with randint(1,9). A d×d
product has ~2d digits, so the cap binds once d > 2:
| operand width | lines produced | fallback |
|---|---|---|
| 3×3 | 5,000 | 1.6% |
| 4×4 | 81 | 100% |
| 8×8 | 81 | 100% |
That table forces min_d = max_d. In a normal 1–N corpus the cap instead limits how
balanced a product can be: a d×d product needs ~2d digits against a cap of max_d + 2,
so the widest balanced product is about max_d/2 + 1.
| corpus | cap | 5*5 present? |
widest balanced |
|---|---|---|---|
| 1–4 digits | 6 | no | 3×3 |
| 1–6 digits | 8 | no | 4×4 |
| 1–8 digits | 10 | yes | 5×5 |
| 1–16 digits | 18 | yes | 9×9 |
So runs 13/16/18 (1–6 digits) topped out at 4×4 despite containing 6-digit operands —
their wide products are all lopsided (6×2, 5×1). Run 17 (1–8) did include balanced 5×5.
Reaching genuinely wide balanced products requires raising max_d well past the operand
width you care about, or changing the cap.
Division does not generalize past its training range. It appeared to (62.62% at
"9–12 digits" where add/sub scored 0.00%), but the eval was bucketed by dividend width
while a = q·d emits dividends across the whole range — 43% of that set had quotients of
≤8 digits. Scored by quotient length:
| quotient | accuracy |
|---|---|
| 1–8 digits | 100% |
| 9 digits | 13.2% |
| 10 digits | 1.2% |
| 11–12 digits | 0.0% |
All four operations collapse at the same answer-length boundary. Division's near-perfect scores throughout this study are real but measure short-answer arithmetic — the generator's backward construction keeps quotients short relative to dividends.
| operation | symbolic | word form | run 16 |
|---|---|---|---|
| div | 100.00% | 96.75% | 100.00% |
| add | 99.17% | 98.92% | 99.08% |
| sub | 98.67% | 98.17% | 98.83% |
| mul | 40.25% | 41.58% | 33.33% |
| pow (in-dist) | 22.17% | 19.67% | 21.33% |
| multi-op | 5.08% | — | 16.25% |
1.022B tokens in 2740s at 373,104 tok/s. SFT held-out 0.1632, best of the study.
Budget finally moved multiplication — 33.33% → 40.25%, the first time more compute has shifted mul. The length curve extended: 86% at 6-digit answers, 71% at 7-digit, reaching 11-digit answers at all.
Wider digits help operations and hurt composition. Multi-op fell to 5.08%. Re-scored
against run 16's exact answer-length profile it stays at 5.08%, so it is not an eval
artifact; per-template scoring shows the drop is uniform (old N*(N+N)-N at 0.0%, same as
the new shapes), so the four added templates are not the cause. At 1–8 digits the
intermediate values inside compositional expressions grow, and each sub-result must be
carried correctly.
Multiplication was rendered four ways, inconsistently. --mul-symbol governed only
single-op lines; build_multi hardcoded *. No line ever contained both x and *. On
400 identical problems:
| rendering | accuracy |
|---|---|
x |
41.75% |
times |
41.75% |
multiplied by |
37.00% |
* |
22.00% |
The model inferred * means multiply from compositional context alone — far above chance
— but 20 points worse than the form it saw standalone. Generator now fixed:
build_multi honours --mul-symbol, verified 0 stray * in an 80k-line sample. All
reported mul figures are x-only.
Pow needs an in-distribution measurement at this corpus size. The 45M-line corpus contains 100% of pow problems below 9 digits (all 10,674) and 0% above, so a length-matched held-out pow set is impossible — the two conditions are mutually exclusive. Measured in-distribution and length-matched to training, labeled as recall.
50% word-form operators, spaces kept, vocab 20 → 40 (19 letters + space). Same operations, same digit ranges, matched SFT exposure — the only variable is how the operator is written.
| operation | symbolic | word form | gap |
|---|---|---|---|
| div | 100.00% | 99.86% | 0.14 |
| add | 99.08% | 99.00% | 0.08 |
| sub | 98.83% | 98.83% | 0.00 |
| mul | 33.33% | 35.58% | −2.25 (word higher) |
| multi-op | 16.25% | — | — |
Encoding is essentially free. Every operation lands within ~2 points across the two
encodings; subtraction is identical to two decimals. The model learned that multiplied by — a 13-token phrase — and x denote the same operation, rather than treating them as
separate tasks.
Both encodings fail identically, which is the stronger evidence: 7 ^ 3 → 327 and
7 to the power of 3 → 327. Same wrong digits from the same model on the same problem
written two ways. Parallel lookup tables would diverge.
Best add/sub/div of the study (99.08% / 98.83% / 100.00%) while carrying five operations, compositional expressions, two encodings and a 40-token vocab. SFT held-out loss 0.1905, best of any multi-operation run. Throughput 371,209 tok/s — doubling the vocab cost nothing measurable.
Mul (33–36%) and multi-op (16.25%) sit where they have all study, with the same by-length collapse (mul: 86% at 5-digit answers, 6% at 9-digit). Encoding, composition, operation count and vocabulary size all leave the digit-carrying ceiling untouched.
Note: dropping --compact reformatted every line (73 + 5 = 78, not 73+5=78), so run
16's absolute numbers are not comparable to runs 1–15. The internal symbolic-vs-word
comparison is the controlled one.
| split | run 14 | run 15 |
|---|---|---|
| div | 99.83% | 99.89% |
| add | 98.53% | 97.67% |
| sub | 98.40% | 95.87% |
| mul | 43.53% | 28.47% |
| multi-op | — | 15.27% |
Adds --multi-op 0.20: compositional lines like 52*(46+33)-91=4017 requiring nested
sub-expressions and operator precedence. Every earlier run was single-operation only.
Vocab 17 → 20 (*, (, )).
Composition is not the barrier — digit carrying is.
| metric | multi-op |
|---|---|
| correct answer length | 91.2% |
| correct first digit | 83.0% |
| exact match | 15.27% |
189+(532132-432349)*6 → 588003 (gold 598887); 429*7^3+8 → 145635 (gold 147155).
Right magnitude, right leading digits, wrong middle. (818-(1463-645))*7=0 is exactly
right including the nested subtraction. The model resolves parentheses and applies
precedence correctly; it runs out of capacity carrying digits through long results, and
compositional answers are long (63% are 6+ digits). The by-length curve mirrors
multiplication.
Two comparisons in this run are unsafe. Pow scored 0.00%, but the 38M corpus absorbed every short-answer pow problem, so the surviving held-out set contains only 7–12 digit answers — none of the 1–6 digit range where run 14 scored 73–100%. It measured pow only where every operation is near zero. The mul drop (43.53% → 28.47%) is partly the same effect: this eval split skews longer. Matched eval sets would be needed to claim either number.
| operation | run 13 (four ops) | run 14 (five ops) |
|---|---|---|
| add | 97.80% | 98.53% |
| sub | 95.93% | 98.40% |
| div | 100.00% | 99.83% |
| mul | 42.00% | 43.53% |
| pow (3,229 held out) | — | 20.00% |
Adding a fifth operation cost nothing — add and sub improved, mul edged up, div held.
511M tokens in 1373s at 372,196 tok/s; SFT 58s; vocab 16 → 17 (^).
The generator's pow had to be fixed first. It was hardcoded to base 2–9 / exp 2–4 —
24 distinct problems in the entire space, smaller than the 136-problem single-digit set
that earlier runs showed gets memorized. It now samples (base, exp) uniformly over every
pair whose answer fits the digit budget: 21,533 problems, 9,998 bases, exponents 2–39,
and the space scales with --max-digits like the other operations. That turned pow from a
lookup table into a real generalization test, with 3,229 problems held out of both
pretraining and SFT.
Pow's 20% is answer length, not exponentiation. By answer length: 3-digit 100%, 6-digit 73%, 7-digit 55%, 8-digit 13%, 12-digit 1%. 68% of held-out pow problems have 8+ digit answers — the band where multiplication is already at 19%. Where answers are short, exponentiation works as well as any other operation.
It is learned, not memorized. Held-out pow vs the training portion converges at 8+ digits (13% vs 16%, 1% vs 2%, 1% vs 4%). A memorizing model would keep the training portion high; instead the same length-limited computation runs on both.
2^32 → 4144961696 (gold 4294967296): right length, right leading digit, wrong
middle — the multiplication signature — despite 84 exposures in training. 2^10, 2^16,
3^5, 7^3 are all correct.
| operation | run 11 (1–9d, 20tpp) | run 12 (1–4d, 100tpp) | run 13 (1–6d, 20tpp) |
|---|---|---|---|
| div | 99.93% | 68.27% | 100.00% |
| add | 94.73% | 92.60% | 97.80% |
| sub | 93.53% | 88.20% | 95.93% |
| mul | 32.27% | 0.40% | 42.00% |
| SFT held-out loss | 0.2165 | 0.4065 | 0.0953 |
Run 12 was a negative result that located the real constraint. Narrowing to 1–4 digits to shorten products made multiplication worse (0.40%), because the 1–4 digit four-op space is only ~10⁸ problems and a 38M-line corpus exhausts it. The model saturated at step 1600 of 15600 — ~10 tpp effective, not 100 — then memorized: train loss plateaued at 1.31 while eval drifted 1.62 → 1.83 and never recovered.
Run 13 kept four operations but widened to 1–6 digits (~2×10¹² problems) at 20 tpp. Train and eval tracked within 0.007 through step 3000 and it ran to completion. Same architecture, same SFT recipe, a fifth of the nominal budget — better on every operation.
Multiplication degrades smoothly with product length rather than failing outright (post-SFT: 4-digit 79%, 5-digit 77%, 6-digit 60%, 7-digit 41%, 8-digit 8%), consistent with a capacity limit on carrying partial products. Run 12's 0.40% was an artifact: saturation forced its held-out set entirely into the 7–8 digit band where mul is weakest.
First partial length generalization in the study. On never-seen 7–8 digit operands run 13 scores 12.27% — 81% where the answer is 5 digits, 74% at 6, 0% at 8. Every run through 12 scored exactly 0.00% outside its training range. It does not extrapolate to longer answers, but it handles longer operands when the answer stays familiar.
The same recipe (55s, ~0.6M supervised tokens) applied to run 8's checkpoint. The SFT set is balanced by (operation, answer length) across 39 buckets, since mul/div answer ranges differ from add/sub.
| operation | pre-SFT | post-SFT |
|---|---|---|
| div | 60.53% | 99.93% |
| add | 75.13% | 94.73% |
| sub | 83.20% | 93.53% |
| mul | 5.20% | 32.27% |
Strict equals lenient on all four afterwards. Division is 100% at every answer length 1–9 digits — it was almost entirely a formatting failure.
Multiplication's diagnosis changed. Pre-SFT it got the length right 54% and the first
digit 12%, which read as pure computation failure; post-SFT those are 99.0% and 95.7%,
and small products are perfect (9x9=81, 12x11=132). What remains is a size gradient —
2-digit answers 100%, 4-digit 70%, 7-digit 36%, 11-digit 2% — failing in the middle digits
of long products (345x67= → 22415, gold 23115). That part is real capacity and
output supervision does not touch it.
Multiplication is not learnable at this scale, and not because of length. It gets the
answer length right 54% of the time and the first digit right only 12% — it knows how big
the answer should be and cannot compute it (9x9 → 1011). Addition is digit-local with
a carry; multiplication needs every partial product simultaneously.
Exact division reaching 60.53% is a property of the corpus, not of division: the
generator builds it backward as a = q·d with d in 2–9, so recovering q is closer to
a digit-local scan than to a full product.
Splitting the same 5.1M params and token budget across four operations costs ~10 points on add/sub versus run 7.
Findings:
- Train on the lengths you need. Extending the training range moves the wall with it; it never removes it. 10–12 digit is 0.00% across four runs and a 5× token sweep (12.8 → 100 tpp).
- Filling the short end lifts the whole range. Adding 1-digit operands gained +14 to +22 points at every answer length, not just the short ones.
- More tokens are not uniformly good. 100 tpp beat 20 tpp on 5–6 digit (80.85% vs 59.25%) but lost ground on 2–4 digit (34.85% vs 47.50%). At 5.1M params the model reallocates capacity toward wherever the data mass sits.
- The short-side wall was a formatting bug, not a capability limit. 40s of SFT with prompt-masked loss took the 2–4 digit eval from 60.15% to 99.05%, finally beating run 3's 78.70% by 20 points. The long-side wall did not move at all.
- Two different walls. Long side: cannot compute. Short side: computes
perfectly and cannot stop — on 1-digit answers the first emitted digit is
correct 91/91 = 100%, but
<eos>ranks 10th on average, so it runs on (1+2=→3633). Different failures, different fixes. - Do not select checkpoints by loss.
ckpt_finalbeat the loss-selectedckpt_beston every split in both runs where both were scored, by up to 10 points, on loss differences of 0.0005 — noise. See Checkpoint selection.
Every run from 1 to 26 held head_dim = 32, scaling head count with model width. That
constant kept the scaling series clean, and it also locked in an untested choice. Head
count is parameter-free — attention is 4·d² regardless of how it splits — so this
sweep changes accuracy at fixed model size.
A four-point sweep at 0.25M (dim 64, L5, ~270.6K params in every arm, spread 280 params):
| head_dim | heads | eval | SFT held-out | add | sub | div | tr_carry | tr_reduce |
|---|---|---|---|---|---|---|---|---|
| 32 | 2 | 0.7262 | 0.3657 | 45.25% | 53.75% | 82.25% | 48.00% | 98.25% |
| 16 | 4 | 0.7062 | 0.3485 | 65.00% | 78.50% | 99.25% | 66.00% | 98.00% |
| 8 | 8 | 0.7259 | 0.3546 | 61.25% | 40.00% | 100.00% | 53.33% | 25.60% |
| 4 | 16 | 0.7421 | 0.3683 | 56.25% | 70.50% | 87.75% | 56.00% | 34.00% |
The curve is U-shaped — 16 is a floor, not a direction. head_dim=8 lands back at the
head_dim=32 baseline; head_dim=4 is worse than the original. RoPE frequency pairs halve
with head_dim (16, 8, 4, 2), so two forces trade off: more heads give more positional-routing
subspaces, fewer dimensions per head give coarser positional resolution. tr_reduce is the
tell — ~98% at head_dim 32 and 16, then collapses to 25.60%/34.00%. The longest-range chains
break first when frequency resolution runs out.
| 5M · dim 256, L6 | baseline (8 heads) | run 27 (16 heads) | Δ |
|---|---|---|---|
| eval loss | 0.6825 | 0.6820 | −0.0005 |
| algebra | 44.25% | 64.25% | +20.00 |
| add | 88.50% | 93.50% | +5.00 |
| tr_partial | 60.40% | 67.60% | +7.20 |
| sub / div / mul | 95.75 / 100.00 / 6.00% | 96.50 / 100.00 / 6.75% | +0.75 / 0 / +0.75 |
| tr_sum / tr_reduce / tr_carry | 77.20 / 58.00 / 60.67% | 75.20 / 56.40 / 59.33% | −2.0 / −1.6 / −1.3 |
| throughput | 373,985 tok/s | 353,879 tok/s | −5.4% |
This is the clearest case in the study for judging on accuracy, not loss. The 5M arm's loss difference is 0.0005 — indistinguishable. Stopping there would have discarded the largest algebra gain in twenty-seven runs. The three small trace regressions sit inside the ~1.6pp SFT seed variance measured in the adjacent repo (an inherited estimate, not measured here).
More heads buy relational binding, not arithmetic. At 0.25M, 2 → 4 heads lifted a broad capability floor — add, sub, div and carry traces all moved 17–25 points — because 2 heads was genuinely starved. At 5M the arithmetic is already past that floor and the same change moves only algebra: the one task requiring a variable bound across an expression rather than digit-local computation.
Checkpoints in checkpoints/run27_heads/.
vocab_110.py (108 tokens — full upper/lowercase, punctuation, brackets, whitespace) was
built during run 26 and never wired in. Run 27 promoted it to back synth_word.py, a
dependency-free generator of grammatical basic-English sentences (imported from
~/.unsloth/studio/sandbox/__LOCALID_J6Xwkgq/). It stays under 255 tokens so the corpus
cache remains uint8.
A 274K-parameter model at 100 tpp — 815 steps, 29 seconds — reaches eval 0.7628 and produces correct subject–verb agreement, determiner–number agreement and clause structure:
Some quick doctors closed these chairs quickly at that sweet mountain now, but he never sold you from…
He had known the moons at the villages later, so we drink badly late…
She is drinking these angry cakes.
Orthography is not fully locked in at this budget (taall, earlyy, 7rass) — the obvious
thing more compute fixes. The English loss is not comparable to the arithmetic losses:
different vocab (108 vs 55) and a different task.
Two traps worth knowing:
encode()is not vocab-portable. The version in the older trainers special-cases<eos>/<think>/<ans>/<end>by literal character offset (i += 5,i += 7), so a vocab with different multi-char tokens is mis-tokenized silently.bench_eng_hd16.pyuses a generic longest-match loop; copy that when changing vocab.- Cross-file contamination again.
synth_word.pydedups within a file but not across files, leaking 629 eval lines (3.15%) from train — the same failure this project already documents forgen_splits*.py. Always diff the splits.
The generator's README documents -n/-o short flags that its argparse did not define;
added on import.
The original train_fast.py ran at ~53,700 tok/s. The same model now runs at
~372,000 tok/s — a 6.9x speedup, and about 1.9x the reference repo's 5M run
(../llm-modern-arch-experiment, 193,236 tok/s). A full 20-tpp run went from a
projected ~20 minutes to 276 seconds.
Three causes, in order of impact:
The original carried the comment "no torch.compile: overhead > gain at 5M". That
is backwards. At dim=256 every kernel is tiny and launch-bound, so eager mode
pays kernel-launch overhead on ops that take microseconds. This is exactly the
size where compile helps most. The reference repo compiles both the model and the
Newton-Schulz kernel (train.py:443, muon.py:17).
The ordering trap: torch.compile prefixes every parameter name with
_orig_mod., and build_optimizer splits Muon vs AdamW by matching names starting
with blocks.. Compiling before building the optimizer routes zero parameters
to Muon and dies with ValueError: optimizer got an empty parameter list.
Compile after build_optimizer. Verify the split: 5,111,808 params to Muon,
7,296 to AdamW. The same trap is noted at reference muon.py:132.
Checkpoints must also be saved from the unwrapped module or the _orig_mod. prefix
leaks into state_dict and breaks loading:
torch.save(getattr(model, "_orig_mod", model).state_dict(), ...).
random.shuffle(train_seqs) # 258k-element Python list, ~70 ms/step
batch = torch.tensor(train_seqs[:MICRO], device=dev)It shuffled 258,000 entries to select 128, every step. Replaced with a shuffled
randperm walked in order, which is both O(MICRO) and gives a real no-repeat
guarantee (STEPS*MICRO <= corpus, asserted at startup) rather than
sampling-with-replacement.
Walked all eval sequences — 26% as many tokens as the training steps between evals. Now a fixed 512-sequence subset; holding it fixed also tightens step-to-step comparison.
Also: the corpus is materialized once as a resident GPU tensor (int16 — the
vocab is 14 tokens, so it is lossless and cuts gather bandwidth 4x), instead of
rebuilding CPU tensors per step.
The 1.9x figure over llm-modern-arch-experiment is not an apples-to-apples
win. Measured directly, with identical body, batch, GPU and compile settings:
| vocab | tok/s |
|---|---|
| 14 (ours) | 385,339 |
| 16,384 (reference) | 251,045 |
Vocab size alone is worth 1.53x. At dim=256 the lm_head dominates: it is
0.07% of forward FLOPs at V=14 versus 45% at V=16,384, and the logits tensor
is 0.92 MB versus 1,073 MB. Correcting for vocab, our advantage is ~1.2x, most of
which is the GPU-resident corpus. The reference streams a 16 GB file and cannot do
that. Tied vs untied embeddings is negligible (~1.5%).
Note the reference README counts "body" params to exclude the vocab as doing "no matmul work" — true for the embedding lookup, but the lm_head projection is very much matmul work at this size.
No python on PATH — use python3 (3.12). Torch 2.14+cu130 is system-wide and
sees the GPU (NVIDIA GB10). No venv, no requirements file, no test suite.
Scripts read and write bare filenames in the current working directory. Work
in a scratch dir with the data files flat, as run4/ does (symlinked scripts,
generated corpora local). Nothing reads from data/.
python3 gen_splits1d.py # corpus: 1-9 digit, 6M lines, + oversampled 1-digit
python3 make_reversed.py --in train1d.txt --out train1d_rev.txt --seed 41
python3 train_fast.py # 20 tpp = STEPS 3120, ~276s
python3 eval_any.py ckpt_final.pt --rev --n 2000 \
--splits "eval=eval1d.txt" "gentest=gentest1d.txt"
python3 audit_corpus.py # generator correctness; expect TOTAL ERRORS: 0eval_any.py supersedes evaluate_math.py: it takes any checkpoint and any
splits, batches the decode, and reports a per-answer-length breakdown plus both
scorers. evaluate_math.py is kept for provenance — it reproduces the original
78.70% exactly.
Held-out sets can be contaminated by construction. The 10–12 digit
gentest_ops.txt scored 9.67%, which looked like the first crack in the length wall. It
was an artifact: --min-digits 10 bounds the dividend, but division is built as
a = q·d with small d, so quotients land back inside the trained range. Restricted to
genuine 10–12 digit add/sub the score is 0.00%. When holding out by operand length, check
the answer length too.
Cross-file contamination. The generator dedups within a file, not across
them. Every split pair generated so far leaked train lines into eval — 533 at 3M
lines, 744 at 6M, 1,344 at 30M. Always diff eval against train and drop the
overlap. gen_splits*.py does not do this; the audit step in the workflow above
does.
Scoring caveat. The scorer accepts an answer in forward or reversed order, because training uses 50% reversed answers. This matches the convention behind the published 78.70%, but it cannot distinguish "right digits, right order" from "right digits, wrong order". Strict forward-only accuracy is much lower and swings wildly with which emission mode a run happens to settle into (run 3 emitted mostly forward; later runs mostly reversed). Compare runs on the either-order number, and do not read the strict number as a capability measure.
Checkpoint selection. ckpt_best.pt is chosen by eval loss, and eval loss is a
poor proxy here. In the 100-tpp run ckpt_final beat it by 8–10 points on every
split while being 0.0005 worse on loss; in run 8 the ordering reversed. The selection is effectively arbitrary — score both.
Single-digit is exhaustive. Only 136 distinct 1-digit add/sub problems exist, so train/test overlap in that band is total by construction. The 11.03% figure measures "can it emit a 1-digit answer at all", not generalization.
STEPS also sets the LR schedule. The cosine spans STEPS, so changing the
token budget rescales the schedule — which is correct, but means two runs at
different budgets are not comparable mid-run, only at completion.
math_corpus_generator.py generator (add/sub/mul/div/pow, algebra, multi-op, word forms)
audit_corpus.py re-parses 20k generated lines and asserts every answer
gen_splits*.py split drivers: 9 = 2-9 digit, 20 = 20 tpp, 100 = 100 tpp, 1d = 1-9 digit
make_reversed.py train.txt -> train_rev.txt (50% reversed answers)
train_fast.py trainer (ModernGPT + Muon + AdamW, bf16, compiled)
train_math.py run 1 baseline (GPT-2-ish, AdamW, fp32) - different architecture
eval_any.py parameterized evaluator
checkpoints/run7_1d_20tpp/ best model
results/ per-run JSON + logs