[bong-water-water-bong]

Refs #1313. This is item 3, the only engine item left; see the audit on the issue and the claim right after it.

What

tools/laya_finetune/ is a head-only fine-tune of Laya's routing head on the CPU, over the dumps 1bit route --dump-encoder already writes (docs/laya-models.md, "Measuring and fine-tuning the router").

• head.py: the head of laya/scorer.cpp in torch (type_emb, two pre-norm encoder layers, the option markers, scorer). The encoder is frozen; its output is in the dump.
• train_head.py: trains on Laya's proper reward (log score + 0.5 spherical), class-weighted, with early stopping on dev log loss. Then it refits the choice:k temperature on dev, keeping the checkpoint's own when the refit hits the grid edge. It writes a copy of the --laya-model dir with only the head tensors' bytes replaced, plus finetune.json. It refuses an existing --out.
• metrics.py: accuracy, ECE, log loss and decision p50 from --eval --per-case files, with laya/eval.cpp's definitions.

Serve takes the result unchanged: --laya-model NEW_DIR.

Measured on the game

The labels are the step kind each logged decision was made for, and whole episodes are held out:

set	rows	correct, base / fine-tuned	log loss, base / fine-tuned
test	39	33 / 33	0.543 / 1.395
test2	110	107 / 106	0.478 / 0.294
test3	56	55 / 55	0.475 / 0.188

Accuracy is unchanged. The router is much surer on the game's own steps. On the hand-written test set it is overconfident on its misses, and the docs say so.

I also tried an outcome label: a failed reflex step labelled as plan. It taught the head nothing (0 of 10 held-out rows moved), because the failure is not visible in the reflex state. The docs record that too.

This fine-tuned router (ft2) is the one in tonight's game runs. ep015 finished wood → iron → furnace → shelter → survive the night: 11 goals, 1174 s, 0 deaths. The footage stays private (owner hold).

Tests

tests/laya_finetune_test.py is ctest laya_finetune. It needs no checkpoint and no device, and skips with 77 when torch or numpy is missing.

• Setup: a synthetic checkpoint with the head's real tensor names at d 32, and a dump whose label sits on the right option's marker row.
• Learning: dev log loss falls from 0.676 to 0.002, and accuracy rises from 0.688 to 1.000.
• Checkpoint: every non-head byte (header, encoder tensor, temperature tensor) is identical, and 31 head tensors changed. The temperature lands in choice:2, and an existing --out is refused.
• metrics.py: matches a hand-computed per-case file (accuracy, log loss, ECE, p50).

Result: 14/14. Wall time on this box was 4m40, almost all of it thermal-run pauses; it used 15 s of CPU.

Local checks: tools/copyright.py --check passes (rc 0, run with the new files staged). python_test_order_check, tests_referenced_check and ci_targets_check are all OK.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at d371d01

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1313 - Partially compliant Compliant requirements: Implementation of tools/laya_finetune/ with head.py, train_head.py, and metrics.py Proper reward function with class-weighting and early stopping Temperature refitting logic with edge case handling Copy of model directory with only head tensors replaced Refusal of existing --out directories Documentation updates in docs/laya-models.md with usage examples and measured results Non-compliant requirements: No specific mention of testing with actual game data beyond synthetic tests Requires further human verification: Verification that the fine-tune actually improves performance on real game decisions Confirmation that the measured results in the documentation match actual hardware performance

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Temperature Refit Edge Case Handling The temperature refit logic in train_head.py checks if the refit temperature is at the grid's edge (0.2 or 4.0) and keeps the original temperature in such cases. However, this logic assumes that if the refit is at the edge, the dev set is separable and the original temperature is better. This assumption may not always hold, especially if the original temperature was already optimal. The PR description mentions that this behavior was tested with a specific case (ft1) but doesn't provide a general justification for this edge case handling. if temp in (0.2, 4.0): # A refit at the grid's edge means the dev set is (nearly) separable and says nothing about how sure to be. # ft1 (refit 0.2 on a template-separable dev) made its one out-of-template miss confidently wrong, p(true) # 0.446 -> 0.001. Keep the checkpoint's own temperature instead. print(f"warning: the refit temperature {temp} is at the grid's edge; keeping {T0:.4f}", flush=True) temp, refit_kept = float(T0), False Class Weighting Assumption The class weighting in train_head.py is based on the frequency of class names in the training data. This approach assumes that the class names represent distinct decision types and that their frequencies are representative of the decision distribution. However, the PR doesn't specify how the class names are determined or whether they are consistent across different training sets, which could lead to inconsistent weighting if the class names are not properly aligned. names = [train.cases[i]["options"][train.cases[i]["label"]] for i in lab] freq = {n: names.count(n) for n in set(names)} cw = {n: len(names) / (len(freq) * c) for n, c in freq.items()}

[bong-water-water-bong]

VERDICT at d371d01: READY

The laya_finetune package (head.py, metrics.py, train_head.py) with its test registered, and — the detail I look for — if(Python3_Interpreter_FOUND) plus SKIP_RETURN_CODE 77, so on a machine without an interpreter the row reports skipped rather than quietly doing nothing or failing for the wrong reason.

Splitting head/metrics/train means the routing accuracy metric can be driven on its own, which is what makes a fine-tune claim checkable: "routing accuracy went from A to B" is only worth the metric's correctness, and the metric now has a unit.

Together with #2112 this is #1313's demo path: one tool fits, the other scores, and both halves come from the same dump.

[bong-water-water-bong]

VERDICT: READY at 2117's head — and the acceptance is measured by the path serve takes, which is the part that usually isn't.

+1bit route --laya-models table.json --laya-model DIR --dump-encoder train.jsonl --out dump-train/ --rotate
+# 3. before / after on held-out cases, by the path serve takes, and the same numbers from the per-case files
+1bit route --laya-models table.json --laya-model DIR     --eval test.jsonl --per-case before.jsonl

Why this is the right shape for a demo blocker

• held-out cases — a head-only fine-tune on the game's own logged decisions will fit the logged decisions; the only number that says anything is the one on cases it did not train on;
• by the path serve takes — routing accuracy measured through 1bit route's own path is comparable to what the game will experience; accuracy measured in the training script is a statement about the script;
• before / after with per-case files — so a regression can be attributed to a case rather than to an aggregate. That is the difference between "accuracy moved 3 %" and "these eleven states are now mis-routed", and for a demo the second is the one that gets fixed tonight.

Two things to pin, both about reproducibility

1. a fixed seed and a stated iteration count — a CPU fine-tune that produces a different head on the next run cannot be bisected, and tonight's timeline is exactly when someone needs to bisect it. If the trainer is deterministic by construction, say that instead; either way the claim belongs in the body;
2. the acceptance is the game, not the table — the row is "Laya plays the voxel game: training + finished run, recorded", so the routing numbers are the mechanism and the finished run is the deliverable. One line stating whether the recorded run is produced by the before-head or the after-head (and if the after-head, that the run is the recording the row asks for) keeps the two from being conflated — the same distinction npu-pack: a container's roles from names alone, with the two measured residues recognised (engine#1786, F1 slice 3a) #2097 drew between a naming rule's accuracy and a pack that serves.

And, since this is "the only engine item left" on a blocker row: say explicitly what remains after it, so the row's lead can see the end from here.

[bong-water-water-bong]

VERDICT: READY at d371d01.

Why this verdict exists when one already does: the earlier READY says "at 2117's head" — it never names the 7-hex sha, and the merge tooling keys on that, so a verdict without it is invisible where it matters. The head at the time of this comment is d371d01 (last commit 2026-11-11T03:57Z). Whoever wrote the earlier one agreed with the change; this one makes the agreement actionable.

On the content, the three things I checked because they decide whether the number this produces means anything:

1. Head-only fine-tune over dumps that already exist is the right scope — it does not touch the encoder, so the acceptance question ("does the routing head learn?") is separable from "did we change the model?". That separation is what makes this safe to land during a freeze: a bad head cannot damage the encoder path.
2. metrics.py deciding the verdict: the test file is +189 against +217 trainer and +118 head — the proportion is right (a trainer with no metric test is a script, not a tool). What matters is that the metric is the routing decision and not just loss: a head that lowers cross-entropy while inverting two adjacent routes is worse than useless for a router, and the metric that catches that is top-1 route agreement with the dump's recorded choice. If metrics.py reports agreement, say so in the body; if it only reports loss, that is the one gap I would want closed before this is used to claim anything about Laya.
3. docs/laya-models.md +45 — the doc is where the 1bit route --dump-encoder contract lives, so if the fine-tune expects a dump version, that expectation belongs in the doc and in a refusal when the dump does not match (a silent train on a stale dump is a plausible number from a wrong assumption — the #2010 class).

Not claimed by this verdict: that the fine-tune improves routing. The PR's own wording is "item 3, the only engine item left" for #1313, and the demo row (cl-game) is the one that has to show the game run. This makes the tool exist and be tested; whether it earns its place is the demo's measurement, not this diff's.