Repository navigation
[bitnet] Maturity gate: golden-token parity, model-gated smoke, entry-point parity + CLI knobs #360
Description
Problem
#346's maturity gate stands at 0/3 for BitNet:
- No llama.cpp golden-token parity test.
- No
tests/smoke/smoke-models.jsonrow — PR SKaiNET-transformers → engine 0.51: full engine-loader migration, MAPPED default, BitNet fold-in #353's "3/3 pass … BitNet-2B4T" run was manual, nothing gates regressions. - Entry-point parity 1/3 — only the unified
skainet-cli; no chat-model facade, no family CLI (the facade arrives with [bitnet] Full #346 conformance: BitNetWeightLoader/RuntimeWeights naming + llm-runtime/kbitnet facade #359'sllm-runtime/kbitnet).
Plus CLI rough edges on the BitNet path:
i2sLayouthardcoded toGROUP_128at the call site — theI2sGgufLayoutknob (GROUP_64 / SEQUENTIAL) is not exposed, and a wrong flavor only "fails fast on code 3 where possible".--contextsilently ignored (Main.kt:246-248).- The explicit
NativeTernaryF32GemvKernel.install/NativeTernaryLmheadKernel.installatMain.kt:249-250are the only kernel bootstraps that survived the 0.52 sweep — load-bearing (the ternary packs are not in the engine's self-healing SPI, Ternary kernel packs are missing from the self-healing dispatch SPI — consumers silently get the int8 fallback SKaiNET#1240) but uncommented, so they look like a missed cleanup.
Tasks
- llama.cpp golden-token parity probe for BitNet (reuse the gemma eval-callback pattern, commit 8a7f1f1)
- BitNet row in
tests/smoke/smoke-models.json(time-bounded, model-gated) - Entry-point parity once
llm-runtime/kbitnet([bitnet] Full #346 conformance: BitNetWeightLoader/RuntimeWeights naming + llm-runtime/kbitnet facade #359) exists -
--i2s-layoutCLI flag (GROUP_128 | GROUP_64 | SEQUENTIAL) - Honor
--contexton the BitNet path - Comment the explicit ternary kernel installs as load-bearing now; remove them once Ternary kernel packs are missing from the self-healing dispatch SPI — consumers silently get the int8 fallback SKaiNET#1240 ships in a consumed release
- README family table + a BitNet docs page (
weight-quantization.adoccarries two lines today)
Verification
- Gate: parity + smoke jobs green in CI with the model present, skipped-not-failed without it.
--i2s-layout sequentialloads a NeoGPU-converted GGUF; wrong-layout run fails with a clear message.
Refs: #346 (maturity gate), #335, #359, SKaiNET-developers/SKaiNET#1240.
Activity
michalharakal commentedon Aug 31, 2026
Status after PR #364: parity probe, smoke row, and CLI knobs are in. Remaining here:
llm-runtime/kbitnetfacade landed in refactor(bitnet): full #346 conformance — BitNetWeightLoader, RuntimeWeights, kbitnet facade (#359) #363; a chat-facade/family-CLI entry point is still open.Main.ktonce SKaiNET#1240 (PR SKaiNET#1241) ships in a consumed BOM.microsoft/bitnet-b1.58-2B-4Tsafetensors) is unverified. Proposal: an eval-callback-style probe against HF transformers BF16 (the gemma 8a7f1f1 method), gated like the parity test. Details and the oracle's top-5 at the divergence position are in the committed fixture header.Merge pull request #364 from SKaiNET-developers/feat/bitnet-maturity-…
michalharakal commentedon Aug 31, 2026
E5 (BF16 arbitration) is resolved — and it found a real bug. The HF BF16 reference sided with bitnet.cpp: SKaiNET was the diverging implementation, and the culprit was the RoPE pairing (
INTERLEAVEDinstead of the NEOX-styleSPLIT_HALFbitnet.cpp assigns all three BITNET arches). Fixed in PR #365; with it, SKaiNET, bitnet.cpp, and the BF16 reference agree token-for-token on all 32 greedy tokens of the parity prompt, and the committed fixture now gates on that full equality. Details, culprit-isolation steps, and the one documented dense-head near-tie are in the PR and fixture header.Remaining on this issue: entry-point parity (E3) and dropping the CLI's explicit kernel installs once a BOM containing SKaiNET#1241 is consumed.
michalharakal commentedon Sep 1, 2026
E3 done in PR #371: a real BitNet chat provider (
BitNetChatTemplateper the HF checkpoint's authoritative format — the GGUF's embedded template is a conversion artifact — plusstopTokenStrings()so the loop stops at<|eot_id|>despite the GGUF declaring 128001 as EOS). Verified end-to-end on 2B4T: clean auto-detected chat, correct answer, proper stop.E4's last piece (dropping the CLI's explicit kernel installs) is in PR #370 (the 0.52.1 BOM bump — merge after the engine release publishes). With #370 + #371 merged, everything on this issue is done and it can close.
Merge pull request #371 from SKaiNET-developers/feat/bitnet-chat-support
michalharakal commentedon Sep 2, 2026
Closing — every item is in:
--i2s-layout/--contextknobs: PR test(bitnet): maturity gate — bitnet.cpp parity probe, model-gated smoke row, CLI knobs (#360) #364BitNetChatTemplate+ stop tokens, PR feat(chat): BitNet chat support — correct template, eot_id stop, resolver entry (#360 E3) #371;llm-runtime/kbitnetfacade, PR refactor(bitnet): full #346 conformance — BitNetWeightLoader, RuntimeWeights, kbitnet facade (#359) #363NativeTernary*Kernel.install(...)calls are insideBitNetWeightLoaderTest/BitNetGoldenTokenParityTest, where pinning the kernel under test is intentional.Released in transformers 0.53.0.