Native C++20 & OpenXLA Inference Engine for Google Cloud TPUs (Gemma 4 + Qwen3)
Video Explainer (9:04, 720p) & Audio Overview (6:16): Watch the Video Explainer on YouTube and Listen to the Audio Deep Dive (
TPUForge_Audio_Overview.m4a), covering Cloud TPUv6eTrillium's256x256MXU batch-1 paradox (99.61%idle rows), C++20 PJRTStaticArena(1vs5compiles,39.30sdown to4.46son Gemma 412B), collapsing Gemma 4's268 MB/stepPCIe logit wall to1 KBvia HBMjnp.argmax, hybridDFlash+ zero-KV AST drafting, andINT8single-chip fit (167.54 tok/s/30.30xon26B-A4BMoE and328.97 tok/son12Bwith128/128exact parity).
Most LLM serving stacks were built around GPU warps, asynchronous CUDA streams, and dynamic-shape runtime graphs. On Google Cloud TPUs, standard batch-1 token decode leaves 127 of 128 rows idle on a v5e Matrix Multiply Unit (255 of 256 on v6e Trillium and v7), while unaligned prompt shapes stall behind multi-second XLA recompilations.
TPUForge aligns static HBM allocation, 128/256-row systolic tree verification, native INT8/FP8 quantized dot_general, host-mmap MoE streaming, and optical torus Ring-Attention directly to TPU silicon geometry. Evaluated live on Cloud TPU v6e-1 Trillium (31.24 GiB HBM, 256x256 MXU) with late-2026 Google Gemma 4 (Apache 2.0, ungated), TPUForge hits 167.54 tok/s (30.30x over Vanilla JAX / 12.17x over Static Arena, 128/128 exact parity) on google/gemma-4-26B-A4B-it Sparse MoE (INT8 single-chip HBM), 328.97 tok/s (12.07x over Vanilla INT8, 18.00x over BF16) on google/gemma-4-12B-it INT8, 186.95 tok/s on 12B BF16, and 900.55 tok/s on Qwen3-4B, alongside verified Cloud TPU v5e-1 baselines (202.80 tok/s on Qwen2.5-Coder-7B, 328.79 tok/s on 3B).
| Mode | Needs Cloud TPU? | Command | What You Verify |
|---|---|---|---|
| 1. Build C++20 runtime & CLI | No | cmake -B build -S . -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build build && ./build/tpuforge_bench --tile-size 128 |
Native pjrt_c_api.h driver, StaticArena, and 128/256-row systolic tree packing. |
| 2. Run offline verification suite | No | python -m pytest -q |
All 59 offline C++/nanobind, PJRT arena, Serve + HUD parity, Gemma 4 (dense/MoE/DFlash/TP=2), INT8/FP8, MoE mmap, and AST-seeding checks. |
| 3. Run OpenXLA CPU parity bench | No | python -m benchmarks.run_qwen_bench --model google/gemma-4-12B-it --draft-model z-lab/gemma4-12B-it-DFlash --weight-quant int8 --ast-seed-path . |
1-compile static arena (M-002), 128-node hybrid DFlash + AST tree parity (M-003/M-006/M-007), and exact match vs Vanilla JAX. |
4. Run on Cloud TPU (v5e / v6e) |
Yes | Open colab/TPUForge_Benchmark.ipynb or standalone benchmarks/results/ |
Multi-chip Ring-Attention (128k), Gemma 4 12B/26B-A4B/31B, INT8 single-chip fit, mmap MoE streaming, and Qwen3-4B check. |
| 5. Launch Preset Server & Live HUD | No | python -m benchmarks.tpuforge_serve gemma4:12b --ast-seed-path . |
Ollama-style TPU preset autofit, OpenAI /v1/chat/completions SSE API, and interactive 128/256-lane Systolic Matrix HUD. |
# Local native C++20 build + offline verification suite (Clang/GCC/MSVC + CMake + Ninja)
cmake -B build -S . -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
./build/tpuforge_bench --tile-size 128
./build/tpuforge_bench --tile-size 256
pytest -q- 1. Target Lineup: Google Gemma 4 on Cloud TPUs
- 2. Verified Cloud TPU Silicon Telemetry (
v6e-1Trillium &v5e-1) - 3. Cold-Prompt Drafting, Quantization, & MoE Extensions (
M-006..M-008) - 4. Architecture and Subsystem Design
- 5. Command Reference and Hardware Matrix
- 6. License
All flagship runs use ungated Apache 2.0 Google Gemma 4 weights implemented natively in pure JAX/OpenXLA (benchmarks/gemma4_jax.py) without transformers or torch runtime overhead:
| Target Pair | Role & Architecture | Weight Footprint | Primary TPU Mapping & Hardware Physics | Cloud TPU Status |
|---|---|---|---|---|
google/gemma-4-26B-A4B-it + z-lab/gemma-4-26B-A4B-it-DFlash |
Flagship Sparse MoE (128 experts, top-8, 4B active) |
48.07 GiB BF16 -> 24.95 GiB target HBM at INT8 |
Fits 100% inside 31.24 GiB v6e-1 HBM (INT8), runs TP=8 on v5e-8, or streams via host mmap on 15.75 GiB v5e-1 (3.81 GiB HBM trunk + static KV). |
Verified v6e-1 Silicon (167.54 tok/s, 30.30x vs Vanilla / 12.17x vs Static) |
google/gemma-4-12B-it + z-lab/gemma4-12B-it-DFlash |
Single-Chip Dense (48 layers, 5 sliding : 1 full) |
22.18 GiB BF16 -> 12.97 GiB target INT8 (0.585x) |
Fits inside 15.75 GiB v5e-1 HBM via native INT8 systolic dot_general; benchmarks both BF16 and INT8 on 31.24 GiB v6e-1. |
Verified v6e-1 Silicon (328.97 tok/s INT8, 186.95 tok/s BF16) |
google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash |
Multi-Chip Flagship Dense (TP=8 Megatron GSPMD) |
58.25 GiB BF16 (2.86 GiB DFlash) -> 9.80 GiB/chip at TP=8 |
Reuses tied embed_tokens directly as lm_head (transpose_lm_head=True), avoiding a duplicate 2.62 GiB projection across v5e-8. |
Queued on Kaggle v5e-8 (Multi-chip TP=8) |
Qwen/Qwen3-4B + z-lab/Qwen3-4B-DFlash-b16 |
Secondary Cross-Arch Check (4B dense) |
8.22 GiB BF16 |
Cross-architecture verification alongside M-005 Optical Torus Ring-Attention. |
Verified v6e-1 Silicon (900.55 tok/s, 11.64 tok/pass) |
262,144-Vocabulary PCIe Avoidance: With a262,144-token vocabulary, returning raw[tile, vocab]logits over PCIe transfers134 MB/step(128tile) or268 MB/step(256tile). Performingjnp.argmaxdirectly on TPU HBM shrinks host-to-device synchronization to512 B(tile=128) or1 KB(tile=256).- Pure-JAX Periodic Scan & Mixed Attention Geometry: Nested
jax.lax.scancompiles Gemma 4's5:1sliding_attention/full_attentionpattern (attention_k_eq_v=True, proportional RoPE withpartial_rotary_factor=0.25on global layers, per-layer residual scalar,30.0softcapping, and3Dstacked MoE banks withrouter.scale/per_expert_scale) down to a compact XLA HLO graph with automatic KV replication whennum_global_key_value_heads < tp_size. - Explicit Architectural Scope Boundaries: Edge variants with Per-Layer Embeddings or cross-layer KV sharing (
gemma-4-E2B/gemma-4-E4B, wherehidden_size_per_layer_input != 0ornum_kv_shared_layers != 0) raise an explicitValueErrorat config load. Hybrid recurrent/linear-attention models such asQwen 3.6/Qwen 3.8(linear_attentionDeltaNet state) are excluded because fixed-memory recurrent states cannot roll back zero-copy across128-node speculative tree branches.
Measured live on Google Cloud TPU v6e-1 via benchmarks/gemma4_jax.py, benchmarks/quantization_jax.py, benchmarks/block_diffusion_jax.py, and benchmarks/ast_seed_jax.py (archived in benchmarks/results/colab_gemma4_results.json):
Model & Precision (v6e-1, tile=256) |
Workload (gen=128, 114 on Qwen3 Codegen) |
Vanilla JAX (tok/s) | Static Arena (tok/s) | TPUForge Tree (tok/s) | Block-Tree (tok/s) | Tok / Pass | Speedup (vs Static / vs Vanilla Dec / Total) | Exact Parity vs Vanilla |
|---|---|---|---|---|---|---|---|---|
Gemma 4 26B-A4B MoE (INT8, 24.95 GiB) |
Code Refactor (P=285) |
5.53 |
13.77 |
64.64 |
167.54 |
12.80 |
12.17x / 30.30x / 32.78x |
128 / 128 |
Gemma 4 26B-A4B MoE (INT8, 24.95 GiB) |
Cold Codegen + AST (P=63, 78K AST) |
5.53 |
20.52 |
125.83 |
9.85 |
9.14x* / 22.76x / 25.80x |
128 / 128 |
|
Gemma 4 26B-A4B MoE (INT8, 24.95 GiB) |
Cold Chat (P=51, 0 AST) |
5.52 |
15.85 |
36.53 |
2.78 |
2.65x* / 6.62x / 7.89x |
46/128 Tree prefix (BF16 tie) |
|
Gemma 4 12B Dense (INT8, 12.97 GiB) |
Code Refactor (P=285) |
27.26 |
28.73 |
134.69 |
328.97 |
12.80 |
11.45x / 12.07x / 31.23x |
128 / 128 |
Gemma 4 12B Dense (INT8, 12.97 GiB) |
Cold Codegen + AST (P=63, 78K AST) |
27.21 |
43.12 |
244.28 |
9.85 |
8.50x* / 8.98x / 24.54x |
128 / 128 |
|
Gemma 4 12B Dense (INT8, 12.97 GiB) |
Cold Chat (P=51, 0 AST) |
27.24 |
33.88 |
92.61 |
3.56 |
3.22x* / 3.40x / 9.72x |
Tree == Block-Tree 100% |
|
Gemma 4 12B Dense (BF16, 22.18 GiB) |
Code Refactor (P=285) |
18.28 |
17.54 |
82.50 |
186.95 |
11.64 |
10.66x / 10.23x / 20.63x |
128 / 128 |
Gemma 4 12B Dense (BF16, 22.18 GiB) |
Cold Codegen + AST (P=63, 78K AST) |
18.25 |
26.46 |
154.19 |
9.85 |
8.79x* / 8.45x / 18.10x |
128 / 128 |
|
Gemma 4 12B Dense (BF16, 22.18 GiB) |
Cold Chat (P=51, 0 AST) |
18.26 |
20.66 |
57.30 |
3.56 |
3.27x* / 3.14x / 7.02x |
Tree == Block-Tree 100% |
|
Qwen3-4B (BF16, 8.22 GiB) |
Code Refactor (P=223) |
127.49 |
104.09 |
386.12 |
900.55 |
11.64 |
8.65x / 7.06x / 15.48x |
128 / 128 |
Qwen3-4B (BF16, 8.22 GiB) |
Cold Codegen + AST (P=59, 63K AST) |
126.48 |
141.57 |
764.05 |
11.40 |
7.34x* / 6.04x / 10.16x |
114 / 114 |
|
Qwen3-4B (BF16, 8.22 GiB) |
Cold Chat (P=54, 0 AST) |
126.50 |
128.02 |
242.77 |
3.13 |
2.33x* / 1.92x / 3.32x |
114 / 121 prefix match |
*Compared against the measured single-token Static Arena decode rate on the same model and precision.
v6e-1 Zero-Recompile Prompt Sweep (4 Unaligned Lengths) |
Vanilla Compiles | Static Arena Compiles | Full Sweep Time | Speedup |
|---|---|---|---|---|
Gemma 4 26B-A4B MoE (INT8, [38, 70, 104, 224]) |
5 |
1 |
13.27s vs 26.94s |
2.03x |
Gemma 4 12B Dense (BF16, [38, 70, 104, 224]) |
5 |
1 |
12.05s vs 39.38s |
3.27x |
Gemma 4 12B Dense (INT8, [38, 70, 104, 224]) |
5 |
1 |
4.46s vs 39.30s |
8.80x |
Qwen3-4B (BF16, [36, 68, 99, 199]) |
5 |
1 |
1.67s vs 4.51s |
2.71x |
- Native
INT8Mixed-Precision Proof (M-007onGemma 4 12B): Symmetrically quantizing12Bweights from22.18 GiB(BF16) to12.97 GiB(INT8,0.585x) saves9.20 GiBHBM (fitting inside a single15.75 GiBv5e-1chip as well) while speeding upStatic Arenasingle-token decode1.64x(28.73vs17.54 tok/s) and liftingBlock-Treecode refactor throughput to328.97 tok/s(18.00xoverBF16Vanilla JAX). - Open-Ended Prose Numerics Note: On open-ended
Cold Chat,[1, D]unpadded Vanilla JAX matmuls and[256, D]systolic MXU tiled matmuls break near-tieBF16logits (1 ULP) differently at token2on12B(46/128prefix on26B-A4B,114/121onQwen3-4B). Across all fixed-tile[256, D]modes on12B,tpuforge_tree == tpuforge_block_treeremains 100% exact, and everyCode RefactorandCold Codegenworkload matches Vanilla JAX token for token (128/128/114/114). M-005Optical Torus Flash-Ring Attention (28,672 / 32,768ctx,1.75 GiBKV,ring=1, TP=1): Onv6e-1(tile=256), Flash-Ring reaches638,908 tok/sprefill (44.88 ms,1.23xfaster thanv5e-1) and1.116 msper256-node systolic tree step (229,305tree-nodes/s,1.85xv5e-1node throughput,collective-permuteHLO verified).
All Phase 1 silicon numbers below were measured on Cloud TPU v5e-1 (TPU v5 lite, JAX 0.11.2, bfloat16) using on-device jnp.argmax (verify_greedy_tokens) on Qwen2.5-Coder (M-002..M-005), matching Vanilla JAX greedy output token for token (100%):
Model (v5e-1, BF16) |
Workload | Vanilla JAX (tok/s) | Static Arena (tok/s) | TPUForge Tree (tok/s) | Tokens / Pass | Decode Speedup | Total Speedup | Exact Match |
|---|---|---|---|---|---|---|---|---|
Qwen2.5-Coder-7B-Instruct (tile=128, gen=64) |
Code Refactor | 49.25 |
45.82 |
202.80 |
4.57 |
4.12x |
7.50x (0.358s vs 2.685s) |
64 / 64 |
Qwen2.5-Coder-7B-Instruct (tile=128, gen=64) |
Cold Codegen | 49.30 |
62.38 |
1.27x |
1.84x |
64 / 64 |
||
Qwen2.5-Coder-3B-Instruct (tile=128, gen=128) |
Code Refactor | 94.88 |
328.79 |
3.88 |
3.47x |
5.68x (0.411s vs 2.334s) |
128 / 128 |
|
Qwen2.5-Coder-3B-Instruct (tile=128, gen=128) |
Cold Codegen | 95.13 |
119.80 |
1.26x |
1.79x |
128 / 128 |
Prompt-Shape Sweep ([32, 64, 95, 195], tile=128) |
Vanilla Compiles | TPUForge Compiles | Full Sweep Time | Post-Warmup (64..195) |
Post-Warmup Speedup |
|---|---|---|---|---|---|
Qwen2.5-Coder-3B-Instruct (v5e-1) |
5 |
1 |
1.215s vs 3.342s (2.75x) |
0.180s vs 2.273s |
12.60x |
Qwen2.5-Coder-7B-Instruct (v5e-1) |
5 |
1 |
1.633s vs 3.869s (2.37x) |
0.347s vs 2.679s |
7.72x |
- Eliminating Host Logits Stalls (
M-003): Movingjnp.argmaxonto HBM removed~78 MB(tile=128) and~156 MB(tile=256) PCIe transfers onQwen2.5-Coder, shrinking a full128-node verification pass to11.80 msvs10.62 msper1-token Vanilla step on3B(22.54 msvs20.63 mson7B). That gained3.38xover pre-argmax97.22 tok/s(3B) and1.76xover115.27 tok/s(7B). - Hardware-Matched MXU Tile Width (
infer_mxu_dim): Onv5e-1(128x128physical MXU),tile=128(328.79 tok/s,11.80 ms/pass) beatstile=256(268.39 tok/s,14.45 ms/pass) by1.23xwhiletile=256matches256x256MXUs onv6e-1Trillium andv7. v5e-1Blockwise Flash-Ring Attention (M-005,ring=1,TP=1,BF16): Reached2,680,640 tok/sprefill (1.337 ms) and0.353 mstree step (362,752tree-nodes/s;3.91e-03prefill /1.22e-04treeBF16max abs error) at3584 / 4096, plus519,735 tok/sprefill (55.167 ms) and1.032 mstree step (124,047tree-nodes/s,1.75 GiBKV,collective-permuteHLO verified) at28672 / 32768.
Context-only suffix drafting drops to 1.27x (7B) and 1.26x (3B) when prompts share no prior n-grams. TPUForge implements four extensions to handle cold codegen, single-chip memory caps, and sparse MoE capacity across both Gemma 4 and Qwen3:
| Extension | Implemented Mechanism | Offline Verification Status (59/59 pytest) |
Cloud TPU Status |
|---|---|---|---|
M-006: Hybrid Block-Diffusion Tree (tpuforge_block_tree) |
Gemma4BlockTreeRunner (benchmarks/gemma4_jax.py) + QwenBlockTreeRunner (benchmarks/block_diffusion_jax.py) fuse parallel DFlash block-drafter top-k predictions with O(1) SuffixIndex continuations via TreeBuilder::build_from_block_top_k. |
Verified (test_block_tree_and_suffix.py & test_gemma4.py): sub-15 ms tree packing at 32K+, 4-mode parity, transpose_lm_head tied reuse, TP=2 HLO green. |
Verified v6e-1 Silicon: 328.97 tok/s (12B INT8), 167.54 tok/s (26B-A4B INT8), 900.55 tok/s (Qwen3-4B) |
M-009: Out-of-Band Repository AST Seeding |
benchmarks/ast_seed_jax.py, SuffixIndex::seed_sequences, ForgeEngine::seed_ast_text, --ast-seed-path, and HTTP "seed_snippets" pre-warm cold codegen drafting at zero prompt/KV cost. |
Verified (test_ast_seed_drafter.py): Python/C++ AST extraction, exact greedy parity, 1-token fallback. |
Verified v6e-1 Silicon (78,096 AST tokens): 244.28 tok/s (12B INT8), 125.83 tok/s (26B-A4B), 764.05 tok/s (Qwen3-4B) |
M-007: Native INT8 / FP8 Systolic Kernels |
benchmarks/quantization_jax.py computes pre-sharding symmetric per-channel int8 / float8_e4m3fn weights and scales (>=2D N-D periodic scan specs) executed via constant-free jax.lax.dot_general. |
Verified (test_quantized_systolic.py & test_gemma4.py): < 0.65x linear HBM footprint vs BF16, Gemma 4 INT8/FP8 4-mode parity, TP=2 all-reduce HLO. |
Verified v6e-1 Silicon: 12.97 GiB vs 22.18 GiB 12B (0.585x), 24.95 GiB 26B-A4B, 1.64x Static decode |
M-008: Host mmap Sparse MoE Streaming |
benchmarks/moe_streaming_jax.py + Gemma4MoeStreamRunner pin attention, routers, and KV caches in HBM while ExpertHostStore streams routed 3D stacked or 2D experts from mmap into E_stage slots amortized across 128 tree nodes. |
Verified (test_moe_streaming.py & test_gemma4.py): < 30% Gemma 4 HBM trunk ratio (< 45% OLMoE), 0 steady-state recompiles, Windows mmap release verified. |
Verified offline (59/59 pytest); staged for 15.75 GiB v5e-1 (26B-A4B INT8 fit 100% in v6e-1 HBM) |
(Multi-chip v5e-8 TP=8 runs and M-008 v5e-1 host mmap streaming remain labeled honestly above until their own Cloud TPU JSON telemetry completes.)
+-----------------------------------------------------------------+
| Client / OpenAI /v1/chat/completions (`seed_snippets` + SSE) |
+-----------------------------------------------------------------+
|
v
+-----------------------------------------------------------------+
| TPUForge Native C++20 Engine |
| `ForgeEngine` | `O(1)` `SuffixIndex` + Out-of-Band AST Pack |
| Hybrid Block-Diffusion `TreeBuilder` (128 / 256 tile) |
+-----------------------------------------------------------------+
| | |
+------------+ | +------------+
v v v
+-----------------------+ +-------------------------+ +------------------------+
| 128/256-Row Systolic | | Zero-Recompile Static | | Host mmap/DRAM Sparse |
| Tree Verifier & | | HBM Arena & Optical | | MoE Expert Stager |
| INT8/FP8 MXU Kernels | | Torus ICI Ring-Attn | | (E_stage -> 128 nodes) |
+-----------------------+ +-------------------------+ +------------------------+
| | |
+--------------------------------+---------------------------------+
|
v
+-----------------------------------------------------------------+
| OpenXLA / PJRT C ABI (`pjrt_c_api.h` Driver) |
+-----------------------------------------------------------------+
| |
v v
[Local PJRT CPU Plugin] [Cloud TPU PJRT Plugin]
(Development / Simulation) (`libpjrt_tpu.so` / Silicon)
- Serving Ingress & Live Systolic HUD (
M-004): Ollama-style hardware-autofit launcher (benchmarks.tpuforge_serve) and Live Systolic Matrix HUD (benchmarks.hud_ui) alongside the native C++20 HTTP/1.1/v1/chat/completionsserver (OpenAIServer) with SSE telemetry streaming,nanobind_tpuforge_extPython bindings, and out-of-band/api/seedAST indexing. O(1)SuffixIndex& Hybrid Block-Diffusion Packing (M-003,M-006,M-009): Incremental 1-to-4-gram suffix hash index merges repository AST declarations and parallelDFlashtop-kcandidates into fixed128- or256-row causal tiles viaTreeBuilder::build_from_block_top_k.- Mixed-Precision & Tied-Embedding Kernels (
M-002,M-007):StaticArena(128-byte alignment) and constant-free weight tuples executebfloat16activations againstint8/float8_e4m3fnper-channel weights while reusing tiedembed_tokensdirectly on Gemma 4 (transpose_lm_head=True). - Host
mmapSparse MoE Expert Streaming (M-008):ExpertHostStorestreams active routed experts (3Dstacked Gemma 4 /Qwen3-MoEor2DOLMoE) frommmapinto fixed-shapeE_stageHBM buffers amortized across all128speculative candidates simultaneously. - Optical Torus Ring-Attention & GSPMD Tensor Parallelism (
M-005): NativeIciRingPlanner+jax.shard_map+jax.lax.ppermutering kernels alongside MegatronP(..., "tp")sharding (TP=8with replicated global KV whennum_global_key_value_heads < tp_size) and persistent OpenXLA compilation caching. - PJRT C ABI Backend (
pjrt_c_api.h): Dispatches fixed-shape compiled programs across locallibpjrt_cpuand Cloud TPUlibpjrt_tpu.so.
# Google Gemma 4 26B-A4B Sparse MoE + DFlash + AST Seeding in INT8 (32 GiB v6e-1 HBM or TP=8 v5e-8)
python -m benchmarks.run_qwen_bench --model google/gemma-4-26B-A4B-it --draft-model z-lab/gemma-4-26B-A4B-it-DFlash --weight-quant int8 --ast-seed-path . --tile-size auto --tp-size auto
# Google Gemma 4 26B-A4B Sparse MoE via host mmap streaming on 16 GiB v5e-1
python -m benchmarks.run_qwen_bench --model google/gemma-4-26B-A4B-it --host-store mmap --stage-experts 16 --weight-quant int8 --ast-seed-path . --tile-size auto --tp-size 1
# Google Gemma 4 12B Dense + DFlash in INT8 (fits on single 15.75 GiB v5e-1 chip)
python -m benchmarks.run_qwen_bench --model google/gemma-4-12B-it --draft-model z-lab/gemma4-12B-it-DFlash --weight-quant int8 --ast-seed-path . --tile-size auto --tp-size auto
# Google Gemma 4 31B Dense + DFlash via Megatron TP=8 (pooled 128 GiB v5e-8)
python -m benchmarks.run_qwen_bench --model google/gemma-4-31B-it --draft-model z-lab/gemma-4-31B-it-DFlash --ast-seed-path . --tile-size auto --tp-size 8
# Ollama-style TPU Forge Serve presets, OpenAI /v1 endpoint, and Live Systolic HUD
python -m benchmarks.tpuforge_serve --list-presets
python -m benchmarks.tpuforge_serve gemma4:12b --ast-seed-path .
python -m benchmarks.tpuforge_serve gemma4:26b --ast-seed-path .
# M-005 Optical Torus Ring-Attention & standalone Kaggle/Colab bundle builder
python -m benchmarks.ring_attention_jax --max-seq-len 8192 --valid-seq-len 6144 --ring-size auto --tp-size 1
python -m benchmarks.package_kaggle_bundle| Hardware Target | Status | Scope |
|---|---|---|
| Local CPU (PJRT CPU) | Verified (59/59 pytest) |
128/256 tile simulation, Preset Serve + Live HUD SSE parity, Gemma 4 periodic 5:1 scan + 3D MoE + DFlash + INT8/FP8 + mmap + TP=2, O(1) SuffixIndex + AST seeding, OpenXLA GSPMD TP parity |
Cloud TPU v5e-1 (15.75 GiB HBM) |
Verified Silicon (Qwen2.5) / Staged (Gemma 4) |
Verified: 32K Flash-Ring (519,735 tok/s), 3B (328.79 tok/s), 7B (202.80 tok/s), tile=128 vs 256 (1.23x). Staged: Gemma 4 12B INT8 (13.97 GiB), 26B-A4B mmap INT8, Qwen3-4B check |
Cloud TPU v6e-1 (31.24 GiB HBM, Trillium) |
Verified Silicon (Gemma 4 + Qwen3-4B) |
tile=256: Gemma 4 26B-A4B INT8 (24.95 GiB, 167.54 tok/s, 30.30x vs Vanilla), 12B INT8 (12.97 GiB, 328.97 tok/s), 12B BF16 (22.18 GiB, 186.95 tok/s), Qwen3-4B (900.55 tok/s), M-005 (638,908 tok/s prefill, 229,305 tree-nodes/s) |
Kaggle Free TPU v5e-8 (128 GiB HBM) |
Queued / Staged | 8x16 GiB pooled HBM: 128K ICI Ring-Attention (M-005), Gemma 4 26B-A4B TP=8, 31B TP=8 (9.80 GiB/chip tied BF16), and 12B tile sweep |
include/: Public C/C++ engine headers (pjrt_c_api.h,kv_cache.h,tree_speculator.h,openai_server.h,engine.h).src/: Core C++20 implementation (core/,ops/,speculative/,runtime/).benchmarks/: OpenXLA/JAX modules (tpuforge_serve.py,hud_ui.py,gemma4_jax.py,qwen_jax.py,quantization_jax.py,moe_streaming_jax.py,ast_seed_jax.py,block_diffusion_jax.py,ring_attention_jax.py,run_qwen_bench.py,package_kaggle_bundle.py).tests/:59offline unit, C++/nanobind, Serve + HUD parity, XLA sharding, Gemma 4, quantized systolic, sparse MoE streaming, AST-seeding, block-tree, and ring-attention tests (pytest).docs/: Architecture and TRC proposal notes (docs/TRC_APPLICATION.md).assets/slides/: Dark-mode architectural slide deck (slide_01.png..slide_08.png) and Gemma 4v6e-1Trillium benchmark chart (gemma4_v6e_benchmark_chart.png).assets/audio/: Systems engineering audio overview (TPUForge_Audio_Overview.m4a,6:16).
Licensed under the Apache License, Version 2.0. See LICENSE for details.








