Before you start
What happened
Qwen3.8-Flash-Next-FP8 does not finish loading on a single RTX 5090 32 GB with 128 GB RAM using the default serve command. Automatic disk PLE works, but the routed FP8 expert banks still consume host RAM. The automatic serial fallback reduces temporary reader memory without making the final banks fit safely.
Expected: either fit the existing host/GPU memory or reject loading clearly before exhausting allocatable host RAM. Actual: main reached at least 45/48 expert layers, memory allocation/reclaim and retained-volume I/O stalled, and the host hung-task handler panicked. This is a loading-capacity defect, not a request to implement the model architecture from scratch.
A candidate keeps the minimum necessary FP8 source-bank layers on GPU, refreshes available host RAM before each untouched layer, skips registration of those unused host banks, and deducts the GPU source bytes from the existing cache budget. It enabled two complete loads and real inference without requantizing the checkpoint.
RTX 5090 FP8 results
The candidate preserves the official checkpoint precision and uses the same command: ft serve --model /tmp/freetoken-eval/checkpoint. Native automatic choices: TP1, disk PLE, offload MoE, hybrid radix cache, and four request slots. No server tuning flag was added for the two fresh-load runs.
| Test |
Result |
| Unmodified main |
Serial expert loading reached 45/48 layers; host allocation/reclaim and I/O stalled; no generation |
| Candidate fresh loads |
Both completed all 48 layers and genuine generation; serial loading phases approximately 411 s and 469 s |
| Arithmetic, natural stop |
Returned 4 for 2+2; second load 2.5900 s, two output tokens |
| Short input, first load |
29.86 tok/s, TTFT 3.81 s; repeated prompt 29.71 tok/s, TTFT 2.91 s |
| Short input, second load |
32.84 tok/s, TTFT 2.43 s; repeated prompt 32.63 tok/s, TTFT 2.39 s |
| Two concurrent requests, second load |
Both completed: 18.43 / 18.58 tok/s per request; 256 output tokens in 11.60 s including prefill |
| Four concurrent requests, first load |
All completed: approximately 5.92 tok/s per request; 512 output tokens in 26.52 s including prefill |
| Default KV capacity |
130 pages x 64 = 8,320 tokens; oversized 8,588-token input returned HTTP 400 context_length_exceeded |
| Rebuilt 65,536-token KV pool |
65,178 actual input; TTFT first/repeat 20.62 / 2.95 s; decode 28.60 / 28.68 tok/s; marker retrieval succeeded |
| Rebuilt 91,968-token KV pool |
91,610 actual input; TTFT first/repeat 29.65 / 3.04 s; decode 27.86 / 28.59 tok/s; marker retrieval succeeded |
| Prefix reuse |
Repeated 72-token prompt logged 64 cached tokens and eight new tokens; long-prefix reuse reduced TTFT substantially |
| GPU temperature |
Per-run maxima 56 C and 51 C |
| Memory |
OOM/kill counters stayed zero in both successful runs; second-run max sampled cgroup current 123,064,684,544 bytes including reclaimable file cache; GPU allocation 31,184,781,312 bytes |
Short/concurrent throughput uses the upstream streaming benchmark helper with 128 output tokens and ignore_eos=true, so it measures throughput rather than natural answer completion. Long-context tests use synthetic repeated-word marker prompts with natural stop, not general agent-quality evaluation. Their cache pool was temporarily rebuilt to 1,024 expert slots and 16 usable mamba slots, then restored to the exact default geometry. These results describe two successful candidate runs, not a speedup over main (main could not serve here).
Loader/copy/engine tests: 154 passed, two skipped. A changing-RAM placement regression fails on the earlier candidate and passes on the adaptive candidate. A real checkpoint layer test compared gate/up/down weights and scales across eight experts before/after existing copy paths, with exact byte equality. Tests and measurements were AI-assisted and run on real hardware in a dedicated Coder Workspace. Full repository tests were not run. No quality, vision, tool-use, 262K or speculative-decoding result is asserted here.
How did you install FreeToken
Built from source
FreeToken version
0.1.3; main d3512b4, remote main rechecked before reporting
OS
Ubuntu
OS details
Ubuntu 24.04.3 LTS; Linux 7.3.0-070300rc3-generic x86_64; Python 3.12.3; torch 2.11.0+cu130; source build in a dedicated Coder Kubernetes Workspace
GPU and driver
One NVIDIA RTX 5090 32 GB; driver 610.57.04; CUDA toolkit 13.0.88
CPU and system RAM
Intel Core Ultra 9 285K; 128 GB physical RAM (134,490,562,560 bytes reported); no finite container RAM cap; ordinary shared host workloads
Checkpoint
Qwen/Qwen3.8-Flash-Next-FP8; revision 236dfdf285828023ca3bcd3f37366c58a3469b13
Command
ft serve --model /tmp/freetoken-eval/checkpoint
Existing virtualenv and CUDA compiler on PATH; CUDA_HOME=/usr/local/cuda. No model tuning flags.
Full log
[2026-10-03|20:19:00] INFO Parsed arguments:
ServerArgs(model_path='/tmp/freetoken-eval/checkpoint', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, hf_overrides={}, max_running_req=4, attention_backend='auto', moe_strategy='auto', quant_backend=None, ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, mm=MultimodalConfig(disabled_encoders=frozenset(), embed_cache_device='cpu', encoder_weights='host', image_min_tokens=None, image_max_tokens=None, processor_kwargs={}), max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=13692', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='checkpoint', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, allowed_media_domains='', allowed_local_media_path='', cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=(), gpu_assigned=None)
[2026-10-03|20:19:00|FrontendAPI] INFO Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
INFO: Started server process [13692]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
/workspaces/repository/python/freetoken/engine/engine.py:1168: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[2026-10-03|20:19:01|core|rank=0] INFO Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[2026-10-03|20:19:02|core|rank=0] INFO Auto-selected attention backend: qsa_sparse
[2026-10-03|20:19:02|core|rank=0] WARNING Page size is overridden to 64 for the qsa_sparse backend
[2026-10-03|20:19:02|core|rank=0] INFO Auto-selected MoE strategy: offload
[2026-10-03|20:19:02|core|rank=0] INFO No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected strategy 'offload'
[2026-10-03|20:19:02|core|rank=0] INFO Resolved config: moe_strategy='offload', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
[2026-10-03|20:19:02|core|rank=0] INFO Free memory before loading model: 30.82 GiB
Loading weights: 0%| | 0/131 [00:00<?, ?it/s]
Loading weights: 2%|▏ | 2/131 [00:00<00:07, 16.40it/s]
Loading weights: 8%|▊ | 10/131 [00:00<00:02, 48.23it/s]
Loading weights: 14%|█▎ | 18/131 [00:00<00:06, 18.75it/s]
Loading weights: 22%|██▏ | 29/131 [00:01<00:03, 31.45it/s]
Loading weights: 32%|███▏ | 42/131 [00:01<00:01, 45.45it/s]
Loading weights: 40%|███▉ | 52/131 [00:01<00:01, 54.10it/s]
Loading weights: 46%|████▌ | 60/131 [00:01<00:01, 55.72it/s]
Loading weights: 55%|█████▍ | 72/131 [00:01<00:00, 66.11it/s]
Loading weights: 69%|██████▊ | 90/131 [00:01<00:00, 90.91it/s]
Loading weights: 77%|███████▋ | 101/131 [00:01<00:00, 76.09it/s]
Loading weights: 87%|████████▋ | 114/131 [00:01<00:00, 83.41it/s]
Loading weights: 95%|█████████▍| 124/131 [00:02<00:00, 78.33it/s]
Loading weights: 100%|██████████| 131/131 [00:02<00:00, 52.29it/s]
[2026-10-03|20:19:07|core|rank=0] INFO PLE disk backend: io_uring, O_DIRECT, wait-sync
[2026-10-03|20:19:07|core|rank=0] INFO MoE experts: fp8_block via triton
[2026-10-03|20:19:07|core|rank=0] WARNING expert banks: low free RAM -> serial build (avoids parallel-reader OOM; override with --expert-load parallel)
[2026-10-03|20:19:07|core|rank=0] INFO expert banks: slow path (serial build)
Loading fp8 experts (serial): 0%| | 0/48 [00:00<?, ?it/s]
Loading fp8 experts (serial): 2%|▏ | 1/48 [00:07<05:47, 7.39s/it]
Loading fp8 experts (serial): 4%|▍ | 2/48 [00:15<06:00, 7.83s/it]
Loading fp8 experts (serial): 6%|▋ | 3/48 [00:23<05:49, 7.77s/it]
Loading fp8 experts (serial): 8%|▊ | 4/48 [00:31<05:57, 8.12s/it]
Loading fp8 experts (serial): 10%|█ | 5/48 [00:40<05:57, 8.32s/it]
Loading fp8 experts (serial): 12%|█▎ | 6/48 [00:48<05:50, 8.35s/it]
Loading fp8 experts (serial): 15%|█▍ | 7/48 [00:57<05:38, 8.26s/it]
Loading fp8 experts (serial): 17%|█▋ | 8/48 [01:04<05:25, 8.13s/it]
Loading fp8 experts (serial): 19%|█▉ | 9/48 [01:12<05:09, 7.93s/it]
Loading fp8 experts (serial): 21%|██ | 10/48 [01:21<05:10, 8.18s/it]
Loading fp8 experts (serial): 23%|██▎ | 11/48 [01:29<05:01, 8.15s/it]
Loading fp8 experts (serial): 25%|██▌ | 12/48 [01:37<04:53, 8.15s/it]
Loading fp8 experts (serial): 27%|██▋ | 13/48 [01:45<04:44, 8.12s/it]
Loading fp8 experts (serial): 29%|██▉ | 14/48 [01:53<04:32, 8.03s/it]
Loading fp8 experts (serial): 31%|███▏ | 15/48 [02:02<04:34, 8.30s/it]
Loading fp8 experts (serial): 33%|███▎ | 16/48 [02:09<04:20, 8.14s/it]
Loading fp8 experts (serial): 35%|███▌ | 17/48 [02:16<04:00, 7.74s/it]
Loading fp8 experts (serial): 38%|███▊ | 18/48 [02:23<03:43, 7.46s/it]
Loading fp8 experts (serial): 40%|███▉ | 19/48 [02:30<03:34, 7.41s/it]
Loading fp8 experts (serial): 42%|████▏ | 20/48 [02:37<03:24, 7.30s/it]
Loading fp8 experts (serial): 44%|████▍ | 21/48 [02:44<03:11, 7.07s/it]
Loading fp8 experts (serial): 46%|████▌ | 22/48 [02:51<03:04, 7.08s/it]
Loading fp8 experts (serial): 48%|████▊ | 23/48 [02:58<02:59, 7.20s/it]
Loading fp8 experts (serial): 50%|█████ | 24/48 [03:06<02:57, 7.40s/it]
Loading fp8 experts (serial): 52%|█████▏ | 25/48 [03:15<02:56, 7.68s/it]
Loading fp8 experts (serial): 54%|█████▍ | 26/48 [03:24<02:56, 8.04s/it]
Loading fp8 experts (serial): 56%|█████▋ | 27/48 [03:31<02:42, 7.74s/it]
Loading fp8 experts (serial): 58%|█████▊ | 28/48 [03:37<02:27, 7.39s/it]
Loading fp8 experts (serial): 60%|██████ | 29/48 [03:46<02:27, 7.76s/it]
Loading fp8 experts (serial): 62%|██████▎ | 30/48 [03:54<02:21, 7.84s/it]
Loading fp8 experts (serial): 65%|██████▍ | 31/48 [04:02<02:13, 7.86s/it]
Loading fp8 experts (serial): 67%|██████▋ | 32/48 [04:11<02:10, 8.17s/it]
Loading fp8 experts (serial): 69%|██████▉ | 33/48 [04:21<02:13, 8.91s/it]
Loading fp8 experts (serial): 71%|███████ | 34/48 [04:31<02:09, 9.22s/it]
Loading fp8 experts (serial): 73%|███████▎ | 35/48 [04:40<01:57, 9.04s/it]
Loading fp8 experts (serial): 75%|███████▌ | 36/48 [04:50<01:51, 9.27s/it]
Loading fp8 experts (serial): 77%|███████▋ | 37/48 [05:02<01:51, 10.09s/it]
Loading fp8 experts (serial): 79%|███████▉ | 38/48 [05:12<01:42, 10.30s/it]
Loading fp8 experts (serial): 81%|████████▏ | 39/48 [05:21<01:29, 9.90s/it]
Loading fp8 experts (serial): 83%|████████▎ | 40/48 [05:29<01:14, 9.32s/it]
Loading fp8 experts (serial): 85%|████████▌ | 41/48 [05:43<01:14, 10.60s/it]
Loading fp8 experts (serial): 88%|████████▊ | 42/48 [05:57<01:09, 11.50s/it]
Loading fp8 experts (serial): 90%|████████▉ | 43/48 [06:14<01:05, 13.18s/it]
Loading fp8 experts (serial): 92%|█████████▏| 44/48 [06:30<00:56, 14.16s/it]
Loading fp8 experts (serial): 94%|█████████▍| 45/48 [07:06<01:01, 20.56s/it]
Anything else
The log above is the entire retained default serve log, with terminal colors removed and progress carriage returns normalized. It ends during expert loading because the host stopped making progress.
Measured checkpoint: 185,523,317,458 total weight bytes; 120,810,700,800 routed-expert bytes; 51,200,245,762 PLE bytes. PLE disk offload does not remove the expert banks. Last default-run sample: cgroup peak 122,319,216,640 bytes, shmem 119,192,018,944, GPU allocation 11,725,307,904, MemAvailable 2,308,886,528; observed OOM counters zero. Kernel dump: Normal free 1,845,484 kB versus CmaFree 1,773,372 kB; pstore reports Kernel panic - not syncing: hung_task: blocked tasks. The registration/CMA/reclaim interaction is not yet a proven generic driver bug.
Related: #214 / #79 cover model support; this report isolates the 32 GB GPU / 128 GB host loading failure. #441 was an FTW loader problem reportedly fixed on earlier main. #333 / #334 concern finite cgroup limits, absent here. #563 addresses hot-expert pinning on a 48 GB GPU. The separate stats issue/PR #598 / #599 does not fix loading.
Question for maintainers: is mixed host/GPU FP8 source-bank placement acceptable for this hardware, or should the default loader reject before registration? The candidate is FP8-specific and future cache/graph minima still need consideration. Performance profiling should identify decode bottlenecks before claiming a speedup.
Prepared and tested with AI assistance under human direction. No credentials or private user prompts are included.
Before you start
mainwhen building from source.What happened
Qwen3.8-Flash-Next-FP8 does not finish loading on a single RTX 5090 32 GB with 128 GB RAM using the default serve command. Automatic disk PLE works, but the routed FP8 expert banks still consume host RAM. The automatic serial fallback reduces temporary reader memory without making the final banks fit safely.
Expected: either fit the existing host/GPU memory or reject loading clearly before exhausting allocatable host RAM. Actual: main reached at least 45/48 expert layers, memory allocation/reclaim and retained-volume I/O stalled, and the host hung-task handler panicked. This is a loading-capacity defect, not a request to implement the model architecture from scratch.
A candidate keeps the minimum necessary FP8 source-bank layers on GPU, refreshes available host RAM before each untouched layer, skips registration of those unused host banks, and deducts the GPU source bytes from the existing cache budget. It enabled two complete loads and real inference without requantizing the checkpoint.
RTX 5090 FP8 results
The candidate preserves the official checkpoint precision and uses the same command:
ft serve --model /tmp/freetoken-eval/checkpoint. Native automatic choices: TP1, disk PLE, offload MoE, hybrid radix cache, and four request slots. No server tuning flag was added for the two fresh-load runs.Short/concurrent throughput uses the upstream streaming benchmark helper with 128 output tokens and ignore_eos=true, so it measures throughput rather than natural answer completion. Long-context tests use synthetic repeated-word marker prompts with natural stop, not general agent-quality evaluation. Their cache pool was temporarily rebuilt to 1,024 expert slots and 16 usable mamba slots, then restored to the exact default geometry. These results describe two successful candidate runs, not a speedup over main (main could not serve here).
Loader/copy/engine tests: 154 passed, two skipped. A changing-RAM placement regression fails on the earlier candidate and passes on the adaptive candidate. A real checkpoint layer test compared gate/up/down weights and scales across eight experts before/after existing copy paths, with exact byte equality. Tests and measurements were AI-assisted and run on real hardware in a dedicated Coder Workspace. Full repository tests were not run. No quality, vision, tool-use, 262K or speculative-decoding result is asserted here.
How did you install FreeToken
Built from source
FreeToken version
0.1.3; main d3512b4, remote main rechecked before reporting
OS
Ubuntu
OS details
Ubuntu 24.04.3 LTS; Linux 7.3.0-070300rc3-generic x86_64; Python 3.12.3; torch 2.11.0+cu130; source build in a dedicated Coder Kubernetes Workspace
GPU and driver
One NVIDIA RTX 5090 32 GB; driver 610.57.04; CUDA toolkit 13.0.88
CPU and system RAM
Intel Core Ultra 9 285K; 128 GB physical RAM (134,490,562,560 bytes reported); no finite container RAM cap; ordinary shared host workloads
Checkpoint
Qwen/Qwen3.8-Flash-Next-FP8; revision 236dfdf285828023ca3bcd3f37366c58a3469b13
Command
ft serve --model /tmp/freetoken-eval/checkpoint Existing virtualenv and CUDA compiler on PATH; CUDA_HOME=/usr/local/cuda. No model tuning flags.Full log
Anything else
The log above is the entire retained default serve log, with terminal colors removed and progress carriage returns normalized. It ends during expert loading because the host stopped making progress.
Measured checkpoint: 185,523,317,458 total weight bytes; 120,810,700,800 routed-expert bytes; 51,200,245,762 PLE bytes. PLE disk offload does not remove the expert banks. Last default-run sample: cgroup peak 122,319,216,640 bytes, shmem 119,192,018,944, GPU allocation 11,725,307,904, MemAvailable 2,308,886,528; observed OOM counters zero. Kernel dump: Normal free 1,845,484 kB versus CmaFree 1,773,372 kB; pstore reports Kernel panic - not syncing: hung_task: blocked tasks. The registration/CMA/reclaim interaction is not yet a proven generic driver bug.
Related: #214 / #79 cover model support; this report isolates the 32 GB GPU / 128 GB host loading failure. #441 was an FTW loader problem reportedly fixed on earlier main. #333 / #334 concern finite cgroup limits, absent here. #563 addresses hot-expert pinning on a 48 GB GPU. The separate stats issue/PR #598 / #599 does not fix loading.
Question for maintainers: is mixed host/GPU FP8 source-bank placement acceptable for this hardware, or should the default loader reject before registration? The candidate is FP8-specific and future cache/graph minima still need consideration. Performance profiling should identify decode bottlenecks before claiming a speedup.
Prepared and tested with AI assistance under human direction. No credentials or private user prompts are included.