Repository navigation
DeepSeek-V4-Flash-0731-NVFP4 (nvidia) fails on 0.1.3+gfa0e9537d: WeightLoadError KeyError 'layers.0.ffn.experts.0.w1.scale' #606
Description
Summary
nvidia/DeepSeek-V4-Flash-0731-NVFP4 (repo sha f1caa71142bd0be02f728c79f75042ac1e461579, unchanged since 2026-09-02) fails to load on engine 0.1.3+gfa0e9537d with:
[core|rank=0] INFO Auto-selected attention backend: dsv4_sparse
[core|rank=0] INFO Resolved config: moe_strategy='offload', attention_backend='dsv4_sparse', cache_type='swa_radix', page_size=128
[core|rank=0] INFO Free memory before loading model: 10.76 GiB
[FrontendAPI] ERROR Backend supervisor: WeightLoadError: KeyError: 'layers.0.ffn.experts.0.w1.scale'
[FrontendAPI] ERROR Backend worker is gone and cannot be restarted; stopping the API server
Kernel selection passes (the DSV4-specific path is selected, unlike the RedHatAI NVFP4 export which is rejected for e4m3 scales), so the ModelArgs in inference/config.json are accepted — the failure is in the expert weight lookup itself: the loader expects a w1.scale tensor naming/format that this checkpoint does not contain.
Repro
ft serve --model C:\llm\models\<nvidia NVFP4 folder> \
--moe-strategy offload --moe-cpu-layers auto \
--kv-reserve-tokens 65536 --max-prefill-length 4096 \
--host 0.0.0.0 --port 1919
Model files SHA256-verified against HF LFS hashes (48 shards + inference/config.json + configs/tokenizer). Fails 8–10 s after weight load starts, on the first expert layer.
History across builds (all on this machine, Windows 11, RTX 5070 12 GB, 31.1 GB RAM)
| checkpoint | engine | result |
|---|---|---|
| official BF16 (deepseek-ai) | 0.1.3+gc8ed699cb | WeightLoadError: invalid python storage (#597 class) |
| nvidia NVFP4 | 0.1.3+gc8ed699cb | same invalid python storage |
| nvidia NVFP4 | 0.1.3+gfa0e9537d | kernel selection OK → KeyError: 'w1.scale' |
| RedHatAI NVFP4 | any | unusable by design (no inference/config.json; e4m3 scales → no usable kernel) |
So the Windows storage bug (#597) is gone in gfa0e9537d, but the DSV4 reader now fails on the only NVFP4 export that satisfies the documented inference/config.json requirement. Either the reader expects a newer modelopt export layout (per-tensor *.scale entries) than the published checkpoint, or the key format it derives from model.safetensors.index.json doesn't match this repo's naming.
Happy to test a fix build or run targeted probes (e.g. dumping the expert tensor names the loader derives vs. what the checkpoint contains) if that helps.
Activity
jason-fxz commentedon Oct 6, 2026
DeepSeek-V4 currently supports only the official checkpoint: deepseek-ai/DeepSeek-V4-Flash-0731 (https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731). Its routed experts are MXFP4 and the rest is FP8. The nvidia export stores its experts as ModelOpt NVFP4 (weight_scale / weight_scale_2 / input_scale), which the DSV4 loader doesn't read, hence the KeyError.
NVFP4 doesn't save memory here either. Its experts are actually larger than MXFP4: one e4m3 scale per 16 values instead of one e8m0 scale per 32. That makes the routed experts about 155.8 GB vs 147.2 GB, and the whole repo 175.6 GB vs 166.9 GB.
Memory needed for DeepSeek-V4-Flash today:
So on a 12 GB GPU / 31 GB RAM machine, DSV4-Flash won't run with either checkpoint, unfortunately.