A hybrid model's cached prefix cannot be resumed far back: no GDN snapshot survives below a prompt's last chunk #569

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

A hybrid GDN model resumes a cached prefix only at a radix node that holds a snapshot of its recurrent state: HybridRadixCache.match_prefix truncates the match to the deepest such node. FreeToken keeps two snapshots per request, the last x64 boundary of the prompt's final prefill chunk and the request's end when it falls on a page (page_size is 64 under qsa_sparse, so rarely), and the card holds four spare ones in LRU order. Intermediate chunks are not cached at all, deliberately, since caching them under overlap scheduling double-freed pages.

So a prompt that leaves a cached conversation far back resumes from nothing. A client's compaction does exactly that: it replays the conversation up to the tail it keeps, then appends its instruction. Serving Qwen3.8-Flash-Next with a 1M-token window, a 760,534-token replay of a cached 985,577-token prompt prefilled every token again: 798.9 s to the first token, #cached-token: 0 on every chunk.

How did you install FreeToken

Built from source

FreeToken version

v0.1.3 with my Windows and 1M-window patches. The code involved (HybridRadixCache, the chunked-prefill drain in scheduler.py, LinearStatePool) is the same on 0d652e7 but for one line each in scheduler.py and core.py.

OS

Windows 11

OS details

Windows 11 Pro 26200, Python 3.13.3. Nothing here is platform-specific: it is the radix cache and the scheduler.

GPU and driver

RTX 3090 Ti 24 GB, driver 616.92

CPU and system RAM

AMD Ryzen 9 9950X3D, 256 GB

Checkpoint

RadixArk/Qwen3.8-Flash-Next-NVFP4

Command

ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 --max-running-requests 1
# 1. send a prompt several prefill chunks long, and let it finish
# 2. send the same conversation cut a few chunks before its end, with one more user message after the cut

My run served the 1M window through the patches in #568; the behaviour needs only a prompt that spans several prefill chunks.

Full log

The replay, every chunk prefilled from scratch:

Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.72, #mamba-slot: 3/8, ...
Prefill batch, #new-seq: 1, #new-token: 8192, #cached-token: 0, token usage: 0.73, #mamba-slot: 3/8, ...
Prefill batch, #new-seq: 1, #new-token: 6870, #cached-token: 0, token usage: 0.73, #mamba-slot: 4/8, ...

With the patch below, the same replay:

Prefill batch, #new-seq: 1, #new-token: 6934, #cached-token: 753600, token usage: 0.73, #mamba-slot: 4/8, ...

Anything else

I have a patch that keeps these snapshots, and would like to know whether it fits before opening a PR:

• LinearStatePool(host_slots=N) behind --mamba-host-slots N: a bank of snapshot slots in page-locked host memory with the device slots' layout, so the PLE conv and n-gram states resume too.
• A radix node's snapshot lives on the device (mamba_value) or in the bank (mamba_host). evict_mamba moves an evicted device snapshot into the bank instead of dropping it; a full bank drops the least recently used path first, and on the path the last request walked, where every node ties, keeps the deepest and drops the one nearest the snapshot above it, so the survivors stay spread along the conversation.
• At each intermediate chunk's drain its x64 track is copied to a bank slot, before the chunk after next reuses the ping-pong slot, and the final chunk's commit attaches them once the prefix is in the tree. Nothing touches the handle or the pages mid-prompt, so the double-free the no-cache rule prevents cannot come back.

Measured with 128 slots (13.79 GiB of pinned host memory, no VRAM): the replay above reaches its first token in 12.3 s instead of 798.9 s; over a 72-turn conversation of about 6,100 tokens a turn, replaying all but its last 131,072 tokens takes 3.7 s, from a snapshot the card had moved to host memory 22 turns earlier. A cold 985,577-token prefill takes 1,168.8 s with the bank against 1,126-1,215 s without. The radix harness's reference model states host residency and the eviction order independently of the implementation.

#525 moves evicted snapshots and KV pages to host memory behind environment flags, with its own token-keyed index; it does not keep intermediate chunks, so a long prompt still has no resume point before its end. The patch carries #505 and #559 unchanged, and without #505, at page size 64 a prompt whose last chunk is 64 tokens or fewer caches nothing.