chizamd · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026 · Oct 11, 2026
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,6 @@
 build
 build_*/
+.worktrees/
 .vscode
 __pycache__
 _site

diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md
@@ -74,7 +74,7 @@ The `src/test/phi4_rai` suite is still Windows-only and is not configured on Lin
 
 A rai build (`-DFLM_ENABLE_RAI=ON`) **links corelib in**, because the NPU device the whole process shares is the one behind corelib's hardware context rather than a device FastFlowLM opens itself. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_RAI_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked rai build it is ignored, and `flm` says so if it is set.
 
-The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.11.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. Within 0.11.0, FastFlowLM needs a corelib from after the weight API became one `*_weights_create` per operator (corelib commit `ab0bfee`); an older 0.11.0 header fails the build on the missing `*_weights_create` symbols. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux).
+The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.26.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_<name>` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows rai build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux).
 
 ```powershell
 flm pull phi4-mini-it-rai:4b

diff --git a/docs/docs/models/qwen.md b/docs/docs/models/qwen.md
@@ -107,9 +107,17 @@ flm run qwen3:8b
 - **Backend:** `rai` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it.
 - **Source format:** GGUF `Q8_0`, read directly. Every weight is requantized to **group 64** while it is packed, except the 0.6B lm_head, which packs at **group 32**.
 - **Usable generation window:** 4095 tokens, prompt plus output. Over-capacity requests are rejected with HTTP 400 before any work reaches the device.
-- **Availability:** developer build against corelib 0.11.0 — see [Phi-4 on the rai backend](phi.md).
+- **Availability:** developer build against corelib 0.26.0 — see [Phi-4 on the rai backend](phi.md).
 
-The same corelib path also runs `gemma4-it-rai:e2b`, `gemma4-it-rai:e4b` (text only) and `qwen3.5-rai:9b`.
+The same corelib path also runs `gemma4-it-rai:e2b` and `gemma4-it-rai:e4b` (text only), `qwen3.5-rai:9b` and `qwen3.6-moe-rai:35b-a3b` (text only). Qwen3 and Qwen3.5 stay at 4096; the other two go further:
+
+- **`qwen3.6-moe-rai:35b-a3b` — up to 262144 tokens.** The default is 4096; raise it with `--ctx-len` (`-c`). Long contexts have three costs:
+    - **Memory, up front.** The KV cache is allocated when the model is created, at the full capacity you ask for (rounded up to whole 4096-token pages), whether or not a conversation ever fills it: about 20 KiB per token (10 attention layers × 2 KV heads × 256 × BF16, for K and V), so 262144 tokens is about 5 GiB on top of the weights.
+    - **Long prompts run at decode speed.** A prompt is one forward call, but only its first 4096 tokens from position 0 run as a single pass; every token after them, and every later turn's prompt, is stepped one row at a time, at about decode speed. corelib measures decode at about 26 tokens/s at a shallow context, slowing by about 2.4 ms per token for each further 4096-token page (about 18 tokens/s at 32k). So a 100k-token prompt takes on the order of an hour: about 64 minutes even at the shallow rate, and nearer two hours once the per-page slowdown is counted (an extrapolation; corelib's measurements stop at 32k). FastFlowLM also still sends a prompt in chunks of at most 4096 tokens (`max_prefill_len`), so a long prompt is a one-pass first chunk followed by stepped ones.
+    - **Checkpoints take host memory.** A checkpoint (what lets a conversation rewind without replaying) copies the KV cache into host memory, in whole 4096-token pages up to the current position: about 80 MiB per filled page, so up to about 5 GiB at a full 262144-token context.
+- **`gemma4-it-rai:e2b` / `gemma4-it-rai:e4b` — up to 16384 tokens**, the length corelib has validated paged Gemma 4 to, and never past the GGUF's own `gemma4.context_length`. The default is 16384. Up to 4096 any length works; above 4096 the context must be a multiple of 4096. The KV cache for the whole context is allocated up front, when the model loads.
+
+Qwen3, Qwen3.5, Qwen3.6 and Gemma 4 run as corelib's own models, which pack their weights on every load: there is no packed-weight cache beside the GGUF. A `<gguf name>.rai-weights.bin` / `.json` pair left there by an earlier build is no longer read and can be deleted.
 
 ```powershell
 flm pull qwen3-rai:4b

diff --git a/docs/rai-setup-linux.md b/docs/rai-setup-linux.md
@@ -8,10 +8,10 @@ repository, so most of the work is standing up that prefix.
 Written against the working install on this machine. Paths use `$P` for the
 dependency prefix (here `/home/alfxu/ddprefix`); substitute your own.
 
-> **Status:** at corelib 0.11.0 the rai backend has been run on hardware on
-> Windows only. The Linux build configures and links against this layout, but
-> it has not been run end to end on Linux at this revision. The
-> [Known issues](#known-issues) are from an earlier Linux bring-up.
+> **Status:** the rai backend has been run on hardware on Windows only. The
+> steps below are updated for corelib 0.26.0, but they have not been run end
+> to end on Linux at this revision. The [Known issues](#known-issues) are from
+> an earlier Linux bring-up.
 
 ---
 
@@ -29,48 +29,44 @@ step 4 and installing XRT itself.
 
 ---
 
-## 1. Dependency prefix: DynamicDispatch, then corelib
+## 1. Dependency prefix: corelib
 
-This is the hard part and it is already written up in **`$P/BUILD-NOTES.md`** —
-read that first. It records the two things that actually matter (the required
-DD branch, and `BUILD_SHARED_LIBS=ON` for DD), the sudo-free dependency prefix,
-the non-PIC link traps, and the exact `cmake` command lines for both projects.
-
-Do not re-derive it. The short version:
+corelib no longer links DynamicDispatch, so there is no DD prefix to
+build first. **`$P/BUILD-NOTES.md`** still records the sudo-free dependency
+prefix and the non-PIC link traps; its DynamicDispatch steps no longer apply.
 
 ```bash
 P=/home/alfxu/ddprefix      # your dependency prefix
 
-# DynamicDispatch -> $P/install-dds     (see BUILD-NOTES.md for the full line)
-cmake -S DynamicDispatch -B build-dds ... -DCMAKE_INSTALL_PREFIX=$P/install-dds
-cmake --build build-dds -j"$(nproc)" --target install
-
 # ryzenai-corelib -> $P/install-corelib
 cmake -S ryzenai-corelib -B build-corelib -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_INSTALL_PREFIX=$P/install-corelib \
-  -DRYZENAI_CORELIB_STAGE_RUNTIME_DLLS=OFF \
-  -DCMAKE_PREFIX_PATH="$P/install-dds;$P/shim;$P/root/usr;$P/root/usr/lib/x86_64-linux-gnu;/opt/xilinx/xrt"
-cmake --build build-corelib -j"$(nproc)"
+  -DCMAKE_PREFIX_PATH="$P/shim;$P/root/usr;$P/root/usr/lib/x86_64-linux-gnu;/opt/xilinx/xrt"
+cmake --build build-corelib -j"$(nproc)" --target install
 ```
 
+The default build installs `libryzenai_corelib.so` and one
+`libryzenai_corelib_model_<name>.so` per model, each carrying that model's
+kernels. corelib loads a model library only from the directory it was itself
+loaded from, so those libraries must sit beside it; nothing else is needed at
+run time, no kernel store and no environment variable.
+
 **Version must match exactly.** `src/include/rai/corelib_api.hpp` has a
-`#error` for anything other than corelib **0.11.0**:
+`#error` for anything other than corelib **0.26.0**:
 
 ```bash
 grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \
   $P/install-corelib/include/ryzenai/corelib.h
 ```
 
-Must print `0`, `11`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening
-line — that text is stale; the installed headers are 0.11.0.) The headers must
-also be from after the weight API became one `*_weights_create` per operator;
-an older 0.11.0 checkout fails the FastFlowLM build on those symbols.
+Must print `0`, `26`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening
+line — that text is stale.)
 
 ### Verify corelib before going further
 
 ```bash
 source /opt/xilinx/xrt/setup.sh
-export LD_LIBRARY_PATH=$P/install-dds/lib:$P/install-corelib/lib:/opt/xilinx/xrt/lib:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=$P/install-corelib/lib:/opt/xilinx/xrt/lib:$LD_LIBRARY_PATH
 $P/build-corelib/tests/ryzenai_corelib_tests
 ```
 
@@ -98,8 +94,9 @@ What the flags do:
 
 - `--rai` selects the `linux-rai-on` preset (`-DFLM_ENABLE_RAI=ON`) and installs
   to `/scratch/$USER/flm_exe_rai` rather than `~/flm_exe`. A rai prefix stages
-  corelib and DynamicDispatch, which are far too large for a home directory, and
-  a rai build and a stock one would otherwise overwrite each other.
+  corelib and every `libryzenai_corelib_model_*.so` beside it, whose model
+  kernels are far too large for a home directory, and a rai build and a stock
+  one would otherwise overwrite each other.
 - `FLM_PREFIX=/path` overrides the install location.
 - `--no-build` installs an existing `build/` tree without rebuilding.
 
@@ -121,10 +118,11 @@ from the corelib *build tree*, not a stripped or patched copy:
 
 ```bash
 objdump -p src/lib/libryzenai_corelib.so | grep RUNPATH
-# -> /home/alfxu/ddprefix/install-dds/lib:/opt/xilinx/xrt/lib:
+# -> should name /opt/xilinx/xrt/lib
 ```
 
-Without that RUNPATH it will not find DynamicDispatch at run time.
+Without that RUNPATH it will not find XRT at run time. Copy every
+`libryzenai_corelib_model_*.so` into `src/lib/` beside it too.
 
 ### Rebuilding after a source change
 
@@ -150,7 +148,7 @@ rm -f /scratch/$USER/flm_exe_rai/lib/libxrt_coreutil.so.2
 ```
 
 This is a known bug in the installer's `ldd` staging walk
-(`home_install.sh:251-261`), not yet fixed. A rai prefix deliberately bundles no
+(`home_install.sh:247-256`), not yet fixed. A rai prefix deliberately bundles no
 XRT — `flm_env.sh` pins `XILINX_XRT` to the real root instead — so the staged
 copy is wrong by construction.
 
@@ -239,6 +237,9 @@ shape is a dead end.
 | `RYZENAI_CORELIB_SAFE_MODE` | force staging buffers, no host-pointer import |
 | `RYZENAI_CORELIB_NO_RUN_POOL` | disable run-object reuse |
 | `RYZENAI_CORELIB_NO_CRASH_HANDLER` | leave crashes to the debugger |
+| `RYZENAI_CORELIB_FALLBACK_PROFILE=1` | when a stream is released, report host (CPU fallback) time per artifact; no output means nothing fell back |
+| `RYZENAI_CORELIB_MAX_RUNS_IN_FLIGHT` | how many runs of one kernel a stream lets run ahead (64 unset, `0` for no cap) |
+| `RYZENAI_CORELIB_KERNELS_ROOT` | a kernel store to load from, for corelib development only; every model library already carries its kernels, so a normal run sets nothing |
 
 ### Other rough edges
 

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
@@ -357,6 +357,61 @@ if(FLM_ENABLE_RAI)
     # to resolve even though nothing calls them here.
     target_link_libraries(flm_rai PUBLIC "${RYZENAI_CORELIB_LIBRARY}" ${CMAKE_DL_LIBS})
     target_link_libraries(flm PRIVATE flm_rai)
+    # corelib loads each ryzenai_corelib_model_<name>.dll, which carries that
+    # model's kernels, only from the directory ryzenai_corelib.dll itself was
+    # loaded from, so both have to land beside flm.exe.
+    if(WIN32)
+        get_filename_component(FLM_CORELIB_LIB_DIR "${RYZENAI_CORELIB_LIBRARY}" DIRECTORY)
+        file(GLOB FLM_CORELIB_RUNTIME_DLLS
+            "${FLM_CORELIB_LIB_DIR}/../bin/ryzenai_corelib.dll"
+            "${FLM_CORELIB_LIB_DIR}/../bin/ryzenai_corelib_model_*.dll")
+        list(FILTER FLM_CORELIB_RUNTIME_DLLS EXCLUDE REGEX "ryzenai_corelib_model_dummy\\.dll$")
+        if(FLM_CORELIB_RUNTIME_DLLS)
+            add_custom_command(TARGET flm POST_BUILD
+                COMMAND ${CMAKE_COMMAND} -E copy_if_different
+                    ${FLM_CORELIB_RUNTIME_DLLS} $<TARGET_FILE_DIR:flm>)
+        else()
+            message(WARNING "No ryzenai_corelib DLLs under ${FLM_CORELIB_LIB_DIR}/../bin; "
+                            "copy them beside flm.exe by hand")
+        endif()
+    endif()
+endif()
+
+# ———————————————————————————————————————————————
+# Opt-in Windows dependency prefix.
+#
+# Windows otherwise has exactly two routes for Boost/curl/FFmpeg: a vcpkg
+# toolchain, or the bare CI runner's hardcoded C:/dev paths and CI-specific
+# library names. A developer box with neither can point this at one prefix
+# laid out as <prefix>/include + <prefix>/lib -- in practice a conda env's
+# Library dir (conda-forge boost, libcurl, ffmpeg) -- and each block that
+# applies the CI runner's include dirs, definitions, link dirs and library
+# names switches to it.
+# Empty (the default) leaves both existing routes exactly as they were.
+# FFTW is not taken from the prefix on any route: its header (include/fftw3.h)
+# and import libs (lib/libfftw3*-3.lib) are vendored in this tree.
+# ———————————————————————————————————————————————
+set(FLM_WIN_DEPS_PREFIX "" CACHE PATH
+    "Windows only: take Boost/curl/FFmpeg from <prefix>/include and <prefix>/lib (e.g. a conda env's Library dir) instead of the CI runner's C:/dev paths")
+if(FLM_WIN_DEPS_PREFIX)
+    if(NOT WIN32)
+        message(FATAL_ERROR "FLM_WIN_DEPS_PREFIX is a Windows-only route")
+    endif()
+    if(VCPKG_TOOLCHAIN)
+        message(FATAL_ERROR "FLM_WIN_DEPS_PREFIX and a vcpkg toolchain are two "
+                            "different dependency routes; use one")
+    endif()
+    # The library names a conda-forge Library dir ships: unversioned Boost and
+    # curl import libs (both are DLL builds), FFmpeg import libs.
+    set(FLM_WIN_DEPS_PREFIX_LIBS
+        avformat avcodec avutil swscale swresample libcurl boost_program_options)
+    foreach(_flm_dep IN LISTS FLM_WIN_DEPS_PREFIX_LIBS)
+        if(NOT EXISTS "${FLM_WIN_DEPS_PREFIX}/lib/${_flm_dep}.lib")
+            message(FATAL_ERROR "FLM_WIN_DEPS_PREFIX: missing "
+                                "${FLM_WIN_DEPS_PREFIX}/lib/${_flm_dep}.lib")
+        endif()
+    endforeach()
+    message(STATUS "Windows deps from prefix: ${FLM_WIN_DEPS_PREFIX}")
 endif()
 
 if(WIN32)
@@ -373,6 +428,8 @@ if(WIN32)
         find_package(FFTW3f CONFIG REQUIRED)
         find_package(FFTW3l CONFIG REQUIRED)
     endif()
+    # A third, opt-in route (FLM_WIN_DEPS_PREFIX, declared above) takes Boost,
+    # curl and FFmpeg from one local prefix, wired next to each CI-runner block.
     # Otherwise (the bare self-hosted CI runner) there is no vcpkg toolchain and
     # no Boost in vcpkg: Boost is a standalone b2 build under C:/dev/boost_1_88_0
     # and curl/ffmpeg/fftw are linked by raw name from the vcpkg lib dir. That
@@ -424,12 +481,14 @@ if(NOT FLM_USE_HRX)
     endif()
 endif()
 
-if(WIN32 AND NOT VCPKG_TOOLCHAIN)
+if(WIN32 AND NOT VCPKG_TOOLCHAIN AND NOT FLM_WIN_DEPS_PREFIX)
     # Bare CI runner: standalone Boost + vcpkg headers by absolute path.
     target_include_directories(flm PUBLIC
         C:/dev/boost_1_88_0
         C:/dev/vcpkg/installed/x64-windows/include/
     )
+elseif(FLM_WIN_DEPS_PREFIX)
+    target_include_directories(flm PUBLIC "${FLM_WIN_DEPS_PREFIX}/include")
 endif()
 
 target_compile_definitions(flm PUBLIC
@@ -456,13 +515,22 @@ if(WIN32)
     )
 endif()
 
-if(WIN32 AND NOT VCPKG_TOOLCHAIN)
+if(WIN32 AND NOT VCPKG_TOOLCHAIN AND NOT FLM_WIN_DEPS_PREFIX)
     # Bare CI runner: statically link the standalone Boost + curl.
     target_compile_definitions(flm PUBLIC
         CURL_STATICLIB
         BOOST_ALL_NO_LIB
         BOOST_ALL_STATIC_LINK
     )
+elseif(FLM_WIN_DEPS_PREFIX)
+    # conda-forge Boost and curl are DLL builds. program_options exports data
+    # (boost::program_options::arg, options_description::m_default_line_length)
+    # that only resolves through dllimport, so BOOST_ALL_STATIC_LINK would leave
+    # it unresolved at link time; CURL_STATICLIB would mis-declare curl likewise.
+    target_compile_definitions(flm PUBLIC
+        BOOST_ALL_NO_LIB
+        BOOST_ALL_DYN_LINK
+    )
 endif()
 
 target_link_directories(flm PUBLIC
@@ -480,12 +548,14 @@ if(NOT FLM_USE_HRX)
     endif()
 endif()
 
-if(WIN32 AND NOT VCPKG_TOOLCHAIN)
+if(WIN32 AND NOT VCPKG_TOOLCHAIN AND NOT FLM_WIN_DEPS_PREFIX)
     # Bare CI runner: standalone Boost stage libs + vcpkg import libs.
     target_link_directories(flm PUBLIC
         C:/dev/boost_1_88_0/stage/lib
         C:/dev/vcpkg/installed/x64-windows/lib
     )
+elseif(FLM_WIN_DEPS_PREFIX)
+    target_link_directories(flm PUBLIC "${FLM_WIN_DEPS_PREFIX}/lib")
 endif()
 
 # Link static libraries first
@@ -602,6 +672,17 @@ if(WIN32 AND VCPKG_TOOLCHAIN)
         FFTW3::fftw3f
         FFTW3::fftw3l
     )
+elseif(WIN32 AND FLM_WIN_DEPS_PREFIX)
+    # Local prefix: the names it ships (checked at configure time above). FFTW
+    # is the tree's vendored lib/libfftw3*-3, as on the CI runner. flm itself
+    # calls only fftwf_* and no prebuilt engine DLL imports FFTW, so the double
+    # and long-double libs contribute no import; they stay for parity.
+    target_link_libraries(flm PUBLIC
+        ${FLM_WIN_DEPS_PREFIX_LIBS}
+        libfftw3-3
+        libfftw3f-3
+        libfftw3l-3
+    )
 elseif(WIN32)
     # Bare CI runner: link the native deps by raw name from the standalone Boost
     # + vcpkg lib dirs wired above (no CMake package configs there).

diff --git a/src/common/AutoModel/builtin_backends.cpp b/src/common/AutoModel/builtin_backends.cpp
@@ -13,6 +13,7 @@
 #include "models/phi4/rai/aie_next/phi4_rai_backend.hpp"
 #include "models/qwen3/rai/aie_next/qwen3_rai_backend.hpp"
 #include "models/qwen35/rai/aie_next/qwen35_rai_backend.hpp"
+#include "models/qwen36/rai/aie_next/qwen36_rai_backend.hpp"
 #endif
 
 namespace flm::backend {
@@ -70,6 +71,9 @@ void register_builtin_backends(BackendRegistry& registry) {
     registry.register_backend("qwen3.5", flm::backend::kRaiBackendId,
                               flm::qwen35::rai_factory(),
                               flm::qwen35::rai_traits());
+    registry.register_backend("qwen3.6-moe", flm::backend::kRaiBackendId,
+                              flm::qwen36::rai_factory(),
+                              flm::qwen36::rai_traits());
 #endif
 }