The GGUF translation models (Gemma E4B Q6/Q8, VNTL Llama-3 Q5_K_M/Q8_0,
Qwen3.5 9B Q6_K/Q8_0 and 4B Q8_0) run through
llama_cpp, a ctypes wrapper around a shared library that does not exist
until someone compiles it. This document covers which version we compile,
for which devices, with which flags — and why each choice is the way it is.
The implementation lives in packaging/llamacpp.py. The build driver that
calls it is packaging/build.py (install_llama_cpp, step 5/15).
Qwen3.5 uses a hybrid Gated DeltaNet/attention architecture, so an older
llama.cpp build that supports Qwen3 is not enough. Use the pinned backend
below with Qwen3.5 support. Fox Reader runs these models as text translators,
without a vision encoder (mmproj), and explicitly disables thinking using
Qwen3.5's non-thinking prompt format. The runtime window defaults to 8192 tokens
and can be set to 16384 per model in /settings, with up to 1024 generated
tokens. The change applies when that model is next loaded. Gemma and Qwen
support both choices; VNTL remains at 8192. The verified upstream GGUF limits
are 131072 for Gemma4, 262144 for Qwen3.5 and 8192 for VNTL/Llama3. Fox Reader
checks the selected file's metadata before allocation and never extends its
declared context limit. Context and character metadata remain optional; plain
text requests follow the same translation path.
| Item | Value | Where |
|---|---|---|
| Package | llama-cpp-python |
pyproject.toml, gguf extra |
| Pinned version | ==0.3.36 |
uv.lock |
| Vendored llama.cpp | ggml 0.25.3 |
inside the locked sdist, vendor/llama.cpp |
| Torch for cu118 | 2.7.1+cu118 (last release on the cu118 index) |
uv.lock |
| Torch for cu129 on Windows | 2.8.0+cu129 |
uv.lock |
| Torch for everything else | 2.13.0 (+cpu/cu126/cu129/cu130/cu132/rocm7.1/rocm7.2) |
uv.lock |
Three rules follow from the fact that PyPI ships no wheels for this package — only an sdist:
- Every install compiles llama.cpp. What the binary can do is decided
entirely by
CMAKE_ARGS. There is no "CUDA wheel" to download. - The version is the locked one.
locked_version()reads it out ofuv.lock; an install with no pinned version is refused rather than resolved fresh, so two builds of one commit never compile different llama.cpp releases. - No cache. Installs use
uv pip install --no-cache-dir, because uv's wheel cache is keyed on the sdist, not onCMAKE_ARGS— without the flag a CUDA build silently reuses yesterday's CPU wheel and segfaults on the first model load.
--device picks a torch build (via a [tool.uv.sources] index) and a
llama.cpp backend (via generated CMAKE_ARGS). Getting the second one
wrong is quiet — a CUDA build with a CPU-only libllama loads, answers, and
translates at a tenth of the speed with nothing in the log — so an argument
list that cannot be satisfied is a build failure, never a downgrade.
--device |
Torch index | Torch | Needs toolkit | CI runner |
|---|---|---|---|---|
cpu |
CPU | 2.13.0 | none | ubuntu-24.04 / windows-latest |
macos |
PyPI | 2.13.0 | Apple Clang + macOS SDK | GitHub-hosted macos-15 ARM64 |
cu118 |
cu118 | 2.7.1 | CUDA 11.8 + gcc-11 | ubuntu-22.04 / windows-latest |
cu126 |
cu126 | 2.13.0 | CUDA 12.6 + gcc-13 | ubuntu-24.04 / windows-latest |
cu129 (Linux) |
cu129 | 2.13.0 | CUDA 12.9 + gcc-13 | ubuntu-24.04 |
cu129 (Windows) |
cu129 | 2.8.0 | CUDA 12.9 + MSVC | windows-latest |
cu130 |
cu130 | 2.13.0 | CUDA 13.0 + gcc-14 | ubuntu-24.04 / windows-latest |
cu132 |
cu132 | 2.13.0 | CUDA 13.2 + gcc-14 | — (no CI row yet; see notes below) |
rocm71 / rocm72 |
rocm7.1 / rocm7.2 | 2.13.0 | ROCm 6.1+ | Linux only |
Notes:
cu118is stuck on torch 2.7.1 because that is the newest release the cu118 index carries;cu129on Windows is locked to 2.8.0. Everything else tracks 2.13.0.cu129is one extra with platform-specific requirements. Linux pins Torch 2.13.0 and torchvision 0.28.0; Windows requires Torch >=2.7.1 and torchvision >=0.22.1, with the lock retaining 2.8.0+cu129 / 0.23.0+cu129. Both platforms usehttps://download.pytorch.org/whl/cu129and the CUDA 12.9 toolkit. The compiler selection and GPU architecture coverage are unchanged. Replace the former--extra cu129_winwith--extra cu129when installing, and--device cu129_winwith--device cu129when building.- 13.x torch landscape (PyTorch 2.11+). Plain
pip install torchnow ships CUDA 13.0 wheels by default;cu126stays as the fallback for older drivers and pre-Turing GPUs;cu128is deprecated in favour ofcu130.cu132is wired throughpyproject.toml(extra, index, sources) sopython packaging/build.py --device cu132works locally, but.github/workflows/build.ymlhas no matrix row for it yet — add{ name: cu132-gguf-linux, os: ubuntu-24.04, device: cu132, cuda: "13.2.x", gcc: "14", extra: "" }(plus-windows/--no-gguftwins) to enable it in CI. It compiles the identical arch list to 13.0 (13.2 adds no new architectures, only toolchain fixes). - ROCm wheels exist for Linux only, so
--device rocm*is refused on Windows — but--gguf-device rocm72with--device cpuworks there for the translator backend alone. - The name is pattern-based, not a lookup table: a future
cu133extra inpyproject.tomlworks with no code change; the CUDA version is read out of the digits.
| Flag | Effect |
|---|---|
--no-gguf |
Skip the backend entirely (no OpenBLAS/CUDA needed; GGUF models won't load) |
--gguf-device DEVICE |
Build llama.cpp for another device than --device, e.g. CUDA torch + CPU translator for a machine whose toolkit can't compile the CUDA backend |
--openblas {auto,download,off} / --no-blas |
OpenBLAS for ggml's CPU backend (auto = system copy, else fetch into nc/) |
--cpu-baseline {avx2,avx,sse42} |
Oldest x86-64 CPU the result must run on (default avx2, ≈2013+) |
--cuda-arch LIST / --hip-arch LIST |
Override the arch lists below, e.g. --cuda-arch "75-real;86-real" |
--device macos builds natively for ARM64, macOS 14.0 and newer. The
locked Torch and torchvision wheels set that OS floor. The hardware target
includes M1 and later Apple Silicon generations, including Pro, Max and
Ultra variants. Intel Macs and Rosetta processes are rejected.
The Mac branch uses these independent portability controls:
| Setting | Purpose |
|---|---|
GGML_NATIVE=OFF |
Do not tune native CPU code to the build host |
GGML_CPU_ARM_ARCH=armv8.2-a+fp16+dotprod |
Explicit M1-compatible CPU instruction baseline |
CMAKE_OSX_ARCHITECTURES=arm64 |
Native Apple Silicon output |
CMAKE_OSX_DEPLOYMENT_TARGET=14.0 |
Keep the minimum OS independent of the build host's OS |
GGML_METAL=ON |
Include the Metal backend |
GGML_METAL_EMBED_LIBRARY=ON |
Embed shader source, compiled by the destination GPU's Metal driver |
GGML_ACCELERATE=ON |
Use Apple's system Accelerate framework for CPU operations |
GGML_OPENMP=OFF |
Avoid a dependency on the build host's OpenMP runtime |
In the locked source, “embed library” embeds the Metal kernel source and its
headers into the native library; it does not freeze a precompiled library for
the build machine's GPU. The Metal runtime chooses the shader language and
feature macros for the destination OS and GPU. This keeps runtime capability
selection while avoiding a separate .metallib file or an offline Metal
compiler requirement. CPU fallback kernels retain the M1 baseline even when
the build host has newer ARM instructions.
With GPU hardware available, the installer verifies that the Metal backend supports GPU offload and initializes it to exercise embedded shader compilation. Torch's MPS backend is checked separately; Torch MPS availability alone does not prove that llama.cpp supports Metal.
GitHub-hosted ARM64 runners may expose no GPU. The Mac workflow sets
FOX_BUILD_ALLOW_NO_GPU=1 so those runners can build the same Metal-enabled
release. Verification still loads the Metal libraries, checks their backend
symbols and embedded shader source, and initializes the CPU backend. The
relocated distribution must also start successfully and select the available
device. Logs report GPU execution as untested when no GPU is present.
The variable changes verification policy only; the CPU instruction baseline, Metal compilation flags and resulting backend support remain the same. Missing Metal support, missing embedded source or CPU initialization failures are fatal. If a GPU is available, shader and GPU initialization failures are also fatal. Local builds remain strict by default and require GPU execution unless the variable is explicitly set.
The distribution builder restores the native libraries needed by ctypes, relocates non-system dependencies within the bundle, checks ARM64 support and minimum OS load commands, and ad-hoc signs the final native files. OpenBLAS is not downloaded for the Metal branch. The Windows/Linux CPU and CUDA build paths retain their existing behavior.
Use macos for both Torch and llama.cpp in a Mac distribution; --no-gguf
can omit llama.cpp. CPU/CUDA/HIP architecture overrides are for other build
variants. At runtime, selecting cpu in Fox Reader settings still uses the
CPU backend included in the Metal build.
Build checks and a fixed baseline do not replace inference testing. Hardware verification requires testing the same relocated archive on an M1 Mac running macOS 14 and a newer Apple Silicon Mac, including GGUF translation and PaddleOCR-VL. Hosted builds can produce and publish releases without a personally owned Mac; they do not establish GPU inference coverage. The Mac build guide describes the automated and manual checks.
Every CUDA build passes an explicit CMAKE_CUDA_ARCHITECTURES. Entries are
mostly -virtual (PTX, JIT-compiled by the driver on first load, forward
compatible — one PTX target covers every newer card) with -real (machine
code) for the architectures people actually have.
| Entry | Codename | Example cards | CUDA range |
|---|---|---|---|
50-virtual |
Maxwell (GM10x) | GTX 750 Ti / 900 series, Tesla M60 | 6.5 – 12.x (dropped in 13.0) |
61-virtual |
Pascal (GP10x) | GTX 10 series, Tesla P100 / P40 | 8.0 – 12.x (dropped in 13.0) |
70-virtual |
Volta (GV100) | Tesla V100, Titan V | 9.0 – 12.x (dropped in 13.0) |
75-real |
Turing (TU10x) | RTX 20 series, GTX 16 series, Tesla T4 | 10.0 – current (13.x floor) |
80-virtual |
Ampere datacenter (GA100) | A100 / A30 | 11.0 – current |
86-real |
Ampere consumer (GA10x) | RTX 30 series, A40 | 11.1 – current |
89-real |
Ada Lovelace (AD10x) | RTX 40 series, L40 | 11.8 – current |
90-virtual |
Hopper (GH100/GH200) | H100 / H200 | arch since 11.8, buildable since 12.x (see §5) |
100 / 103 |
Blackwell datacenter | B100 / B200 / B300, GB200 | 12.x – current |
120a-real |
Blackwell consumer | RTX 50 series | 12.8 – current |
121a-real |
Blackwell refresh | newer 50-series SKUs | 12.9 – current |
107 |
Rubin (preview) | Vera Rubin | 13.4+ (preview, GA later) |
Reference: NVIDIA's CUDA Toolkit, Driver, and Architecture Matrix, Table 3 (First/Last Toolkit columns); per-release arch sets at https://gpucompat.com/cuda/. Latest release at time of writing: 13.4.x.
| Toolkit | Arch set (sm_XX) |
Floor / notes |
|---|---|---|
| 13.0 (Aug 2025) | 75, 80, 86, 89, 90, 100, 103, 120, 121 | Drops everything below 7.5 (Maxwell/Pascal/Volta gone from offline compilation); min driver R580; GCC 15 support |
| 13.1 (Dec 2025) | same as 13.0 | bugfix release, no arch changes |
| 13.2 (Mar 2026) | same as 13.1 | cu132 wired in pyproject.toml, still experimental upstream; compiler/CUBLAS correctness fixes over 13.0–13.1 |
| 13.3 (May 2026) | same as 13.2 | CUDA-graph recapture improvements |
| 13.4 (Jul–Sep 2026) | above + 107 (Rubin, preview) |
Windows-on-Arm support; Rubin GA in a later toolkit |
So for this repo: cu130 (13.0) is the current stable CUDA build and covers
Turing through Blackwell; cu132 compiles the identical arch list
(75-real … 121a-real) — 13.2 adds no new architectures over 13.0, only
toolchain fixes. Pre-Turing cards (Maxwell/Pascal/Volta) can only use
cu118/cu126 builds, which is why those rows stay in the matrix.
What cuda_architectures() (packaging/llamacpp.py:473-503) generates from that table:
- CUDA 11.x keeps
50-virtual(deprecated loudly in 12.x, but an 11.x build exists for old cards) and drops it from 12.x on. - CUDA 13 drops everything below 75 — asking for 50 there is a hard nvcc error.
120a/121aarrive with 12.8/12.9. They must be-real: family-specific targets are not forward compatible, so PTX from 9.0 is what covers anything newer.filter_cuda_architectures()then asks the installed nvcc (--list-gpu-arch) what it knows and drops anything unknown — so a stale table costs a slightly smaller binary, never a failed build. (It cannot save a build from missing headers, only from unknown archs; see §5.)
nvcc compiles no host code itself; it hands it to the system compiler and
refuses one newer than its toolkit knew about. The CI matrix pins one GCC
per toolkit (one step under the ceiling, matching
packaging/build_with_docker.py):
| Toolkit | GCC | Runner |
|---|---|---|
| 11.8 | gcc-11 / g++-11 |
ubuntu-22.04 (NVIDIA's 11.8 repos stop there) |
| 12.6 / 12.9 | gcc-13 / g++-13 |
ubuntu-24.04 |
| 13.0 / 13.2 | gcc-14 / g++-14 (13.x also accepts GCC 15) |
ubuntu-24.04 |
On Windows nvcc drives MSVC only (never MinGW). Each toolkit has an
_MSC_VER ceiling read out of its own crt/host_config.h — e.g. 11.8 stops
at 1940, so the build picks toolset 14.29 when the runner ships a newer one.
A machine with only an incompatible pair (CUDA 11.8 + MSVC 14.44, which wants
CUDA ≥ 12.4 from the other direction) cannot build CUDA at all until one
side changes; the probe .cu compile in _cuda_host_compiler() proves the
pair works before CMake ever runs.
llama-cpp-python 0.3.35 (ggml 0.20.0) calls
cudaGridDependencySynchronize() /
cudaTriggerProgrammaticLaunchCompletion() in
ggml/src/ggml-cuda/common.cuh. Its own guard:
#if (CUDART_VERSION >= 12030 || (!(defined(_MSC_VER) && !defined(__clang__)) && CUDART_VERSION >= 11080))
# define GGML_CUDA_USE_PDL
- Linux + GCC + CUDA 11.8 → PDL on → device code for arch 90 calls
functions the 11.8 headers don't declare →
acc.cufails, whole build fails. Any90target, real or PTX, triggers it. - Windows + MSVC + CUDA 11.8 → PDL off (the MSVC carve-out) → the same arch list compiles to no-ops → build passes, minus that optimization.
Hence cuda_architectures() emits 89-real for 11.8 and only adds
90-virtual from CUDA 12.0 on. Hopper cards on an 11.8 build still run via
the 89-PTX fallback. Upstream's own 11.8 wheels cover 6.0–8.9 for the same
reason. If a future llama-cpp moves its minimum toolkit again, this gate
(packaging/llamacpp.py:497) — not the arch filter — is where to move it.
A GPU backend never replaces ggml's CPU backend; it sits beside it and runs whatever the device has no kernel for. So every device but Metal wants BLAS:
auto(default): use the system OpenBLAS if usable, else fetch intonc/without asking (Windows release zip /scipy-openblas32wheel on Linux).download: same fetch, explicitly.off/--no-blas: build without it — slower prompt processing, nothing else changes. Never interactive: CI must work unattended.
Linux note: the distro package hides headers in a multiarch subdirectory, so
CI symlinks them into one prefix and names it via OPENBLAS_HOME — the
layout packaging/llamacpp.py looks for. python packaging/llamacpp.py --find-openblas reports what would be used.
# What would this device build with (no toolchain needed for the answer)?
python packaging/llamacpp.py --device cu129 --print-args
python packaging/llamacpp.py --device cpu --openblas auto --print-args
python packaging/llamacpp.py --device macos --print-args
# What OpenBLAS is here?
python packaging/llamacpp.py --find-openblas
# Install into an environment (note --no-cache-dir is mandatory, see §1):
python packaging/llamacpp.py --device cu129 --install --python .venv/bin/python
CMAKE_ARGS="<printed line>" uv pip install --no-cache-dir llama-cpp-python==0.3.36On Apple Silicon, prefer python3.12 packaging/llamacpp.py --device macos --install --python .venv/bin/python so the installer also sets the Apple
toolchain environment and performs Metal verification.
--print-args answers even on a machine that couldn't build the result
(exit 1 with the reason first, then the unfiltered args). verify() afterwards
checks the install actually built what was asked: CUDA backend present for
CUDA devices, lib/ layout intact, linked runtimes shippable.
| Symptom | Cause | Fix |
|---|---|---|
common.cuh: error: cudaGridDependencySynchronize is undefined |
Toolkit too old for this llama-cpp (see §5) | Build that device with --gguf-device cpu, or a newer toolkit |
nvcc refuses the host compiler (unsupported GNU/MSVC version) |
Wrong GCC/MSVC for the toolkit (§4) | Install the table's compiler; the build picks versioned g++-N itself |
cblas_sgemm was not declared deep in ggml |
System OpenBLAS missing/misdetected | apt install libopenblas-dev, or --openblas download |
| CUDA build fails on MinGW (Windows) | Expected — nvcc only drives MSVC | Visual Studio Build Tools, "Desktop development with C++" |
| Rebuild behaves oddly | Stale packaging/build/ or reused venv compiled for another device |
--clean; installs always --no-cache-dir |
| GGUF problems you don't need | — | --no-gguf builds everything else |
| No Metal GPU on a hosted runner | The VM exposes no GPU hardware | Hosted CI sets FOX_BUILD_ALLOW_NO_GPU=1; backend/source and CPU checks remain mandatory, GPU execution is reported untested |
| Metal library present but initialization fails on an available GPU | Unusable embedded shaders or backend failure | Inspect the Metal probe output; allowing absent GPUs does not suppress this failure |
| Mac binary fails on an older Apple chip | Host-specific CPU flags or stale native libraries | Rebuild with the Mac branch's fixed M1 baseline and --clean; do not add -mcpu=native |
| Mac binary requires a newer OS than advertised | A dependency has a higher minimum OS load command | Use dependencies compatible with macOS 14; the bundle audit rejects this instead of silently raising the release floor |