Run open-weight models on your own hardware, from ServerKit.
Anyone can start an inference container. What nobody tells you — before you spend two hours and 600 GB of disk — is whether the model you picked will actually run on your box. That is what this extension is for.
- Scores every model against this host before you deploy: ✅ fits in VRAM · 🟡 fits with CPU offload · 🟠 RAM-only (very slow) · ❌ needs N GPUs you don't have.
- Preflights GPU passthrough. A compose file with
deploy.resources.reservations.devicesstill starts without the NVIDIA Container Toolkit — it just silently runs on the CPU, 15–20× slower. That is the single worst failure mode here, so it is checked rather than assumed. - Picks the engine — Ollama, llama.cpp, vLLM or SGLang — from the verdict and the host, then deploys it through ServerKit's normal template path.
- Registers the result as an endpoint the panel's assistant and other extensions can use.
For a Mixture-of-Experts model, memory scales with total parameters and compute scales with active parameters. Every expert must be resident even though only a few run per token.
So "Kimi K3 activates only 104B of its 2.8T parameters" does not mean 104B worth of VRAM. It means ~1.4 TB of weights and a two-node, 16-GPU deployment. The advisor says this out loud instead of returning a bare refusal:
Will not fit. Needs about 1.44 TB; this host has 48.0 GB of VRAM across 2 GPUs
and 64.0 GB of available RAM. Only 104B of 2.8T parameters are active per
token, but every expert must be resident — memory scales with the total, not
the active count. Documented deployments: B300 1x8; GB300 2x4; B200 2x8
(DP16 + EP16, 56 experts per GPU); H200 2x8 (vLLM default recipe, TP16);
H100 4x8; MI350X/MI355X 1x8.
It advises; it never blocks. A ❌ row still renders and still installs behind an acknowledgment — detected hardware is never a hard gate.
backend/hardware.py host probes (CPU/RAM/disk/GPU/container toolkit)
backend/advisor.py the sizing arithmetic and its verdicts
backend/catalog.py catalog loader + structural validation
catalog/models.json measured constants per model and quantization
templates/ engine templates, CPU and GPU variants
scripts/ catalog validator (dependency-free, runs in CI)
Deriving weight sizes and KV footprints at runtime from each model's
config.json is a research project with a long tail of architecture-specific
special cases. A reviewable, dated, sourced JSON file is honest and testable
instead. Every entry carries verified_on and sources, and CI fails on a
malformed one.
Where a model's attention shape is not confidently known, the kv block is
omitted rather than guessed — and the advisor then flags the KV cache as
unaccounted instead of quietly returning a number that is too low.
Phases 0–2 of the plan are implemented and unit-tested. No part of this has been verified against real GPU hardware — the box it was written on has none. Anything touching passthrough, weight pulls or multi-node serving is code-complete and untested in the field.
MIT.