poolrun

The local-model runtime for unified-memory boxes — DGX Spark, GB10, Jetson. An OpenAI-compatible proxy that sits in front of Ollama and does the one thing Ollama and vLLM structurally don't: it refuses to start a model load that would freeze the machine.

your app  ──OpenAI /v1/chat/completions──▶  poolrun (:11453)  ──▶  ollama (:11434)
                                              │
                                    admission gate · single-resident
                                    mutex · hot-swap · thinking-strip · PSI watchdog

Why this exists (the differentiator)

On a normal box the GPU has its own VRAM, separate from system RAM, and the GPU driver returns a clean out-of-memory error when a model won't fit. On a unified-memory box (Spark / GB10 / Jetson) the GPU and CPU share one physical pool, and a GPU allocation is charged through the driver where the Linux cgroup cannot see it — a verified 8 GiB cudaMalloc moved memory.current by only 87 MiB. nvidia-smi reports N/A. So when you start a model onto a nearly-full pool, the kernel can't reclaim the GPU pages, drags in the slow swapfile, memory pressure (PSI full) climbs, and the whole box livelocks before any clean GPU-OOM ever fires. It just freezes. You reboot.

Ollama and vLLM have no defense against this — they'll happily try the load. poolrun's insight (productized from the memguard methodology): you can't cap your way to safety on UMA — you have to not start the load that won't fit. The only gauges that tell the truth are /proc/meminfo (MemAvailable + SwapFree) and /proc/pressure/memory. poolrun reads exactly those, never nvidia-smi.

What it does

1. Admission control — before loading or switching a model, read MemAvailable from /proc/meminfo, estimate the model's footprint generously (on-disk weights × 1.25 + overhead — because the GPU side is uncounted, under-guessing is the only way to defeat the guard), and refuse with a clear 503 + human reason if the load would breach the configured headroom, the hard floor, or the swap-min floor.
2. Single-resident mutex + hot-swap — exactly one heavy model resident at a time. A request for a different model evicts the incumbent, then loads the requested one — serialized under a lock so two requests can't race a double-load onto the shared pool.
3. Auto thinking-mode strip — injects think:false on chat completions, removing the known local-chat latency tax (thinking-mode adds many seconds per turn on reasoning models) unless you opt out.
4. PSI watchdog — a background thread polls /proc/pressure/memory; when full pressure goes red (or the pool + swap both bottom out), it freezes admissions (non-destructive) and auto-thaws after sustained calm.
5. poolrun status — a truthful live view of the shared pool and which model is actually resident (from Ollama's /api/ps, not a guess).

Requirements

• Python 3.11+ (uses stdlib tomllib). Zero pip dependencies.
• Ollama running at localhost:11434.

Run

# 1. start the proxy (foreground)
cd ~/poolrun
./bin/poolrun serve
#   poolrun listening on http://127.0.0.1:11453  ->  http://localhost:11434

# 2. point any OpenAI client at it
curl http://127.0.0.1:11453/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gemma4:latest","messages":[{"role":"user","content":"hi"}]}'

# 3. truthful pool + residency view
./bin/poolrun status

# 4. dry-run the admission ladder for a model WITHOUT loading it
./bin/poolrun check qwen3.6:latest
./bin/poolrun check monster:200b --sim-footprint-gb 200   # -> REFUSE

Symlink bin/poolrun into ~/bin for a global command. As an OpenAI base URL, use http://127.0.0.1:11453/v1.

Endpoints

Method	Path	Purpose
POST	/v1/chat/completions	admission-gated, thinking-stripped chat (non-streaming)
GET	/v1/models	model list, flags the resident one
GET	/status	same JSON as poolrun status --json
GET	/healthz	liveness

Testing the guard without risking the box

Every admission decision is a pure function fed a memory snapshot and an estimated footprint, so you can prove the refusal path without loading anything:

• Header hook: send X-Poolrun-Sim-Footprint-GB: 300 on a chat request for a non-resident model — poolrun runs the full ladder against the simulated footprint, returns the decision, and loads nothing.
• Phantom ledger: the [phantom_models] table in poolrun.toml registers fake models with fake footprints (e.g. monster:200b = 200.0) that a plain request will exercise the gate against.

Configuration

Defaults live in poolrun.toml (tuned for a 128 GB Spark) and map 1:1 to poolrun/config.py. Override any field with a POOLRUN_<FIELD> env var, e.g. POOLRUN_MIN_HEADROOM_GB=24 ./bin/poolrun serve.

Key knobs (all GB): min_headroom_gb (16 — projected free must clear this), floor_gb (4 — abort below), swap_min_gb (4 — never load below), and footprint_factor / footprint_base_gb for the estimate.

Design notes

• Stack: Python + stdlib only, for the fastest path to a dogfooded runtime. A Rust rewrite is the eventual "franken-line" target (matching the Dicklesworthstone / Jeffrey-Emanuel pattern of a clean Python proof followed by a hardened Rust reimplementation) — but that is explicitly not built yet.
• Relationship to memguard: memguard is the machine-wide launch guardian (wraps any heavy process in a cgroup scope). poolrun productizes the same admission-control core into a model-serving runtime — one long-lived proxy that gates every model load through the OpenAI API surface your apps already speak. They compose: run poolrun under memguard run for belt-and-suspenders.
• Layered safety (from the memguard methodology): the admission gate + mutex is the primary defense (it prevents the freeze); the PSI watchdog is an early-warning freeze on unwrapped pressure; earlyoom remains the one kernel-adjacent reaper of last resort. poolrun does not add a second OOM daemon (double-kills).

Not yet built (walking skeleton)

Streaming responses (stream:true), request queueing instead of outright refuse (the memguard DEGRADE/QUEUE rungs), embeddings/completions endpoints, and the Rust franken-line. See the top of TODO in the final report.

Layout

poolrun/
  meminfo.py    truthful /proc/meminfo + /proc/pressure/memory reads
  config.py     Config dataclass, TOML + env loading
  admission.py  the pure decision ladder (unit-tested, no I/O)
  ollama.py     tiny stdlib Ollama client (tags / ps / chat / load / unload)
  pool.py       single-resident mutex, hot-swap, PSI watchdog thread
  server.py     OpenAI-compatible proxy (http.server)
  cli.py        serve / status / check
tests/
  test_admission.py   10 tests, simulate oversized loads, allocate nothing
bin/poolrun     launcher (symlink into ~/bin)
poolrun.toml    default config

License

MIT. Clean-room: implements the admission-control method from the memguard skill; contains no third-party code.