local-llm-stack

Small Linux-focused scripts for running Open WebUI with a local llama.cpp Vulkan backend.

The project has one job: start and manage one complete local stack. ./llm-stack start launches Open WebUI and llama.cpp together, waits until both are healthy, and returns to the shell.

Requirements

• Linux and Bash
• Docker
• A Vulkan-capable GPU exposed as /dev/dri/renderD*
• curl, python3, realpath, readlink, mktemp, cp, mkdir, date, du, df, grep, and stat

ss and findmnt improve port and storage diagnostics when installed. Port checks fall back to /proc/net/tcp* when ss is unavailable.

If the current user cannot access Docker, commands that change Docker resources can ask for sudo. Status, models, and storage remain non-interactive.

Do not run ./llm-stack itself with sudo. Setup and uninstall refuse it because changing users can point configuration or data at the wrong home directory.

Quick Start

git clone https://github.com/MSC2016/local-llm-stack
cd local-llm-stack

./llm-stack setup
./llm-stack start

When startup succeeds, open:

http://localhost:3000

The llama.cpp OpenAI-compatible host API is available at:

http://localhost:8080/v1

Both host ports bind to 127.0.0.1 only. Open WebUI reaches llama.cpp through a private Docker network, so the backend does not need to be exposed to the LAN.

Commands

./llm-stack setup
./llm-stack start
./llm-stack status
./llm-stack models
./llm-stack storage
./llm-stack logs
./llm-stack logs webui
./llm-stack stop
./llm-stack update
./llm-stack uninstall

Setup

./llm-stack setup

Setup validates required commands, ports, the data path, and models.ini before writing configuration. Runtime data defaults to:

$HOME/.local/share/local-llm-stack

Options:

./llm-stack setup --defaults
./llm-stack setup --reconfigure
./llm-stack setup --pull-images
./llm-stack setup --no-pull-images

LLM_ROOT is canonicalized before it is saved. /, the home directory, the repository, repository subdirectories, and parent paths that contain the home directory or repository are rejected as deletion targets. Normal data directories inside the home directory remain valid.

Reconfiguration backs up the previous file and preserves every recognized setting, including image digests, resource names, model limits, and the API key. If the runtime models.ini is invalid, --reconfigure backs it up and restores the repository preset.

Start

./llm-stack start

Start performs the following checks before launching anything:

• Docker is installed, running, and accessible.
• Vulkan render devices exist.
• configured ports are available.
• existing containers, volume, and network carry the local-llm-stack ownership label.
• existing containers match the pinned images, ports, mounts, volume, network, restart policy, and runtime settings.
• models.ini is readable and structurally valid.

Missing pinned images are downloaded after local preflight succeeds. The backend is started and health-checked before Open WebUI is created or started. Both containers run detached with unless-stopped restart policies. Startup waits by wall-clock time: up to 60 seconds for llama.cpp, then up to five minutes for Open WebUI. A progress message is printed every 30 seconds.

Backend or container-creation failure prints an actionable error and removes only the containers, volume, and network created by that start attempt. Existing persistent data is never rolled back. If Open WebUI is still initializing after five minutes, its container and persistent volume remain in place, and the command prints the exact logs and status commands to use.

Automatic downloading of Open WebUI's local RAG embedding model is disabled by default so normal chat startup does not depend on Hugging Face. Normal chat and llama.cpp are unaffected. Document upload, knowledge-base search, and other local RAG features require choosing and downloading an embedding model later in Admin Settings > Documents.

The stack never removes or replaces a Docker resource that has a configured name but lacks its ownership label. It reports the collision and asks you to rename the foreign resource or choose another resource name.

Status

./llm-stack status
./llm-stack status --verbose

Status is read-only and non-interactive. It shows:

• owned container state and configuration conflicts
• service and Docker-network health
• the Open WebUI connection default installed by this project
• dependencies, images, Vulkan devices, ports, and storage
• models currently loading, loaded, or running

Status exits with code 0 only when the complete managed stack is healthy and matches configuration. It exits 1 for a confirmed health, configuration, dependency, GPU, image, storage, or preset problem. It exits 2 when Docker inspection is unavailable, because responsive ports alone cannot prove that the managed containers are present.

Open WebUI is configured on first container creation with the internal llama.cpp URL and API key. Settings later changed by the user in Open WebUI's Admin UI are intentionally preserved and can override that installation default. Status verifies the container default and Docker connection; it does not rewrite Open WebUI's database.

The active-model table appears below the system checks. Idle configurations and sleeping models are omitted. For every backend entry marked loaded, the report requests its non-loading per-model properties: ready means the model child is responding, while initializing or not ready means it cannot yet do so. Active rows are bold magenta when output is connected to a terminal; set NO_COLOR=1 to disable color.

Models

./llm-stack models
./llm-stack models --active

models first validates the runtime preset, then shows one parent row per backing model and indents its launch configurations underneath. Disk size is measured from matching GGUF files in the local cache, with live backend metadata as a fallback. A ready status requires both llama.cpp's loaded state and a successful per-model /props response; the probe uses autoload=false and does not wake or load idle models. Invalid or missing presets and missing formatter dependencies produce an actionable nonzero exit.

The default report includes all configured models plus cache-only backing models that are not represented by a configuration. Configuration rows use available because they describe settings rather than a process consuming memory. --active uses a compact table without KIND or group spacing: an active configuration is flattened onto one row with its disk size, context, output, and quantization; a raw model selection uses the backing-model name. Status uses the same shared formatter. Sleeping models are reported as cached, idle by the complete report.

Configuration rows have no disk size because they reuse their parent model file.

Logs

./llm-stack logs
./llm-stack logs backend
./llm-stack logs webui

Logs follow the last 100 lines. Missing, stopped, or unowned containers produce an actionable error rather than a raw Docker failure.

Stop

./llm-stack stop

Stop removes the two owned containers. Open WebUI chat data and the private Docker network are kept for the next start.

Storage

./llm-stack storage

Shows local paths, disk usage, free space, and Open WebUI volume state. It does not create or delete anything.

Update

./llm-stack update

Downloads and verifies the configured immutable Open WebUI and llama.cpp digests. It never moves to a newer build automatically. To upgrade, intentionally replace the two digests in configuration, run update, then restart:

./llm-stack stop
./llm-stack start

Uninstall

./llm-stack uninstall

Uninstall is interactive and has no dry-run mode. It can remove owned containers, Open WebUI chat data, local preferences, LLM_ROOT, and the two configured image references. Unowned Docker resources are never modified. Image data shared through another tag or digest is reported and retained.

At completion it lists every retained local path and its size. The repository is always retained. A retained Docker volume is also reported, with the command needed to inspect its Docker-managed size.

Configuration

Defaults are committed in config/config.env.default. Setup writes all recognized values to the Git-ignored config/config.env, so later reconfiguration can preserve them exactly.

Important values:

LLM_ROOT="$HOME/.local/share/local-llm-stack"
BACKEND_PORT="8080"
WEBUI_PORT="3000"
MODELS_MAX="3"
SLEEP_IDLE_SECONDS="300"

MODELS_MAX is a count limit, not a memory limit. The default permits up to three model processes at once. llama.cpp may unload a model when another request would exceed that count, and models idle for SLEEP_IDLE_SECONDS are put to sleep automatically. It does not calculate whether a combination fits in RAM or VRAM; choosing models and context sizes that fit is the user's responsibility. An oversized combination can fail to load, in which case inspect ./llm-stack logs backend.

Changing MODELS_MAX or SLEEP_IDLE_SECONDS requires ./llm-stack stop followed by ./llm-stack start because these are backend startup settings.

Both Docker images use immutable @sha256: references. A fresh setup therefore runs the same artifacts that were tested for this repository revision. ./llm-stack update only ensures those exact artifacts are present; upgrading requires changing the configured digests deliberately.

Docker resource names can also be overridden:

OPENWEBUI_CONTAINER="local-llm-stack-webui"
VULKAN_CONTAINER="local-llm-stack-backend"
OPENWEBUI_VOLUME="local-llm-stack-openwebui"
STACK_NETWORK="local-llm-stack"

config/config.env is sourced by Bash. Do not put untrusted content in it.

Model configuration

The repository preset is copied during setup:

config/models.ini -> $LLM_ROOT/config/models.ini

The runtime file is the one used by llama.cpp:

nano "$LLM_ROOT/config/models.ini"

Models are downloaded lazily into $LLM_ROOT/cache. There is no separate models directory because the default workflow uses Hugging Face repositories from models.ini.

Runtime layout:

$LLM_ROOT/
├── cache/
└── config/
    └── models.ini

Older installations

Old unlabelled resources named open-webui or llamacpp-vulkan are never adopted or deleted automatically. Stop those containers before starting this stack, and retain the old open-webui volume until any required chat history has been migrated and verified.

Safety

• Host ports bind only to 127.0.0.1.
• Docker resources are ownership-labelled before lifecycle commands manage them.
• Destructive paths are canonicalized and validated again immediately before removal.
• The repository is never removed by uninstall.
• Open WebUI's first account may become its administrator.

Only ./llm-stack is a public command. Files under scripts/ are internal implementation details and depend on environment prepared by the wrapper.

Development Checks

bash -n llm-stack lib/*.sh scripts/*.sh
shellcheck -x -P SCRIPTDIR llm-stack lib/*.sh scripts/*.sh
shfmt -i 2 -d llm-stack lib/*.sh scripts/*.sh

CI also parses the embedded Python, validates the repository preset, rejects ambiguous leading-zero ports, and smoke-tests every public help route.