quickthink is a local-first CLI and Python library that wraps Ollama-backed LLM calls with a compressed plan-then-answer scaffold and latency-aware routing.
quickthink is developed by Hermes Labs.
Hermes Labs is an agentic infrastructure company building the reliability layer for autonomous systems.
It adds a short, validated planning step before the answer for prompts that look multi-step, and routes simple prompts straight through to the model.
It currently ships as a lightweight scaffolding layer for local LLMs with three modes:
lite(default): one-pass inline plan prefix + answer in a single generationtwo_pass: separate plan call then answer calldirect: no planning pass, raw prompt to model
The plan can be logged as metadata while hidden from normal UI output.
quickthink shapes the inference call; sibling tools cover other layers — for example, LintLang statically lints agent-config files, which is complementary to (not a substitute for) quickthink's runtime planning scaffold.
quickthink is useful when you need:
- local LLM routing for local-first inference pipelines
- small model optimization for constrained hardware and low-latency workflows
- latency-aware inference via routing, bypass, and planning-budget controls
- structured output reliability through strict planning grammar and eval gates
- Ollama middleware for practical local deployment
- agent runtime compatibility for CLI and automation-driven execution contexts
What this is:
- A local middleware layer for Ollama-backed LLM calls.
- A small CLI for planned-answer generation, routing diagnostics, and local benchmarking.
- A canonical eval harness for reproducible project-level quality checks.
What this is not:
- Not a hosted API service.
- Not a model training framework.
- Not a replacement for full agent orchestration platforms.
Small/local models are fast but often underperform on multi-step tasks.
quickthink adds a strict planning pass (6–16 keyword tokens by default), intended to help with multi-step response quality without full verbose reasoning traces; whether it helps depends on the model and task.
- Ollama-first integration
- Model profiles:
qwen2.5:1.5b,mistral:7b,gemma3:27b - Three execution modes:
lite(default),two_pass,direct - Preset routing profiles:
fast,balanced,strict - Lane policy:
defaultorstrict_safe(routes strict-format tasks to direct path) - Hidden plan by default, optional plan display/logging
- Bypass mode for short prompts (latency control)
- Adaptive routing (
skip,12-token,max-tokenplanning lanes) - Strict plan grammar:
g:<...>;c:<...>;s:<...>;r:<...> - Local eval UI server (
quickthink ui) athttp://127.0.0.1:7860 - Canonical eval harness: run → judge → validate → report
Prerequisite: install and start Ollama locally.
# 1) Install the released package
python -m pip install "quickthink==0.2.2"
# 2) Pull one supported model
ollama pull qwen2.5:1.5b
# 3) Run your first command
quickthink ask "Give me a 3-step plan to learn SQL basics" --model qwen2.5:1.5bIf this command works, your local setup is ready.
Copilot CLI users can install the repository as a plugin to make the existing
quickthink skill available in any project:
copilot plugin marketplace add roli-lpci/quickthink
copilot plugin install quickthink@quickthink
copilot plugin listIn Copilot CLI, invoke /quickthink when you want to route a prompt through a
local Ollama model. The plugin supplies usage instructions, not the runtime:
install quickthink==0.2.2, start Ollama, and pull a supported model as shown
above. The skill also offers quickthink ask ... --dry-run to inspect routing
without calling a model. Plugin installation does not send prompts to a hosted
inference service.
For development, clone the repository and install the editable development extras:
git clone https://github.com/roli-lpci/quickthink.git
cd quickthink
python -m pip install -e '.[dev]'- Docs index:
docs/README.md - First-time setup:
docs/GETTING_STARTED.md - Common failures and fixes:
docs/TROUBLESHOOTING.md - Known limitations:
docs/KNOWN_LIMITATIONS.md - Quick demo script:
docs/demo/QUICK_DEMO.md - OSS readiness scorecard:
docs/release/OSS_READINESS_SCORECARD_2026-02-25.md - OSS standards alignment (with external references):
docs/release/OSS_STANDARDS_ALIGNMENT_2026.md - Agent operating notes:
AGENTS.md
src/quickthink/ Runtime package (CLI, engine, prompts, routing, UI server)
scripts/eval_harness/ Canonical evaluation pipeline (run/judge/validate/report)
scripts/evals/ Legacy smoke/demo helpers (non-canonical)
scripts/demo/ One-command local demo runner
docs/evals/ Prompt sets, rubrics, harness specs, deployment gate notes
docs/release/ Release process and repository audit notes
tests/ Unit tests for runtime and harness safety checks
See full architecture + publishability audit:
docs/release/REPO_STRUCTURE_AND_PUBLISHABILITY_AUDIT_2026-02-20.md.
Canonical project workflows:
scripts/eval_harness/*: maintained evaluation pipeline for run/judge/validate/report.scripts/demo/quickstart.sh: canonical end-to-end local smoke/demo flow.
Legacy helpers (kept for compatibility and ad-hoc smoke checks):
scripts/evals/*: non-canonical helpers; do not treat as release gate source of truth.
When in doubt, use scripts/eval_harness/* and scripts/demo/quickstart.sh.
List supported profiles:
quickthink list-modelsList preset routing profiles:
quickthink list-presetsShow officially supported compatibility models:
quickthink compatibilityAsk with compressed planning:
quickthink ask "How would a cow round up a border collie?" --model qwen2.5:1.5b --preset balancedShow plan in terminal:
quickthink ask "How would a cow round up a border collie?" --model mistral:7b --show-planSwitch to two-pass mode:
quickthink ask "How would a cow round up a border collie?" --mode two_pass --show-route --show-planShow routing diagnostics:
quickthink ask "Design a robust parser with tradeoffs and a JSON output schema" --show-route --show-planSkip planning entirely (direct mode):
quickthink ask "What is the capital of France?" --mode directInspect routing and the exact prompt(s) without calling Ollama (--dry-run works offline):
quickthink ask "Design a retry strategy for a flaky payments API: compare exponential backoff versus a circuit breaker, list the tradeoffs, and return a JSON schema for the config" --mode two_pass --dry-runOptional continuity hint (tiny, off by default):
quickthink ask "Continue the previous structure" --continuity-hint "ctx:prior_goal,format_json"Strict-format-safe lane policy (routes strict format tasks to direct path first):
quickthink ask "json only: {\"ok\":true,\"why\":\"short\"}" --lane-policy strict_safe --show-routeBenchmark with strict-safe lane policy:
quickthink bench "Answer with YES or NO only: Is 2+2=4?" --lane-policy strict_safe --runs 3Log plan + metrics as JSONL metadata:
quickthink ask "Design a tiny retry strategy" --log-file ./logs/quickthink.jsonlBenchmark all three modes (lite, two_pass, direct):
quickthink bench "Design a robust parser for CSV with malformed quotes" --model qwen2.5:1.5b --runs 3Run full local demo setup and artifact generation:
bash scripts/demo/quickstart.shIt does:
- Python env + package install
ollama pullfor supported models- Sample A/B/C eval run
- Result validation
- Markdown/HTML report generation
- Compatibility snapshot update
For a one-minute terminal walkthrough command set, see docs/demo/QUICK_DEMO.md.
Optional environment flags:
QUICKTHINK_PRESET=fast|balanced|strictQUICKTHINK_LIMIT=<n>(number of prompts from canonical set)QUICKTHINK_RUNS=<n>QUICKTHINK_RUN_JUDGE=1(switch judge backend fromruletoollama)QUICKTHINK_JUDGE_MODEL=<model>
For common setup/runtime failures and fixes, see docs/TROUBLESHOOTING.md.
Canonical report flow:
python3 scripts/eval_harness/run_suite.py \
--prompt-set docs/evals/prompt_set.jsonl \
--out docs/evals/results/run-<timestamp>.jsonl \
--manifest-out docs/evals/results/manifest-<timestamp>.json \
--runs 3
python3 scripts/eval_harness/judge_suite.py \
--prompt-set docs/evals/prompt_set.jsonl \
--results docs/evals/results/run-<timestamp>.jsonl \
--out docs/evals/results/judged-<timestamp>.jsonl \
--backend rule
python3 scripts/eval_harness/validate_judged_results.py \
--path docs/evals/results/judged-<timestamp>.jsonl
python3 scripts/eval_harness/report_suite.py \
--runs docs/evals/results/run-<timestamp>.jsonl \
--judged docs/evals/results/judged-<timestamp>.jsonl \
--out-json docs/evals/results/report-<timestamp>.json \
--out-md docs/evals/results/report-<timestamp>.md \
--out-html docs/evals/results/report-<timestamp>.htmlLegacy helpers in scripts/evals/* remain available for smoke/demo use only.
- Supported models are fixed to:
qwen2.5:1.5bmistral:7bgemma3:27b
- Experimental evaluations may include additional models (for example
llama3.2:latest) in deployment-gate or variant-gate workflows. Treat those as research lanes unless promoted intoSUPPORTED_MODELSin runtime config. - Regenerate matrix + snapshot with:
python3 scripts/evals/compat_matrix_snapshot.pyLaunch local web UI (for eval/scaffolding testing):
quickthink uiThen open http://127.0.0.1:7860 if it does not open automatically.
UI lane control:
Lane policydropdown supportsdefaultandstrict_safefor single-prompt runs and 3-mode comparisons.
UI eval safety gates:
- Preflight is required before any eval run (
validate_prompt_set.pymust returnstatus=OK). - Run-file ingestion is blocked unless
validate_results.pyreturnsstatus=OK. - UI displays validator output and dataset SHA256 for reproducible/comparable runs.
Included in this public repository:
- runtime source code (
src/quickthink) - reusable evaluation harness (
scripts/eval_harness,docs/evalsprompt/spec files) - tests and release process notes
Excluded from public tracking:
- internal multi-agent comms logs
- generated eval result dumps and ad-hoc local traces
- private experiment workspaces under
experiments-local/
- Keep version tracks isolated in
codex/*branches. - Merge to
mainonly after benchmarks and notes are updated. - See
docs/VERSION_NOTES.mdfor version-to-version differences.
Install (editable + dev):
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'Test:
PYTHONPATH=src .venv/bin/pytest -qLint (same commands as CI):
.venv/bin/ruff check src/ tests/ scripts/
python -m compileall src tests scriptsRelease docs + checklist:
make release-check VERSION=x.y.zFollow:
docs/release/RELEASE_CHECKLIST.mddocs/release/RELEASE_PROCESS.mddocs/release/SUPPLY_CHAIN_BASELINE_2026.md
Grounded in how the code actually behaves:
- It does not improve every answer. Whether the planning scaffold helps is model- and task-dependent; run the eval harness before claiming improvements.
- It does not add an LLM of its own. The routing, plan grammar, and validation/repair logic are plain Python; the answer (and, in
two_passmode, the plan) still come from your Ollama model, sotwo_passadds one extra model call versusdirect. - It does not verify correctness. When a generated plan fails the grammar check, the engine substitutes a fixed fallback plan (
g:solve;c:constraints;s:direct_reasoning;r:verify_output); this keeps the format valid but does not make the answer correct. - It only targets the three pinned models in
SUPPORTED_MODELS(qwen2.5:1.5b,mistral:7b,gemma3:27b). Other models may run but are untuned. - It is local-first only: it talks to a local Ollama HTTP endpoint and is not a hosted API, a training framework, or an agent-orchestration platform.
- Hidden planning is still logged for transparency; keep
--log-fileoutput auditable if you rely on the plan.
- lintlang — Static analysis for AI agent tool descriptions and workflows.
- little-canary — Prompt injection detection through a powerless sacrificial model.
- fidelis — Semantic memory for long-running agents with local retrieval.
- hermeneutic — Detect recurring AI drift from correction history.
- zer0dex — Local agent recall without burdening the context window.
- Hermes Labs plugins marketplace — Plugin system for Claude Code extensibility.
Browse the open-source catalog or contact roli@hermes-labs.ai.
Apache-2.0