Skip to content

About

quickthink is a local-first CLI and Python library that wraps Ollama-backed LLM calls with a compressed plan-then-answer scaffold and latency-aware routing. It adds a short validated planning step for multi-step prompts and routes simple ones straight through. Local inference control for small models.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

67 Commits

Folders and files

quickthink

quickthink is a local-first CLI and Python library that wraps Ollama-backed LLM calls with a compressed plan-then-answer scaffold and latency-aware routing.

quickthink is developed by Hermes Labs.

Hermes Labs is an agentic infrastructure company building the reliability layer for autonomous systems.

CI License: Apache-2.0 Python PyPI

It adds a short, validated planning step before the answer for prompts that look multi-step, and routes simple prompts straight through to the model.

It currently ships as a lightweight scaffolding layer for local LLMs with three modes:

  • lite (default): one-pass inline plan prefix + answer in a single generation
  • two_pass: separate plan call then answer call
  • direct: no planning pass, raw prompt to model

The plan can be logged as metadata while hidden from normal UI output.

quickthink shapes the inference call; sibling tools cover other layers — for example, LintLang statically lints agent-config files, which is complementary to (not a substitute for) quickthink's runtime planning scaffold.

Where it fits

quickthink is useful when you need:

  • local LLM routing for local-first inference pipelines
  • small model optimization for constrained hardware and low-latency workflows
  • latency-aware inference via routing, bypass, and planning-budget controls
  • structured output reliability through strict planning grammar and eval gates
  • Ollama middleware for practical local deployment
  • agent runtime compatibility for CLI and automation-driven execution contexts

What this is / what this is not

What this is:

  • A local middleware layer for Ollama-backed LLM calls.
  • A small CLI for planned-answer generation, routing diagnostics, and local benchmarking.
  • A canonical eval harness for reproducible project-level quality checks.

What this is not:

  • Not a hosted API service.
  • Not a model training framework.
  • Not a replacement for full agent orchestration platforms.

Why

Small/local models are fast but often underperform on multi-step tasks. quickthink adds a strict planning pass (6–16 keyword tokens by default), intended to help with multi-step response quality without full verbose reasoning traces; whether it helps depends on the model and task.

Features

  • Ollama-first integration
  • Model profiles: qwen2.5:1.5b, mistral:7b, gemma3:27b
  • Three execution modes: lite (default), two_pass, direct
  • Preset routing profiles: fast, balanced, strict
  • Lane policy: default or strict_safe (routes strict-format tasks to direct path)
  • Hidden plan by default, optional plan display/logging
  • Bypass mode for short prompts (latency control)
  • Adaptive routing (skip, 12-token, max-token planning lanes)
  • Strict plan grammar: g:<...>;c:<...>;s:<...>;r:<...>
  • Local eval UI server (quickthink ui) at http://127.0.0.1:7860
  • Canonical eval harness: run → judge → validate → report

5-minute quickstart

Prerequisite: install and start Ollama locally.

# 1) Install the released package
python -m pip install "quickthink==0.2.2"

# 2) Pull one supported model
ollama pull qwen2.5:1.5b

# 3) Run your first command
quickthink ask "Give me a 3-step plan to learn SQL basics" --model qwen2.5:1.5b

If this command works, your local setup is ready.

GitHub Copilot CLI plugin

Copilot CLI users can install the repository as a plugin to make the existing quickthink skill available in any project:

copilot plugin marketplace add roli-lpci/quickthink
copilot plugin install quickthink@quickthink
copilot plugin list

In Copilot CLI, invoke /quickthink when you want to route a prompt through a local Ollama model. The plugin supplies usage instructions, not the runtime: install quickthink==0.2.2, start Ollama, and pull a supported model as shown above. The skill also offers quickthink ask ... --dry-run to inspect routing without calling a model. Plugin installation does not send prompts to a hosted inference service.

For development, clone the repository and install the editable development extras:

git clone https://github.com/roli-lpci/quickthink.git
cd quickthink
python -m pip install -e '.[dev]'

Documentation Map

Repository Layout

src/quickthink/         Runtime package (CLI, engine, prompts, routing, UI server)
scripts/eval_harness/   Canonical evaluation pipeline (run/judge/validate/report)
scripts/evals/          Legacy smoke/demo helpers (non-canonical)
scripts/demo/           One-command local demo runner
docs/evals/             Prompt sets, rubrics, harness specs, deployment gate notes
docs/release/           Release process and repository audit notes
tests/                  Unit tests for runtime and harness safety checks

See full architecture + publishability audit: docs/release/REPO_STRUCTURE_AND_PUBLISHABILITY_AUDIT_2026-02-20.md.

Canonical vs Legacy Scripts

Canonical project workflows:

  • scripts/eval_harness/*: maintained evaluation pipeline for run/judge/validate/report.
  • scripts/demo/quickstart.sh: canonical end-to-end local smoke/demo flow.

Legacy helpers (kept for compatibility and ad-hoc smoke checks):

  • scripts/evals/*: non-canonical helpers; do not treat as release gate source of truth.

When in doubt, use scripts/eval_harness/* and scripts/demo/quickstart.sh.

Usage

List supported profiles:

quickthink list-models

List preset routing profiles:

quickthink list-presets

Show officially supported compatibility models:

quickthink compatibility

Ask with compressed planning:

quickthink ask "How would a cow round up a border collie?" --model qwen2.5:1.5b --preset balanced

Show plan in terminal:

quickthink ask "How would a cow round up a border collie?" --model mistral:7b --show-plan

Switch to two-pass mode:

quickthink ask "How would a cow round up a border collie?" --mode two_pass --show-route --show-plan

Show routing diagnostics:

quickthink ask "Design a robust parser with tradeoffs and a JSON output schema" --show-route --show-plan

Skip planning entirely (direct mode):

quickthink ask "What is the capital of France?" --mode direct

Inspect routing and the exact prompt(s) without calling Ollama (--dry-run works offline):

quickthink ask "Design a retry strategy for a flaky payments API: compare exponential backoff versus a circuit breaker, list the tradeoffs, and return a JSON schema for the config" --mode two_pass --dry-run

Optional continuity hint (tiny, off by default):

quickthink ask "Continue the previous structure" --continuity-hint "ctx:prior_goal,format_json"

Strict-format-safe lane policy (routes strict format tasks to direct path first):

quickthink ask "json only: {\"ok\":true,\"why\":\"short\"}" --lane-policy strict_safe --show-route

Benchmark with strict-safe lane policy:

quickthink bench "Answer with YES or NO only: Is 2+2=4?" --lane-policy strict_safe --runs 3

Log plan + metrics as JSONL metadata:

quickthink ask "Design a tiny retry strategy" --log-file ./logs/quickthink.jsonl

Benchmark all three modes (lite, two_pass, direct):

quickthink bench "Design a robust parser for CSV with malformed quotes" --model qwen2.5:1.5b --runs 3

One-Command Quickstart Demo

Run full local demo setup and artifact generation:

bash scripts/demo/quickstart.sh

It does:

  • Python env + package install
  • ollama pull for supported models
  • Sample A/B/C eval run
  • Result validation
  • Markdown/HTML report generation
  • Compatibility snapshot update

For a one-minute terminal walkthrough command set, see docs/demo/QUICK_DEMO.md.

Optional environment flags:

  • QUICKTHINK_PRESET=fast|balanced|strict
  • QUICKTHINK_LIMIT=<n> (number of prompts from canonical set)
  • QUICKTHINK_RUNS=<n>
  • QUICKTHINK_RUN_JUDGE=1 (switch judge backend from rule to ollama)
  • QUICKTHINK_JUDGE_MODEL=<model>

Troubleshooting

For common setup/runtime failures and fixes, see docs/TROUBLESHOOTING.md.

Reports

Canonical report flow:

python3 scripts/eval_harness/run_suite.py \
  --prompt-set docs/evals/prompt_set.jsonl \
  --out docs/evals/results/run-<timestamp>.jsonl \
  --manifest-out docs/evals/results/manifest-<timestamp>.json \
  --runs 3

python3 scripts/eval_harness/judge_suite.py \
  --prompt-set docs/evals/prompt_set.jsonl \
  --results docs/evals/results/run-<timestamp>.jsonl \
  --out docs/evals/results/judged-<timestamp>.jsonl \
  --backend rule

python3 scripts/eval_harness/validate_judged_results.py \
  --path docs/evals/results/judged-<timestamp>.jsonl

python3 scripts/eval_harness/report_suite.py \
  --runs docs/evals/results/run-<timestamp>.jsonl \
  --judged docs/evals/results/judged-<timestamp>.jsonl \
  --out-json docs/evals/results/report-<timestamp>.json \
  --out-md docs/evals/results/report-<timestamp>.md \
  --out-html docs/evals/results/report-<timestamp>.html

Legacy helpers in scripts/evals/* remain available for smoke/demo use only.

Compatibility Matrix

  • Supported models are fixed to:
    • qwen2.5:1.5b
    • mistral:7b
    • gemma3:27b
  • Experimental evaluations may include additional models (for example llama3.2:latest) in deployment-gate or variant-gate workflows. Treat those as research lanes unless promoted into SUPPORTED_MODELS in runtime config.
  • Regenerate matrix + snapshot with:
python3 scripts/evals/compat_matrix_snapshot.py

Launch local web UI (for eval/scaffolding testing):

quickthink ui

Then open http://127.0.0.1:7860 if it does not open automatically.

UI lane control:

  • Lane policy dropdown supports default and strict_safe for single-prompt runs and 3-mode comparisons.

UI eval safety gates:

  • Preflight is required before any eval run (validate_prompt_set.py must return status=OK).
  • Run-file ingestion is blocked unless validate_results.py returns status=OK.
  • UI displays validator output and dataset SHA256 for reproducible/comparable runs.

Public Repo Scope

Included in this public repository:

  • runtime source code (src/quickthink)
  • reusable evaluation harness (scripts/eval_harness, docs/evals prompt/spec files)
  • tests and release process notes

Excluded from public tracking:

  • internal multi-agent comms logs
  • generated eval result dumps and ad-hoc local traces
  • private experiment workspaces under experiments-local/

Branching

  • Keep version tracks isolated in codex/* branches.
  • Merge to main only after benchmarks and notes are updated.
  • See docs/VERSION_NOTES.md for version-to-version differences.

Maintainer Commands

Install (editable + dev):

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Test:

PYTHONPATH=src .venv/bin/pytest -q

Lint (same commands as CI):

.venv/bin/ruff check src/ tests/ scripts/
python -m compileall src tests scripts

Release docs + checklist:

make release-check VERSION=x.y.z

Follow:

  • docs/release/RELEASE_CHECKLIST.md
  • docs/release/RELEASE_PROCESS.md
  • docs/release/SUPPLY_CHAIN_BASELINE_2026.md

Limitations / What it does not do

Grounded in how the code actually behaves:

  • It does not improve every answer. Whether the planning scaffold helps is model- and task-dependent; run the eval harness before claiming improvements.
  • It does not add an LLM of its own. The routing, plan grammar, and validation/repair logic are plain Python; the answer (and, in two_pass mode, the plan) still come from your Ollama model, so two_pass adds one extra model call versus direct.
  • It does not verify correctness. When a generated plan fails the grammar check, the engine substitutes a fixed fallback plan (g:solve;c:constraints;s:direct_reasoning;r:verify_output); this keeps the format valid but does not make the answer correct.
  • It only targets the three pinned models in SUPPORTED_MODELS (qwen2.5:1.5b, mistral:7b, gemma3:27b). Other models may run but are untuned.
  • It is local-first only: it talks to a local Ollama HTTP endpoint and is not a hosted API, a training framework, or an agent-orchestration platform.
  • Hidden planning is still logged for transparency; keep --log-file output auditable if you rely on the plan.

More from Hermes Labs

  • lintlang — Static analysis for AI agent tool descriptions and workflows.
  • little-canary — Prompt injection detection through a powerless sacrificial model.
  • fidelis — Semantic memory for long-running agents with local retrieval.
  • hermeneutic — Detect recurring AI drift from correction history.
  • zer0dex — Local agent recall without burdening the context window.
  • Hermes Labs plugins marketplace — Plugin system for Claude Code extensibility.

Browse the open-source catalog or contact roli@hermes-labs.ai.

License

Apache-2.0

About

quickthink is a local-first CLI and Python library that wraps Ollama-backed LLM calls with a compressed plan-then-answer scaffold and latency-aware routing. It adds a short validated planning step for multi-step prompts and routes simple ones straight through. Local inference control for small models.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages