Skip to content

About

LLM cost router: scores prompt difficulty, routes to the cheapest capable Claude model, caches repeats - 27.5% cheaper on a mixed benchmark with a live savings dashboard

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

78 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

TokenMeter ⚑

CI

Route every prompt to the cheapest LLM that can handle it. TokenMeter is an Anthropic-compatible proxy that scores each prompt's difficulty with a zero-cost heuristic, routes it to the right model tier β€” across Anthropic, OpenAI or Gemini β€” caches repeats (exact, semantic, fuzzy), streams responses, and shows you exactly how much money you didn't spend.

Most teams run every prompt through their best model. That's like taking a taxi to your mailbox. On a mixed benchmark workload TokenMeter cuts costs 27.5% with the hard prompts still going to the frontier model β€” and real workloads (support bots, extraction pipelines) skew easier, so savings run higher.

πŸ”΄ Live demo: tokenmeter.onrender.com β€” try the routing playground (free tier: first visit after idle takes ~30s to wake)

Benchmark (15 mixed prompts, scripts/benchmark.py)

Routed by TokenMeter Everything on Opus 4.8
Easy prompts (lookups, classification) β†’ Haiku 4.5 (7) β†’ Opus
Medium prompts (explanations, how-tos) β†’ Sonnet 5 (4) β†’ Opus
Hard prompts (design, debugging, proofs) β†’ Opus 4.8 (4) β†’ Opus
Cost $0.0497 $0.0685
Saved 27.5% β€”

Repeated and near-duplicate prompts cost $0 β€” an exact + fuzzy cache short-circuits the API entirely. The cheapest token is the one you never buy.

How routing works

prompt ──► difficulty score (0..1)  ──►  tier  ──►  model
             zero-cost heuristic:         < 0.35     claude-haiku-4-5   ($1/$5 per MTok)
             length, code, reasoning      < 0.65     claude-sonnet-5    ($3/$15)
             keywords, question shape     β‰₯ 0.65     claude-opus-4-8    ($5/$25)
                     β”‚
                     └─► cache checked first β†’ repeat = $0
                         exact match β†’ embedding similarity β†’ token overlap

The same three tiers map onto every provider β€” pick one per request with X-TokenMeter-Provider: anthropic|openai|gemini (or TOKENMETER_PROVIDER server-wide):

Tier Anthropic OpenAI Gemini
haiku claude-haiku-4-5 gpt-5-nano gemini-2.5-flash-lite
sonnet claude-sonnet-5 gpt-5-mini gemini-2.5-flash
opus claude-opus-4-8 gpt-5 gemini-2.5-pro

The classifier is deliberately not an LLM call β€” routing overhead must never eat the savings. Every request is logged to SQLite with cost vs the "all-frontier" baseline, and the dashboard renders the ledger live.

Two modes: free demo vs real API

Demo mode (default) Real mode
Trigger ANTHROPIC_API_KEY not set key set in backend/.env
Completions Simulated text Real Claude responses
Routing, cache, pricing math Real β€” identical code path Real
Cost β‚Ή0 Pay per request (fractions of a cent)

The deployed demo stays up 24/7 at zero cost; add the key only when showcasing real completions.

Run locally

cd backend
python -m venv .venv && .venv\Scripts\activate   # Windows
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Open http://localhost:8000 β€” the dashboard has a live playground: type a prompt, watch it get scored and routed.

# reproduce the benchmark table
python scripts/benchmark.py

Drop-in Anthropic-compatible proxy

TokenMeter also speaks the Anthropic Messages API shape β€” point any existing app at it and every request gets routed, cached, and metered with zero code changes beyond base_url:

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="unused-locally")
response = client.messages.create(
    model="claude-opus-4-8",        # your requested model...
    max_tokens=256,
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.model)               # ...but this ran on claude-haiku-4-5

The response carries a tokenmeter extension field with the provider, tier, difficulty score, cost, and baseline cost. Pin a tier per-request with X-TokenMeter-Tier: opus|sonnet|haiku, or a provider with X-TokenMeter-Provider.

Streaming works too β€” pass stream=True (or use client.messages.stream(...)) and TokenMeter emits the full Anthropic SSE event sequence (message_start β†’ content_block_deltas β†’ message_stop). Anthropic-routed requests stream token-by-token straight from the API; cached, demo and cross-provider answers are chunked. The final message_delta event carries usage and the tokenmeter extension, and streamed requests land in the ledger like any other.

Prompt caching is priced honestly: the proxy forwards your full message list (so cache_control blocks reach Anthropic intact) and the ledger bills cache reads at 10% of the input rate and cache writes at 125% β€” the same rates Anthropic charges β€” instead of pretending they're plain input tokens.

Semantic cache

Repeats don't have to be verbatim. Cache lookup escalates through three matchers, cheapest first:

  1. Exact β€” normalized string match, free
  2. Semantic β€” embedding cosine similarity (OpenAI text-embedding-3-small, ~$0.02/MTok β€” thousands of lookups per cent), so "What's the capital of France?" and "Tell me France's capital city" share one answer. Activates when OPENAI_API_KEY is set; silently skipped otherwise.
  3. Token overlap β€” Jaccard similarity fallback, free

All three are scoped per user β€” one account's completions are never served to another.

Quality evals (LLM-as-judge)

Cheap routing only counts if the answers hold up. scripts/eval_quality.py answers every eval prompt twice β€” routed tier vs all-frontier β€” and an Opus judge scores both 1-10 on correctness, completeness and clarity:

python scripts/eval_quality.py    # writes scripts/eval_results.json

Without an API key the judge scores are simulated (deterministic, clearly labeled) so the harness runs free in CI; with a key you get a real quality-parity table. The headline metric is quality parity β€” routed points as a share of frontier points β€” alongside the cost saved.

Budget guardrails

Set a daily spend cap and TokenMeter degrades gracefully instead of blowing through it:

TOKENMETER_DAILY_BUDGET_USD=1.50

When the projected request would cross the cap, the router downgrades the tier (opus β†’ sonnet β†’ haiku) rather than failing the request. Responses carry downgraded: true so you can alert on it.

API

Endpoint What it does
POST /v1/messages Anthropic-compatible proxy β€” Messages API shape in, routed completion out; stream=true for SSE; X-TokenMeter-Provider to route cross-provider
POST /v1/complete {prompt, tier?, provider?} β†’ routed completion with model, difficulty, cost, baseline_cost
GET /api/stats Totals, savings %, per-tier distribution, today's spend vs budget, recent requests β€” scoped to the signed-in user
GET /api/health {status, mock}
POST /api/reset Clear your own ledger
POST /api/auth/register Β· login Β· logout Β· GET /api/auth/me Email + password accounts with cookie sessions
POST /api/keys Β· GET /api/keys Β· DELETE /api/keys/{id} Mint, list, and revoke tm_... API keys
GET /api/account Β· POST/DELETE /api/account/keys/{provider} Β· PUT /api/account/budget BYOK keys per provider (verified, encrypted) and personal daily budget

Accounts & API keys

Sign up on /account.html, mint a tm_... key, and use it as the SDK's api_key β€” every request lands in your ledger and the dashboard shows your savings. Keys are hashed at rest (shown once at mint), passwords use PBKDF2-HMAC-SHA256, and sessions are server-side revocable cookies.

client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="tm_...")

The proxy also accepts Authorization: Bearer tm_.... Unauthenticated requests fall into a shared anonymous ledger so the public demo stays open; set TOKENMETER_REQUIRE_KEY=1 to reject them instead. For production, point DATABASE_URL at Postgres (Neon free tier works; pip install psycopg[binary]) β€” SQLite remains the zero-config default.

Bring your own key (BYOK)

Add your Anthropic, OpenAI and/or Gemini keys on the account page and your requests run real completions billed to your accounts β€” TokenMeter never pays for your tokens. Every key is health-checked on save (a free 1-token or list-models call, with clear error states), Fernet-encrypted at rest with a server-side secret (TOKENMETER_SECRET_KEY), and never returned by any API β€” only its last four characters. Users without a key get demo mode; the router picks whichever key matches the provider it routed to.

Budgets are per-user too: set a daily USD cap on the account page and the router downgrades tiers (opus β†’ sonnet β†’ haiku) instead of failing your requests; responses carry downgraded: true and the dashboard shows your spend against your own cap. The response cache is scoped per user, so one account's completions are never served to another.

Tests

61 unit + API tests run on every push via GitHub Actions (plus the routing benchmark and simulated quality eval):

pip install -r backend/requirements-dev.txt
cd backend && python -m pytest tests -q

Deploy (Render, free tier)

render.yaml blueprint included: render.com β†’ New β†’ Blueprint β†’ connect this repo β†’ deploy. Runs in demo mode (zero API cost) until you add ANTHROPIC_API_KEY in the dashboard.

Roadmap: demo β†’ product

TokenMeter is evolving from a single-user demo into a multi-user, bring-your-own-key cost router. Phases ship in order; each phase is usable on its own.

Phase 1 β€” Core router βœ… (shipped)

  • Zero-cost difficulty classifier β†’ Haiku / Sonnet / Opus tiers
  • Exact + fuzzy response cache (token-set similarity)
  • Daily budget guardrails with tier auto-downgrade
  • Anthropic-compatible /v1/messages proxy endpoint
  • SQLite ledger + live savings dashboard with playground
  • 21-test suite + GitHub Actions CI
  • Demo mode (zero API cost) + Render deploy blueprint

Phase 2 β€” Accounts & real multi-user βœ… (shipped)

The jump from "demo app" to "working product": every request belongs to a user.

  • Auth: email + password signup/login β€” PBKDF2-HMAC-SHA256 hashing (stdlib, OWASP iteration count), server-side revocable cookie sessions, FastAPI dependency-injected current-user
  • Database: SQLAlchemy 2.0 models (users, sessions, api_keys, requests); SQLite by default, Postgres via DATABASE_URL; in-place schema patch for Phase-1 databases
  • Per-user ledger & dashboard: stats, reset, and recent requests scoped to the signed-in user; anonymous demo scope keeps the public playground open
  • TokenMeter API keys: mint/list/revoke tm_... keys (hash-at-rest, shown once); the proxy authenticates via x-api-key or Authorization: Bearer tm_...; TOKENMETER_REQUIRE_KEY=1 rejects unauthenticated proxy requests
  • Account page (signup / login / key management) matching the existing light design system

Phase 3 β€” Bring your own key (BYOK) βœ… (shipped)

Real completions without TokenMeter ever paying for tokens β€” each user's requests bill their Anthropic account.

  • Store each user's Anthropic key encrypted at rest (Fernet, server-side secret); never returned by any API β€” only the last four characters for display
  • Router uses the requesting user's key; users without a key get demo mode; response cache scoped per user
  • Per-user daily budgets with tier-downgrade guardrails and spend-vs-cap display (global env-var budget remains as the operator fallback)
  • Key health check on save (1-token count_tokens call) with clear error states

Phase 4 β€” Router intelligence βœ… (shipped)

The router grows past "one provider, buffered responses, verbatim cache".

  • OpenAI / Gemini adapters behind the same tiers and price table β€” pick per request (X-TokenMeter-Provider) or server-wide (TOKENMETER_PROVIDER); called over plain HTTP, no extra SDKs
  • Streaming on /v1/messages β€” full Anthropic SSE event sequence, token-by-token from Anthropic, chunked for cached/demo/cross-provider answers, ledger intact
  • Semantic cache: embedding cosine similarity (text-embedding-3-small) between exact match and token-overlap fallback; per-user scoped; inactive without an embedding key
  • Prompt-caching awareness: full message list forwarded (cache_control reaches Anthropic), cache reads/writes billed at real rates (10% / 125% of input)
  • LLM-judge quality evals (scripts/eval_quality.py): routed vs all-frontier scored by an Opus judge, quality-parity table in JSON; simulated (labeled) in demo mode so CI runs free
  • BYOK extended to all three providers (encrypted slots, health checks, account-page cards); 61-test suite

Phase 5 β€” Production polish

  • Rate limiting per key, structured request logging
  • Usage export (CSV) + weekly spend summary email
  • Public landing page with live aggregate savings counter
  • Docs site: 5-minute integration guide per SDK

Why this project

Cost optimization is one of the most-screened skills in 2026 AI engineering interviews β€” everyone can call a model, few can run one economically. TokenMeter is the whole discipline in miniature: difficulty-aware routing, caching, per-request cost accounting, and a savings baseline you can defend with numbers.

About

LLM cost router: scores prompt difficulty, routes to the cheapest capable Claude model, caches repeats - 27.5% cheaper on a mixed benchmark with a live savings dashboard

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages