Route every prompt to the cheapest LLM that can handle it. TokenMeter is an Anthropic-compatible proxy that scores each prompt's difficulty with a zero-cost heuristic, routes it to the right model tier β across Anthropic, OpenAI or Gemini β caches repeats (exact, semantic, fuzzy), streams responses, and shows you exactly how much money you didn't spend.
Most teams run every prompt through their best model. That's like taking a taxi to your mailbox. On a mixed benchmark workload TokenMeter cuts costs 27.5% with the hard prompts still going to the frontier model β and real workloads (support bots, extraction pipelines) skew easier, so savings run higher.
π΄ Live demo: tokenmeter.onrender.com β try the routing playground (free tier: first visit after idle takes ~30s to wake)
| Routed by TokenMeter | Everything on Opus 4.8 | |
|---|---|---|
| Easy prompts (lookups, classification) | β Haiku 4.5 (7) | β Opus |
| Medium prompts (explanations, how-tos) | β Sonnet 5 (4) | β Opus |
| Hard prompts (design, debugging, proofs) | β Opus 4.8 (4) | β Opus |
| Cost | $0.0497 | $0.0685 |
| Saved | 27.5% | β |
Repeated and near-duplicate prompts cost $0 β an exact + fuzzy cache short-circuits the API entirely. The cheapest token is the one you never buy.
prompt βββΊ difficulty score (0..1) βββΊ tier βββΊ model
zero-cost heuristic: < 0.35 claude-haiku-4-5 ($1/$5 per MTok)
length, code, reasoning < 0.65 claude-sonnet-5 ($3/$15)
keywords, question shape β₯ 0.65 claude-opus-4-8 ($5/$25)
β
βββΊ cache checked first β repeat = $0
exact match β embedding similarity β token overlap
The same three tiers map onto every provider β pick one per request with X-TokenMeter-Provider: anthropic|openai|gemini (or TOKENMETER_PROVIDER server-wide):
| Tier | Anthropic | OpenAI | Gemini |
|---|---|---|---|
| haiku | claude-haiku-4-5 | gpt-5-nano | gemini-2.5-flash-lite |
| sonnet | claude-sonnet-5 | gpt-5-mini | gemini-2.5-flash |
| opus | claude-opus-4-8 | gpt-5 | gemini-2.5-pro |
The classifier is deliberately not an LLM call β routing overhead must never eat the savings. Every request is logged to SQLite with cost vs the "all-frontier" baseline, and the dashboard renders the ledger live.
| Demo mode (default) | Real mode | |
|---|---|---|
| Trigger | ANTHROPIC_API_KEY not set |
key set in backend/.env |
| Completions | Simulated text | Real Claude responses |
| Routing, cache, pricing math | Real β identical code path | Real |
| Cost | βΉ0 | Pay per request (fractions of a cent) |
The deployed demo stays up 24/7 at zero cost; add the key only when showcasing real completions.
cd backend
python -m venv .venv && .venv\Scripts\activate # Windows
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000Open http://localhost:8000 β the dashboard has a live playground: type a prompt, watch it get scored and routed.
# reproduce the benchmark table
python scripts/benchmark.pyTokenMeter also speaks the Anthropic Messages API shape β point any existing app at it and every request gets routed, cached, and metered with zero code changes beyond base_url:
import anthropic
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="unused-locally")
response = client.messages.create(
model="claude-opus-4-8", # your requested model...
max_tokens=256,
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.model) # ...but this ran on claude-haiku-4-5The response carries a tokenmeter extension field with the provider, tier, difficulty score, cost, and baseline cost. Pin a tier per-request with X-TokenMeter-Tier: opus|sonnet|haiku, or a provider with X-TokenMeter-Provider.
Streaming works too β pass stream=True (or use client.messages.stream(...)) and TokenMeter emits the full Anthropic SSE event sequence (message_start β content_block_deltas β message_stop). Anthropic-routed requests stream token-by-token straight from the API; cached, demo and cross-provider answers are chunked. The final message_delta event carries usage and the tokenmeter extension, and streamed requests land in the ledger like any other.
Prompt caching is priced honestly: the proxy forwards your full message list (so cache_control blocks reach Anthropic intact) and the ledger bills cache reads at 10% of the input rate and cache writes at 125% β the same rates Anthropic charges β instead of pretending they're plain input tokens.
Repeats don't have to be verbatim. Cache lookup escalates through three matchers, cheapest first:
- Exact β normalized string match, free
- Semantic β embedding cosine similarity (OpenAI
text-embedding-3-small, ~$0.02/MTok β thousands of lookups per cent), so "What's the capital of France?" and "Tell me France's capital city" share one answer. Activates whenOPENAI_API_KEYis set; silently skipped otherwise. - Token overlap β Jaccard similarity fallback, free
All three are scoped per user β one account's completions are never served to another.
Cheap routing only counts if the answers hold up. scripts/eval_quality.py answers every eval prompt twice β routed tier vs all-frontier β and an Opus judge scores both 1-10 on correctness, completeness and clarity:
python scripts/eval_quality.py # writes scripts/eval_results.jsonWithout an API key the judge scores are simulated (deterministic, clearly labeled) so the harness runs free in CI; with a key you get a real quality-parity table. The headline metric is quality parity β routed points as a share of frontier points β alongside the cost saved.
Set a daily spend cap and TokenMeter degrades gracefully instead of blowing through it:
TOKENMETER_DAILY_BUDGET_USD=1.50When the projected request would cross the cap, the router downgrades the tier (opus β sonnet β haiku) rather than failing the request. Responses carry downgraded: true so you can alert on it.
| Endpoint | What it does |
|---|---|
POST /v1/messages |
Anthropic-compatible proxy β Messages API shape in, routed completion out; stream=true for SSE; X-TokenMeter-Provider to route cross-provider |
POST /v1/complete |
{prompt, tier?, provider?} β routed completion with model, difficulty, cost, baseline_cost |
GET /api/stats |
Totals, savings %, per-tier distribution, today's spend vs budget, recent requests β scoped to the signed-in user |
GET /api/health |
{status, mock} |
POST /api/reset |
Clear your own ledger |
POST /api/auth/register Β· login Β· logout Β· GET /api/auth/me |
Email + password accounts with cookie sessions |
POST /api/keys Β· GET /api/keys Β· DELETE /api/keys/{id} |
Mint, list, and revoke tm_... API keys |
GET /api/account Β· POST/DELETE /api/account/keys/{provider} Β· PUT /api/account/budget |
BYOK keys per provider (verified, encrypted) and personal daily budget |
Sign up on /account.html, mint a tm_... key, and use it as the SDK's api_key β every request lands in your ledger and the dashboard shows your savings. Keys are hashed at rest (shown once at mint), passwords use PBKDF2-HMAC-SHA256, and sessions are server-side revocable cookies.
client = anthropic.Anthropic(base_url="http://localhost:8000", api_key="tm_...")The proxy also accepts Authorization: Bearer tm_.... Unauthenticated requests fall into a shared anonymous ledger so the public demo stays open; set TOKENMETER_REQUIRE_KEY=1 to reject them instead. For production, point DATABASE_URL at Postgres (Neon free tier works; pip install psycopg[binary]) β SQLite remains the zero-config default.
Add your Anthropic, OpenAI and/or Gemini keys on the account page and your requests run real completions billed to your accounts β TokenMeter never pays for your tokens. Every key is health-checked on save (a free 1-token or list-models call, with clear error states), Fernet-encrypted at rest with a server-side secret (TOKENMETER_SECRET_KEY), and never returned by any API β only its last four characters. Users without a key get demo mode; the router picks whichever key matches the provider it routed to.
Budgets are per-user too: set a daily USD cap on the account page and the router downgrades tiers (opus β sonnet β haiku) instead of failing your requests; responses carry downgraded: true and the dashboard shows your spend against your own cap. The response cache is scoped per user, so one account's completions are never served to another.
61 unit + API tests run on every push via GitHub Actions (plus the routing benchmark and simulated quality eval):
pip install -r backend/requirements-dev.txt
cd backend && python -m pytest tests -qrender.yaml blueprint included: render.com β New β Blueprint β connect this repo β deploy. Runs in demo mode (zero API cost) until you add ANTHROPIC_API_KEY in the dashboard.
TokenMeter is evolving from a single-user demo into a multi-user, bring-your-own-key cost router. Phases ship in order; each phase is usable on its own.
- Zero-cost difficulty classifier β Haiku / Sonnet / Opus tiers
- Exact + fuzzy response cache (token-set similarity)
- Daily budget guardrails with tier auto-downgrade
- Anthropic-compatible
/v1/messagesproxy endpoint - SQLite ledger + live savings dashboard with playground
- 21-test suite + GitHub Actions CI
- Demo mode (zero API cost) + Render deploy blueprint
The jump from "demo app" to "working product": every request belongs to a user.
- Auth: email + password signup/login β PBKDF2-HMAC-SHA256 hashing (stdlib, OWASP iteration count), server-side revocable cookie sessions, FastAPI dependency-injected current-user
- Database: SQLAlchemy 2.0 models (users, sessions, api_keys, requests); SQLite by default, Postgres via
DATABASE_URL; in-place schema patch for Phase-1 databases - Per-user ledger & dashboard: stats, reset, and recent requests scoped to the signed-in user; anonymous demo scope keeps the public playground open
- TokenMeter API keys: mint/list/revoke
tm_...keys (hash-at-rest, shown once); the proxy authenticates viax-api-keyorAuthorization: Bearer tm_...;TOKENMETER_REQUIRE_KEY=1rejects unauthenticated proxy requests - Account page (signup / login / key management) matching the existing light design system
Real completions without TokenMeter ever paying for tokens β each user's requests bill their Anthropic account.
- Store each user's Anthropic key encrypted at rest (Fernet, server-side secret); never returned by any API β only the last four characters for display
- Router uses the requesting user's key; users without a key get demo mode; response cache scoped per user
- Per-user daily budgets with tier-downgrade guardrails and spend-vs-cap display (global env-var budget remains as the operator fallback)
- Key health check on save (1-token count_tokens call) with clear error states
The router grows past "one provider, buffered responses, verbatim cache".
- OpenAI / Gemini adapters behind the same tiers and price table β pick per request (
X-TokenMeter-Provider) or server-wide (TOKENMETER_PROVIDER); called over plain HTTP, no extra SDKs - Streaming on
/v1/messagesβ full Anthropic SSE event sequence, token-by-token from Anthropic, chunked for cached/demo/cross-provider answers, ledger intact - Semantic cache: embedding cosine similarity (
text-embedding-3-small) between exact match and token-overlap fallback; per-user scoped; inactive without an embedding key - Prompt-caching awareness: full message list forwarded (
cache_controlreaches Anthropic), cache reads/writes billed at real rates (10% / 125% of input) - LLM-judge quality evals (
scripts/eval_quality.py): routed vs all-frontier scored by an Opus judge, quality-parity table in JSON; simulated (labeled) in demo mode so CI runs free - BYOK extended to all three providers (encrypted slots, health checks, account-page cards); 61-test suite
- Rate limiting per key, structured request logging
- Usage export (CSV) + weekly spend summary email
- Public landing page with live aggregate savings counter
- Docs site: 5-minute integration guide per SDK
Cost optimization is one of the most-screened skills in 2026 AI engineering interviews β everyone can call a model, few can run one economically. TokenMeter is the whole discipline in miniature: difficulty-aware routing, caching, per-request cost accounting, and a savings baseline you can defend with numbers.