tiergate

Data classification — not cost — picks the model.

tiergate is a privacy-tier enforcement layer for LLM routing. It sits between your application and your model gateway (LiteLLM) and guarantees, in code, that sensitive data only ever reaches endpoints you've approved for it. Not as a system prompt. Not as a convention. As a runtime invariant that raises before any network I/O happens.

Reference implementation distilled from a production system I operate privately. The pattern has been validated across two production workloads with different privacy requirements — one cloud-primary handling regulated customer data, one local-first handling personal data — running the same enforcement core with different tier policies.

The problem

Most teams bolting LLMs onto systems that hold sensitive data handle the privacy boundary in one of two ways: a line in the system prompt ("never include customer PII"), or a team convention ("we only use the local model for that"). Both fail the same way — silently. Prompts can be ignored or jailbroken; conventions break the day a new developer wires a new feature to the cheapest endpoint.

The fix is architectural: make the privacy boundary a property of the call path, so that violating it is a loud error instead of a quiet leak.

How it works

Three tiers, ordered by trust

Tier	Meaning	PII allowed	Example endpoints
TIER_0_LOCAL	On-device only. No network egress.	Yes	Ollama, llama.cpp
TIER_1_RESTRICTED	Cloud endpoints with a contractual data agreement (e.g., BAA, ZDR)	Yes	Approved vendor endpoints
TIER_2_GENERAL	General cloud. Cheapest/fastest available.	No	Anything else

Routes are workload-semantic, not model-semantic

Application code never names a model. It names a route that describes the sensitivity and shape of the workload:

# routes.yaml — the policy, versioned and reviewable
routes:
  local:          { tier: 0 }   # on-device, interactive
  embed:          { tier: 0 }   # local embeddings
  pii-bulk:       { tier: 1 }   # high-volume sensitive workloads
  pii-critical:   { tier: 1 }   # sensitive + high cost-of-error
  nonpii-turbo:   { tier: 2 }   # fast, cheap, nothing sensitive
  nonpii-long:    { tier: 2 }   # long-context, nothing sensitive

The route → model mapping lives in your LiteLLM config. Swap the model behind pii-critical from one vendor to another and zero application code changes — the route name is the policy anchor, the model is an implementation detail. The router is route-aware, not model-aware.

Enforcement happens before the network call

Each deployment declares the tiers it is allowed to touch:

router = TierRouter(allowed_tiers={Tier.TIER_1_RESTRICTED, Tier.TIER_2_GENERAL})

# fine — pii-bulk is tier 1, tier 1 is allowed
await router.call("pii-bulk", messages)

# raises TierViolationError BEFORE any network I/O
await router.call("local", messages)

Two details that matter:

1. TierViolationError is designed to be unswallowable. It deliberately does not subclass ValueError or RuntimeError, so broad except clauses in application code won't quietly eat it. A tier violation is a configuration bug, not a recoverable runtime condition.
2. Restricted routes require double acknowledgment. A route can't become tier 1 by YAML edit alone — it must also appear in a hardcoded frozenset in the policy module. One change in config, one change in code, two independent reviews before sensitive data flows anywhere new.

No cross-tier fallback. Ever.

If the endpoint behind a tier-1 route is down, tiergate retries with exponential backoff (transient errors are classified and retried; auth and validation errors propagate immediately). What it will never do is "gracefully degrade" to a different tier. The privacy boundary is not negotiable under load, and a fallback path that crosses tiers is just a leak with extra steps.

Always-on audit log

Every call — success or failure — writes a structured audit row: route, resolved tier, token counts, estimated cost, outcome. Audit writes are fire-and-forget so logging can never block or break the request path. The audit table is the answer to "prove to me nothing sensitive went to a general endpoint last quarter."

What tiergate deliberately does not do

No automatic PII detection. Callers classify their workloads explicitly by choosing a route. Content-sniffing classifiers fail open — they miss things, and the miss is silent. Explicit classification fails closed: choosing a route is choosing a policy, the choice is visible in code review, and a wrong choice is auditable. This is the load-bearing design decision in the whole project.

Not a gateway replacement. tiergate enforces policy in front of LiteLLM; LiteLLM handles providers, keys, and load balancing. Use both.

Quickstart

git clone https://github.com/ConsultRuss/tiergate && cd tiergate
docker compose up -d        # LiteLLM gateway + Postgres (audit log) + demo config
python demo.py              # runs an allowed call, then a blocked one

demo.py output shows the round trip: a nonpii-turbo call routed to a cheap model, then a deliberately misrouted sensitive call stopped cold with a TierViolationError — including what the audit log recorded for both.

Proving the block blocks

A policy that isn't tested is a comment. tiergate ships its enforcement test suite, and the README of every release reports its results:

Test class	What it proves
Tier gate happy path	Allowed tier + allowed route → call proceeds
Tier violations	Disallowed tier raises before network I/O, for every route × policy combination
Restricted-route acknowledgment	Tier-1 route missing from the code-level frozenset raises even when tier 1 is allowed
Mislabel resistance	Policy fields with wrong types (e.g., a quoted "1" where an integer tier is required) are rejected, not coerced
Fallback containment	Endpoint failure triggers same-tier retry, never a cross-tier downgrade
Audit completeness	Every outcome — success, violation, transient failure — produces an audit row

pytest tests/ -v

Design notes

• Why enforcement in code, not prompts: prompts can be ignored or jailbroken; a permission gate has to be architectural or it's theater. In the strictest deployment of this pattern, the tier-0 service doesn't even import the cloud-routing module — cloud egress is impossible by absence, not forbidden by instruction.
• Why declarative routes.yaml: tier taxonomy is a configuration concern, not logic. Versioned YAML makes policy reviewable in a diff, lets model swaps skip code review, and lets multiple deployments inherit one route set with different allowed-tier policies.
• Why per-deployment tier policies: the same core serves a deployment that excludes the local tier (no offline requirement; local fallback added operational burden without reducing risk) and one that requires it. Policy is data; enforcement is code.
• Cost posture: route defaults favor capability over cost where the cost of error dominates — a few hundred dollars a year of "over-routing" is cheap insurance against a compliance mistake. The depth of the route taxonomy (-bulk / -critical / -escalation) exists so that trade-off is a per-workload dial, not a global setting.

Roadmap

• OpenTelemetry export for audit events
• Policy linter: CI check that fails on route additions without matching tests
• TypeScript client

License

MIT