Small models that know how sure they are.
certo is a toolkit for building narrow, calibrated decision models — models that don't generate text, they read a state and a set of options and return calibrated probabilities (the confidence means what it says). Calibration comes from the training recipe — learn the full answer distribution, not just the winning label — so it's by construction, not patched on afterward. You train on whatever labels you have: a known-answer synthetic world (where calibration can be measured exactly), your own labeled data, or a mix.
Status: research preview. The code here reproduces the experiments and results below. The packaged
pip install certointerface (inference API, checkpoint save/load, tuning helpers) and a demo checkpoint are in progress — see Roadmap.
Project page: https://altslate-labs.github.io/certo/ · Technical report: https://altslate-labs.github.io/certo/report.html · Synthetic dataset: https://huggingface.co/datasets/rajpdus/certo-synthetic-decisions
Inspired by Jev / "System-1" decision models. Independent project — not affiliated with TypeSafe.
A decision model earns its keep when it says "route to billing, 0.86" and the 0.86 is trustworthy — so you can escalate on doubt or trade quality against cost. Train on hard labels (the usual way) and the model still picks well but turns overconfident.
certo trains against the full answer distribution (a proper scoring rule), and because the data comes from a world with a known posterior, every model is graded against the exact answer — not just accuracy, but posterior fidelity (how close the probabilities are to the truth), which you can never measure on real data. That known-answer world is our measurement environment — on real data you train the same way and grade with calibration + accuracy against outcomes.
Both — the recipe is provenance-aware. It trains on exact posteriors from a known-answer
world (soft targets → measurable KL/TV), real observed / gold labels (ordinary supervised loss),
or teacher estimates (optional distillation) — each with the right loss, and kept distinct so
they're never confused (the canonical schema + validator live in prepare/). Synthetic
known-answer worlds are how we can measure calibration exactly and one convenient way to generate
data — not the only source. The v2 corpus mixes rule-based generation with public decision
datasets (NLI, intent, emotion, reasoning) under strict licensing/leakage/split discipline — see
prepare/SPEC.md.
Honest note: the current default checkpoint was trained only on synthetic data — a demonstrator. Training on your own data and real-dataset mixes is the toolkit's intended use and the v2 focus.
Measured against the exact answer on held-out wording; lower KL is better.
| finding | result |
|---|---|
| training on the full distribution recovers the exact answer from English | KL 0.004 |
| … robust across seeds; single-label targets fail every time (0.6–1.1) | see figure |
| hardening targets to the winning label wrecks the probabilities at ~matched accuracy | KL 0.67 vs ~0 |
| the model learns interaction structure a per-feature rule provably can't | KL 0.04 vs 0.62 |
| runtime options — generalizes to options / wording / counts it never trained on | KL 0.13, acc 0.82 |
| multi-label routing — calibration is what enables cost-aware choice | near-oracle routing |
The full method, experiments, and honest limitations are in the technical report.
A ModernBERT-large decision model trained on soft targets — altslate/certo-decision-model.
Held-out (graded against the exact answer): KL 0.008 · acc 0.844 · ECE 0.004 on options it never saw; option-order invariance exact. It generalizes to unseen options/wording/counts — and it fails safely (abstains) on free-flowing real prose it wasn't trained on (see the report). Use it locally:
from huggingface_hub import snapshot_download
from certo import DecisionModel
m = DecisionModel.load(snapshot_download("altslate/certo-decision-model"))
r = m.decide(
state="We measured tempo as gale, texture as dawn, density as ember.",
options=[{"id": "Loam", "description": "typically texture dawn, tempo gale, density ember"},
{"id": "Dune", "description": "typically texture gale, tempo dawn, density dawn"},
{"id": "Moor", "description": "typically texture ember, tempo frost, density frost"}],
abstain_below=0.6)
r["probs"] # {'Loam': 0.99, 'Dune': 0.0, 'Moor': 0.0} — real output, calibrated
r["top"] # 'Loam'
r["abstain"] # False (it abstains when the evidence is ambiguous)More real runs — confident matches, calibrated uncertainty, many options, order-invariance, and the
honest real-language boundary — are in examples/.
A calibrated encoder is cheap and trustworthy, but on hard, reasoning-heavy decisions (multi-step
arithmetic, timezone math, nested exceptions) a single forward pass hits a wall — those decisions need
generation. So certo now includes Certo-R1
(altslate/certo-r1-qwen3-4b, Apache-2.0): a Qwen3-4B fine-tuned for concise, verified reasoning — it
reasons in one or two sentences and returns the chosen option. It runs behind the same API:
from certo import ReasoningDecisionModel
m = ReasoningDecisionModel.load("altslate/certo-r1-qwen3-4b", device="cuda")
r = m.decide(
state="A parcel weighs 2.4 kg. Shipping is $3 per kg, any part of a kg rounds up. Charged $9.",
options=[{"id": "yes", "description": "the amount charged is correct"},
{"id": "no", "description": "the amount charged is wrong"}],
instructions="Was the customer charged correctly?")
r["top"] # 'yes'
r["rationale"] # 'Round 2.4->3 kg; correct = 3x3 = $9; paid $9 -> correct. FINAL ANSWER: A'On a held-out set of ten short calc/compositional families, Certo-R1 is non-inferior to full verbose reasoning at ~15× fewer tokens and ~8–11× lower latency. On JevBench 1.4.2 (public 231) it scores easy 0.98 / standard 0.90 / hard 0.60 — far above an encoder on the hard tier, trading calibration for intelligence (it emits an answer, not a probability). The natural architecture is a cascade: the calibrated encoder decides the easy majority; low-confidence cases escalate to Certo-R1.
We also built the single-model decider the cascade points toward: one Qwen3-4B backbone with one scoring head, serving all three primitives — Choice, Score (ordinal), and Noul (yes/no) — with probabilities as the primary output. The head reads the backbone at the last prompt token (fast path) or after a short rationale (reasoning path), so both paths return a calibrated distribution and software assembles the typed response. This fixes the reasoner's one-hot output — the reasoning path is now distributional too.
What we found — honestly (held-out, temperatures fit on a separate split):
| capability | result |
|---|---|
| Choice — trained + unseen domain | acc 0.93–0.97 · ECE 0.02–0.04 |
| Noul | acc 0.88–0.92 · ECE 0.008–0.03 |
| Score — trained rubric | acc up to 1.00 · ECE 0.05 |
| Score — unseen rubric / different source | acc ~0.46–0.61 · no transfer |
| different-source overall (JevBench public 231) | acc 0.71 · ECE 0.22 |
Two results are worth stating plainly. Calibration transfers only when the calibration data matches deployment — fast-path ECE is ~0.03 on matched data but ~0.22 on a genuinely different source. And across three independent attempts to buy generalization — broader supervised data, an RLCR-style calibration-reward RL run, and a Score-focused update — the model learns the decision types it is trained on very well but does not self-generalize to unseen rubrics, sources, or labels. (A related trap we hit and fixed: training a task with one label held out teaches the model to suppress that label — include every label you want it to be able to emit.)
We tested this directly on a different source (NVIDIA Nemotron tool-calling + science decisions, converted to Choice/Noul). Two clean results: tool-selection Choice transfers almost perfectly zero-shot (0.99) — picking the right action from described options generalizes across sources — but a novel decision type it had never seen ("should I call a tool now, or ask / decline?") fails zero-shot (~0.57) and only becomes reliable (0.88, well-calibrated) once trained on examples of it. Adding that large new source didn't lift accuracy on other unseen rubrics either. Same lesson, four different ways.
So the honest positioning: train it on the decision types you will actually serve — there it is strong and its probabilities are trustworthy — rather than expecting zero-shot transfer to arbitrary new specifications. The exception is Choice over described options, which does generalize to unseen domains and sources. Checkpoints are currently internal; the recipe and full capability study are in progress.
Requires torch, transformers, numpy (a CUDA box for the language-model runs; the synthetic
world + verification run anywhere).
python datagen.py selftest # 7-check verification of the synthetic world
python train.py compare # 2 models x 3 target types, graded on posterior fidelity
python train.py compare --world ix # world with interaction structure
python realize.py demo # see states rendered as text
python train_text.py run --arm soft --device cuda # recover the answer from English
python train_route.py run --arm soft --device cuda # multi-label expert routing
python train_generic.py run --arm soft --device cuda # generic runtime-option modelGood fit: narrow decisions you can define or simulate (routing, triage, intent, verification, ordinal scoring); anywhere trustworthy confidence matters (abstention, escalation, cost-aware choice); local, on a small encoder — no large model, no API.
Not this: a general world-knowledge decision model (a small encoder is calibrated, not omniscient); open-ended generation (certo returns typed probabilities); tasks you can neither generate nor label.
-
pip install certo—DecisionSpec→generate→train, andDecisionModel.load().decide()/route()/calibrate() - checkpoint save/load + inference API (
infer.py) + a model on the HF Hub - Certo-R1 — concise generative reasoner for the hard tier, same API (
infer_reasoner.py); cascade over the calibrated encoder - Unified model (experimental) — one backbone + one head, three primitives, probabilities-first; capability study done (strong on trained decision types; does not self-generalize zero-shot)
- real-data validation (accuracy + calibration)
- broaden decision structures (extraction, temporal, verification) toward a general base
- baseline: apply certo's soft-target recipe to a GLiClass-family backbone and measure the calibration gain
The runtime-label classifier family — GLiClass, GLiNER, and open Jev rebuilds — nails the mechanism of scoring runtime options in one pass. certo's contribution is calibration by construction on top of that interface.
MIT — see LICENSE.