jev-sim

A wire-compatible re-implementation of TypeSafe's Jev (POST /v1/systemone) that reads typed decisions straight out of an LLM's next-token logits — plus a reproducible benchmark of that approach against Jev itself, on the same public items, through the same scorer (JevBench).

Question answered: how far does "an off-the-shelf LLM + constrained readout, zero training" get you on accuracy, calibration, latency and cost, compared with Jev?

system	JevBench hard (111)	JevBench original (72)	Nimble holdout (324)	p50 latency	$ / 1k decisions
Jev 1.13.0 (TypeSafe, published)	73.0%	98.6%	93.2%	0.65 s	$0.040
jev-sim · DeepSeek-Flash logprob readout	74.8%	97.2%	84.3%	1.0 s	$0.17
jev-sim · Qwen3.5-4B, 4-bit, RTX 4070 laptop	53.2%	76.4%	65.7%	0.14–0.30 s	≈ electricity
jev-sim · Qwen3.5-0.8B, bf16, RTX 4070 laptop	39.6%	52.8%	43.2%	0.06–0.10 s	≈ electricity
DeepSeek-Flash asked to verbalize probabilities (control)	71.2%	91.7%	80.6%	2.7 s	$0.20

Jev's numbers are JevBench's own per-item results on identical item ids and Bespoke's published Nimble holdout figure; everything else was run here. Full tables, calibration (ECE/Brier), per-family breakdown and methodology: bench/REPORT.en.md · 中文报告.

Short version. With a large enough model behind it, plain logit readout matches Jev's accuracy on the hardest public items and costs no training. It does not match Jev's calibration out of the box, it is 4× more expensive and 1.5× slower per decision than Jev's production API, and small local models (0.8B–4B) fall 20–30 points short. Jev's moat is the training, not the architecture.

How it works

No text is ever generated. For each question the model sees the state and lettered options, and the backend reads the next-token distribution at the answer position, restricted to the option letters:

state + question + "A. billing  B. technical  C. sales" ──► one forward pass
                                                             ──► softmax over logits[A], logits[B], logits[C]

• noul → P(yes); choice → distribution + argmax; score → distribution + probability-weighted level.
• confidence uses TypeSafe's documented statistic (n·p_max − 1)/(n − 1) (uniform → 0, one-hot → 1).
• Local backend packs all questions of a request into one sequence and answers them in a single forward pass (state read once): 8 questions cost 1.5× one question on a 4B model — but answers drift vs. separate evaluation (74% argmax agreement at k = 8), see the report. API backends do one call per question.
• Optional post-hoc temperature scaling fitted on a labelled split (jev-sim calibrate).

Quick start

uv sync --extra dev                # core: FastAPI server + OpenAI-compatible backend
uv sync --extra dev --extra local  # + torch/transformers for local weights

# Any OpenAI-compatible server that returns top_logprobs: DeepSeek, OpenAI, vLLM, Ollama, ...
export DEEPSEEK_API_KEY=...
uv run jev-sim smoke --backend openai --provider deepseek
uv run jev-sim serve --backend openai --provider deepseek --port 8080

# Local weights on a laptop GPU (Qwen3.5-4B in 4-bit fits in 8 GB)
uv run jev-sim serve --backend hf --model Qwen/Qwen3.5-4B --load-in-4bit --port 8080

Then talk to it exactly like Jev. The official typesafe-sdk works unmodified (this is a test in CI):

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient(api_key="unused", base_url="http://127.0.0.1:8080") as client:
    r = client.system_one(
        state="I was charged twice, fix it ASAP",
        questions={"team": Choice(instructions="Which team?", criteria={"billing": "money", "technical": "bugs"})},
    )
    print(r.answers["team"].choice, r.answers["team"].confidence)

or plain HTTP:

curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "jev-latest",
  "state": "Help! My payouts have been failing for 3 days.",
  "questions": {
    "is_urgent": {"type": "noul",   "instructions": "Does this convey urgency?"},
    "team":      {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": "Pricing"}},
    "mood":      {"type": "score",  "instructions": "How frustrated is the customer?",
                  "criteria": ["Calm", "Frustrated", "Very angry"]}
  }}'

{"model": "jev-sim/openai:deepseek-flash",
 "answers": {"is_urgent": {"type": "noul", "noul": 0.99999},
             "team": {"type": "choice", "choice": "billing",
                      "probabilities": {"billing": 0.914, "technical": 0.085, "sales": 0.001}, "confidence": 0.87},
             "mood": {"type": "score", "score": 1.95, "legend": {"0": "Calm", "1": "Frustrated", "2": "Very angry"},
                      "probabilities": {"0": 0.001, "1": 0.046, "2": 0.954}, "confidence": 0.93}},
 "usage": {"input_tokens": 328, "output_tokens": 3}}

Wire compatibility with TypeSafe

	TypeSafe API	jev-sim
POST /v1/systemone request/response	✓	✓ same shape, incl. legend, usage
GET /v1/models	{"models": [{name, description, release_date}]}	✓
Validation failure	422	422, {"error": {"type", "message"}}
Missing / bad key	401	401 with --require-key (off by default for local use)
x-typesafe-request-id header	✓	✓
Context budget	64k / request, 32k / (state + longest question)	enforced → 422
Choice ≤ 255 options, Score 2–10 levels	✓	Score ✓; Choice ≤ 26 options (single-letter codes)
GET /v1/limits	—	extension: budgets + backend info
Answers within one request	evaluated independently	independent; packed layout lets later questions see earlier question text (not answers)

Backends

backend	reads the distribution from	notes
openai	logprobs.content[0].top_logprobs of any OpenAI-compatible chat API	profiles: deepseek (thinking off, temperature 1.0 — at 0 DeepSeek clamps logprobs to −9999), openai, vllm, ollama, custom
hf	exact logits from a local transformers model	packing on by default (self-checked per tokenizer, --no-pack to disable); --load-in-4bit; --dtype fp32 for bit-exact packed = separate

Benchmark

uv run jev-sim convert-nimble --src-dir data --out-dir data            # Nimble 324 holdout + calibration split
uv run jev-sim calibrate --backend openai --provider deepseek --tasks data/nimble_calib.jsonl
uv run python scripts/run_bench.py --system jev-sim-deepseek-raw --adapter typesafe \
    --endpoint http://127.0.0.1:8080 --price-in 0.15 --price-out 0.6
uv run python scripts/recalibrate.py --system jev-sim-deepseek-raw --out-system jev-sim-deepseek-cal \
    --key openai:deepseek:deepseek-flash                                 # temperature-scaled run, offline
uv run python scripts/extract_published.py                              # Jev & clones on the same public items
uv run python scripts/report.py bench/summaries/*.json > bench/TABLES.md
uv run python scripts/bench_packing.py --model Qwen/Qwen3.5-4B --load-in-4bit   # packed vs separate

Everything goes through JevBench's typesafe adapter and scorer (vendored as a submodule, MIT), so the numbers are directly comparable with its published leaderboard. Raw responses land in bench/raw/ (git-ignored); aggregates in bench/summaries/.

Acknowledgements

JevBench (harness + datasets), Bespoke Nimble (holdout set), open-alternative-jev (letter-readout + packed layout idea), and TypeSafe for publishing a clean enough API to imitate.

License

MIT — see LICENSE. JevBench datasets keep their own licenses (see third_party/jevbench).