A client-side codec that turns a JSON Schema into a baked skeleton plus an ordered slot list, so the language model only emits the genuinely variable values instead of the entire JSON envelope.
When you ask an LLM to produce structured output, you pay per output token for every brace, quote, comma, key name, and known constant. For envelopes with a high deterministic share (versioned wrappers, tool call protocols, classification heads), most of those tokens carry no information that the model could possibly choose. They are dictated by the schema, but the model still has to write them. JSON mode and grammar-constrained decoding remove the chance of malformed output, they do not remove the cost of emitting it.
schema-skip-decoder takes a JSON Schema and partitions it into two pieces:
- A skeleton: a partial JSON value that contains everything the schema already pins down (
const, single-elementenum,null, fixedrequiredkeys). - An ordered slot list: every spot in the document where the schema permits more than one value.
You hand the skeleton to your code and the slot descriptors to the model. The model emits one value per slot in order (one JSON literal per line). The decoder zips skeleton and fills back into the full structured instance, validates it, and you keep going. The over-the-wire payload from the model is the slot stream, which is strictly a subset of the full document.
It is one of the few output-side optimizations that does not require provider cooperation. The schema is yours, the slot stream is yours, the merge is yours. Any chat completion endpoint can be wrapped. The savings scale with the deterministic share of the schema, which for typical agent and API envelopes is high.
The compiler is conservative: anything it cannot prove is constant becomes a slot. That keeps round-trip equality intact. The benchmark in this repo measures real tokenizer counts on real sample JSON, not a simulation.
The schema flows into the compiler, which splits it into a deterministic skeleton (cached locally) and an ordered slot list. The slot list drives the prompt sent to the LLM. The model returns one JSON literal per slot, one per line. The decoder merges fills with the skeleton to produce a full instance, which is then validated by jsonschema.
flowchart LR
Schema --> Compiler
Compiler --> Skeleton
Compiler --> Slots
Slots --> Prompt
Prompt --> Model
Model --> Stream
Skeleton --> Decoder
Stream --> Decoder
Decoder --> Instance
The chart shows total tokens emitted by the model across every sample for each of six realistic schemas, counted with the cl100k_base BPE tokenizer used by recent OpenAI models. The "naive" bar is the JSON the model would have to produce in the standard structured-output flow. The "skip" bar is the slot stream the model produces when the schema-skip codec owns the skeleton. Same valid instances in both cases, identical after the decoder reassembles them.
| schema | slots | naive tokens | skip tokens | saved | saved % |
|---|---|---|---|---|---|
| tool_call | 2 | 201 | 111 | 90 | 44.8% |
| classification | 2 | 144 | 42 | 102 | 70.8% |
| log_entry | 5 | 323 | 208 | 115 | 35.6% |
| api_response | 4 | 312 | 194 | 118 | 37.8% |
| bug_report | 6 | 297 | 212 | 85 | 28.6% |
| product_listing | 6 | 346 | 262 | 84 | 24.3% |
| overall | - | 1623 | 1029 | 594 | 36.6% |
The metric is the number of output tokens the model is asked to produce. Baseline is the full JSON document encoded with json.dumps(separators=(",", ":")), since that is the most token-efficient form a provider would generate. The "after" number is the same content rendered as one JSON literal per slot, separated by newlines. Both numbers go through tiktoken.get_encoding("cl100k_base").encode. The savings come from not asking the model to re-emit field names, version strings, fixed constants, and the bracketing punctuation that the schema already determines.
The schemas closest to a pure data carrier (classification, tool_call) save the most. Schemas dominated by free text (product_listing) save the least, because the variable content already accounts for most tokens. Across this six-schema suite, average savings is 36.6%.
git clone https://github.com/Joshuaakaspace/schema-skip-decoder
cd schema-skip-decoder
python -m venv .venv
.venv\Scripts\activate # on Windows
pip install -r requirements.txt
python examples/demo.py
pytest -qexamples/demo.py re-runs the full benchmark end-to-end, validates that every sample round-trips through the codec, prints the per-schema table, and rewrites docs/before_after.png.
Compiled skeleton for the tool call envelope:
{
"protocol": "openagent.tool_call",
"version": "1",
"kind": "function",
"tool": "<<slot 0: tool>>",
"args": "<<slot 1: args>>"
}Sample slot stream the model would emit instead of the full JSON:
"create_calendar_event"
{"title":"1:1 with Mara","start":"2026-05-12T10:00:00Z","duration_minutes":30}
After the decoder merges skeleton and stream:
{
"protocol": "openagent.tool_call",
"version": "1",
"kind": "function",
"tool": "create_calendar_event",
"args": {"title": "1:1 with Mara", "start": "2026-05-12T10:00:00Z", "duration_minutes": 30}
}schema-skip-decoder/
README.md
BLOG.md
requirements.txt
src/schema_skip_decoder/
compiler.py
decoder.py
tokens.py
examples/
demo.py
schemas/
samples/
tests/
docs/before_after.png
