regbench

A small benchmark for register-level embedded systems reasoning, implemented with Inspect. Every question exists in three paired forms, so a run reports not only how often a model is right but how much of that score survives a change of wording or a change of numbers.

78 items across 26 families in two difficulty tiers.

The first 20 families are single-step calculations: I2C addressing, SPI configuration, UART framing and baud generation, ADC quantization, PWM and timer configuration, GPIO read-modify-write, register field extraction, real-time scheduling, and QNX Neutrino message-passing IPC.

The remaining 6 present C source and are harder: integer promotion in bitwise expressions, write-1-to-clear register semantics under read-modify-write, sign extension of sub-word two's-complement samples, Q15 fixed-point requantisation, struct padding under a stated ABI, and unsigned modular arithmetic across a timer wrap. They exist because the first tier saturates; see docs/RESULTS.md.

The idea

A single accuracy number hides two distinct failure modes:

• Surface-form dependence. The response tracked how the question was worded rather than what it asked. Rewording moves the score.
• Instance dependence. The response reproduced a worked example rather than applying the method. Changing the input values moves the score.

regbench separates them by construction. Each family contains:

Variant	What changes	Answer
base	nothing, this is the item as written	-
rename	identifiers, peripheral names, phrasing	unchanged
renumber	the input values	recomputed

A rename that costs accuracy is surface-form dependence: the question did not change. A renumber that costs accuracy means the method did not transfer to values that cannot have been memorised in that combination. Both are reported as signed deltas in percentage points against the base items.

Metrics

Metric	Meaning
accuracy, stderr	overall, across all 60 items
base, rename, renumber, all	accuracy per variant
easy, medium, hard	accuracy per difficulty tier
rename_delta	percentage points lost from base to rename
renumber_delta	percentage points lost from base to renumber
rename_consistency	fraction of families graded alike on both variants
renumber_consistency	same, for base against renumber
no_answer_rate	fraction of responses with no parsable ANSWER: line

The consistency metrics exist because the deltas can be fooled. A model that answers a different half of the item set correctly on each variant reports a delta near zero while agreeing on almost no individual family. Only the paired figure shows that.

no_answer_rate stays out of accuracy: ignoring the output format is a failure of instruction following, not of embedded systems reasoning.

Scoring

Items declare one of three answer types, each with its own equality rule:

• integer - compared by numeric value, so 0x90, 144 and 0b10010000 are the same answer. Explicit notations (0x, 0b, an h suffix) are always honoured. Digits with no notation are read in the base the question asked for, which each item declares, so A6 answering a hexadecimal item is correct and 166 answering it means 0x166.
• quantity - converted to a base SI unit and compared within a per-item relative tolerance, so 1.5 ms, 1500 us and 0.0015 s are the same answer. A bare number is read as being in the unit the question asked for.
• choice - compared after case folding, with per-item aliases for spellings such as SEND-blocked against SEND blocked.

Grading is deterministic and uses no grader model. Scores are therefore reproducible, and the whole benchmark runs with no API credentials.

Answer key verification

Every computational item stores the formula and inputs that produce its answer alongside the hand-written answer itself. tools/verify_answers.py recomputes the key and reports disagreements, and it runs in CI:

$ python tools/verify_answers.py
verified 54 of 60 items (6 have no check block)
answer key agrees with recomputation

This checks transcription, unit prefixes and arithmetic, not the physics. One author wrote both paths, so a conceptual error in a formula appears in both and passes; docs/REVIEW_CHECKLIST.md covers that case by review. The first run of this script caught a wrong hand-computed bit-field answer.

Running it

git clone https://github.com/jSubbz/regbench && cd regbench
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

No credentials needed, against the built-in mock model:

python tools/smoke_run.py

Against a real provider:

export OPENAI_API_KEY=...
inspect eval src/regbench/task.py --model openai/gpt-4o
inspect view

Subsets:

inspect eval src/regbench/task.py --model openai/gpt-4o -T domains=i2c,spi
inspect eval src/regbench/task.py --model openai/gpt-4o -T variants=base

Tests and lint:

python tools/verify_answers.py
pytest
ruff check . && ruff format --check .

docs/ADDING_ITEMS.md covers the run and test loop in more detail, and the procedure for adding a family of items.

Limitations

• Small. 26 families demonstrates the method. The standard error on 78 items is too wide to rank models against each other.
• One author. A consistent misconception would be invisible. A second domain reviewer is the next step.
• Perturbations are hand-written. Keeps the key auditable, caps the item count. Generating them would scale it and move the correctness risk into the generator.
• One perturbation per family. No variance estimate. Read the aggregate over 20 families, not an individual delta.
• No contamination claim. renumber makes a memorised instance less useful. It does not prove a model never saw the material.
• Publication starts the clock. These items are public, so future models may train on them, and the renumber signal weakens as that happens. Results are dated and tied to a commit for that reason. A held-out split is the standard fix and is not implemented here.
• Difficulty is self-labelled, not calibrated against measured performance.

How this went

I built this to measure whether a model's answers survive rewording and renumbering. It has not measured that yet, and the reason is worth stating plainly: a strong model answers the first twenty families correctly every time, and a metric with no variance cannot carry information.

What the benchmark did find, four times over, was defects in itself. An item whose wording collided with a definition in Freescale's own documentation. A scorer that rejected a correct answer written in bare hexadecimal. A sampling fluctuation that dissolved once it was run more times. And one item that discriminates for a reason other than the one on its label.

That last one took three controlled probes to pin down. The short version is that the model understands the hardware semantics the item was built around and cannot reliably assemble eight bits into a byte, which are different problems with different fixes. The long version, with the numbers, is in docs/RESULTS.md.

None of this is what I set out to measure. It is what the instrument was actually able to see, and telling those two apart turned out to be most of the work.

Scope

Built in about a week as a self-directed project, and my first using Inspect. The embedded content comes from two years of computer engineering technology coursework and a capstone build; the evaluation methodology is newer to me than the subject matter. Design decisions are in docs/METHODOLOGY.md.

Results are in docs/RESULTS.md. The work was done in two stages and the repository has been extended since the first stage's conclusions were written; RESULTS.md opens with what was concluded when, so a reader arriving from earlier material can tell which claims they are looking at.

Licence

MIT. See LICENSE.