Parallel inference runner for SpreadsheetBench that uses GRID's spreadsheet agent API to solve tasks and evaluate results inline.
SpreadsheetBench's built-in inference scripts ask the LLM to generate Python code (using openpyxl) to manipulate spreadsheets. The generated code runs in a Jupyter kernel to produce output .xlsx files. Because openpyxl cannot evaluate Excel formulas, the pipeline then opens each output file in Excel or LibreOffice to recalculate formulas before evaluation can compare cell values.
GRID's agent operates directly on a live spreadsheet engine, so output files are fully calculated .xlsx workbooks. This eliminates two steps from the pipeline:
- No Python/openpyxl code generation — the agent manipulates spreadsheets directly.
- No Excel/LibreOffice recalculation pass — output files already contain computed values.
The evaluation logic (cell-by-cell comparison with type coercion) is ported from SpreadsheetBench/evaluation/evaluation_verified.py to ensure compatible results.
Requires Python 3.12+ and uv.
make installOr run directly without installing:
uv run sheetbench-runner --helpPoint the runner at a SpreadsheetBench dataset directory and an output directory:
sheetbench-runner \
--dataset data/spreadsheetbench_verified_400/ \
--run-dir data/runs/2026-02-05-my-run \
--solve-profile profiles/anthropic-profile.json \
--concurrency 10To run a specific subset of tasks, use a task file:
sheetbench-runner \
--dataset data/spreadsheetbench_verified_400/ \
--run-dir data/runs/2026-02-05-my-run \
--task-file task-sets/all_verified_tasks.txt \
--solve-profile profiles/anthropic-profile.json \
--concurrency 10Runs are resumable — if interrupted, re-running the same command skips already-completed tasks and retries any that failed due to transient errors (5xx, timeouts). Resume with the same solve profile: the runner compares the profile's configuration and default model with the run metadata before it contacts the server.
To stop a run early without losing work in flight, press Ctrl-C (or send
SIGTERM): the runner finishes and grades the running tasks but starts no new
ones, and a resume picks up the rest. Press Ctrl-C again to abort the running
tasks too.
The v2 test set is split into category directories, each with its own dataset.json.
Point --dataset at one category per run:
sheetbench-runner \
--dataset data/spreadsheetbench-v2/Template \
--run-dir data/runs/2026-08-10-template \
--concurrency 10Debugging, Financial_Model and Template are supported. v2 entries reference
their input workbook (spreadsheet_path) and golden file (golden_response_path)
directly, have no instruction_type, and always sheet-qualify answer_position.
The prompt for v2 tasks omits the instruction_type section accordingly. Grading
uses the upstream SpreadsheetBench 2 semantics: cells in answer_position are
classified by comparing input to golden — unchanged cells are regression cells,
changed cells are modification cells — and the output is compared to golden with
1% numeric tolerance, "not meaningful" equivalence (#DIV/0!/#N/A vs N/A,
NM, dashes), and a formula-level fallback for error values. A task passes when
all modification cells match and ≥ 99.8% of regression cells are intact. Each
v2 entry in results.json records regression_accuracy and
modification_accuracy, and the run summary reports their averages.
By default, numeric comparison uses a combined 1% relative or 0.01 absolute
tolerance. This treats tiny iterative-calculation residuals as equivalent. Pass
--numeric-tolerance-mode relative to preserve the upstream rule: nonzero values
use relative tolerance, while absolute tolerance applies only when either value
is exactly zero.
Caveats:
Visualizationis not supported — its tasks are graded against a rubric (criteria), not cell ranges, and need a different evaluator.- The five
Financial_Model/spreadsheet/06_Project DigiMark/*_input.xlsxfiles cannot be opened by openpyxl (malformed XML namespace). These tasks fail evaluation with a load error and score 0.0 on both accuracy ratios — the same outcome as the upstream evaluator, which also reads inputs with openpyxl.
If the evaluation logic changes (e.g. a parser fix for edge-case Excel references), you can re-evaluate existing output files without re-running inference:
sheetbench-runner \
--dataset data/spreadsheetbench_verified_400/ \
--run-dir data/runs/2026-02-05-my-run \
--numeric-tolerance-mode combined \
--reevaluateThe selected mode is recorded in run.json. Normal resume requires the recorded
mode; --reevaluate may switch modes and records the new mode after regrading.
Switching modes requires an unfiltered re-evaluation with every recorded output
file present, preventing one run directory from mixing results graded under
different modes.
Each run directory is bound to exactly one dataset (task ids overlap across
v2 categories, so mixing them would collide). To run all three gradable
categories in one command, each into its own <prefix>-<category> run dir,
with a combined summary at the end:
make bench-all PREFIX=data/runs/<date> [ARGS="--concurrency 2"]
make parity-all PREFIX=data/runs/<date> # LO-parity pass over all threeNew run dirs record their dataset in run.json; regrading a run dir
against a different dataset is refused, and lo_parity.sh uses the
recorded dataset automatically.
The official SpreadsheetBench 2 protocol recalculates workbooks with LibreOffice before evaluation: model outputs before grading, and (per the upstream README) the dataset's input and golden files before any experiment. GRID outputs are already calculated, so native grading skips this — the honest measure of the engine, but it systematically differs from published numbers. To produce an upstream-comparable number:
make lo-parity RUN=data/runs/<run-dir> [DATASET=<dataset-dir>](equivalent to docker build -t lo-recalc docker/lo-recalc/ followed by
scripts/lo_parity.sh <run-dir> [dataset-dir])
Run it from the repo root; the regrade uses uv run so it always grades
with the current evaluator source (not the installed CLI snapshot). The
script clones the run dir to <run-dir>-parity, applies upstream's
open_spreadsheet.py in a Linux container (macOS cannot run it natively —
and the container runs as root because LibreOffice cannot create its user
profile under an unmapped UID), regrades the copies, and prints the
native-vs-parity comparison. Report both numbers; lo-version.txt in the
parity dir records the LibreOffice version used.
The LibreOffice version matters: versions before 24.8 lack XLOOKUP and cache
#NAME? for it, which zeroes every dependent value. The image builds on
Debian trixie (LibreOffice 25.2). The dataset's own input and golden files
were last written by LibreOffice 24.2, so they must be recalculated too before
their cached values are comparable:
make lo-recalc-all DIR=<dataset-dir>/spreadsheetrewrites every .xlsx under DIR in place (upstream's script only processes
files named *output.xlsx; the vendored copy adds --suffix, see
docker/lo-recalc/UPSTREAM.md). Do this on a copy or a version-controlled
checkout, never on the only copy of a dataset.
Usage: sheetbench-runner [OPTIONS]
Parallel inference runner for SpreadsheetBench with inline evaluation.
Options:
--dataset PATH Path to SpreadsheetBench dataset directory
(containing dataset.json) [required]
--run-dir PATH Directory to store results (creates if missing,
resumes if exists) [required]
--task-ids TEXT Comma-separated list of specific task IDs to run
--task-file PATH File with task IDs to run (one per line)
--config PATH Path to config.toml file
--solve-server-url TEXT Override solve server URL from config
--solve-profile FILE Solve profile JSON file
--concurrency INTEGER Number of parallel tasks (default: 4)
--timeout INTEGER Timeout per task in seconds (default: 3600)
--numeric-tolerance-mode [relative|combined]
V2 numeric comparison: combined relative/absolute
(default) or relative
-v, --verbose Enable verbose logging
--reevaluate Re-evaluate all tasks that have output files
(useful after parser fixes)
--help Show this message and exit.
Copy config.example.toml to config.toml and adjust as needed:
[solve]
url = "http://localhost:3000"
profile = "profiles/anthropic-profile.json"
[runner]
concurrency = 4
timeout_seconds = 3600
[evaluation]
numeric_tolerance_mode = "combined"CLI options (--solve-server-url, --solve-profile, --concurrency, --timeout,
--numeric-tolerance-mode) override their config file equivalents.
The solve profile is JSON containing models and modelRoles. Each model
names a transport, an apiKeyEnv and a request. request is the
provider's own request body for that transport (Anthropic Messages, OpenAI
Responses or OpenAI Chat Completions), without the messages, system prompt,
tools and streaming fields the solve server sets on every turn. model is
required in every request. The runner sends the body to the solve server
unchanged; the server validates it against the provider with one short
generation per model before the run starts, so a rejected parameter or key
fails the run before the first task.
apiKeyEnv names the environment variable that holds the model's API key.
Models may share an environment variable or name different variables. The
runner reads those variables and sends each key with the context request as
that model's apiKey.
For openai-compatible, an optional baseUrl beside transport, apiKeyEnv
and request selects the model's API root (for example,
https://inference.example.com/v1). Do not include /chat/completions.
The runner sends it through the solve context; no solve-server environment
variable is needed. Different models may use different endpoints. When omitted,
the solve server uses https://api.openai.com/v1. Native anthropic and
openai-responses transports do not accept baseUrl.
The URL must be absolute HTTP(S), at most 2048 characters, without credentials,
query parameters, fragments, whitespace or backslashes. It is recorded in
run.json and compared on resume, so do not put secrets in the URL (including
its path). Local/private endpoints are supported: use a trusted solve server,
since it makes requests to the configured URL and sends that model's key there.
The URL is resolved from the server's network, not the runner's.
Standard profiles are checked in:
| Profile | Transport | Model | Environment variable |
|---|---|---|---|
profiles/anthropic-profile.json |
anthropic |
claude-sonnet-5, adaptive thinking |
ANTHROPIC_API_KEY |
profiles/openai-profile.json |
openai-responses |
gpt-5.2, medium reasoning effort |
OPENAI_API_KEY |
profiles/qwen-compatible-profile.json |
openai-compatible |
qwen3.8-27b, thinking enabled |
COMPATIBLE_API_KEY |
The Qwen profile uses a placeholder endpoint; replace it with your API root.
It enables thinking but leaves effort at the endpoint's default. To request
both thinking and an explicit effort level on a LiteLLM-backed Qwen endpoint,
set the model's request to:
{
"model": "qwen3.8-27b",
"max_tokens": 16000,
"reasoning_effort": "medium",
"allowed_openai_params": ["reasoning_effort"],
"extra_body": {
"chat_template_kwargs": {
"enable_thinking": true
}
}
}- Thinking:
enable_thinkingenables the model's thinking mode. - Effort:
reasoning_effortselects the effort level. The tested Qwen deployment acceptslow,medium, andxhigh(its default); it rejectshigh. Confirm the supported values with your endpoint operator rather than assuming they are the same across models or deployments. - Proxy forwarding:
allowed_openai_paramsis a LiteLLM-specific override that permits forwardingreasoning_effortwhen its parameter validation would otherwise reject it. It does not enable thinking by itself or add effort support to a model that lacks it. Other endpoints may not need or accept this field. - Thinking visibility: enabling thinking and setting effort do not guarantee
a separate reasoning channel. In the tested deployment, both work while
reasoning and the answer arrive together in ordinary
content. The current compatible adapter does not translate separate reasoning fields into thinking blocks. Inspect a saved transcript before relying on structured thinking.
These fields belong inside the profile model's request, not beside baseUrl.
The nested extra_body above is part of the forwarded JSON body; do not assume
it has the same semantics as an SDK's extra_body keyword argument. This
example was exercised with both non-streaming and streaming requests. Effort
is not a fixed token budget, and a higher setting need not produce a longer
response to every prompt. Changing effort changes the saved solve configuration;
use a new run directory rather than resuming a run with different settings.
run.json documents with a schema_version below 3 (runs created before
profiles carried request bodies, and released-format runs) are read as
released metadata: regrading needs no profile, and resuming requires a
profile whose default model matches the recorded one, after which run.json
is rewritten with that profile's configuration.
For every non-reevaluation invocation, the runner creates one ephemeral solve
context before running tasks, uses it for all workbook uploads and solves, and
deletes it on exit. Run metadata records the profile's configuration, which the
runner compares against the profile on resume. A pure --reevaluate run makes
no server requests and does not require a solve profile. It rewrites run.json
only when changing the numeric tolerance mode.
A run directory contains:
run-dir/
├── run.json # Run metadata
├── results.json # Task results sorted by task_id
├── run.log # Execution log
├── 13-1-output.xlsx # Output workbook for task 13-1
├── 13-1-transcript.json # Agent transcript for task 13-1
├── 203-15-output.xlsx
├── 203-15-transcript.json
└── ...
Each entry in results.json records the task outcome, timing, and token usage:
{
"task_id": "13-1",
"duration_seconds": 45.2,
"turns": 5,
"tool_calls": 8,
"input_tokens": 12500,
"output_tokens": 3200,
"input_file": "spreadsheet/13-1/1_13-1_init.xlsx",
"output_file": "13-1-output.xlsx",
"transcript_file": "13-1-transcript.json",
"result": "pass",
"message": ""
}make test # run tests with coverage
make lt # lint + typecheck