Skip to content

About

Parallel inference runner for SpreadsheetBench with inline evaluation.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

sheetbench-runner

Parallel inference runner for SpreadsheetBench that uses GRID's spreadsheet agent API to solve tasks and evaluate results inline.

How it differs from the default SpreadsheetBench pipeline

SpreadsheetBench's built-in inference scripts ask the LLM to generate Python code (using openpyxl) to manipulate spreadsheets. The generated code runs in a Jupyter kernel to produce output .xlsx files. Because openpyxl cannot evaluate Excel formulas, the pipeline then opens each output file in Excel or LibreOffice to recalculate formulas before evaluation can compare cell values.

GRID's agent operates directly on a live spreadsheet engine, so output files are fully calculated .xlsx workbooks. This eliminates two steps from the pipeline:

  1. No Python/openpyxl code generation — the agent manipulates spreadsheets directly.
  2. No Excel/LibreOffice recalculation pass — output files already contain computed values.

The evaluation logic (cell-by-cell comparison with type coercion) is ported from SpreadsheetBench/evaluation/evaluation_verified.py to ensure compatible results.

Install

Requires Python 3.12+ and uv.

make install

Or run directly without installing:

uv run sheetbench-runner --help

Usage

Point the runner at a SpreadsheetBench dataset directory and an output directory:

sheetbench-runner \
  --dataset data/spreadsheetbench_verified_400/ \
  --run-dir data/runs/2026-02-05-my-run \
  --solve-profile profiles/anthropic-profile.json \
  --concurrency 10

To run a specific subset of tasks, use a task file:

sheetbench-runner \
  --dataset data/spreadsheetbench_verified_400/ \
  --run-dir data/runs/2026-02-05-my-run \
  --task-file task-sets/all_verified_tasks.txt \
  --solve-profile profiles/anthropic-profile.json \
  --concurrency 10

Runs are resumable — if interrupted, re-running the same command skips already-completed tasks and retries any that failed due to transient errors (5xx, timeouts). Resume with the same solve profile: the runner compares the profile's configuration and default model with the run metadata before it contacts the server.

To stop a run early without losing work in flight, press Ctrl-C (or send SIGTERM): the runner finishes and grades the running tasks but starts no new ones, and a resume picks up the rest. Press Ctrl-C again to abort the running tasks too.

SpreadsheetBench v2 datasets

The v2 test set is split into category directories, each with its own dataset.json. Point --dataset at one category per run:

sheetbench-runner \
  --dataset data/spreadsheetbench-v2/Template \
  --run-dir data/runs/2026-08-10-template \
  --concurrency 10

Debugging, Financial_Model and Template are supported. v2 entries reference their input workbook (spreadsheet_path) and golden file (golden_response_path) directly, have no instruction_type, and always sheet-qualify answer_position. The prompt for v2 tasks omits the instruction_type section accordingly. Grading uses the upstream SpreadsheetBench 2 semantics: cells in answer_position are classified by comparing input to golden — unchanged cells are regression cells, changed cells are modification cells — and the output is compared to golden with 1% numeric tolerance, "not meaningful" equivalence (#DIV/0!/#N/A vs N/A, NM, dashes), and a formula-level fallback for error values. A task passes when all modification cells match and ≥ 99.8% of regression cells are intact. Each v2 entry in results.json records regression_accuracy and modification_accuracy, and the run summary reports their averages.

By default, numeric comparison uses a combined 1% relative or 0.01 absolute tolerance. This treats tiny iterative-calculation residuals as equivalent. Pass --numeric-tolerance-mode relative to preserve the upstream rule: nonzero values use relative tolerance, while absolute tolerance applies only when either value is exactly zero.

Caveats:

  • Visualization is not supported — its tasks are graded against a rubric (criteria), not cell ranges, and need a different evaluator.
  • The five Financial_Model/spreadsheet/06_Project DigiMark/*_input.xlsx files cannot be opened by openpyxl (malformed XML namespace). These tasks fail evaluation with a load error and score 0.0 on both accuracy ratios — the same outcome as the upstream evaluator, which also reads inputs with openpyxl.

Re-evaluation

If the evaluation logic changes (e.g. a parser fix for edge-case Excel references), you can re-evaluate existing output files without re-running inference:

sheetbench-runner \
  --dataset data/spreadsheetbench_verified_400/ \
  --run-dir data/runs/2026-02-05-my-run \
  --numeric-tolerance-mode combined \
  --reevaluate

The selected mode is recorded in run.json. Normal resume requires the recorded mode; --reevaluate may switch modes and records the new mode after regrading. Switching modes requires an unfiltered re-evaluation with every recorded output file present, preventing one run directory from mixing results graded under different modes.

Running every category

Each run directory is bound to exactly one dataset (task ids overlap across v2 categories, so mixing them would collide). To run all three gradable categories in one command, each into its own <prefix>-<category> run dir, with a combined summary at the end:

make bench-all PREFIX=data/runs/<date> [ARGS="--concurrency 2"]
make parity-all PREFIX=data/runs/<date>   # LO-parity pass over all three

New run dirs record their dataset in run.json; regrading a run dir against a different dataset is refused, and lo_parity.sh uses the recorded dataset automatically.

Upstream-parity grading (LibreOffice pass)

The official SpreadsheetBench 2 protocol recalculates workbooks with LibreOffice before evaluation: model outputs before grading, and (per the upstream README) the dataset's input and golden files before any experiment. GRID outputs are already calculated, so native grading skips this — the honest measure of the engine, but it systematically differs from published numbers. To produce an upstream-comparable number:

make lo-parity RUN=data/runs/<run-dir> [DATASET=<dataset-dir>]

(equivalent to docker build -t lo-recalc docker/lo-recalc/ followed by scripts/lo_parity.sh <run-dir> [dataset-dir])

Run it from the repo root; the regrade uses uv run so it always grades with the current evaluator source (not the installed CLI snapshot). The script clones the run dir to <run-dir>-parity, applies upstream's open_spreadsheet.py in a Linux container (macOS cannot run it natively — and the container runs as root because LibreOffice cannot create its user profile under an unmapped UID), regrades the copies, and prints the native-vs-parity comparison. Report both numbers; lo-version.txt in the parity dir records the LibreOffice version used.

The LibreOffice version matters: versions before 24.8 lack XLOOKUP and cache #NAME? for it, which zeroes every dependent value. The image builds on Debian trixie (LibreOffice 25.2). The dataset's own input and golden files were last written by LibreOffice 24.2, so they must be recalculated too before their cached values are comparable:

make lo-recalc-all DIR=<dataset-dir>/spreadsheet

rewrites every .xlsx under DIR in place (upstream's script only processes files named *output.xlsx; the vendored copy adds --suffix, see docker/lo-recalc/UPSTREAM.md). Do this on a copy or a version-controlled checkout, never on the only copy of a dataset.

All options

Usage: sheetbench-runner [OPTIONS]

  Parallel inference runner for SpreadsheetBench with inline evaluation.

Options:
  --dataset PATH           Path to SpreadsheetBench dataset directory
                           (containing dataset.json)  [required]
  --run-dir PATH           Directory to store results (creates if missing,
                           resumes if exists)  [required]
  --task-ids TEXT          Comma-separated list of specific task IDs to run
  --task-file PATH         File with task IDs to run (one per line)
  --config PATH            Path to config.toml file
  --solve-server-url TEXT  Override solve server URL from config
  --solve-profile FILE     Solve profile JSON file
  --concurrency INTEGER    Number of parallel tasks (default: 4)
  --timeout INTEGER        Timeout per task in seconds (default: 3600)
  --numeric-tolerance-mode [relative|combined]
                           V2 numeric comparison: combined relative/absolute
                           (default) or relative
  -v, --verbose            Enable verbose logging
  --reevaluate             Re-evaluate all tasks that have output files
                           (useful after parser fixes)
  --help                   Show this message and exit.

Configuration

Copy config.example.toml to config.toml and adjust as needed:

[solve]
url = "http://localhost:3000"
profile = "profiles/anthropic-profile.json"

[runner]
concurrency = 4
timeout_seconds = 3600

[evaluation]
numeric_tolerance_mode = "combined"

CLI options (--solve-server-url, --solve-profile, --concurrency, --timeout, --numeric-tolerance-mode) override their config file equivalents.

The solve profile is JSON containing models and modelRoles. Each model names a transport, an apiKeyEnv and a request. request is the provider's own request body for that transport (Anthropic Messages, OpenAI Responses or OpenAI Chat Completions), without the messages, system prompt, tools and streaming fields the solve server sets on every turn. model is required in every request. The runner sends the body to the solve server unchanged; the server validates it against the provider with one short generation per model before the run starts, so a rejected parameter or key fails the run before the first task.

apiKeyEnv names the environment variable that holds the model's API key. Models may share an environment variable or name different variables. The runner reads those variables and sends each key with the context request as that model's apiKey.

For openai-compatible, an optional baseUrl beside transport, apiKeyEnv and request selects the model's API root (for example, https://inference.example.com/v1). Do not include /chat/completions. The runner sends it through the solve context; no solve-server environment variable is needed. Different models may use different endpoints. When omitted, the solve server uses https://api.openai.com/v1. Native anthropic and openai-responses transports do not accept baseUrl.

The URL must be absolute HTTP(S), at most 2048 characters, without credentials, query parameters, fragments, whitespace or backslashes. It is recorded in run.json and compared on resume, so do not put secrets in the URL (including its path). Local/private endpoints are supported: use a trusted solve server, since it makes requests to the configured URL and sends that model's key there. The URL is resolved from the server's network, not the runner's.

Standard profiles are checked in:

Profile Transport Model Environment variable
profiles/anthropic-profile.json anthropic claude-sonnet-5, adaptive thinking ANTHROPIC_API_KEY
profiles/openai-profile.json openai-responses gpt-5.2, medium reasoning effort OPENAI_API_KEY
profiles/qwen-compatible-profile.json openai-compatible qwen3.8-27b, thinking enabled COMPATIBLE_API_KEY

The Qwen profile uses a placeholder endpoint; replace it with your API root. It enables thinking but leaves effort at the endpoint's default. To request both thinking and an explicit effort level on a LiteLLM-backed Qwen endpoint, set the model's request to:

{
  "model": "qwen3.8-27b",
  "max_tokens": 16000,
  "reasoning_effort": "medium",
  "allowed_openai_params": ["reasoning_effort"],
  "extra_body": {
    "chat_template_kwargs": {
      "enable_thinking": true
    }
  }
}
  • Thinking: enable_thinking enables the model's thinking mode.
  • Effort: reasoning_effort selects the effort level. The tested Qwen deployment accepts low, medium, and xhigh (its default); it rejects high. Confirm the supported values with your endpoint operator rather than assuming they are the same across models or deployments.
  • Proxy forwarding: allowed_openai_params is a LiteLLM-specific override that permits forwarding reasoning_effort when its parameter validation would otherwise reject it. It does not enable thinking by itself or add effort support to a model that lacks it. Other endpoints may not need or accept this field.
  • Thinking visibility: enabling thinking and setting effort do not guarantee a separate reasoning channel. In the tested deployment, both work while reasoning and the answer arrive together in ordinary content. The current compatible adapter does not translate separate reasoning fields into thinking blocks. Inspect a saved transcript before relying on structured thinking.

These fields belong inside the profile model's request, not beside baseUrl. The nested extra_body above is part of the forwarded JSON body; do not assume it has the same semantics as an SDK's extra_body keyword argument. This example was exercised with both non-streaming and streaming requests. Effort is not a fixed token budget, and a higher setting need not produce a longer response to every prompt. Changing effort changes the saved solve configuration; use a new run directory rather than resuming a run with different settings.

run.json documents with a schema_version below 3 (runs created before profiles carried request bodies, and released-format runs) are read as released metadata: regrading needs no profile, and resuming requires a profile whose default model matches the recorded one, after which run.json is rewritten with that profile's configuration.

For every non-reevaluation invocation, the runner creates one ephemeral solve context before running tasks, uses it for all workbook uploads and solves, and deletes it on exit. Run metadata records the profile's configuration, which the runner compares against the profile on resume. A pure --reevaluate run makes no server requests and does not require a solve profile. It rewrites run.json only when changing the numeric tolerance mode.

Output

A run directory contains:

run-dir/
├── run.json                  # Run metadata
├── results.json              # Task results sorted by task_id
├── run.log                   # Execution log
├── 13-1-output.xlsx          # Output workbook for task 13-1
├── 13-1-transcript.json      # Agent transcript for task 13-1
├── 203-15-output.xlsx
├── 203-15-transcript.json
└── ...

Each entry in results.json records the task outcome, timing, and token usage:

{
  "task_id": "13-1",
  "duration_seconds": 45.2,
  "turns": 5,
  "tool_calls": 8,
  "input_tokens": 12500,
  "output_tokens": 3200,
  "input_file": "spreadsheet/13-1/1_13-1_init.xlsx",
  "output_file": "13-1-output.xlsx",
  "transcript_file": "13-1-transcript.json",
  "result": "pass",
  "message": ""
}

Development

make test   # run tests with coverage
make lt     # lint + typecheck

About

Parallel inference runner for SpreadsheetBench with inline evaluation.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages