NatureBench

Can coding agents match the published SOTA of Nature-family papers?

   

📰 News • 📖 Overview • 🏅 Submit Results • 🔧 Installation • 🚀 Quick Start
🌱 NatureGym • 📚 Documentation • ⚖️ License • 🎈 Citation

📰News

• ⚖️ [2026-07-30] We update the validity judge and use GPT-5.5 to reassess all runs; the results have been comprehensively updated.
• 🏆 [2026-07-07] We update the results with 12 coding-agent configurations, adding GLM-5.2 and MiniMax-M3.
• 📦 [2026-07-03] We release the full NatureBench agent traces on Hugging Face.
• 🚀 [2026-06-24] We introduce NatureBench, a benchmark for scientific ML coding agents on Nature-family tasks.

📖Overview

NatureBench is a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, spanning 6 scientific domains, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery. Each task asks an agent to solve a real scientific machine-learning problem and is scored against the source paper's reported state of the art.

NatureBench is built on NatureGym, an automated pipeline that converts a published paper into a containerized task package comprising a task brief, the paper's dataset, a held-out test set with hidden ground truth, and an automated evaluator.

📊Results

Across twelve coding-agent configurations, the strongest reaches a 15.6% Surpass-SOTA rate, and success remains uneven across the six scientific domains NatureBench spans.

🏅Submit Results

We welcome results from researchers and developers who have evaluated models or agents on NatureBench. Submissions are currently accepted by email; see submit-results/ for the result templates, validation and scoring scripts, required raw artifacts, and submission email.

🔧Installation

git clone https://github.com/FrontisAI/NatureBench.git
cd NatureBench

conda env create -f conda_env.yml
conda env create -f conda_env_eval.yml
conda activate naturebench

This creates two environments: naturebench (the main orchestration environment that runs run_naturebench.py, agent adapters, Docker scheduling, and result aggregation) and naturebench-eval (the evaluation service environment that runs scoring logic).

The base Docker image is built automatically on the first run via --ensure-base-image (used in the Quick Start below). To build it manually:

bash scripts/ensure_naturebench_base.sh

🚀Quick Start

Set credentials for your agent. Claude Code is shown here; for Codex, Gemini CLI, the post-hoc judge, and network proxy, see docs/configuration.md.

export ANTHROPIC_API_KEY=...
export ANTHROPIC_BASE_URL=...

Run end-to-end. This single command downloads the dataset, builds the base image if needed, starts the evaluation service, and evaluates:

python run_naturebench.py \
  --tasks gpu_low \
  --agent claude \
  --model <model-name> \
  --out-dir ./results/claude_<model-name>_gpu_low \
  --gpu-devices 0,1,2,3 \
  --max-workers 4 \
  --start-eval-services \
  --eval-env-mapping ./eval_env_mapping.json \
  --ensure-base-image

This lists only the parameters you set explicitly; options with sensible defaults are omitted (see Quick Start defaults for the full list and their values). Adjust --gpu-devices / --max-workers to your hardware. When running without GPUs, omit --gpu-devices and any other GPU scheduling options. The complete parameter reference is in docs/usage.md.

The built-in --agent options are claude, codex, and gemini; to run your own agent on NatureBench instead, see docs/custom-agents.md.

Evaluation Setup

For comparability, our reported runs use the following setup:

Setting	Configuration
Per-task timeout	A 4-hour agent solve budget (--timeout 14400).
Compute	Each gpu_low task uses one NVIDIA RTX 3090/4090 (24 GB), each gpu_high task uses one NVIDIA A800 (80 GB, A100-class), and the 3 cpu tasks use no GPU. Task-group lists are provided in task-set/.
Web search	Disabled for all evaluated agents.

🌱NatureGym

The task packages are built by NatureGym, an automated, Skills-based pipeline that turns a published Nature-family paper into a containerized, runnable task. It filters papers, acquires and verifies the data, and assembles the task package (brief, data, evaluator, environment, metadata), while an information firewall removes the source method so that agents must discover solutions rather than reproduce them.

The pipeline runs as a chain of Claude Code skills driven by batch scripts, all under naturegym/. See naturegym/README.md for the stage-by-stage flow, the construction skills, and how to run them.

📦Repository Contents

• run_naturebench.py — one-command entry point: download data and launch evaluation
• solve.py — main evaluation orchestrator
• eval_service.py — host-side evaluation service
• judge.py — post-hoc validity judge
• agent/ — agent adapters and registry for Claude Code / Codex CLI / Gemini CLI (and custom agents)
• evaluator/ — evaluator interface
• docker/Dockerfile.base — NatureBench base Docker image
• scripts/ — helper scripts
  • ensure_naturebench_base.sh — build the NatureBench base image if it is missing
  • start_eval_services.sh — start evaluation service from the mapping file
• task-set/ — task lists grouped by resource demand
• submit-results/ — leaderboard submission guide, templates, and validation/scoring tools
• docs/ — detailed configuration, usage, task-packages, and custom-agents references
• naturegym/ — NatureGym construction pipeline: skills + batch drivers that build task packages from papers
• conda_env.yml — main orchestration environment
• conda_env_eval.yml — evaluation service environment
• eval_env_mapping.json — task-to-evaluation-service port mapping
• config.example.yaml — example configuration file
• LICENSE, NOTICE — MIT license for original work; NOTICE defines the scope

📚Documentation

Document	Contents
docs/configuration.md	Agent authentication (Claude Code / Codex CLI / Gemini CLI), the post-hoc judge, network proxy, and the evaluation service.
docs/usage.md	More run examples (CPU, GPU batch, Codex login, resume), the complete parameter reference, and output formats.
docs/task-packages.md	Task package structure and the resource-grouped task lists.
docs/custom-agents.md	Plugging in a custom agent: How to run your own agent on NatureBench.
submit-results/	Email-submission guide, result templates, validation and scoring tools, and the publication process.

⚖️License

The top-level LICENSE is the MIT License and applies only to original NatureBench contributions; see NOTICE for the exact scope. Third-party data bundled in each task package is governed by the notices in that task's tasks/<case_id>/licenses/ directory.

🎈Citation

If you use NatureBench in your research, please cite our work:

@misc{wang2026naturebench,
  title         = {NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?},
  author        = {Yuru Wang and Lejun Cheng and Yuxin Zuo and Sihang Zeng and Bingxiang He and Che Jiang and Junlin Yang and Yuchong Wang and Kaikai Zhao and Weifeng Huang and Kai Tian and Zhenzhao Yuan and Jincheng Zhong and Weizhi Wang and Ning Ding and Bowen Zhou and Kaiyan Zhang},
  year          = {2026},
  eprint        = {2606.24530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24530}
}