JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

[中文版] | [English]

🎉 News: JADE has been accepted to ICML 2026! Check out the paper on OpenReview or arXiv.

JADE is an innovative dual-layer evaluation framework designed to assess Agent performance on complex, open-ended professional tasks (e.g., market research, strategic sourcing). It effectively addresses the "stability-adaptability" dilemma in evaluation by decoupling general expert principles from specific evidence verification.

---

🖼️ JADE Framework Overview

Given a user Query and an Agent-generated Response, JADE first activates corresponding Expert-Authored Skills to guide the generation of Query-Specific Checklists. Subsequently, it derives Report-Specific Checklists for verifiable Factual Claims and Reasoning Quality.

Factual claims are verified via real-time web verification, while the reasoning component is evaluated by an LLM conditioned on the Query-Specific Checklists. Additionally, an evidence-based gating mechanism is introduced to ensure that subsequent dependency judgments are automatically invalidated if the underlying facts are unverified.

---

📊 BizBench Leaderboard

JADE evaluates the performance of various models and agents on BizBench.

Model	Type	Tool	Final (%)	Reasoning (%)	Evidence (%)	Credibility (%)	Density	Tokens
Agentic Deep Research Systems
Gemini Deep Research	Prop.	✓	57.1	63.4	89.1	43.8	0.063	7590
Shopping Research	Prop.	✓	56.2	59.1	95.3	44.4	0.067	4772
o4-mini Deep Research	Prop.	✓	47.7	54.5	87.6	41.2	0.061	2907
ChatGPT (Web Search)	Prop.	✓	38.8	46.0	84.6	48.6	0.051	2244
API-Based Models with Tool Use
GPT-5.2	Prop.	✓	55.7	59.2	93.5	50.2	0.071	3167
DeepSeek V3.2	Open	✓	55.8	59.2	93.5	48.1	0.071	2877
Gemini 3 Pro	Prop.	✓	41.8	48.5	86.5	44.7	0.059	1419
Claude Opus 4.5	Prop.	✓	36.2	45.8	80.2	44.8	0.049	1824
Claude Sonnet 4.5	Prop.	✓	32.9	45.5	73.3	48.1	0.046	1975
Qwen3-Max	Open	✓	34.0	39.9	83.8	45.5	0.047	1340
GPT-4.1	Prop.	✓	32.7	40.2	81.4	45.7	0.046	1434
API-Based Models (No Tool)
GPT-5.2	Prop.	–	49.0	52.6	93.1	53.7	0.064	2516
DeepSeek V3.2	Open	–	46.1	50.7	90.9	52.1	0.061	2299
Gemini 3 Pro	Prop.	–	44.8	52.7	84.8	50.7	0.063	1495
GPT-4.1	Prop.	–	42.7	54.6	80.0	51.2	0.060	1380
Qwen3-Max	Open	–	34.2	46.0	74.0	50.9	0.051	1040
Claude Opus 4.5	Prop.	–	32.1	49.3	68.0	52.2	0.044	2314
Claude Sonnet 4.5	Prop.	–	29.1	41.8	71.6	52.2	0.041	1585

---

🌟 Core Features

• Dual-Layer Decomposition Architecture: Layer 1 encodes expert knowledge into stable evaluation skills; Layer 2 dynamically generates assertion-level checkpoints for specific reports.
• Evidence-Aware Gating: A unique mechanism where subsequent logical conclusions are automatically invalidated if underlying facts are judged as hallucinations or errors.
• Real-Time Web Verification: Integrates a Verification Agent to verify the authenticity of data, prices, certificates, and quantitative descriptions via real-time web searches.
• BizBench Benchmark: Contains 150 high-quality strategic sourcing queries derived from real-world B2B scenarios, offering high practical relevance (see bizbench.json).

---

🛠️ Installation

# python version >= 3.10

git clone https://github.com/smiling-world/JADE.git
# or 'git clone git@github.com:smiling-world/JADE.git'
# or download the zip file and unzip
cd JADE
pip install -r requirements.txt

---

🚀 Quick Start

Follow these four steps to quickly start using JADE to evaluate your Agent reports:

1. Configure Environment Variables

First, clone the environment configuration template and fill in your API keys (Note: OPENROUTER_API_KEY, SERPAPI_KEY, and JINA_API_KEY are rarely used and can optionally be left blank).

cp env.example .env

2. Prepare Evaluation Data

Fill the reports generated by your Agent into the report attribute in the data input template.

Template path: data/input/base_template.json

💡 Tip: You can fill in multiple reports and use one of the following methods for small-scale testing:

1. Delete unfilled samples in base_template.json.
2. Set the item_ids attribute in configs/bizbench_eval.yaml to evaluate only specific samples.

3. Run Evaluation Script

python scripts/run_jade.py --config configs/bizbench_eval.yaml

4. View Results

Results are saved in output/base_template by default. If you need to change the output directory, you can use one of the following methods:

1. Modify the output_dir attribute in configs/bizbench_eval.yaml.
2. Specify the output directory via command line arguments:

python scripts/run_jade.py --config configs/bizbench_eval.yaml \
    --output_dir output/base_template

---

👀 Advanced Customization

1. You are not limited to bizbench questions or our given classification standards. You can define classification standards for questions in different domains, generate expert experience files following the format in rubrics/bizbench, and modify the rubric_dir path in configs/bizbench_eval.yaml.
2. You can explore the performance of the dual-layer structure in JADE by modifying the use_skill and use_report_specific attributes in configs/bizbench_eval.yaml. You can also try modifying score_fusion_mode, reasoning_weight, evidence_weight, and credibility_weight to explore the impact of weights on the final score.

---

📖 Citation

If you find JADE or BizBench useful in your research, please cite our paper:

@inproceedings{
lin2026jade,
title={{JADE}: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks},
author={Lanbo Lin and Jiayao Liu and Tianyuan Yang and Li Cai and Yuanwu Xu and Lei Wei and Sicong Xie and Guannan Zhang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=SoilRyCv1i}
}