A curated list of awesome works in benchmark compression for efficient LLM evaluation.
🤩👉 Welcome to submit your contributions to this code repository.
| Paper | Conference/Journal | Code |
|---|---|---|
| Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models | arXiv 2026 | Code |
Reduce the cost of evaluating one benchmark using a subset chosen before the main evaluation.
Choose evaluation items using benchmark content or a sampling design, without historical model responses or a target-specific signal.
Reuse earlier models' evaluation results to select items or estimate the results of a new model.
Use a low-cost signal from the model being evaluated to choose its evaluation subset.
Reduce evaluation across multiple benchmarks through shared selection, prediction, or evidence reuse.
Use results observed during evaluation to decide what to test next or when to stop.
Choose the next evaluation item using the results collected so far.
Allocate evaluation effort to reach a final decision, eliminate candidates, or stop with sufficient evidence.
Study benchmark redundancy, measurement quality, and the reliability of reduced evaluations.
Contributions and corrections are welcome. Please provide the paper title, an official paper link, publication venue and year, application domains, and an author-released code link if available. For new papers, suggest the closest category and briefly explain the fit.
If you find this collection useful, please consider giving this repository a ⭐ or citing our survey:
@misc{ling2026testingmuchsurveybenchmark,
title={Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models},
author={Guoming Ling and Yupei Lin and Zhongzhan Huang and Junxin Li and Junjie Liu and Shanshan Zhong and Hefeng Wu and Liang Lin},
year={2026},
eprint={2610.18248},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2610.18248},
}