Official Codebase for "METAL: Towards Multilingual Meta-Evaluation" (published in Findings of ACL: NAACL 2024)
-
Updated
Jun 18, 2024 - Python
Official Codebase for "METAL: Towards Multilingual Meta-Evaluation" (published in Findings of ACL: NAACL 2024)
Automatic validation of grammatical-error-correction metrics using MAEGE lattices (ACL 2018).
A local LLM benchmarking framework designed to evaluate model performance across multiple backends (LM Studio, Ollama, etc.), including metrics for speed, quality, and instruction adherence. Supports structured test runs, result analysis, and reproducible evaluations.
A criterion-validity test of LLM-as-judge evaluation: does a rubric that passes reliability and inter-judge agreement checks actually track reader behaviour? 840 held-out Chinese web novels; judges qwen3-max, o3, deepseek-v4-pro, qwen-turbo.
A meta-eval that grades the quality of other evals: discrimination, contamination, difficulty spread, statistical power
Cross-Family LLM-Judge Agreement for Institutional RAG: 5 families, 9 judges. Validated on TREC RAG 2024 (kappa=0.4941) + BEIR scifact.
Meta-evaluation of LLM judges: measuring judge/human agreement and where judges miss
Correlate automatic metric or grader scores with human ratings, with the grouping stated and an interval on every number
Meta-evaluation system that detects bias, inconsistency, and hallucinated rationales in LLM judges using PubMedQA
Kappa Labs: independent assurance for AI evaluations. Meta-evaluation, public audits and field notes.
Control the adequacy–fluency balance in MT meta-evaluation. Code for the WMT2026 paper "Mind Which Bird You Favour".
To associate your repository with the meta-evaluation topic, visit your repo's landing page and select "manage topics."