Joss by gmartinon-ed · Pull Request #469 · EmertonData/glide

gmartinon-ed · 2026-10-08T19:28:35Z

JOSS

codecov · 2026-10-08T19:29:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

imerad

En dehors des commentaires spécifiques, je pense que l'article est globalement trop qualitatif et ne met pas la technicité de glide suffisamment en valeur. Je recommande de regarder quelques articles récemment publiés sur ce journal. Il y a aussi celui d'MFGlib

Suivant ces exemples, je pense que ce serait bien si on peut mettre quelques formules et éventuellement des figures extraite de nos notebooks. Des snippets de codes seraient bien aussi. On pourra réduire certaines sections textuelles s'il y a une contrainte.

J'ai l'impression que la plupart des articles du journal déclarent une intervention de l'IA faible ou nulle. Ça me fait penser que le journal y est peut être peu tolérant.

imerad · 2026-10-12T14:47:09Z

+
+# Statement of need
+
+Evaluating a deployed GenAI system under a fixed annotation budget requires combining a few reliable human annotations with a large set of proxy annotations, such as LLM-as-judge labels. The literature has addressed the resulting complications one strand at a time: heterogeneous strata, handled by stratified sampling [@NEURIPS2024_c9fcd02e; @fogliato2024framework]; small labeled sets, handled by bootstrap intervals [@kluger2025prediction]; correlated units, handled by clustered inference; multiple judges, handled by multi-proxy aggregation [@shan2025sada; @cowen2026multiple]; judge uncertainty, exploited by active sampling [@pmlr-v235-zrnic24a; @gligoric2025can]; unequal human and LLM costs, handled by cost-aware sampling [@angelopoulos2025cost]; and continuous monitoring, made safe from false alarms by anytime-valid confidence sequences [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].


Ce paragraphe est trop condensé. Le lecteur lambda n'a aucune chance de comprendre toutes ces notions à ce niveau de concision. Il faudrait soit étoffer, soit assumer que c'est une liste sans description des éléments

Le début du paragraphe considère de facto qu'on évalue la GenAI avec labels humains+proxys, mais il ne faut pas oublier que ça ne se fait souvent pas comme ça. Quelques phrases de cheminement ne feraient pas de mal (on veux évaluer fiablement, on utilise des labels humains, c'est trop cher, on se tourne vers un proxy, il est biaisé, ppi donne le meilleur des deux mondes)

imerad · 2026-10-12T15:03:16Z

+
+Evaluating a deployed GenAI system under a fixed annotation budget requires combining a few reliable human annotations with a large set of proxy annotations, such as LLM-as-judge labels. The literature has addressed the resulting complications one strand at a time: heterogeneous strata, handled by stratified sampling [@NEURIPS2024_c9fcd02e; @fogliato2024framework]; small labeled sets, handled by bootstrap intervals [@kluger2025prediction]; correlated units, handled by clustered inference; multiple judges, handled by multi-proxy aggregation [@shan2025sada; @cowen2026multiple]; judge uncertainty, exploited by active sampling [@pmlr-v235-zrnic24a; @gligoric2025can]; unequal human and LLM costs, handled by cost-aware sampling [@angelopoulos2025cost]; and continuous monitoring, made safe from false alarms by anytime-valid confidence sequences [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].
+
+These methods share one template, yet they are scattered across papers with heterogeneous notation and partial reference implementations [@song2026demystifying], some in R and some in Python, none covering more than a slice of the design space. Combining these methods currently means stitching together several repositories and verifying their consistency by hand. `GLIDE` closes that gap by treating sampling, estimation and monitoring as one consistent family, and it quantifies the return on the judge: the effective sample size says how many extra expert annotations the proxy is worth.


La description ici reste très qualitative, j'imagine que tu as bien tenu compte de la structure attendue, mais je recommande de regarder (avec Claude) quelques article publiés sur ce journal pour avoir une idée du niveau de détail technique attendu. Accessoirement, la dernière phrase sur l'effective sample size ne colle pas très bien avec l'idée du paragraphe.

Reformulé. le niveau technique n'est pas si élevé dans les 10 derniers articles publiés.

imerad · 2026-10-12T15:12:47Z

+
+# Software design
+
+`GLIDE`'s design is organized around three sequential steps inherited from survey theory (\autoref{fig:workflow}): sampling, which selects the observations worth a human label; annotation, which only domain experts can perform; and estimation, which combines labels and proxy predictions into a debiased estimate with a confidence interval. Additionally, monitoring wraps per-batch estimates in an anytime-valid bound. Samplers expose `sample`, estimators `estimate` and monitors `detect`, so a full workflow fits in a handful of lines and a new component plugs in by implementing one of these interfaces. Data follows a simple contract: one aligned array per collection of labels, with `numpy.nan` marking unlabeled entries of the ground-truth array, in the spirit of scikit-learn conventions [@Pedregosa_Scikit-learn_Machine_Learning_2011]. Each estimator family factors its computation into a shared `MeanEstimationEngine`, called once over the whole dataset by an estimator and once per batch by a monitor, so contributors can extend an estimator into its monitor counterpart without reimplementing its statistics. In terms of quality gates, every new component passes three lines of defense, all run in continuous integration: unit tests with 100% coverage and executed docstring examples, functional tests of statistical properties (e.g., a stratified estimator on a single stratum must equal the uniform one), and Monte Carlo validation notebooks assessing coverage, interval width, false-alarm and miscoverage rates.


Cette formulation "Samplers expose sample, estimators estimate and monitors detect ..." est un peu trop sèche ...

Attention le terme coverage est utilisé deux fois avec des sens différents. On pourrait parler "statistical coverage" ou "CI coverage" pour la 2e

imerad · 2026-10-12T15:25:29Z

+
+`GLIDE` has been developed in public since its first commit in March 2026, with roughly two releases per month and a dozen contributors. It has attracted more than a hundred stars and is disseminated through a monthly newsletter. Beyond the open-source community, the package has already been used in several missions delivered by Emerton Data, a consulting firm, to evaluate industrialized GenAI systems with limited human annotation budgets.
+
+The statistical framework behind the library is described in a companion methods paper presented at the ICML 2026 workshop on statistical frameworks for uncertainty in agentic systems [@martinon2026industrializing]. Validation notebooks cover all prediction-powered estimators and monitors: each attains its nominal coverage across proxy quality and confidence levels, never yields an interval wider than the human labeled-only baseline, and shows the effective sample size growing with proxy quality. Three reproducible case studies in the documentation run full workflows on public benchmarks, whose proxy-labeled versions we release as documented datasets: agentic safety evaluation on R-Judge [@yuan-etal-2024-r], text-to-SQL accuracy on Spider [@yu-etal-2018-spider], and multilingual retrieval-augmented generation faithfulness on MEMERAG [@cruz-blandon-etal-2025-memerag].


Cette section détaille bien l'impact que Glide a pu avoir. Néanmoins, ce paragraphe qui parle des validation notebooks et des case study qu'on a fait me semble un peu hors sujet.

imerad · 2026-10-12T15:27:42Z

+
+# Code availability
+
+GLIDE can be installed from PyPI with `pip install glide-py`. The source, test suite, scientific validation notebooks and released datasets are on GitHub (https://github.com/EmertonData/glide). The documentation, including tutorials, user guides and case studies, is at https://glide-py.readthedocs.io, and a landing page (https://emertondata.github.io/glide/) links to these resources and hosts an interactive simulator comparing prediction-powered and human-only confidence intervals. Contributions are welcome via issues or pull requests.


La première url dépasse sur sa ligne

imerad · 2026-10-12T15:34:59Z

+@article{martinon2026industrializing,
+  title={Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation},
+  author={Martinon, Gr{\'e}goire and Merad, Ibrahim and Raki, Mohammed},
+  journal={arXiv preprint arXiv:2605.31278},
+  doi={10.48550/arXiv.2605.31278},
+  url={https://arxiv.org/abs/2605.31278},
+  year={2026}
+}


J'ai l'impression qu'on cite la publication d'ICML deux fois, je ne trouve pas celle d'AI4good, la voici

@inproceedings{ merad2026industrializing, title={Industrializing Prediction-Powered Inference: The {GLIDE} Library for Reliable Gen{AI} Evaluation}, author={Ibrahim Merad and Gr{\'e}goire Martinon}, booktitle={Trustworthy AI for Good (AI4GOOD) Workshop @ NeurIPS 2026}, year={2026}, url={https://openreview.net/forum?id=Z51G2eCNIA} }

EDIT : je vois qu'elle y est mais ne la trouvais pas sur la PDF, je recommande d'utiliser celle ci dessus générée par openreview

imerad · 2026-10-12T15:42:02Z

+
+# Summary
+
+Evaluating a generative AI (GenAI) system, for example by measuring its hallucination rate or the share of unsafe agent trajectories, requires reliable labels. Human experts provide them at high cost, while LLM-as-judge proxies are cheap but biased [@NEURIPS2023_91f18a12]. `GLIDE` (Generated Label Inference & Debiasing Engine) is a Python library built on prediction-powered inference [@angelopoulos2023prediction; @angelopoulos2023ppiplusplus]. It combines a small set of human labels with a large set of proxy labels to produce debiased metric estimates whose confidence intervals stay valid however poor the proxy is. Behind a scikit-learn-style API, it offers sampling designs that choose which observations humans should label, and estimators that report the effective sample size, meaning the number of human labels saved by the proxy. It also provides monitors that track a metric over batches with anytime-valid bounds, which control false alarms under repeated evaluation [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].


Je n'aurais pas dit "built on prediction-powered inference", plutôt "which implements PPI statistical methods" ou un truc comme ça

gmartinon-ed added 14 commits September 11, 2026 18:22

draft

b38cafc

draft

2e65651

fix

b046b6f

improve summary

b775002

statement of need revised

9e6583e

state of the field revised

783a39b

software design revision

5a57d23

research impact statement revised

de8aee9

AI usage disclosure and acknowledgements revised

c292f5d

Add code availability section

b1f18d5

Summary and state of the field compressed

a6727ec

further compression

035fd6c

AI usage disclosure compressed

d08735c

finition

f21cee7

imerad requested changes Oct 12, 2026

View reviewed changes

gmartinon-ed added 6 commits October 21, 2026 11:12

feedback from ibrahim

bca90b3

fix biblio

992cfbf

first edit

46dc37b

Merge branch 'main' into joss

9c305f1

fix date

5e6d4e5

clean joss folder

b09e84d


		# Statement of need

		Evaluating a deployed GenAI system under a fixed annotation budget requires combining a few reliable human annotations with a large set of proxy annotations, such as LLM-as-judge labels. The literature has addressed the resulting complications one strand at a time: heterogeneous strata, handled by stratified sampling [@NEURIPS2024_c9fcd02e; @fogliato2024framework]; small labeled sets, handled by bootstrap intervals [@kluger2025prediction]; correlated units, handled by clustered inference; multiple judges, handled by multi-proxy aggregation [@shan2025sada; @cowen2026multiple]; judge uncertainty, exploited by active sampling [@pmlr-v235-zrnic24a; @gligoric2025can]; unequal human and LLM costs, handled by cost-aware sampling [@angelopoulos2025cost]; and continuous monitoring, made safe from false alarms by anytime-valid confidence sequences [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].


		Evaluating a deployed GenAI system under a fixed annotation budget requires combining a few reliable human annotations with a large set of proxy annotations, such as LLM-as-judge labels. The literature has addressed the resulting complications one strand at a time: heterogeneous strata, handled by stratified sampling [@NEURIPS2024_c9fcd02e; @fogliato2024framework]; small labeled sets, handled by bootstrap intervals [@kluger2025prediction]; correlated units, handled by clustered inference; multiple judges, handled by multi-proxy aggregation [@shan2025sada; @cowen2026multiple]; judge uncertainty, exploited by active sampling [@pmlr-v235-zrnic24a; @gligoric2025can]; unequal human and LLM costs, handled by cost-aware sampling [@angelopoulos2025cost]; and continuous monitoring, made safe from false alarms by anytime-valid confidence sequences [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].

		These methods share one template, yet they are scattered across papers with heterogeneous notation and partial reference implementations [@song2026demystifying], some in R and some in Python, none covering more than a slice of the design space. Combining these methods currently means stitching together several repositories and verifying their consistency by hand. `GLIDE` closes that gap by treating sampling, estimation and monitoring as one consistent family, and it quantifies the return on the judge: the effective sample size says how many extra expert annotations the proxy is worth.


		# Software design

		`GLIDE`'s design is organized around three sequential steps inherited from survey theory (\autoref{fig:workflow}): sampling, which selects the observations worth a human label; annotation, which only domain experts can perform; and estimation, which combines labels and proxy predictions into a debiased estimate with a confidence interval. Additionally, monitoring wraps per-batch estimates in an anytime-valid bound. Samplers expose `sample`, estimators `estimate` and monitors `detect`, so a full workflow fits in a handful of lines and a new component plugs in by implementing one of these interfaces. Data follows a simple contract: one aligned array per collection of labels, with `numpy.nan` marking unlabeled entries of the ground-truth array, in the spirit of scikit-learn conventions [@Pedregosa_Scikit-learn_Machine_Learning_2011]. Each estimator family factors its computation into a shared `MeanEstimationEngine`, called once over the whole dataset by an estimator and once per batch by a monitor, so contributors can extend an estimator into its monitor counterpart without reimplementing its statistics. In terms of quality gates, every new component passes three lines of defense, all run in continuous integration: unit tests with 100% coverage and executed docstring examples, functional tests of statistical properties (e.g., a stratified estimator on a single stratum must equal the uniform one), and Monte Carlo validation notebooks assessing coverage, interval width, false-alarm and miscoverage rates.


		`GLIDE` has been developed in public since its first commit in March 2026, with roughly two releases per month and a dozen contributors. It has attracted more than a hundred stars and is disseminated through a monthly newsletter. Beyond the open-source community, the package has already been used in several missions delivered by Emerton Data, a consulting firm, to evaluate industrialized GenAI systems with limited human annotation budgets.

		The statistical framework behind the library is described in a companion methods paper presented at the ICML 2026 workshop on statistical frameworks for uncertainty in agentic systems [@martinon2026industrializing]. Validation notebooks cover all prediction-powered estimators and monitors: each attains its nominal coverage across proxy quality and confidence levels, never yields an interval wider than the human labeled-only baseline, and shows the effective sample size growing with proxy quality. Three reproducible case studies in the documentation run full workflows on public benchmarks, whose proxy-labeled versions we release as documented datasets: agentic safety evaluation on R-Judge [@yuan-etal-2024-r], text-to-SQL accuracy on Spider [@yu-etal-2018-spider], and multilingual retrieval-augmented generation faithfulness on MEMERAG [@cruz-blandon-etal-2025-memerag].


		# Code availability

		GLIDE can be installed from PyPI with `pip install glide-py`. The source, test suite, scientific validation notebooks and released datasets are on GitHub (https://github.com/EmertonData/glide). The documentation, including tutorials, user guides and case studies, is at https://glide-py.readthedocs.io, and a landing page (https://emertondata.github.io/glide/) links to these resources and hosts an interactive simulator comparing prediction-powered and human-only confidence intervals. Contributions are welcome via issues or pull requests.


		# Summary

		Evaluating a generative AI (GenAI) system, for example by measuring its hallucination rate or the share of unsafe agent trajectories, requires reliable labels. Human experts provide them at high cost, while LLM-as-judge proxies are cheap but biased [@NEURIPS2023_91f18a12]. `GLIDE` (Generated Label Inference & Debiasing Engine) is a Python library built on prediction-powered inference [@angelopoulos2023prediction; @angelopoulos2023ppiplusplus]. It combines a small set of human labels with a large set of proxy labels to produce debiased metric estimates whose confidence intervals stay valid however poor the proxy is. Behind a scikit-learn-style API, it offers sampling designs that choose which observations humans should label, and estimators that report the effective sample size, meaning the number of human labels saved by the proxy. It also provides monitors that track a metric over batches with anytime-valid bounds, which control false alarms under repeated evaluation [@podkopaev2021tracking; @waudbysmith2024time; @zhang2026prediction].

Conversation

gmartinon-ed commented Oct 8, 2026

Uh oh!

codecov Bot commented Oct 8, 2026

Codecov Report

Uh oh!

imerad left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants