Measure how often two voters of an LLM judge panel or voter ensemble pick the same answer when both break from the plurality, report Kish's effective number of votes by raw agreement, and correct the vote for voters that do, naming the panel shape where it cannot. Closed-form, label-free, numpy only.
evaluation calibration ensemble self-consistency conditional-independence majority-voting effective-sample-size overconfidence llm llm-evaluation llm-as-judge llm-as-a-judge judge-panel correlated-errors design-effect error-correlation dependence-aware-aggregation intra-model-bias agreement-confidence cross-family-correlation
-
Updated
Nov 22, 2026 - Python