Desired outcome
A case that correctly retrieves nothing does not count as recall 0.0, and the dead is not None filter goes away.
Why it matters
Two connected things in app/evaluate.py.
run_case() decides recall consistently for the normal path:
else:
recall = 1.0 # nothing was expected; not retrieving anything is perfectBut the early return for an unresolvable principal never sets recall at all:
if principal is None:
# An unknown user retrieving nothing is correct behaviour, not an error in the eval.
return CaseResult(case_id=case.id, user_id=case.user_id, correct_refusal=not case.should_answer, llm_judged=llm_judged)so it takes the dataclass default of 0.0. The comment says this is correct behaviour, and passed agrees, yet the case is averaged into mean_recall as a total miss. Add one gold case for an unknown user, which is a natural ACL test to write, and the headline recall number drops even though nothing regressed. The docstring calls recall "the metric that quietly dies when you tighten security", so a false drop here is expensive: it looks exactly like the failure mode the harness exists to detect.
Second, mean_recall filters on a condition that cannot be false:
scored = [c.recall for c in self.cases if c.recall is not None]
CaseResult.recall is typed float with default 0.0 and is never assigned None anywhere in the file. The filter reads as if some cases are deliberately excluded from the average, which is misleading given the bug above, where a case genuinely should have been excluded or scored 1.0.
Steps
- In the
principal is None branch, pass recall=1.0 when case.expected_docs is empty and 0.0 otherwise, matching the logic further down. That keeps a genuinely missed case honest.
- Either drop the
is not None filter in mean_recall, or make recall: float | None = None a real "not scored" signal and set it deliberately. Whichever, the type and the filter should agree.
- Add a test: a gold case for a user id that does not exist, with
expected_docs=[], must not drag mean_recall below 1.0.
Claiming this
Comment below to claim it. A reply usually comes within a day.
Desired outcome
A case that correctly retrieves nothing does not count as recall 0.0, and the dead
is not Nonefilter goes away.Why it matters
Two connected things in
app/evaluate.py.run_case()decides recall consistently for the normal path:But the early return for an unresolvable principal never sets
recallat all:so it takes the dataclass default of
0.0. The comment says this is correct behaviour, andpassedagrees, yet the case is averaged intomean_recallas a total miss. Add one gold case for an unknown user, which is a natural ACL test to write, and the headline recall number drops even though nothing regressed. The docstring calls recall "the metric that quietly dies when you tighten security", so a false drop here is expensive: it looks exactly like the failure mode the harness exists to detect.Second,
mean_recallfilters on a condition that cannot be false:CaseResult.recallis typedfloatwith default0.0and is never assignedNoneanywhere in the file. The filter reads as if some cases are deliberately excluded from the average, which is misleading given the bug above, where a case genuinely should have been excluded or scored 1.0.Steps
principal is Nonebranch, passrecall=1.0whencase.expected_docsis empty and0.0otherwise, matching the logic further down. That keeps a genuinely missed case honest.is not Nonefilter inmean_recall, or makerecall: float | None = Nonea real "not scored" signal and set it deliberately. Whichever, the type and the filter should agree.expected_docs=[], must not dragmean_recallbelow 1.0.Claiming this
Comment below to claim it. A reply usually comes within a day.