docs(research): is --for's ranking-confidence calibrated enough to abstain on? A first measurement, and what we'd need to go further

[joyful-ii-V-I]

Agent Retrieval Bench (arXiv:2607.24882) reports that across four retrieval families no retriever tells its caller when its own ranking should not be trusted, and names abstention as the unsolved axis. Its docs/LINEAGE.md row is at L89 — we already shipped the disclosure half: every --for root carries confidence=/margin_pct=, derived from the same relevance-cliff statistic --adaptive cuts on, so the two can never disagree. Full write-up: docs/research/confidence-and-abstention.md.

Disclosure is a first step, not a solution. This branch is the follow-up: is the signal we ship actually calibrated, and what would it take to go further than warning? It changes no code in src/ and adds no gate. (Base note: measured against main at 755f9026, before 0.6.2; the diff still merges cleanly onto today's main, 15a20855, tag 0.6.2.)

Headline finding. Two pre-registered docs/EVALS.md rounds already scored this signal against ARB's unanswerable-query splits and both closed as recorded negatives — they asked does the signal know there is no answer. A CLI that must answer faces a different question: does it know its own answer is wrong? Measured on a held-out LocBench slice (92 instances, repository-disjoint split), the signal is directionally right and weakly discriminating, and unusable at its only operating point: it separates function-grain correctness (+0.375 [+0.132, +0.513], AUROC 0.622) but not file-grain (+0.134 [−0.081, +0.247]) and not the bundle as a whole (+0.012). Warning iff confidence="low" fires on 74 of 92 answers and costs 60 false warnings to catch 14 misses. Structurally, margin_pct= is 0 for every "low" row, so there is no threshold to tune — only a fact to add, which reaches docs/EVALS.md round 2's conclusion from the other direction.

What else is in the document. The four ways a CLI could spell "abstain," each with its cost against this repository's honesty contract; a downstream agent experiment we cannot run here, designed in full with the power arithmetic and the results that would make us not ship; and a pre-registration with bands, a directional-refutation rung, and explicit self-reject rules, including a fire-rate ceiling today's signal fails by 3×.

This is a first pass with a lot further to push, and two places we would like outside help: calibration methodology — --for returns a set, "correct" has at least four grains and they disagree, so which grain should a single confidence attribute be a claim about, and is a two-valued label calibratable at all; and the abstention criterion — we pre-registered a 0.20 false-warn ceiling and can defend the direction but not the value, and we do not know of any measured evidence that agent behavior responds to a retrieval-confidence signal at all.

Our working premise across this line of investigations: algorithmic, deterministic checks applied while an AI writes are the practical way to keep code sound at the speed AI now writes it — no label, no vendor claim, a check that either fires or it doesn't.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}