docs(research): measure --slice line recall on an issue-derived Python corpus

[joyful-ii-V-I]

This runs ripwire's --slice primitive against the question the paper it came from actually asks. --slice is our bounded first rung of the statement-level def-use primitive in ARISE (arXiv:2605.03117), which measured such a primitive at +17 pp Function Recall@1 and argued the binding constraint was the graph's granularity floor rather than its ranking. Its docs/LINEAGE.md row is at L87. docs/EVALS.md has carried the registered contract and, since August, results on four C/C++ corpora; this round asks the harder question on Python, on issue-derived fix patches, with gold taken at the pre-fix tree — the setting an agent is actually in. Full write-up: docs/research/slice-line-recall.md.

The protocol is committed before the numbers exist (results section deliberately empty in the first commit), and the three things the original registration left open — which tree, what counts as gold for a pure insertion, what "surfaces" means when the rows are unordered — are closed as dated amendments rather than decided afterwards. 560 dataset rows, 173 scored, 498 (instance, variable) pairs, deterministic, re-run identical. No src/ changes. (Base note: measured against main at 755f9026, before 0.6.2; the diff still merges cleanly onto today's main, 15a20855, tag 0.6.2.)

What it says, including the parts that do not flatter us.

• The ceiling dominates. 70.8% of the corpus's gold lines are in fixes that span functions, which an intra-procedural slice cannot address at all. Of the reachable remainder, 56.7% of gold lines name a sliceable local — that is what accuracy multiplies against, reported on its own line rather than folded in.
• Accuracy is high and the misses belong to the oracle. Per-variable line recall 0.932 under the registered word-regex relevance oracle, 0.995 under a tokenizer-decided one. All 100 misses were classified rather than sampled; 97 are a name inside a comment, docstring or string. The three that survive are one construct: a keyword-argument name colliding with a local.
• Finer granularity does change retrieval quality, not only presentation — by about 10 points, mostly through the filter rather than the ranking. Given the correct function at a 512-byte budget, whole-function delivery surfaces 17.6% of gold lines, def-use-filtered lines 27.6%, ranked line-level 32.1%. But Recall@1 is 0.048 against a 0.042 chance rate — the primitive re-ranks a shortlist, it does not name the line.
• Our rung 2 is redundant where we unseed it. Unioned over a function's whole inventory, --slice-flow=both reaches no line the flat rows do not — a flow row at depth ≥1 is by construction a line in some other variable's own slice.

This is a first pass with a lot further to push, and the write-up states plainly where we are weakest against the original: name-based occurrences instead of true def-use edges, source lines instead of AST statement nodes, no alias analysis, no lexical scope separation in Python, no inter-procedural tier at all. We would like correction on, first, whether the paper's own gain came from coverage or from precision — our construct loses almost nothing in coverage and pays heavily in precision, and the answer decides whether name-based def-use is worth extending or worth replacing — plus four more questions in the doc's closing section.

Our working premise across this line of investigations: algorithmic, deterministic checks applied while an AI writes are the practical way to keep code sound at the speed AI now writes it — no label, no vendor claim, a check that either fires or it doesn't.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}