research: MULocBench adapter design + pre-registration (no data, no run yet)

[joyful-ii-V-I]

Offers a baseline design for scoring ripwire against MULocBench (arXiv:2509.25242) — a public dataset of 1,100 issues across 46 Python projects with non-code gold locations (config/docs/tests/assets, not just functions), which our existing bench/locbench and bench/multiswe harnesses structurally can't score.

No docs/LINEAGE.md row exists for MULocBench yet, and this PR does not add one. Nothing has been folded into the tool from it — there is no data on disk, no adapter has run, and no number is claimed. The nearest existing kin row is docs/LINEAGE.md L79 (SWE-bench, behind the same bench/ family this adapter would join); a MULocBench row belongs after a real run produces something to fold, not before. Full write-up: docs/research/mulocbench-baseline.md.

(Base note: measured against main at 755f9026, before 0.6.2; the diff still merges cleanly onto today's main, 15a20855, tag 0.6.2.)

What this is. The pre-registration: what an adapter must change versus our existing harnesses (dataset-native gold instead of diff-derived, F1 alongside strict Acc@k, class-level granularity, doc/config/asset/test handling), a verb-mapping table, metric definitions restated in our own words, eligibility/exclusion rules, and what we would report versus withhold — fixed in writing before any row is mined. An untested adapter skeleton (bench/mulocbench/) is included and clearly marked as such; it refuses to run rather than guessing at a schema we have not verified against a real row. The next step is an owner-run fetch, documented in the design doc's own §4.

This is a first pass — design only, nothing measured — with a lot further to push once real rows exist. Six open questions are listed for the paper's authors: the dataset's license (unstated on the HF card, unlike Multi-SWE-bench's explicit CC0), a 1,100-issue vs 3,052-card-row discrepancy we have not reconciled, the F1 partial-credit convention, two field-semantics questions (ass_file_loc, other_rep_loc), and how a deterministic non-LLM tool's numbers should be made legible against the paper's own BM25/LLM-agent baselines. We would welcome correction on any of this before we run it for real.

Our working premise across this line of investigations: algorithmic, deterministic checks applied while an AI writes are the practical way to keep code sound at the speed AI now writes it — no label, no vendor claim, a check that either fires or it doesn't.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}