research(readability): construct-validity investigation for --readability

[joyful-ii-V-I]

This is an investigation, not a behavior change — no src/ changes, docs + bench scripts only.

Two published results motivate this round: LLM judges of code readability lean on surface features rather than the construct itself (CoReEval, arXiv:2510.16579), and prompt-side style constraints on LLM code generation help but plateau (Readability Spectrum, arXiv:2605.13280). Neither is folded into docs/LINEAGE.md as its own row — they are design influences on how we read --readability's claim, not implementations of either paper, so this PR does not invent a row for them. The feature they bear on does have a row: docs/LINEAGE.md L97, Posnett/Hindle/Devanbu, which is where --readability's existing "ORDER, never a grade" disclosure comes from. Full write-up: docs/research/readability-construct-validity.md.

(Base note: measured against main at 755f9026, before 0.6.2; the diff still merges cleanly onto today's main, 15a20855, tag 0.6.2.)

Headline finding, and it is unflattering. We designed a pairwise + rank-correlation validation of the ordering claim (not yet run — needs a blinded human study) and, ahead of that, ran two label-free proxies against the real binary. Refactor-commit direction: 80 refactor/simplify/cleanup commits, 484 matched before/after function pairs, and the lens agrees with the commit's own implied direction only 30.2% of the time — median Δz is negative, i.e. inverted more often than not. Self-consistency under rename/reorder mutations that should not change readability: 95 mutations, 100% exact tie, which the formula guarantees mathematically and so mostly certifies implementation correctness rather than construct validity. We also verified precisely what --readability does and does not feed: it is not one of --quality-delta's ten gating kinds, and the one adjacent kind, verbosity, shares only one of its three formula inputs.

We are not shipping a fix on the strength of one proxy measurement — docs/LINEAGE.md's own withdrawn naming-body-mismatch rule (§9.0 / src/naminglens.h) is used in the doc as the template for what acting on a failed validation would require, and this round stops at "validate first."

This is a first pass with a lot further to push, and the doc closes with four concrete questions for readability researchers: whether the pairwise + rank-correlation design is right, whether the pass/fail bands are calibrated defensibly, whether commit-message mining is too noisy a label for the 30.2% number to stand on without a stricter filter, and whether Posnett 2011 is still the right deterministic formula to be defending in 2026.

Our working premise across this line of investigations: algorithmic, deterministic checks applied while an AI writes are the practical way to keep code sound at the speed AI now writes it — no label, no vendor claim, a check that either fires or it doesn't.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}