docs(research): reading path into --quality-delta + a harness design for gate-vs-degradation

[joyful-ii-V-I]

Responds to Nakka, Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the Paradox (IEEE-ISTAS 2025, arXiv:2506.11022), which finds security getting worse, not better, across iterations of an LLM improving its own code.

This work is not filed as a docs/LINEAGE.md row, and this PR does not add one — --quality-delta's external/deterministic-gate design predates reading this paper, so it would misrepresent the lineage convention (a row records a lesson actually taken from a cited work). The actual design record is src/quality.h's own top-of-file comment, on main at src/quality.h#L7-L20, which the note below points to instead. Full write-up: docs/research/gate-vs-iterative-degradation.md.

This is a first pass, not a finished study, and says so throughout. No src/ changes. (Base note: measured against main at 755f9026, before 0.6.2; the diff still merges cleanly onto today's main, 15a20855, tag 0.6.2.)

Two parts, one of them an unflattering measurement of our own tool.

1. A reading path into --quality-delta for an outside researcher — the ten kinds and where each is computed, the baseline model, the ack ledger, the eval harness, and an honest restatement of our own backtest: --quality-delta behaves as a debt ratchet (58% recall / 40% false-alarm rate at a realistic 5-commit window), not a defect detector. That is already true of the tool's own documentation ("report only what got worse") — this is the first time it has been measured rather than asserted, and the note says so plainly rather than only citing the flattering half.
2. A concrete harness design for the actual question — does a deterministic external gate change the trajectory of repeated self-improvement, not just catch individual regressions — with four pre-registered instruments, two controls for the "the gate also adds prompt length" confound, and an explicit section on what result would count against our own position, including the Goodhart-one-level-down failure mode: the model gaming the gate's visible bar while sub-bar debt still accumulates.

bench/degradeloop/ has the scaffolding — deterministic bookkeeping and measurement calls, with the one seam that needs a real model API left as an explicit NotImplementedError rather than a faked call site. No model was run to produce this; no numbers here are results, and we cannot run them here — that is stated plainly in the doc rather than worked around.

This is a first pass with a lot further to push. We would like help with: a vetted seed-task set, a model-call adapter, a pinned per-language security scanner, and a second opinion on whether the wrong-target confound control (arm D) earns its cost over the neutral-text control alone. Full list at the end of the doc.

Our working premise across this line of investigations: algorithmic, deterministic checks applied while an AI writes are the practical way to keep code sound at the speed AI now writes it — no label, no vendor claim, a check that either fires or it doesn't.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}