From d21807c795ed3274bf8833667f1969ef38fcce2a Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Sun, 20 Sep 2026 14:36:04 -0400 Subject: [PATCH 1/3] docs(research): a reading path into --quality-delta, and a harness design for gate-vs-degradation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Responds to Nakka 2025 (arXiv:2506.11022, IEEE-ISTAS), which measures security getting worse across iterations of an LLM improving its own code. Part 1 is a reading path for an outside researcher into --quality-delta: the ten kinds and where each is computed, the two-floor baseline model, the no-op-diff fix from this round (t12-qd-noop), the ack ledger, the eval harness, and our own honest backtest (a debt ratchet, not a detector: 14/24 recall vs 12/30 control at a 5-commit window). Part 2 designs the actual experiment — a gated vs ungated iterative-improvement loop, four pre-registered instruments, the prompt-length confound and two controls for it, and what result would count against our own position. This is a design and untested scaffolding, not a result: no model calls were made, and bench/degradeloop's call_model() seam raises NotImplementedError on purpose rather than faking a call site. The position (a deterministic external gate as a Goodhart defense) is recorded where it has lived since before this note — src/quality.h's own design comment — and stays there rather than being backdated into docs/LINEAGE.md, which is reserved for a lesson actually taken from a work, not a hypothesis this note argues from. No src/ changes. Co-Authored-By: Claude Sonnet 5 --- bench/degradeloop/README.md | 49 +++ bench/degradeloop/analyze_trajectory.py | 250 ++++++++++++ bench/degradeloop/measure.py | 171 ++++++++ bench/degradeloop/run_degradeloop.py | 206 ++++++++++ .../research/gate-vs-iterative-degradation.md | 383 ++++++++++++++++++ 5 files changed, 1059 insertions(+) create mode 100644 bench/degradeloop/README.md create mode 100644 bench/degradeloop/analyze_trajectory.py create mode 100644 bench/degradeloop/measure.py create mode 100644 bench/degradeloop/run_degradeloop.py create mode 100644 docs/research/gate-vs-iterative-degradation.md diff --git a/bench/degradeloop/README.md b/bench/degradeloop/README.md new file mode 100644 index 000000000..57dd0fe6f --- /dev/null +++ b/bench/degradeloop/README.md @@ -0,0 +1,49 @@ +# bench/degradeloop — scaffolding for the gate-vs-degradation harness + +**UNTESTED. Nothing in this directory has been run.** These scripts were written alongside +[`docs/research/gate-vs-iterative-degradation.md`](../../docs/research/gate-vs-iterative-degradation.md) +(read that first — it's the design these scripts implement) as a starting point for whoever runs the +actual experiment. No model API is wired in anywhere; the one seam that needs one +(`call_model()` in `run_degradeloop.py`) raises `NotImplementedError` on purpose rather than faking a +call site that looks functional. Every number these scripts *could* produce would be synthetic until +that seam is filled in and the scripts are actually exercised against a real model and a real +`build/ripwire` binary. + +## What's here + +- `run_degradeloop.py` — drives one arm (`ungated` / `gated` / `neutral-control` / `wrong-target`, + see the design doc §2.1/§2.4) of the loop for one seed task: calls the model seam each iteration, + commits the result to a scratch git repo, and calls `measure.py`'s functions to snapshot it. Writes + one JSONL trajectory file per (seed task, arm) pair. +- `measure.py` — the deterministic measurement layer. Wraps `ripwire --quality-delta --json` and + `ripwire --test-gate --json` as subprocess calls against a scratch working tree, plus a stub for + the public security-scanner call (`run_security_scanner()`, also unimplemented — pick and pin a + scanner per §2.2 of the design doc before filling this in). Parses only what the design doc's + §2.2/§2.3 instruments need; does not attempt to be a general ripwire-output parser. +- `analyze_trajectory.py` — reads the JSONL trajectory files `run_degradeloop.py` writes (for however + many arms and seed tasks are on disk) and computes the four pre-registered instruments from the + design doc's §2.3: cumulative-regression slope (paired Wilcoxon across seed tasks), terminal-state + comparison, sub-bar growth rate, and — if a scanner ran — the security-finding trajectory. Refuses + to report an instrument it cannot compute rather than silently omitting it from a table (a design + doc's own house rule, `src/quality.h`'s honesty framing applied here to our own output). + +## Before running any of this + +1. Fill in `call_model()` in `run_degradeloop.py` against whatever model access you have. It takes + `(prompt: str, current_code: str) -> str` and returns the new code — that's the whole contract. +2. Pick and pin a security scanner per seed-task language in `measure.py::run_security_scanner()` + (a version and ruleset hash, not just a tool name — see the design doc's §2.2 caveat on this). +3. Assemble a seed-task set (see the design doc's "what we'd like help with" — we don't have one). +4. Point `RIPWIRE_BIN` (env var, read by `measure.py`) at a real `build/ripwire` built from this + repository's `CLAUDE.md` build instructions. `measure.py` refuses to run against a bare `ripwire` + on `PATH` without this being set explicitly, so a stale system install can't silently produce + numbers. + +## Determinism note + +`measure.py`'s ripwire calls are deterministic (same binary, same tree → same output, per this +project's own determinism contract). The loop as a whole is **not** deterministic end to end, because +`call_model()` isn't — that's expected and is not something these scripts try to paper over. Record +the model, its version/date, and the temperature/sampling settings used alongside every trajectory +file; `run_degradeloop.py`'s JSONL header line has fields for exactly this and refuses to run without +them filled in. diff --git a/bench/degradeloop/analyze_trajectory.py b/bench/degradeloop/analyze_trajectory.py new file mode 100644 index 000000000..58c8e82db --- /dev/null +++ b/bench/degradeloop/analyze_trajectory.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +"""analyze_trajectory.py — the four pre-registered instruments from +docs/research/gate-vs-iterative-degradation.md §2.3, computed over whatever trajectory JSONL +files run_degradeloop.py has produced. + +SCAFFOLDING, NOT VALIDATED AGAINST REAL DATA. The statistics below (paired slope comparison, +Wilcoxon signed-rank) are implemented from scratch, zero-dependency, matching this project's own +"no host-installed dependencies" posture (CLAUDE.md G3) rather than reaching for scipy — they have +been sanity-checked against small synthetic inputs (see the __main__ self-test at the bottom, +runnable with `python3 analyze_trajectory.py --selftest`) but NOT against a real trajectory run, +because none exists yet. Do not cite a p-value out of this file in a publishable report without an +independent check against a reference implementation once real data exists. + +INPUT FORMAT. One or more JSONL files as written by run_degradeloop.py: a header row +(kind="degradeloop-trajectory-header") followed by one row per iteration. This script does not +assume a particular per-iteration row schema beyond what each instrument function documents it +reads — see each function's docstring for the exact fields it expects, matching what measure.py's +snapshot dataclasses would serialize. + +REFUSAL, NOT SILENT OMISSION. Per the design doc's §2.3 rule ("the write-up reports … all four +instruments … not whichever one came out favorable"), report_all() below always emits a row for +every instrument, marking it unavailable with a stated reason when the data can't support it, +rather than dropping it from the table. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +# --------------------------------------------------------------------------- +# Minimal zero-dependency statistics (see module docstring on why not scipy) +# --------------------------------------------------------------------------- + +def _ranks( values: list[float] ) -> list[float]: + """Average ranks, ties split evenly — the standard Wilcoxon tie-handling.""" + order = sorted( range( len( values ) ), key=lambda i: values[ i ] ) + ranks = [ 0.0 ] * len( values ) + i = 0 + while i < len( order ): + j = i + while j + 1 < len( order ) and values[ order[ j + 1 ] ] == values[ order[ i ] ]: + j += 1 + avg_rank = ( i + j ) / 2.0 + 1.0 + for k in range( i, j + 1 ): + ranks[ order[ k ] ] = avg_rank + i = j + 1 + return ranks + + +def wilcoxon_signed_rank( a: list[float], b: list[float] ) -> dict: + """Paired Wilcoxon signed-rank test, a vs b (a - b), normal approximation (no exact tables — + fine for the N this design expects to run with being small enough that the approximation's own + documented weakness at very small N should be flagged, which this function does via `note` + rather than silently reporting a p-value the sample size doesn't support). + + Returns W (the smaller of W+/W-), z (normal-approximation statistic), and a two-sided p + computed from a standard-normal survival approximation (erf-based, stdlib only). + """ + diffs = [ x - y for x, y in zip( a, b ) if x != y ] # zero-diffs dropped, standard convention + n = len( diffs ) + if n < 6: + return { + "n_nonzero": n, "W": None, "z": None, "p_two_sided": None, + "note": "n < 6 after dropping zero-diffs — normal approximation is not trustworthy here; " + "report the raw paired differences instead of a p-value.", + } + abs_diffs = [ abs( d ) for d in diffs ] + ranks = _ranks( abs_diffs ) + w_pos = sum( r for r, d in zip( ranks, diffs ) if d > 0 ) + w_neg = sum( r for r, d in zip( ranks, diffs ) if d < 0 ) + w = min( w_pos, w_neg ) + mean_w = n * ( n + 1 ) / 4.0 + std_w = ( n * ( n + 1 ) * ( 2 * n + 1 ) / 24.0 ) ** 0.5 + z = ( w - mean_w ) / std_w if std_w > 0 else 0.0 + p = _two_sided_normal_p( z ) + return { "n_nonzero": n, "W": w, "z": z, "p_two_sided": p, "note": None } + + +def _erf( x: float ) -> float: + # Abramowitz & Stegun 7.1.26 approximation — stdlib-only, adequate for a two-sided p-value + # at the precision this design needs (a pre-registered effect-size floor, not a tight p-value). + sign = 1 if x >= 0 else -1 + x = abs( x ) + a1, a2, a3, a4, a5 = 0.254829592, -0.284496736, 1.421413741, -1.453152027, 1.061405429 + p = 0.3275911 + t = 1.0 / ( 1.0 + p * x ) + y = 1.0 - ( ( ( ( ( a5 * t + a4 ) * t ) + a3 ) * t + a2 ) * t + a1 ) * t * pow( 2.718281828459045, -x * x ) + return sign * y + + +def _two_sided_normal_p( z: float ) -> float: + return 2.0 * ( 1.0 - 0.5 * ( 1.0 + _erf( abs( z ) / ( 2 ** 0.5 ) ) ) ) + + +def _slope( ys: list[float] ) -> float: + """OLS slope of ys against iteration index 0..n-1. Plain least squares, stdlib only.""" + n = len( ys ) + if n < 2: + return 0.0 + xs = list( range( n ) ) + mean_x = sum( xs ) / n + mean_y = sum( ys ) / n + num = sum( ( x - mean_x ) * ( y - mean_y ) for x, y in zip( xs, ys ) ) + den = sum( ( x - mean_x ) ** 2 for x in xs ) + return num / den if den else 0.0 + + +# --------------------------------------------------------------------------- +# Trajectory loading +# --------------------------------------------------------------------------- + +def load_trajectory( path: Path ) -> tuple[dict, list[dict]]: + lines = path.read_text().splitlines() + if not lines: + raise ValueError( f"{path}: empty trajectory file" ) + header = json.loads( lines[ 0 ] ) + if header.get( "kind" ) != "degradeloop-trajectory-header": + raise ValueError( f"{path}: first line is not a degradeloop-trajectory-header row" ) + rows = [ json.loads( l ) for l in lines[ 1: ] ] + return header, rows + + +# --------------------------------------------------------------------------- +# The four instruments (design doc §2.3) +# --------------------------------------------------------------------------- + +def instrument_1_cumulative_regression_slope( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Paired by seed task (same index in both lists = same seed task). Expects each row to carry + 'regressions' (int, from measure.QualityDeltaSnapshot.regressions, baseline-anchored per the + design doc §2.2's instruction to measure against state[0], never state[i-1]).""" + if not gated_trajs or len( gated_trajs ) != len( ungated_trajs ): + return { "available": False, "reason": "need equal, nonzero paired seed-task counts for both arms" } + gated_slopes = [ _slope( [ r[ "regressions" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "regressions" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) # ungated - gated: positive means gate reduced slope + return { + "available": True, + "gated_slopes": gated_slopes, + "ungated_slopes": ungated_slopes, + "paired_test": test, + "interpretation": "test is on (ungated_slope - gated_slope); z > 0 means ungated slopes ranked " + "higher (gate associated with a LOWER regression slope), z < 0 the opposite. " + "Check the sign of z, not just p_two_sided, before claiming a direction.", + } + + +def instrument_2_terminal_state( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 2 — the weaker, paper-shaped claim (end state only).""" + if not gated_trajs or len( gated_trajs ) != len( ungated_trajs ): + return { "available": False, "reason": "need equal, nonzero paired seed-task counts for both arms" } + gated_terminal = [ traj[ -1 ][ "regressions" ] for traj in gated_trajs if traj ] + ungated_terminal = [ traj[ -1 ][ "regressions" ] for traj in ungated_trajs if traj ] + if len( gated_terminal ) != len( gated_trajs ): + return { "available": False, "reason": "at least one trajectory had zero iterations" } + test = wilcoxon_signed_rank( ungated_terminal, gated_terminal ) + return { "available": True, "gated_terminal": gated_terminal, "ungated_terminal": ungated_terminal, "paired_test": test } + + +def instrument_3_subbar_growth( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 3 — expects each row to carry 'subbar_growth_total' (a float/int + the row-producer computed via measure.measure_subbar_growth, itself UNIMPLEMENTED in + measure.py). Refuses rather than guessing when the field is absent, per this project's own + honesty-in-output convention (src/quality.h: "a zero means none found, never none exists").""" + if not gated_trajs or not ungated_trajs: + return { "available": False, "reason": "no trajectories supplied" } + if any( "subbar_growth_total" not in r for traj in ( gated_trajs + ungated_trajs ) for r in traj ): + return { + "available": False, + "reason": "rows are missing subbar_growth_total — measure.measure_subbar_growth is " + "unimplemented in this scaffolding (see measure.py); this is NOT the same as " + "the instrument measuring zero growth.", + } + gated_slopes = [ _slope( [ r[ "subbar_growth_total" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "subbar_growth_total" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) + return { "available": True, "gated_slopes": gated_slopes, "ungated_slopes": ungated_slopes, "paired_test": test } + + +def instrument_4_security_trajectory( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 4 — expects 'security_finding_count' per row, from + measure.run_security_scanner (UNIMPLEMENTED in measure.py until a scanner is pinned).""" + if not gated_trajs or not ungated_trajs: + return { "available": False, "reason": "no trajectories supplied" } + if any( "security_finding_count" not in r for traj in ( gated_trajs + ungated_trajs ) for r in traj ): + return { + "available": False, + "reason": "rows are missing security_finding_count — no scanner is pinned yet " + "(measure.run_security_scanner is unimplemented); see README.md.", + } + gated_slopes = [ _slope( [ r[ "security_finding_count" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "security_finding_count" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) + return { "available": True, "gated_slopes": gated_slopes, "ungated_slopes": ungated_slopes, "paired_test": test } + + +def report_all( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Always emits all four instruments — an unavailable one is reported with its reason, never + dropped from the table. See design doc §2.3's selective-reporting warning.""" + return { + "instrument_1_cumulative_regression_slope": instrument_1_cumulative_regression_slope( gated_trajs, ungated_trajs ), + "instrument_2_terminal_state": instrument_2_terminal_state( gated_trajs, ungated_trajs ), + "instrument_3_subbar_growth": instrument_3_subbar_growth( gated_trajs, ungated_trajs ), + "instrument_4_security_trajectory": instrument_4_security_trajectory( gated_trajs, ungated_trajs ), + } + + +def main() -> int: + ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) + ap.add_argument( "--gated", nargs="+", type=Path, default=[], help="trajectory JSONL files, arm=gated, one per seed task" ) + ap.add_argument( "--ungated", nargs="+", type=Path, default=[], help="trajectory JSONL files, arm=ungated, one per seed task, SAME ORDER as --gated" ) + ap.add_argument( "--selftest", action="store_true", help="run the built-in synthetic sanity check and exit" ) + args = ap.parse_args() + + if args.selftest: + return _selftest() + + if not args.gated or not args.ungated: + print( "error: --gated and --ungated each need at least one trajectory file (or pass --selftest)", file=sys.stderr ) + return 1 + + gated_trajs = [ load_trajectory( p )[ 1 ] for p in args.gated ] + ungated_trajs = [ load_trajectory( p )[ 1 ] for p in args.ungated ] + print( json.dumps( report_all( gated_trajs, ungated_trajs ), indent=2 ) ) + return 0 + + +def _selftest() -> int: + """Sanity check ONLY — synthetic data, not a claim about anything real. Checks that (a) a + clearly-lower-slope gated arm is detected as such in instrument 1, and (b) an + all-fields-missing input correctly reports unavailable for instruments 3/4 rather than + crashing or silently defaulting to zero.""" + gated = [ [ { "regressions": r } for r in [ 0, 1, 1, 2, 2 ] ] for _ in range( 8 ) ] + ungated = [ [ { "regressions": r } for r in [ 0, 2, 4, 6, 8 ] ] for _ in range( 8 ) ] + report = report_all( gated, ungated ) + i1 = report[ "instrument_1_cumulative_regression_slope" ] + assert i1[ "available" ], i1 + assert all( g < u for g, u in zip( i1[ "gated_slopes" ], i1[ "ungated_slopes" ] ) ), i1 + i3 = report[ "instrument_3_subbar_growth" ] + assert i3[ "available" ] is False and "subbar_growth_total" in i3[ "reason" ], i3 + print( "selftest OK (synthetic data only — not a real result)", file=sys.stderr ) + print( json.dumps( report, indent=2 ) ) + return 0 + + +if __name__ == "__main__": + sys.exit( main() ) diff --git a/bench/degradeloop/measure.py b/bench/degradeloop/measure.py new file mode 100644 index 000000000..c791a4a25 --- /dev/null +++ b/bench/degradeloop/measure.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""measure.py — deterministic per-iteration measurement layer for bench/degradeloop. + +UNTESTED. Written alongside docs/research/gate-vs-iterative-degradation.md as scaffolding for +whoever runs the actual harness; no function in this file has been exercised against a real +ripwire binary or a real code tree. Read the design doc (../../docs/research/ +gate-vs-iterative-degradation.md, sections 2.2 and 2.3) before changing what gets measured — +these functions exist to implement that section, not the other way around. + +WHAT THIS DOES. Wraps two ripwire verbs as subprocess calls against a scratch working tree +(--quality-delta --json, --test-gate --json), both of which docs/COMMANDS.md's --json entry +confirms are on the JSON allow-list with keys mirroring the XML attribute names 1:1. Also holds +an unimplemented seam for a public security scanner (run_security_scanner) — deliberately not +guessed at here; see the design doc's "what we'd like help with" for why the choice is left open. + +WHAT THIS DOES NOT DO. It does not interpret a --quality-delta exit code as "a real defect was +introduced." The design doc's Part 1 section 7 backtests --quality-delta against this project's +own commit history and finds it behaves as a debt ratchet (58% recall vs 40% false-alarm rate at +a 5-commit window) rather than a defect detector. Every function here reports counts, not +verdicts, and the analysis layer (analyze_trajectory.py) is responsible for any claim built on +top of them. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from dataclasses import dataclass, field + + +def _resolve_ripwire_bin() -> str: + """Refuse to fall back to a bare 'ripwire' on PATH — a stale system install would silently + produce numbers from a different binary than the one the run's metadata claims. RIPWIRE_BIN + must be set explicitly to a real build/ripwire; see README.md "before running any of this".""" + bin_path = os.environ.get( "RIPWIRE_BIN" ) + if not bin_path: + raise RuntimeError( + "RIPWIRE_BIN is not set. Point it at a build/ripwire built from this repository's " + "CLAUDE.md instructions — measure.py will not guess at a PATH install." + ) + if not os.path.isfile( bin_path ) or not os.access( bin_path, os.X_OK ): + raise RuntimeError( f"RIPWIRE_BIN={bin_path!r} is not an executable file." ) + return bin_path + + +def _run_ripwire_json( tree_dir: str, *verb_args: str ) -> dict: + """Run one ripwire verb with --json against tree_dir and parse the result. Raises on a + nonzero-but-unexpected exit rather than silently returning an empty dict — --quality-delta and + --test-gate both use exit codes as part of their contract (exit 2 / exit 4 respectively are + expected "gating" outcomes, not errors), so this only raises on exit codes neither verb + documents (see docs/COMMANDS.md for the documented exit-code contract of each verb before + changing the accepted set below).""" + binp = _resolve_ripwire_bin() + cmd = [ binp, tree_dir, *verb_args, "--json" ] + proc = subprocess.run( cmd, capture_output=True, text=True ) + # --quality-delta: exit 0 (clean) or exit 2 (gating regressions). --test-gate: exit 0 or exit 4. + if proc.returncode not in ( 0, 2, 4 ): + raise RuntimeError( + f"unexpected exit {proc.returncode} from {cmd!r}\nstdout={proc.stdout!r}\nstderr={proc.stderr!r}" + ) + if not proc.stdout.strip(): + # An empty stdout on a documented-refusal exit path (see --json's ALLOW-list refusal + # shape in docs/COMMANDS.md) is a real outcome, not a parse failure — but the caller needs + # to know, so this is surfaced rather than coerced into {}. + raise RuntimeError( f"empty stdout from {cmd!r} (exit {proc.returncode}); stderr={proc.stderr!r}" ) + return json.loads( proc.stdout ) + + +@dataclass +class QualityDeltaSnapshot: + """One --quality-delta --json reading, trimmed to the fields the design doc's instruments + need. Field names mirror the XML attribute spellings from docs/COMMANDS.md's --quality-delta + section verbatim (including the hyphens, carried as dict-style access below) rather than + renaming them into Python convention — a mismatch between this dataclass and a future + --quality-delta output change should be loud, not silently absorbed by a renamed field.""" + + regressions: int + gating: int + minor: int + acked: int + per_kind_gating: dict = field( default_factory=dict ) # kind -> count, computed from the row list + raw: dict = field( default_factory=dict ) # the full parsed JSON, for anything not modeled above + + +def measure_quality_delta( tree_dir: str, baseline_ref: str | None = None ) -> QualityDeltaSnapshot: + """One snapshot of the ten kinds against tree_dir. + + baseline_ref, if given, is passed as --quality-delta=..HEAD (the ref-pair form) so the + design doc's section 2.2 instruction — measure against the FIXED seed-state baseline, not + against the previous iteration — can be honored without needing a .ripwire_quality_baseline + sidecar dance on every iteration. If None, measures the bare working-tree-vs-HEAD form. + """ + args = [ f"--quality-delta={baseline_ref}..HEAD" ] if baseline_ref else [ "--quality-delta" ] + data = _run_ripwire_json( tree_dir, *args ) + # The JSON root mirrors the XML root's attributes; per-kind breakdown rides on the row list, + # not the root, so it's folded here rather than assumed present at the top level. + per_kind: dict = {} + for row in data.get( "regressions_detail", data.get( "rows", [] ) ): + k = row.get( "kind" ) + if k: + per_kind[ k ] = per_kind.get( k, 0 ) + 1 + return QualityDeltaSnapshot( + regressions=int( data.get( "regressions", 0 ) ), + gating=int( data.get( "gating", 0 ) ), + minor=int( data.get( "minor", 0 ) ), + acked=int( data.get( "acked", 0 ) ), + per_kind_gating=per_kind, + raw=data, + ) + + +@dataclass +class TestGateSnapshot: + tests: int + untested: int + raw: dict = field( default_factory=dict ) + + +def measure_test_gate( tree_dir: str ) -> TestGateSnapshot: + """--test-gate --json against the working tree (default = git diff, per docs/COMMANDS.md).""" + data = _run_ripwire_json( tree_dir, "--test-gate" ) + return TestGateSnapshot( + tests=int( data.get( "tests", 0 ) ), + untested=int( data.get( "untested", 0 ) ), + raw=data, + ) + + +def measure_subbar_growth( tree_dir: str, baseline_ref: str ) -> dict: + """The design doc's §2.2/§2.3 instrument 3: raw ccx/LOC/nest/params growth per touched symbol, + regardless of whether any single step crossed a --quality-delta bar. NOT part of + --quality-delta's own reported kinds (which only fire on a bar crossing or, above the bar, a + material-growth threshold — see src/quality.h's kMaterialGrowthPct/kSubBarGrowthPct) — this is + exactly the accumulation a bar-gated report can be blind to, which is why the design doc treats + it as a separate instrument. + + UNIMPLEMENTED. Computing this needs the per-symbol raw metrics for both trees, which + --quality-delta's own JSON does not expose below its minor/major-severity rows (deliberately — + it reports regressions, not a full metrics dump). The two ways to get it: (a) --metrics --json + on both trees and a symbol-identity join done here, matching quality.h's own canonId scheme + (path::scope::name — see docs/COMMANDS.md's --quality-delta LIMIT clause on renamed/moved + symbols), or (b) a small ripwire patch that exposes sub-bar deltas directly. Left unimplemented + rather than approximated, because an approximated version of exactly the instrument meant to + catch quiet accumulation would be worth less than nothing. + """ + raise NotImplementedError( + "sub-bar growth needs a --metrics-based symbol join (or a ripwire-side addition) that " + "this scaffolding does not implement — see the docstring above before filling this in." + ) + + +def run_security_scanner( tree_dir: str, language: str ) -> dict: + """UNIMPLEMENTED — placeholder for the public, deterministic security scanner named in the + design doc's §2.2 ("Semgrep's default ruleset, or a language-appropriate equivalent"). Left + unimplemented because the choice needs pinning (tool version + ruleset hash) per language + before it means anything as an instrument — see docs/research/gate-vs-iterative-degradation.md + "what we would like help with". A caller that needs this today should pin its own scanner + invocation here rather than treating this function's absence as "no scanner was run" silently: + call sites MUST check for NotImplementedError and record scanner_ran=false in the trajectory + row rather than defaulting a finding count to 0 (a 0 here must never be mistaken for "clean"). + """ + raise NotImplementedError( + f"no security scanner pinned yet for language={language!r} — see README.md before running." + ) + + +if __name__ == "__main__": + print( __doc__, file=sys.stderr ) + sys.exit( 1 ) diff --git a/bench/degradeloop/run_degradeloop.py b/bench/degradeloop/run_degradeloop.py new file mode 100644 index 000000000..71d06d44e --- /dev/null +++ b/bench/degradeloop/run_degradeloop.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""run_degradeloop.py — orchestrates one arm of the iterative-improvement loop for one seed task. + +UNTESTED. Scaffolding for docs/research/gate-vs-iterative-degradation.md §2.1/§2.4. Nobody has +run this — call_model() below raises NotImplementedError on purpose rather than faking a call +site that looks wired up. Filling that one function in is the only thing standing between this +script and a real run; everything else (git bookkeeping, measurement calls, trajectory recording) +is written to work once it is. + +USAGE (once call_model() is implemented and RIPWIRE_BIN is set, per README.md): + python3 run_degradeloop.py \\ + --seed-dir /path/to/seed/task/checkout \\ + --arm gated \\ + --iterations 10 \\ + --model-name "" --model-version "" \\ + --out trajectory_seed1_gated.jsonl + +Arms (design doc §2.1/§2.4): ungated | gated | neutral-control | wrong-target. "wrong-target" +needs --wrong-target-report pointing at a previously recorded --quality-delta JSON report from a +DIFFERENT seed task, per the design doc's arm D. +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +import measure + + +IMPROVE_PROMPT = "Improve this code." # deliberately the paper's own minimal prompt — see design doc §2.1 + + +def call_model( prompt: str, current_code: str, *, model_name: str, model_version: str ) -> str: + """THE seam. Takes a prompt and the current state of the seed task's code, returns the model's + proposed new code. This is the only function in bench/degradeloop that this scaffolding does + not and cannot implement — it needs real model access, which this environment does not have + (see the design doc §2.6: "no model calls were made to produce this document"). + + Contract for whoever fills this in: (1) deterministic sampling settings (fixed seed/temperature + if the API supports it) recorded in the trajectory header, not just in code comments here — + see main()'s header-writing below; (2) the SAME model_name/model_version across every arm and + every iteration of one seed task's comparison, or the arms are not comparable; (3) return the + FULL new code, not a diff — the measurement layer re-ingests the whole tree each iteration. + """ + raise NotImplementedError( + "call_model() is the one unimplemented seam in this harness — wire it to a real model " + "before running. See this function's docstring and README.md." + ) + + +def build_prompt( arm: str, base_prompt: str, previous_report: dict | None, wrong_target_report: dict | None ) -> str: + """Assemble the per-iteration prompt for the given arm. See design doc §2.1 (loop shape) and + §2.4 (the neutral-text / wrong-target confound controls) for why each arm's prompt looks the + way it does — this function is the literal implementation of that section, not a paraphrase of + it, so a change here should be a change there too.""" + if arm == "ungated": + return base_prompt + if arm == "gated": + if previous_report is None: + return base_prompt # iteration 1 has no prior report yet + return ( + base_prompt + + "\n\nThe deterministic quality gate reports the following about your PREVIOUS change " + "to this code. Address what you can before proposing the next change:\n\n" + + json.dumps( previous_report, indent=2 ) + ) + if arm == "neutral-control": + # Matched-LENGTH neutral text, not matched-content — see design doc §2.4 arm C. Using the + # tool's own --legend text (long, dense, describes the checker rather than this code) as + # the filler is the design doc's suggestion; swap in whatever's pinned for the actual run. + filler = _neutral_filler_text() + return base_prompt + "\n\n" + filler + if arm == "wrong-target": + if wrong_target_report is None: + raise ValueError( "arm=wrong-target requires --wrong-target-report" ) + return ( + base_prompt + + "\n\nA deterministic quality gate reports the following (NOTE: about a DIFFERENT " + "codebase, used here only as a matched-content control — see design doc §2.4 arm D):\n\n" + + json.dumps( wrong_target_report, indent=2 ) + ) + raise ValueError( f"unknown arm {arm!r}" ) + + +def _neutral_filler_text() -> str: + """Returns the matched-length filler text for the neutral-control arm. Placeholder: a real run + should call `ripwire --quality-delta --legend` (or any fixed, task-irrelevant, roughly + report-length text) ONCE per run and reuse it, rather than regenerating it per iteration — the + text must be IDENTICAL across iterations and across seed tasks within the control arm, or the + control stops controlling for length and starts introducing its own variable content.""" + raise NotImplementedError( + "pin a fixed neutral-text block before running the neutral-control arm — see design doc §2.4." + ) + + +def run_arm( + seed_dir: Path, + arm: str, + iterations: int, + model_name: str, + model_version: str, + wrong_target_report_path: Path | None, +) -> list[dict]: + """Runs `iterations` rounds of the loop for one seed task under one arm, in a scratch git + clone of seed_dir (never the original — see the warning in main()). Returns the list of + per-iteration trajectory rows; does not write them (main() does, so a caller composing several + arms in one process can hold them all before writing).""" + scratch = Path( tempfile.mkdtemp( prefix=f"degradeloop-{arm}-" ) ) + tree = scratch / "tree" + shutil.copytree( seed_dir, tree ) + subprocess.run( [ "git", "-C", str( tree ), "init", "-q" ], check=True ) + subprocess.run( [ "git", "-C", str( tree ), "add", "-A" ], check=True ) + subprocess.run( + [ "git", "-C", str( tree ), "-c", "user.name=degradeloop", "-c", "user.email=degradeloop@invalid", + "commit", "-q", "-m", "seed" ], + check=True, + ) + baseline_ref = subprocess.run( + [ "git", "-C", str( tree ), "rev-parse", "HEAD" ], check=True, capture_output=True, text=True + ).stdout.strip() + + wrong_target_report = None + if wrong_target_report_path is not None: + wrong_target_report = json.loads( wrong_target_report_path.read_text() ) + + rows: list[dict] = [] + previous_report: dict | None = None + current_code_dir = tree + + for i in range( 1, iterations + 1 ): + prompt = build_prompt( arm, IMPROVE_PROMPT, previous_report, wrong_target_report ) + + # NOTE: single-file-tree simplification. A real seed task with many files needs a real + # (prompt, tree) -> tree contract instead of (prompt, one code string) -> one code string; + # this scaffolding assumes call_model operates over a concatenated or single-entry-point + # representation and that whoever implements call_model() handles multi-file state. Left + # simple on purpose rather than guessing a serialization format nobody has asked for yet. + current_code = "\n".join( p.read_text() for p in sorted( current_code_dir.rglob( "*" ) ) if p.is_file() ) + new_code = call_model( prompt, current_code, model_name=model_name, model_version=model_version ) + # Caller-side responsibility once call_model is real: write new_code back into tree, + # respecting whatever multi-file contract was chosen above, before the commit below. + raise NotImplementedError( + "run_arm() cannot proceed past the first call_model() call in this environment — " + "this RuntimeError is expected until call_model() and the write-back step are filled in." + ) + + return rows # unreachable until the seam above is implemented; kept for the intended shape + + +def main() -> int: + ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) + ap.add_argument( "--seed-dir", required=True, type=Path, help="checkout of the seed task's starting code" ) + ap.add_argument( "--arm", required=True, choices=[ "ungated", "gated", "neutral-control", "wrong-target" ] ) + ap.add_argument( "--iterations", required=True, type=int ) + ap.add_argument( "--model-name", required=True ) + ap.add_argument( "--model-version", required=True, help="date or version string — recorded, not validated" ) + ap.add_argument( "--wrong-target-report", type=Path, default=None, + help="required for --arm=wrong-target; a --quality-delta --json report from a DIFFERENT seed task" ) + ap.add_argument( "--out", required=True, type=Path ) + args = ap.parse_args() + + if not args.seed_dir.is_dir(): + print( f"error: --seed-dir {args.seed_dir} is not a directory", file=sys.stderr ) + return 1 + print( + "WARNING: this harness is scaffolding and has never been run end to end. " + "call_model() will raise NotImplementedError. See README.md.", + file=sys.stderr, + ) + + header = { + "kind": "degradeloop-trajectory-header", + "arm": args.arm, + "seed_dir": str( args.seed_dir ), + "iterations_requested": args.iterations, + "model_name": args.model_name, + "model_version": args.model_version, + "recorded_at_unix": int( time.time() ), + "ripwire_bin": measure._resolve_ripwire_bin(), # fails fast if RIPWIRE_BIN unset — intentional + } + + try: + rows = run_arm( + args.seed_dir, args.arm, args.iterations, args.model_name, args.model_version, + args.wrong_target_report, + ) + except NotImplementedError as e: + print( f"stopped (expected, scaffolding-only): {e}", file=sys.stderr ) + return 2 + + with args.out.open( "w" ) as f: + f.write( json.dumps( header ) + "\n" ) + for row in rows: + f.write( json.dumps( row ) + "\n" ) + return 0 + + +if __name__ == "__main__": + sys.exit( main() ) diff --git a/docs/research/gate-vs-iterative-degradation.md b/docs/research/gate-vs-iterative-degradation.md new file mode 100644 index 000000000..0036afb6a --- /dev/null +++ b/docs/research/gate-vs-iterative-degradation.md @@ -0,0 +1,383 @@ +# Gate vs. iterative degradation — a research note + +**Status: first pass. A reading path plus a harness design, not a result.** Nothing here reports a +measured effect of a gate on a degradation trajectory — we have not run one. What follows is (1) a +reading path into `--quality-delta` for an outside researcher, written to stand alone, and (2) a +concrete design for the experiment that would actually answer the question, including the shape of +result that would prove the design's own premise wrong. + +## The paper this responds to + +Kalyan Nakka, "Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the +Paradox," accepted IEEE-ISTAS 2025, [arXiv:2506.11022](https://arxiv.org/abs/2506.11022). The paper +runs a controlled experiment — 400 code samples, 40 rounds, four prompting strategies — asking an +LLM to iteratively "improve" its own code, and finds critical vulnerabilities increase 37.6% after +five iterations. It names the mechanism "feedback loop security degradation": a model optimizing for +its own stated goal (functionality, readability, the thing the prompt asked for) trades away a +property nobody in the loop is measuring. The paper's own recommendation is human validation between +iterations — an external check, not another self-critique pass. + +That is the premise this note takes seriously, generalized past security to code quality broadly: +if the reason iterative self-improvement degrades is that nothing *external and stable* is measuring +what degrades, the fix is not a better prompt, it is an oracle the model cannot rationalize past. This +is a design **hypothesis**, not a result borrowed from the paper — the paper does not test +`--quality-delta` or anything resembling it, and Part 2 exists because this note is not entitled to +claim otherwise. + +## Where this hypothesis actually lives — and why not `docs/LINEAGE.md` + +`docs/LINEAGE.md` folds a work only when "the lesson taken from it can be named in one sentence +**and** pointed at a real flag or source file" — a citation for something the paper *caused* us to +build. That is not this. `--quality-delta` predates this note; nothing about it changed on reading +Nakka 2025. Filing this as a lineage row would misrepresent the tool's history to buy a citation. + +The actual record of this position is the design comment at the top of `src/quality.h` (the file +that implements `--quality-delta`), which already states the mechanism this note is arguing from — +without citing this specific paper: + +> `quality.h` — `--quality-baseline` / `--quality-delta`: the deterministic oracle for a code-quality +> **CONVERGENCE LOOP**. … the "delta, not absolute" discipline that lets a refine loop target *the +> regression it introduced* instead of chasing absolute numbers (**the defense against Goodhart / +> metric-gaming**). + +That comment is design intent recorded where the code lives, not a claim staked in a document meant +to be an honest ledger of external influence. This note is the first place the two are put side by +side on purpose. If a later change actually adapts something from Nakka 2025 — the vulnerability +taxonomy, the round-count design, a specific finding — *that* change earns a `LINEAGE.md` row at the +time it lands, not retroactively from this note. + +--- + +## Part 1 — a reading path into `--quality-delta`, for an outside researcher + +Read in this order. Each step names the file, and what you're checking for. + +### 1. Start here + +- `./build/ripwire --help` and [`docs/COMMANDS.md`](../COMMANDS.md) (`--quality-delta`, + `--quality-delta=REV|A..B`, `--quality-baseline`, `--quality-ack`, `--ack-only`, `--dmm`) — the + generated, always-current flag reference. If this note disagrees with `--help`, `--help` is right. +- `src/quality.h` lines 1–20 — the file's own one-paragraph design statement (quoted above in full + context). It states the Goodhart defense explicitly: report only the *regression*, never an + absolute score, because an absolute score is a target a model can learn to game without fixing + anything. + +### 2. The ten kinds, and where each is computed + +All ten live in `src/quality.h`. `computeSnapshot` (line 3959) builds the per-symbol/per-group floor +from one tree; `computeDelta` (the function whose body spans roughly line 6700–7200) compares two +snapshots and emits only what got worse. Per-kind entry points (line numbers at `origin/main` +`755f9026`, this branch's base — they drift as the file changes, so treat them as a starting point, +not a pin): + +| kind | bar / threshold | computed at | +| --- | --- | --- | +| `complexity` | ccx > 15 (`kCcxBar`), AND grew | `perSymbolKind( "complexity", … )`, `src/quality.h:6860` | +| `verbosity` | LOC > 60 (`kLocBar`), AND grew | `perSymbolKind( "verbosity", … )`, `src/quality.h:6861` | +| `nesting` | max-nest > 4 (`kNestBar`), AND grew | `perSymbolKind( "nesting", … )`, `src/quality.h:6862` | +| `params` | param count > 5 (`kParamBar`), AND grew | `perSymbolKind( "params", … )`, `src/quality.h:6863` | +| `duplication` | new/grown clone group ≥ `kMinCloneTokens` (18) | `src/quality.h:7000`, clone detection in `src/clones.h` | +| `dead-code` | zero in-edges, not a registered/test/fixture exemption | `isDeadCandidate`, `src/quality.h:648`; emitted `src/quality.h:7029` | +| `api-surface` | a symbol becomes public, or a public signature's arity changes | `src/quality.h:7115` (visibility flip), `:7148` (arity/contract change) | +| `error-masking` | a construct in `findErrorMasking`'s built-in rule table (`src/lintrules.h`) | `errorMaskCountsBySym`, `src/quality.h:765`; emitted `:7183` | +| `short-horizon-churn` | a file rewritten ≥2 times inside a 14-day window AND touched by this diff | `src/quality.h:7310`, mined via `gitmine.h` | +| `new-clone-of-reused-helper` | a fresh clone of a helper whose existing fan-in ≥ 3 | `src/quality.h:7392` | + +Four numeric kinds (`complexity`, `verbosity`, `nesting`, `params`) share one generic +`perSymbolKind` driven by a lambda that reads the metric off `Symbol` — read that one function and +you have read all four. The other six are presence/structural kinds with their own emission sites, +listed above. + +**The materiality design, which the paper's own framing makes relevant.** A regression whose delta +is below a per-kind "minor" threshold (`kMinorCcxDelta = 3`, `kMinorLocDelta = 10`, +`kMinorParamDelta = 2`; nesting and the presence kinds have none — any instance is major) is reported +but does not gate exit 2 on its own. This exists because a +1-ccx edit to a function already over the +bar is technically a regression but is noise that would drown the findings a refine loop should +actually chase (`src/quality.h` comment above `kMinorCcxDelta`). If a degradation trajectory is a +slow accumulation of small unmeasured moves rather than one visible break, this threshold is exactly +where a gate could be *blind* to it by design — see §2.2's growth-rate instrument, added for this +reason. + +### 3. The baseline model + +Two floors, chosen automatically (`docs/COMMANDS.md` `--quality-delta` section, and +`computeHeadSnapshot`, `src/quality.h:3525`): + +- **Sidecar** (`.ripwire_quality_baseline`, written by `--quality-baseline`) — honored only when + pinned at exactly the current `git HEAD` (strict equality; an ancestor is a different tree). Stale + → self-heals by deleting the file and falling back to the second floor. `--quality-baseline` + itself **refuses** (exit 1) to pin on a dirty tree unless you pass `--allow-dirty`, specifically so + the debt already in a working tree cannot be swallowed into the floor and read clean forever after. +- **git-HEAD** (no sidecar, or a stale one) — `computeHeadSnapshot` (`src/quality.h:3525`) does + `git archive HEAD` into a temp directory and re-ingests it (`materializeCommitTree`, + `src/quality.h:3345`), then compares the working tree against that. + +### 4. The fix that just landed — a clean tree must never gate + +`$ORCH/reports/t12-qd-noop.md` (this orchestration round, branch `lane/t12-qd-noop-diff`, head +`d54ce3da`). The mechanism: the git-HEAD floor is built by archiving and re-ingesting HEAD into a +temp directory, which is a **different file population** from the working tree whenever anything is +untracked, gitignored, export-ignored, sparse-checked-out, or otherwise present on one side and not +the other — a shallow clone, `.gitignore`d duplicate, or skip-worktree flag all reproduce it. Because +a dead-code verdict is a property of the *whole population* a symbol is ingested with, not of its own +file, a file present on only one side can flip a same-named definition's dead/alive verdict on a +symbol *both* sides share — and a no-op diff gates. Three review rounds converged on the same defect +class reached three different ways; the final fix (round 3, `d54ce3da`) replaces a hand-written model +of git's file-selection rules with git's own answer, rather than patching the model a third time. This +matters to a researcher reading the tool cold: it is the concrete shape of "the gate itself has bugs," +and it is now closed for the no-op case specifically — read the report for what's still scoped out +(the Django `cls.`/`self.` dispatch half of the same issue, tracked separately, #237). + +### 5. The ack ledger, and why it exists + +`.ripwire_quality_acks` (format documented at the top of the file itself, and in +`--quality-ack`/`--ack-only` in `docs/COMMANDS.md`). An ack records a **reviewed** finding as known, +suppressing it until it *worsens past its acked magnitude* — a ratchet, not a mute. `--ack-only=KIND` +exists because bare `--quality-ack` accepts the whole current report, and accepting one deliberate +change alongside everything else unacked turns a ratchet into a rubber stamp — the tool's own +documentation names this failure mode explicitly and gives the scoped form as the way to avoid it. +For the degradation question this ledger is a hazard to control for: an iteration loop with a +human (or an agent) free to ack findings can make the *gate* report clean while debt still +accumulates under the ack. §2.4 treats acking as something the harness must record, not something it +assumes away. + +### 6. The eval harness + +[`docs/EVALS.md`](../EVALS.md) §1 tables every instrument this project has and what each measures; +§6 has `--quality-delta`'s own kind list restated with the exact `kind=` strings; §7 is a section of +*honest counterexamples* — measured findings that went against the tool's own claims, published on +purpose. `bench/` holds the harness code itself: `bench/recalleval/`, `bench/headtohead/`, +`bench/locbench/`, `bench/ensemblecal/`, each with its own README. `bench/ensemblecal/` is the +closest structural relative to what Part 2 proposes: it separately verifies a calibration hypothesis +(that four evidence families are orthogonal) with a stated honesty contract, and reports what it +would take for the hypothesis to be wrong. + +### 7. What our own backtest says — read this before citing the tool as a detector + +`$ORCH/reports/study-checks.md` §3 (backtest against ripwire's own commit history, `W` = window in +commits): at the realistic five-commit window, `--quality-delta` fires on **14/24 (58%) of states +that provably contained a finding somebody later fixed, and on 12/30 (40%) of control states with no +recorded defect.** 58% vs 40% is not discrimination — it is closer to a coin flip weighted by how +much code moved. The kind firing most on *defect-free* control states is `complexity` (10 of 30 +control reds), the same kind whose per-symbol external-corpus signal (§4 of that report) is the one +structural metric that *does* separate human-authored defects from non-defects at 1.63× (95% CI +[1.11, 2.38]). Read plainly: `--quality-delta` measures accumulated debt, and debt correlates with +defect-proneness in the literature and in our own external check, but firing on a state is not the +same claim as identifying a defect in it. The tool's own documentation says "report only what got +worse … descriptive," and this backtest is the first time that sentence has been measured rather than +asserted. Any harness in Part 2 that treats a `--quality-delta` exit 2 as "a real regression was +introduced" is overclaiming; treat it as "measured debt increased," which is a different, still +useful, and honestly weaker claim. + +### 8. Reading order, summarized + +`docs/COMMANDS.md` (`--quality-delta` family) → `src/quality.h` lines 1–20 → the ten-kind table +above with the file open beside it → `$ORCH/reports/t12-qd-noop.md` → `.ripwire_quality_acks` header +comment → `docs/EVALS.md` §1 and §6 → `$ORCH/reports/study-checks.md` §3. That is roughly 45 minutes +to a working mental model, and it ends exactly where Part 2 starts: with the honest limit of what the +gate has been shown to do. + +--- + +## Part 2 — a harness design for the degradation question + +**The question Part 1's reading path does not answer:** does a deterministic external gate change +the *trajectory* of repeated self-improvement — the shape of the curve across iterations — or does it +only catch individual regressions after they already happened, leaving the underlying trajectory the +same? These are different claims. A gate that catches every regression it sees but never changes what +the model tries next iteration would still let the *next* unmeasured failure mode (the one the gate +doesn't check) degrade freely — Nakka 2025's whole point is that *something* always degrades when +only *some* things are measured. + +### 2.1 The loop + +For each of N seed tasks (real small-to-medium functions or modules, ideally drawn from more than one +language ripwire indexes, since the ten kinds and their bars are language-general but their density +of hits is not measured to be): + +``` +state[0] = seed code, committed +for i in 1..K: + prompt = "improve this code" + (gate feedback from state[i-1], IF gated arm) + state[i] = model(prompt, state[i-1]) + measure(state[i]) # §2.2, run regardless of arm + commit state[i] +``` + +Two arms per seed task, same model, same temperature, same seed code, same K: + +- **UNGATED** — the prompt is "improve this code" (plus whatever the task needs for continuity, e.g. + the running test suite's pass/fail if the task has one) and nothing else. This is the paper's own + setup. +- **GATED** — the prompt additionally receives the previous iteration's `--quality-delta` (and, if + in scope, `--test-gate`) report, unfiltered, as context the model is told to address before + proposing the next change. No human is in the loop; the gate is the only external signal. This is + deliberately the *weakest* form of "external and deterministic" — it tests whether the model + reading the gate's own words changes behavior, not whether a human enforcing exit 2 would (a + stronger, cheaper-to-argue-for design that isn't the interesting question: of course a hard stop + changes what ships. The trajectory question is about the *code the model chooses to write*, not + about a merge gate.) + +K should be at least 10 — Nakka 2025 reports the divergence sharpening between iteration 5 and 10; a +shorter loop cannot see whether a gate changes the *slope* rather than just one round's value. + +### 2.2 What is measured each iteration + +Everything below is computed fresh on `state[i]`, in both arms, whether or not that arm sees it in +its prompt — the ungated arm's own trajectory needs the same instruments or there's nothing to +compare against. + +**Deterministic, ripwire-native (the ten kinds, from `--quality-delta --json` against `state[0]` as +the fixed baseline — not against `state[i-1]`, so a regression that got partially fixed and +re-introduced two iterations later is still visible; see §2.3):** +- All ten kinds' counts (`regressions`, `gating`, `minor`, per-`kind=` breakdown). +- `api-new-surface` and `register-macro-excluded` as printed floors, for context. +- Clone growth specifically (`duplication` + `new-clone-of-reused-helper` counts and the clone + groups' member counts) — Nakka 2025 doesn't measure this, but GitClear's agent-code findings that + motivate `--quality-delta`'s own kinds (cited in `src/quality.h`'s comments) do, and duplication is + cheap to accumulate silently under an "improve this" prompt that never says "don't repeat + yourself." + +**Deterministic, test-native:** +- Test pass/fail count, IF the seed task carries a test suite (`--test-gate --json`'s `tests=` / + `untested=`, or the task's own runner if simpler). A trajectory that improves quality kinds while + breaking tests is not a win for either arm's position. + +**Deterministic, growth-rate (not in `--quality-delta`'s own output, but derivable from the raw +snapshot each iteration writes):** +- Sub-bar growth that `--quality-delta`'s "minor" threshold (§1.2) would not gate: raw ccx/LOC/nest/ + params values per touched symbol, iteration over iteration, regardless of whether any single step + crossed a bar. This is the instrument for "the gate is blind to slow accumulation" — a real + possibility given how the minor-severity design works, and worth measuring even though it isn't + what the shipped tool reports. + +**Public and deterministic, non-ripwire:** +- A public static security scanner appropriate to the seed task's language — e.g. Semgrep's default + ruleset (multi-language, free, deterministic given a pinned ruleset version) or a + language-appropriate equivalent (Bandit for Python, `cargo audit`/clippy for Rust). This is the + instrument that actually answers the paper's own question (security), separate from ripwire's + structural kinds, which do not claim to measure security. **Pin the scanner's version and ruleset + hash in the run's metadata** — an unpinned scanner is not a repeatable instrument. + +**Recorded, not computed:** whether the gated arm's model actually *acted* on the previous report +(a crude proxy: did the flagged symbol change in the next iteration at all) and whether any +`--quality-ack` was exercised (it should not be, in this design — acking belongs to a human reviewer, +and an unattended loop acking its own findings would silently defeat the gate; if a later variant lets +the model ack, that has to be a separate, explicitly labeled arm). + +### 2.3 What "the trajectory changed" would mean, numerically — fixed in advance + +Stated before any run, per the project's own pre-registration discipline (`study-checks.md`'s +external arm is the house precedent: the sampling amendment was written and executed *before* the +first measurement). Candidate instruments, all computed per seed task then aggregated: + +1. **Cumulative-regression slope.** Fit `regressions[i]` (from §2.2, baseline-anchored so + regressions don't wash out when partially fixed) against iteration `i`, per arm, per seed task. + The claim "the gate changes the trajectory" requires the GATED slope to be statistically lower + than the UNGATED slope, paired by seed task (Wilcoxon signed-rank across seed tasks, not a pooled + t-test across iterations — iterations within one task's trajectory are not independent + observations). +2. **Terminal-state comparison.** `regressions[K]` and scanner-finding-count`[K]`, GATED vs UNGATED, + paired by seed task. This is the paper's own headline shape (end state after N rounds) and should + be reported even though it's the weaker claim (it can differ without the *trajectory* differing — + e.g. one late correction). +3. **Sub-bar growth rate** (§2.2's growth-rate instrument): does the GATED arm's raw metric growth + per iteration (summed across touched symbols, below any bar) differ from UNGATED's? This is the + instrument that would catch a gate that looks clean on `--quality-delta`'s own gating count while + debt still accumulates just under every bar — the concern §1.2 raises. +4. **Security-finding trajectory**, same slope test as (1), on the scanner's output — this is the one + that actually tests Nakka 2025's claim in our setting, since none of ripwire's ten kinds are + security checks. + +**Pre-registered thresholds, to be set with the actual seed-task count and iteration count once +chosen (N and K bound the achievable power) — but the form is fixed now:** a slope difference is +"real" only if it clears a pre-specified effect size (not just p < 0.05 — with small N, significance +without an effect-size floor is how the ratchet-vs-detector confusion in §1.7 happens again one level +up), and the write-up reports the ungated-vs-gated comparison for **all four instruments**, not +whichever one came out favorable. Selective reporting of the metric that happened to move is the +single easiest way to manufacture the result this note wants to see. + +### 2.4 The confound: the gate also changes the prompt + +The GATED arm's prompt contains the previous quality-delta report; the UNGATED arm's does not. Any +measured difference could be "the model responded to structured, specific feedback about its own +code" rather than "the model responded to *this particular gate*" — a generic-text confound, not a +gate-specific effect. Two controls, both needed: + +- **Matched-length neutral-text control (arm C).** Same prompt structure as GATED, but the appended + text is a fixed, task-irrelevant block of roughly the same token length as a typical + `--quality-delta` report (e.g. the tool's own `--legend` text, which is long, dense, and describes + nothing about the current code). If GATED and C both outperform UNGATED equally, the effect is + "more context helps," not "the gate helps," and the paper's mitigation claim does not transfer to + this tool specifically. +- **Matched-content wrong-target control (arm D), if resourcing allows a fourth arm.** Feed the GATED + prompt a real `--quality-delta` report — but from a *different* seed task, not this one. This + isolates "specific, on-target feedback about this code" from "quality-delta-shaped feedback about + something." If GATED clearly beats D, the effect is that the gate is *reading the actual code*, not + merely primed by its vocabulary and structure. + +Report GATED against UNGATED, C, and D (as available) side by side. The claim this note is actually +interested in is GATED vs. C, not GATED vs. UNGATED — the paper's baseline is silence, and beating +silence with *any* structured feedback would be a weak, likely-true result that says nothing about +determinism or externality specifically. + +### 2.5 What result would count against our position + +Stated plainly, in advance, because this is the part most likely to be skipped: + +- **GATED does not clearly beat C (matched-length neutral text).** If a same-length block of + unrelated text produces the same trajectory improvement as the actual quality report, the + mechanism is "more tokens of structured-looking feedback," not "deterministic external + measurement," and `src/quality.h`'s design comment (the Goodhart-defense framing) would be + unsupported by this experiment specifically — it might still be true for other reasons, but this + harness would not have shown it. +- **GATED's terminal state is better but its slope is not.** This would mean the gate produces one + correction late in the loop (the model notices and fixes things right before the run ends) rather + than changing the trajectory throughout — closer to "catches regressions after the fact" than + "changes the trajectory," which is the distinction this whole design exists to draw, and the + weaker of the two claims would be the honest one to publish. +- **The security-scanner trajectory (instrument 4) doesn't track the ripwire-kind trajectory + (instrument 1) at all** — e.g. GATED improves structural debt but the scanner's finding count is + flat or worse across both arms. This would mean structural-quality gating and security are close to + orthogonal in this setting, which directly limits how far "a deterministic quality gate" can be + read as an answer to a *security*-framed paper, no matter what instrument 1–3 show. +- **Sub-bar growth (instrument 3) is worse in GATED than UNGATED.** This would be the sharpest + negative result: a model gaming the visible bar by keeping every individual metric just under + threshold while the underlying code gets worse in aggregate — literally the Goodhart failure mode + the gate exists to prevent, reappearing one level down. If this shows up, it belongs in + `src/quality.h`'s own comments as a documented limit, not quietly dropped from the writeup. + +Any one of these, on real data, is a more useful contribution than a clean win — this project's own +`docs/EVALS.md` §7 exists for exactly that reason, and a degradation-harness result belongs beside it +if it runs. + +### 2.6 What we did not, and could not, do here + +This is a design and a set of scaffolding scripts, not a study. Nobody has an LLM API wired into this +worktree, and no model calls were made to produce this document. The scripts in `bench/degradeloop/` +implement the deterministic half — the loop's bookkeeping, the measurement calls into +`--quality-delta`/`--test-gate`, and the trajectory-analysis math from §2.3 — with an explicit, +unimplemented seam where a real model call belongs, and every file says so at the top rather than +faking a call site that looks wired up. **No numbers in this document are results.** Anyone who runs +the harness and gets numbers should report them beside this design, including if they contradict +§2.5's stated position. + +--- + +## What we would like help with + +- **Seed tasks.** A good seed set needs real, non-trivial functions with a genuine "improve this" + prompt that isn't already solved — ideally spanning at least two of ripwire's indexed languages, so + the ten kinds' language-general bars actually get exercised differently. We don't have a vetted set. +- **A model-call adapter.** `bench/degradeloop/run_degradeloop.py`'s `call_model()` seam is written + against a plain (prompt, code) → code interface deliberately — wiring it to a specific API is a + few lines for whoever has one available and wants to run this. +- **The scanner choice per language**, pinned and justified — we picked Semgrep as the + cross-language default in §2.2 but have not evaluated whether its default ruleset has the recall to + see what Nakka 2025's own taxonomy would flag. +- **A second opinion on the confound design (§2.4).** Arm D (matched-content, wrong-target) is the + one we're least sure earns its cost against arm C alone — if C is enough to isolate the effect, + D is one fewer arm to run. +- **Anyone who runs this** — even a small N, even one seed task, one language, K=10 — and reports the + four instruments honestly, favorable or not. That is worth more than a larger version of this + design document. From 8c0fdb9f1e0db4d4cfca9b9314f6b4d29a8ae91b Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Mon, 21 Sep 2026 11:52:19 -0400 Subject: [PATCH 2/3] chore(docs): list docs/research/ in docs/README.md ripwirepubliccheck arm 6b requires every file under `docs/` to be reachable from `docs/README.md`. The row added here is byte-identical to the one on every other research lane, so eight branches adding the same table line merge without a conflict. Co-Authored-By: Claude Opus 5 --- docs/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/README.md b/docs/README.md index a9ba460b8..4d459a24f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ # ripwire documentation -Sixteen entries, each written for one reader. Start with the row that matches why you are here. +Twenty-one entries, each written for one reader. Start with the row that matches why you are here. | File | Who it is for | What it answers | | --- | --- | --- | @@ -22,6 +22,7 @@ Sixteen entries, each written for one reader. Start with the row that matches wh | **[`gatecount_build.py`](gatecount_build.py)** | Maintainers, and anyone adding a gate | The generator behind the **published gate count**. Derives it from the single `for _g in …; do` loop in `test/regression.sh` and rewrites all eight marked sites across `README.md`, `EVALS.md` and the deck; `--check` is the drift comparison that `test/gatecountcheck.sh` runs. Never edit that number by hand — two lanes hand-writing the same N+1 auto-merge clean against a loop of N+2. | | **[`limits_classes.tsv`](limits_classes.tsv)** | Maintainers | Cap name -> INDEXING or OUTPUT, the taxonomy `LIMITS.md` renders in its `class` column: does this cap bound what can EVER be found, or only what is shown from what was found. A sidecar with a known expiry — the tag belongs on the declaration in `src/` — kept honest by `limitstablecheck.sh`, which fails if a row names a cap that no longer exists. | | **[`lineage-paper-dates.tsv`](lineage-paper-dates.tsv)** | Maintainers | arXiv id -> publication date for every 2026 paper in `LINEAGE.md`. The ID stem does not track the date (`2607.09691` was published 2026-06-19), so the README's recency claim is re-derived from this file by `readmedriftcheck.sh` arm (H2) rather than from the ids. Adding a 2026 paper without a date row fails that arm. | +| **[`research/`](research/)** | Anyone working on an open question here, or offering to help with one | Investigation notes: a question this tool has not answered, the measurement run against it, and the pre-registration that says what would license a change. Not a claim surface — a number here is local evidence for a decision not yet taken, and nothing in it is quoted on `README.md`. | | **[`assets/`](assets/)** | The front page | The README banner and tagline artwork (SVG, self-contained). | | **[`captures/`](captures/)** | Maintainers, and the curious | One recorded run of every verb against a real repository — the source of `COMMANDS.md`'s sample output, and the harvest source for the differential argv harness. | From 689acc8a0794f76d55e94d51786e3e38b5489231 Mon Sep 17 00:00:00 2001 From: joyful-ii-V-I Date: Tue, 22 Sep 2026 06:48:14 -0400 Subject: [PATCH 3/3] fix(research): correct author attribution for arXiv:2506.11022 The note cited "Kalyan Nakka" / "Nakka 2025" at 8 sites for the Security Degradation paper. The real authors, per the arXiv abstract page and export API, are Shivani Shukla, Himanshu Joshi and Romilla Syed (Shukla is first author). Fix the citation line and every shorthand reference. Co-Authored-By: Claude Sonnet 5 --- docs/research/gate-vs-iterative-degradation.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/research/gate-vs-iterative-degradation.md b/docs/research/gate-vs-iterative-degradation.md index 0036afb6a..f8bd584cd 100644 --- a/docs/research/gate-vs-iterative-degradation.md +++ b/docs/research/gate-vs-iterative-degradation.md @@ -8,7 +8,7 @@ result that would prove the design's own premise wrong. ## The paper this responds to -Kalyan Nakka, "Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the +Shivani Shukla, Himanshu Joshi and Romilla Syed, "Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the Paradox," accepted IEEE-ISTAS 2025, [arXiv:2506.11022](https://arxiv.org/abs/2506.11022). The paper runs a controlled experiment — 400 code samples, 40 rounds, four prompting strategies — asking an LLM to iteratively "improve" its own code, and finds critical vulnerabilities increase 37.6% after @@ -29,7 +29,7 @@ claim otherwise. `docs/LINEAGE.md` folds a work only when "the lesson taken from it can be named in one sentence **and** pointed at a real flag or source file" — a citation for something the paper *caused* us to build. That is not this. `--quality-delta` predates this note; nothing about it changed on reading -Nakka 2025. Filing this as a lineage row would misrepresent the tool's history to buy a citation. +Shukla et al. 2025. Filing this as a lineage row would misrepresent the tool's history to buy a citation. The actual record of this position is the design comment at the top of `src/quality.h` (the file that implements `--quality-delta`), which already states the mechanism this note is arguing from — @@ -42,7 +42,7 @@ without citing this specific paper: That comment is design intent recorded where the code lives, not a claim staked in a document meant to be an honest ledger of external influence. This note is the first place the two are put side by -side on purpose. If a later change actually adapts something from Nakka 2025 — the vulnerability +side on purpose. If a later change actually adapts something from Shukla et al. 2025 — the vulnerability taxonomy, the round-count design, a specific finding — *that* change earns a `LINEAGE.md` row at the time it lands, not retroactively from this note. @@ -186,7 +186,7 @@ the *trajectory* of repeated self-improvement — the shape of the curve across only catch individual regressions after they already happened, leaving the underlying trajectory the same? These are different claims. A gate that catches every regression it sees but never changes what the model tries next iteration would still let the *next* unmeasured failure mode (the one the gate -doesn't check) degrade freely — Nakka 2025's whole point is that *something* always degrades when +doesn't check) degrade freely — Shukla et al. 2025's whole point is that *something* always degrades when only *some* things are measured. ### 2.1 The loop @@ -218,7 +218,7 @@ Two arms per seed task, same model, same temperature, same seed code, same K: changes what ships. The trajectory question is about the *code the model chooses to write*, not about a merge gate.) -K should be at least 10 — Nakka 2025 reports the divergence sharpening between iteration 5 and 10; a +K should be at least 10 — Shukla et al. 2025 reports the divergence sharpening between iteration 5 and 10; a shorter loop cannot see whether a gate changes the *slope* rather than just one round's value. ### 2.2 What is measured each iteration @@ -233,7 +233,7 @@ re-introduced two iterations later is still visible; see §2.3):** - All ten kinds' counts (`regressions`, `gating`, `minor`, per-`kind=` breakdown). - `api-new-surface` and `register-macro-excluded` as printed floors, for context. - Clone growth specifically (`duplication` + `new-clone-of-reused-helper` counts and the clone - groups' member counts) — Nakka 2025 doesn't measure this, but GitClear's agent-code findings that + groups' member counts) — Shukla et al. 2025 doesn't measure this, but GitClear's agent-code findings that motivate `--quality-delta`'s own kinds (cited in `src/quality.h`'s comments) do, and duplication is cheap to accumulate silently under an "improve this" prompt that never says "don't repeat yourself." @@ -286,7 +286,7 @@ first measurement). Candidate instruments, all computed per seed task then aggre instrument that would catch a gate that looks clean on `--quality-delta`'s own gating count while debt still accumulates just under every bar — the concern §1.2 raises. 4. **Security-finding trajectory**, same slope test as (1), on the scanner's output — this is the one - that actually tests Nakka 2025's claim in our setting, since none of ripwire's ten kinds are + that actually tests Shukla et al. 2025's claim in our setting, since none of ripwire's ten kinds are security checks. **Pre-registered thresholds, to be set with the actual seed-task count and iteration count once @@ -374,7 +374,7 @@ the harness and gets numbers should report them beside this design, including if few lines for whoever has one available and wants to run this. - **The scanner choice per language**, pinned and justified — we picked Semgrep as the cross-language default in §2.2 but have not evaluated whether its default ruleset has the recall to - see what Nakka 2025's own taxonomy would flag. + see what Shukla et al. 2025's own taxonomy would flag. - **A second opinion on the confound design (§2.4).** Arm D (matched-content, wrong-target) is the one we're least sure earns its cost against arm C alone — if C is enough to isolate the effect, D is one fewer arm to run.