diff --git a/bench/degradeloop/README.md b/bench/degradeloop/README.md new file mode 100644 index 000000000..57dd0fe6f --- /dev/null +++ b/bench/degradeloop/README.md @@ -0,0 +1,49 @@ +# bench/degradeloop — scaffolding for the gate-vs-degradation harness + +**UNTESTED. Nothing in this directory has been run.** These scripts were written alongside +[`docs/research/gate-vs-iterative-degradation.md`](../../docs/research/gate-vs-iterative-degradation.md) +(read that first — it's the design these scripts implement) as a starting point for whoever runs the +actual experiment. No model API is wired in anywhere; the one seam that needs one +(`call_model()` in `run_degradeloop.py`) raises `NotImplementedError` on purpose rather than faking a +call site that looks functional. Every number these scripts *could* produce would be synthetic until +that seam is filled in and the scripts are actually exercised against a real model and a real +`build/ripwire` binary. + +## What's here + +- `run_degradeloop.py` — drives one arm (`ungated` / `gated` / `neutral-control` / `wrong-target`, + see the design doc §2.1/§2.4) of the loop for one seed task: calls the model seam each iteration, + commits the result to a scratch git repo, and calls `measure.py`'s functions to snapshot it. Writes + one JSONL trajectory file per (seed task, arm) pair. +- `measure.py` — the deterministic measurement layer. Wraps `ripwire --quality-delta --json` and + `ripwire --test-gate --json` as subprocess calls against a scratch working tree, plus a stub for + the public security-scanner call (`run_security_scanner()`, also unimplemented — pick and pin a + scanner per §2.2 of the design doc before filling this in). Parses only what the design doc's + §2.2/§2.3 instruments need; does not attempt to be a general ripwire-output parser. +- `analyze_trajectory.py` — reads the JSONL trajectory files `run_degradeloop.py` writes (for however + many arms and seed tasks are on disk) and computes the four pre-registered instruments from the + design doc's §2.3: cumulative-regression slope (paired Wilcoxon across seed tasks), terminal-state + comparison, sub-bar growth rate, and — if a scanner ran — the security-finding trajectory. Refuses + to report an instrument it cannot compute rather than silently omitting it from a table (a design + doc's own house rule, `src/quality.h`'s honesty framing applied here to our own output). + +## Before running any of this + +1. Fill in `call_model()` in `run_degradeloop.py` against whatever model access you have. It takes + `(prompt: str, current_code: str) -> str` and returns the new code — that's the whole contract. +2. Pick and pin a security scanner per seed-task language in `measure.py::run_security_scanner()` + (a version and ruleset hash, not just a tool name — see the design doc's §2.2 caveat on this). +3. Assemble a seed-task set (see the design doc's "what we'd like help with" — we don't have one). +4. Point `RIPWIRE_BIN` (env var, read by `measure.py`) at a real `build/ripwire` built from this + repository's `CLAUDE.md` build instructions. `measure.py` refuses to run against a bare `ripwire` + on `PATH` without this being set explicitly, so a stale system install can't silently produce + numbers. + +## Determinism note + +`measure.py`'s ripwire calls are deterministic (same binary, same tree → same output, per this +project's own determinism contract). The loop as a whole is **not** deterministic end to end, because +`call_model()` isn't — that's expected and is not something these scripts try to paper over. Record +the model, its version/date, and the temperature/sampling settings used alongside every trajectory +file; `run_degradeloop.py`'s JSONL header line has fields for exactly this and refuses to run without +them filled in. diff --git a/bench/degradeloop/analyze_trajectory.py b/bench/degradeloop/analyze_trajectory.py new file mode 100644 index 000000000..58c8e82db --- /dev/null +++ b/bench/degradeloop/analyze_trajectory.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +"""analyze_trajectory.py — the four pre-registered instruments from +docs/research/gate-vs-iterative-degradation.md §2.3, computed over whatever trajectory JSONL +files run_degradeloop.py has produced. + +SCAFFOLDING, NOT VALIDATED AGAINST REAL DATA. The statistics below (paired slope comparison, +Wilcoxon signed-rank) are implemented from scratch, zero-dependency, matching this project's own +"no host-installed dependencies" posture (CLAUDE.md G3) rather than reaching for scipy — they have +been sanity-checked against small synthetic inputs (see the __main__ self-test at the bottom, +runnable with `python3 analyze_trajectory.py --selftest`) but NOT against a real trajectory run, +because none exists yet. Do not cite a p-value out of this file in a publishable report without an +independent check against a reference implementation once real data exists. + +INPUT FORMAT. One or more JSONL files as written by run_degradeloop.py: a header row +(kind="degradeloop-trajectory-header") followed by one row per iteration. This script does not +assume a particular per-iteration row schema beyond what each instrument function documents it +reads — see each function's docstring for the exact fields it expects, matching what measure.py's +snapshot dataclasses would serialize. + +REFUSAL, NOT SILENT OMISSION. Per the design doc's §2.3 rule ("the write-up reports … all four +instruments … not whichever one came out favorable"), report_all() below always emits a row for +every instrument, marking it unavailable with a stated reason when the data can't support it, +rather than dropping it from the table. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +# --------------------------------------------------------------------------- +# Minimal zero-dependency statistics (see module docstring on why not scipy) +# --------------------------------------------------------------------------- + +def _ranks( values: list[float] ) -> list[float]: + """Average ranks, ties split evenly — the standard Wilcoxon tie-handling.""" + order = sorted( range( len( values ) ), key=lambda i: values[ i ] ) + ranks = [ 0.0 ] * len( values ) + i = 0 + while i < len( order ): + j = i + while j + 1 < len( order ) and values[ order[ j + 1 ] ] == values[ order[ i ] ]: + j += 1 + avg_rank = ( i + j ) / 2.0 + 1.0 + for k in range( i, j + 1 ): + ranks[ order[ k ] ] = avg_rank + i = j + 1 + return ranks + + +def wilcoxon_signed_rank( a: list[float], b: list[float] ) -> dict: + """Paired Wilcoxon signed-rank test, a vs b (a - b), normal approximation (no exact tables — + fine for the N this design expects to run with being small enough that the approximation's own + documented weakness at very small N should be flagged, which this function does via `note` + rather than silently reporting a p-value the sample size doesn't support). + + Returns W (the smaller of W+/W-), z (normal-approximation statistic), and a two-sided p + computed from a standard-normal survival approximation (erf-based, stdlib only). + """ + diffs = [ x - y for x, y in zip( a, b ) if x != y ] # zero-diffs dropped, standard convention + n = len( diffs ) + if n < 6: + return { + "n_nonzero": n, "W": None, "z": None, "p_two_sided": None, + "note": "n < 6 after dropping zero-diffs — normal approximation is not trustworthy here; " + "report the raw paired differences instead of a p-value.", + } + abs_diffs = [ abs( d ) for d in diffs ] + ranks = _ranks( abs_diffs ) + w_pos = sum( r for r, d in zip( ranks, diffs ) if d > 0 ) + w_neg = sum( r for r, d in zip( ranks, diffs ) if d < 0 ) + w = min( w_pos, w_neg ) + mean_w = n * ( n + 1 ) / 4.0 + std_w = ( n * ( n + 1 ) * ( 2 * n + 1 ) / 24.0 ) ** 0.5 + z = ( w - mean_w ) / std_w if std_w > 0 else 0.0 + p = _two_sided_normal_p( z ) + return { "n_nonzero": n, "W": w, "z": z, "p_two_sided": p, "note": None } + + +def _erf( x: float ) -> float: + # Abramowitz & Stegun 7.1.26 approximation — stdlib-only, adequate for a two-sided p-value + # at the precision this design needs (a pre-registered effect-size floor, not a tight p-value). + sign = 1 if x >= 0 else -1 + x = abs( x ) + a1, a2, a3, a4, a5 = 0.254829592, -0.284496736, 1.421413741, -1.453152027, 1.061405429 + p = 0.3275911 + t = 1.0 / ( 1.0 + p * x ) + y = 1.0 - ( ( ( ( ( a5 * t + a4 ) * t ) + a3 ) * t + a2 ) * t + a1 ) * t * pow( 2.718281828459045, -x * x ) + return sign * y + + +def _two_sided_normal_p( z: float ) -> float: + return 2.0 * ( 1.0 - 0.5 * ( 1.0 + _erf( abs( z ) / ( 2 ** 0.5 ) ) ) ) + + +def _slope( ys: list[float] ) -> float: + """OLS slope of ys against iteration index 0..n-1. Plain least squares, stdlib only.""" + n = len( ys ) + if n < 2: + return 0.0 + xs = list( range( n ) ) + mean_x = sum( xs ) / n + mean_y = sum( ys ) / n + num = sum( ( x - mean_x ) * ( y - mean_y ) for x, y in zip( xs, ys ) ) + den = sum( ( x - mean_x ) ** 2 for x in xs ) + return num / den if den else 0.0 + + +# --------------------------------------------------------------------------- +# Trajectory loading +# --------------------------------------------------------------------------- + +def load_trajectory( path: Path ) -> tuple[dict, list[dict]]: + lines = path.read_text().splitlines() + if not lines: + raise ValueError( f"{path}: empty trajectory file" ) + header = json.loads( lines[ 0 ] ) + if header.get( "kind" ) != "degradeloop-trajectory-header": + raise ValueError( f"{path}: first line is not a degradeloop-trajectory-header row" ) + rows = [ json.loads( l ) for l in lines[ 1: ] ] + return header, rows + + +# --------------------------------------------------------------------------- +# The four instruments (design doc §2.3) +# --------------------------------------------------------------------------- + +def instrument_1_cumulative_regression_slope( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Paired by seed task (same index in both lists = same seed task). Expects each row to carry + 'regressions' (int, from measure.QualityDeltaSnapshot.regressions, baseline-anchored per the + design doc §2.2's instruction to measure against state[0], never state[i-1]).""" + if not gated_trajs or len( gated_trajs ) != len( ungated_trajs ): + return { "available": False, "reason": "need equal, nonzero paired seed-task counts for both arms" } + gated_slopes = [ _slope( [ r[ "regressions" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "regressions" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) # ungated - gated: positive means gate reduced slope + return { + "available": True, + "gated_slopes": gated_slopes, + "ungated_slopes": ungated_slopes, + "paired_test": test, + "interpretation": "test is on (ungated_slope - gated_slope); z > 0 means ungated slopes ranked " + "higher (gate associated with a LOWER regression slope), z < 0 the opposite. " + "Check the sign of z, not just p_two_sided, before claiming a direction.", + } + + +def instrument_2_terminal_state( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 2 — the weaker, paper-shaped claim (end state only).""" + if not gated_trajs or len( gated_trajs ) != len( ungated_trajs ): + return { "available": False, "reason": "need equal, nonzero paired seed-task counts for both arms" } + gated_terminal = [ traj[ -1 ][ "regressions" ] for traj in gated_trajs if traj ] + ungated_terminal = [ traj[ -1 ][ "regressions" ] for traj in ungated_trajs if traj ] + if len( gated_terminal ) != len( gated_trajs ): + return { "available": False, "reason": "at least one trajectory had zero iterations" } + test = wilcoxon_signed_rank( ungated_terminal, gated_terminal ) + return { "available": True, "gated_terminal": gated_terminal, "ungated_terminal": ungated_terminal, "paired_test": test } + + +def instrument_3_subbar_growth( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 3 — expects each row to carry 'subbar_growth_total' (a float/int + the row-producer computed via measure.measure_subbar_growth, itself UNIMPLEMENTED in + measure.py). Refuses rather than guessing when the field is absent, per this project's own + honesty-in-output convention (src/quality.h: "a zero means none found, never none exists").""" + if not gated_trajs or not ungated_trajs: + return { "available": False, "reason": "no trajectories supplied" } + if any( "subbar_growth_total" not in r for traj in ( gated_trajs + ungated_trajs ) for r in traj ): + return { + "available": False, + "reason": "rows are missing subbar_growth_total — measure.measure_subbar_growth is " + "unimplemented in this scaffolding (see measure.py); this is NOT the same as " + "the instrument measuring zero growth.", + } + gated_slopes = [ _slope( [ r[ "subbar_growth_total" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "subbar_growth_total" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) + return { "available": True, "gated_slopes": gated_slopes, "ungated_slopes": ungated_slopes, "paired_test": test } + + +def instrument_4_security_trajectory( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Design doc §2.3 instrument 4 — expects 'security_finding_count' per row, from + measure.run_security_scanner (UNIMPLEMENTED in measure.py until a scanner is pinned).""" + if not gated_trajs or not ungated_trajs: + return { "available": False, "reason": "no trajectories supplied" } + if any( "security_finding_count" not in r for traj in ( gated_trajs + ungated_trajs ) for r in traj ): + return { + "available": False, + "reason": "rows are missing security_finding_count — no scanner is pinned yet " + "(measure.run_security_scanner is unimplemented); see README.md.", + } + gated_slopes = [ _slope( [ r[ "security_finding_count" ] for r in traj ] ) for traj in gated_trajs ] + ungated_slopes = [ _slope( [ r[ "security_finding_count" ] for r in traj ] ) for traj in ungated_trajs ] + test = wilcoxon_signed_rank( ungated_slopes, gated_slopes ) + return { "available": True, "gated_slopes": gated_slopes, "ungated_slopes": ungated_slopes, "paired_test": test } + + +def report_all( gated_trajs: list[list[dict]], ungated_trajs: list[list[dict]] ) -> dict: + """Always emits all four instruments — an unavailable one is reported with its reason, never + dropped from the table. See design doc §2.3's selective-reporting warning.""" + return { + "instrument_1_cumulative_regression_slope": instrument_1_cumulative_regression_slope( gated_trajs, ungated_trajs ), + "instrument_2_terminal_state": instrument_2_terminal_state( gated_trajs, ungated_trajs ), + "instrument_3_subbar_growth": instrument_3_subbar_growth( gated_trajs, ungated_trajs ), + "instrument_4_security_trajectory": instrument_4_security_trajectory( gated_trajs, ungated_trajs ), + } + + +def main() -> int: + ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) + ap.add_argument( "--gated", nargs="+", type=Path, default=[], help="trajectory JSONL files, arm=gated, one per seed task" ) + ap.add_argument( "--ungated", nargs="+", type=Path, default=[], help="trajectory JSONL files, arm=ungated, one per seed task, SAME ORDER as --gated" ) + ap.add_argument( "--selftest", action="store_true", help="run the built-in synthetic sanity check and exit" ) + args = ap.parse_args() + + if args.selftest: + return _selftest() + + if not args.gated or not args.ungated: + print( "error: --gated and --ungated each need at least one trajectory file (or pass --selftest)", file=sys.stderr ) + return 1 + + gated_trajs = [ load_trajectory( p )[ 1 ] for p in args.gated ] + ungated_trajs = [ load_trajectory( p )[ 1 ] for p in args.ungated ] + print( json.dumps( report_all( gated_trajs, ungated_trajs ), indent=2 ) ) + return 0 + + +def _selftest() -> int: + """Sanity check ONLY — synthetic data, not a claim about anything real. Checks that (a) a + clearly-lower-slope gated arm is detected as such in instrument 1, and (b) an + all-fields-missing input correctly reports unavailable for instruments 3/4 rather than + crashing or silently defaulting to zero.""" + gated = [ [ { "regressions": r } for r in [ 0, 1, 1, 2, 2 ] ] for _ in range( 8 ) ] + ungated = [ [ { "regressions": r } for r in [ 0, 2, 4, 6, 8 ] ] for _ in range( 8 ) ] + report = report_all( gated, ungated ) + i1 = report[ "instrument_1_cumulative_regression_slope" ] + assert i1[ "available" ], i1 + assert all( g < u for g, u in zip( i1[ "gated_slopes" ], i1[ "ungated_slopes" ] ) ), i1 + i3 = report[ "instrument_3_subbar_growth" ] + assert i3[ "available" ] is False and "subbar_growth_total" in i3[ "reason" ], i3 + print( "selftest OK (synthetic data only — not a real result)", file=sys.stderr ) + print( json.dumps( report, indent=2 ) ) + return 0 + + +if __name__ == "__main__": + sys.exit( main() ) diff --git a/bench/degradeloop/measure.py b/bench/degradeloop/measure.py new file mode 100644 index 000000000..c791a4a25 --- /dev/null +++ b/bench/degradeloop/measure.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""measure.py — deterministic per-iteration measurement layer for bench/degradeloop. + +UNTESTED. Written alongside docs/research/gate-vs-iterative-degradation.md as scaffolding for +whoever runs the actual harness; no function in this file has been exercised against a real +ripwire binary or a real code tree. Read the design doc (../../docs/research/ +gate-vs-iterative-degradation.md, sections 2.2 and 2.3) before changing what gets measured — +these functions exist to implement that section, not the other way around. + +WHAT THIS DOES. Wraps two ripwire verbs as subprocess calls against a scratch working tree +(--quality-delta --json, --test-gate --json), both of which docs/COMMANDS.md's --json entry +confirms are on the JSON allow-list with keys mirroring the XML attribute names 1:1. Also holds +an unimplemented seam for a public security scanner (run_security_scanner) — deliberately not +guessed at here; see the design doc's "what we'd like help with" for why the choice is left open. + +WHAT THIS DOES NOT DO. It does not interpret a --quality-delta exit code as "a real defect was +introduced." The design doc's Part 1 section 7 backtests --quality-delta against this project's +own commit history and finds it behaves as a debt ratchet (58% recall vs 40% false-alarm rate at +a 5-commit window) rather than a defect detector. Every function here reports counts, not +verdicts, and the analysis layer (analyze_trajectory.py) is responsible for any claim built on +top of them. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from dataclasses import dataclass, field + + +def _resolve_ripwire_bin() -> str: + """Refuse to fall back to a bare 'ripwire' on PATH — a stale system install would silently + produce numbers from a different binary than the one the run's metadata claims. RIPWIRE_BIN + must be set explicitly to a real build/ripwire; see README.md "before running any of this".""" + bin_path = os.environ.get( "RIPWIRE_BIN" ) + if not bin_path: + raise RuntimeError( + "RIPWIRE_BIN is not set. Point it at a build/ripwire built from this repository's " + "CLAUDE.md instructions — measure.py will not guess at a PATH install." + ) + if not os.path.isfile( bin_path ) or not os.access( bin_path, os.X_OK ): + raise RuntimeError( f"RIPWIRE_BIN={bin_path!r} is not an executable file." ) + return bin_path + + +def _run_ripwire_json( tree_dir: str, *verb_args: str ) -> dict: + """Run one ripwire verb with --json against tree_dir and parse the result. Raises on a + nonzero-but-unexpected exit rather than silently returning an empty dict — --quality-delta and + --test-gate both use exit codes as part of their contract (exit 2 / exit 4 respectively are + expected "gating" outcomes, not errors), so this only raises on exit codes neither verb + documents (see docs/COMMANDS.md for the documented exit-code contract of each verb before + changing the accepted set below).""" + binp = _resolve_ripwire_bin() + cmd = [ binp, tree_dir, *verb_args, "--json" ] + proc = subprocess.run( cmd, capture_output=True, text=True ) + # --quality-delta: exit 0 (clean) or exit 2 (gating regressions). --test-gate: exit 0 or exit 4. + if proc.returncode not in ( 0, 2, 4 ): + raise RuntimeError( + f"unexpected exit {proc.returncode} from {cmd!r}\nstdout={proc.stdout!r}\nstderr={proc.stderr!r}" + ) + if not proc.stdout.strip(): + # An empty stdout on a documented-refusal exit path (see --json's ALLOW-list refusal + # shape in docs/COMMANDS.md) is a real outcome, not a parse failure — but the caller needs + # to know, so this is surfaced rather than coerced into {}. + raise RuntimeError( f"empty stdout from {cmd!r} (exit {proc.returncode}); stderr={proc.stderr!r}" ) + return json.loads( proc.stdout ) + + +@dataclass +class QualityDeltaSnapshot: + """One --quality-delta --json reading, trimmed to the fields the design doc's instruments + need. Field names mirror the XML attribute spellings from docs/COMMANDS.md's --quality-delta + section verbatim (including the hyphens, carried as dict-style access below) rather than + renaming them into Python convention — a mismatch between this dataclass and a future + --quality-delta output change should be loud, not silently absorbed by a renamed field.""" + + regressions: int + gating: int + minor: int + acked: int + per_kind_gating: dict = field( default_factory=dict ) # kind -> count, computed from the row list + raw: dict = field( default_factory=dict ) # the full parsed JSON, for anything not modeled above + + +def measure_quality_delta( tree_dir: str, baseline_ref: str | None = None ) -> QualityDeltaSnapshot: + """One snapshot of the ten kinds against tree_dir. + + baseline_ref, if given, is passed as --quality-delta=..HEAD (the ref-pair form) so the + design doc's section 2.2 instruction — measure against the FIXED seed-state baseline, not + against the previous iteration — can be honored without needing a .ripwire_quality_baseline + sidecar dance on every iteration. If None, measures the bare working-tree-vs-HEAD form. + """ + args = [ f"--quality-delta={baseline_ref}..HEAD" ] if baseline_ref else [ "--quality-delta" ] + data = _run_ripwire_json( tree_dir, *args ) + # The JSON root mirrors the XML root's attributes; per-kind breakdown rides on the row list, + # not the root, so it's folded here rather than assumed present at the top level. + per_kind: dict = {} + for row in data.get( "regressions_detail", data.get( "rows", [] ) ): + k = row.get( "kind" ) + if k: + per_kind[ k ] = per_kind.get( k, 0 ) + 1 + return QualityDeltaSnapshot( + regressions=int( data.get( "regressions", 0 ) ), + gating=int( data.get( "gating", 0 ) ), + minor=int( data.get( "minor", 0 ) ), + acked=int( data.get( "acked", 0 ) ), + per_kind_gating=per_kind, + raw=data, + ) + + +@dataclass +class TestGateSnapshot: + tests: int + untested: int + raw: dict = field( default_factory=dict ) + + +def measure_test_gate( tree_dir: str ) -> TestGateSnapshot: + """--test-gate --json against the working tree (default = git diff, per docs/COMMANDS.md).""" + data = _run_ripwire_json( tree_dir, "--test-gate" ) + return TestGateSnapshot( + tests=int( data.get( "tests", 0 ) ), + untested=int( data.get( "untested", 0 ) ), + raw=data, + ) + + +def measure_subbar_growth( tree_dir: str, baseline_ref: str ) -> dict: + """The design doc's §2.2/§2.3 instrument 3: raw ccx/LOC/nest/params growth per touched symbol, + regardless of whether any single step crossed a --quality-delta bar. NOT part of + --quality-delta's own reported kinds (which only fire on a bar crossing or, above the bar, a + material-growth threshold — see src/quality.h's kMaterialGrowthPct/kSubBarGrowthPct) — this is + exactly the accumulation a bar-gated report can be blind to, which is why the design doc treats + it as a separate instrument. + + UNIMPLEMENTED. Computing this needs the per-symbol raw metrics for both trees, which + --quality-delta's own JSON does not expose below its minor/major-severity rows (deliberately — + it reports regressions, not a full metrics dump). The two ways to get it: (a) --metrics --json + on both trees and a symbol-identity join done here, matching quality.h's own canonId scheme + (path::scope::name — see docs/COMMANDS.md's --quality-delta LIMIT clause on renamed/moved + symbols), or (b) a small ripwire patch that exposes sub-bar deltas directly. Left unimplemented + rather than approximated, because an approximated version of exactly the instrument meant to + catch quiet accumulation would be worth less than nothing. + """ + raise NotImplementedError( + "sub-bar growth needs a --metrics-based symbol join (or a ripwire-side addition) that " + "this scaffolding does not implement — see the docstring above before filling this in." + ) + + +def run_security_scanner( tree_dir: str, language: str ) -> dict: + """UNIMPLEMENTED — placeholder for the public, deterministic security scanner named in the + design doc's §2.2 ("Semgrep's default ruleset, or a language-appropriate equivalent"). Left + unimplemented because the choice needs pinning (tool version + ruleset hash) per language + before it means anything as an instrument — see docs/research/gate-vs-iterative-degradation.md + "what we would like help with". A caller that needs this today should pin its own scanner + invocation here rather than treating this function's absence as "no scanner was run" silently: + call sites MUST check for NotImplementedError and record scanner_ran=false in the trajectory + row rather than defaulting a finding count to 0 (a 0 here must never be mistaken for "clean"). + """ + raise NotImplementedError( + f"no security scanner pinned yet for language={language!r} — see README.md before running." + ) + + +if __name__ == "__main__": + print( __doc__, file=sys.stderr ) + sys.exit( 1 ) diff --git a/bench/degradeloop/run_degradeloop.py b/bench/degradeloop/run_degradeloop.py new file mode 100644 index 000000000..71d06d44e --- /dev/null +++ b/bench/degradeloop/run_degradeloop.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""run_degradeloop.py — orchestrates one arm of the iterative-improvement loop for one seed task. + +UNTESTED. Scaffolding for docs/research/gate-vs-iterative-degradation.md §2.1/§2.4. Nobody has +run this — call_model() below raises NotImplementedError on purpose rather than faking a call +site that looks wired up. Filling that one function in is the only thing standing between this +script and a real run; everything else (git bookkeeping, measurement calls, trajectory recording) +is written to work once it is. + +USAGE (once call_model() is implemented and RIPWIRE_BIN is set, per README.md): + python3 run_degradeloop.py \\ + --seed-dir /path/to/seed/task/checkout \\ + --arm gated \\ + --iterations 10 \\ + --model-name "" --model-version "" \\ + --out trajectory_seed1_gated.jsonl + +Arms (design doc §2.1/§2.4): ungated | gated | neutral-control | wrong-target. "wrong-target" +needs --wrong-target-report pointing at a previously recorded --quality-delta JSON report from a +DIFFERENT seed task, per the design doc's arm D. +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +import measure + + +IMPROVE_PROMPT = "Improve this code." # deliberately the paper's own minimal prompt — see design doc §2.1 + + +def call_model( prompt: str, current_code: str, *, model_name: str, model_version: str ) -> str: + """THE seam. Takes a prompt and the current state of the seed task's code, returns the model's + proposed new code. This is the only function in bench/degradeloop that this scaffolding does + not and cannot implement — it needs real model access, which this environment does not have + (see the design doc §2.6: "no model calls were made to produce this document"). + + Contract for whoever fills this in: (1) deterministic sampling settings (fixed seed/temperature + if the API supports it) recorded in the trajectory header, not just in code comments here — + see main()'s header-writing below; (2) the SAME model_name/model_version across every arm and + every iteration of one seed task's comparison, or the arms are not comparable; (3) return the + FULL new code, not a diff — the measurement layer re-ingests the whole tree each iteration. + """ + raise NotImplementedError( + "call_model() is the one unimplemented seam in this harness — wire it to a real model " + "before running. See this function's docstring and README.md." + ) + + +def build_prompt( arm: str, base_prompt: str, previous_report: dict | None, wrong_target_report: dict | None ) -> str: + """Assemble the per-iteration prompt for the given arm. See design doc §2.1 (loop shape) and + §2.4 (the neutral-text / wrong-target confound controls) for why each arm's prompt looks the + way it does — this function is the literal implementation of that section, not a paraphrase of + it, so a change here should be a change there too.""" + if arm == "ungated": + return base_prompt + if arm == "gated": + if previous_report is None: + return base_prompt # iteration 1 has no prior report yet + return ( + base_prompt + + "\n\nThe deterministic quality gate reports the following about your PREVIOUS change " + "to this code. Address what you can before proposing the next change:\n\n" + + json.dumps( previous_report, indent=2 ) + ) + if arm == "neutral-control": + # Matched-LENGTH neutral text, not matched-content — see design doc §2.4 arm C. Using the + # tool's own --legend text (long, dense, describes the checker rather than this code) as + # the filler is the design doc's suggestion; swap in whatever's pinned for the actual run. + filler = _neutral_filler_text() + return base_prompt + "\n\n" + filler + if arm == "wrong-target": + if wrong_target_report is None: + raise ValueError( "arm=wrong-target requires --wrong-target-report" ) + return ( + base_prompt + + "\n\nA deterministic quality gate reports the following (NOTE: about a DIFFERENT " + "codebase, used here only as a matched-content control — see design doc §2.4 arm D):\n\n" + + json.dumps( wrong_target_report, indent=2 ) + ) + raise ValueError( f"unknown arm {arm!r}" ) + + +def _neutral_filler_text() -> str: + """Returns the matched-length filler text for the neutral-control arm. Placeholder: a real run + should call `ripwire --quality-delta --legend` (or any fixed, task-irrelevant, roughly + report-length text) ONCE per run and reuse it, rather than regenerating it per iteration — the + text must be IDENTICAL across iterations and across seed tasks within the control arm, or the + control stops controlling for length and starts introducing its own variable content.""" + raise NotImplementedError( + "pin a fixed neutral-text block before running the neutral-control arm — see design doc §2.4." + ) + + +def run_arm( + seed_dir: Path, + arm: str, + iterations: int, + model_name: str, + model_version: str, + wrong_target_report_path: Path | None, +) -> list[dict]: + """Runs `iterations` rounds of the loop for one seed task under one arm, in a scratch git + clone of seed_dir (never the original — see the warning in main()). Returns the list of + per-iteration trajectory rows; does not write them (main() does, so a caller composing several + arms in one process can hold them all before writing).""" + scratch = Path( tempfile.mkdtemp( prefix=f"degradeloop-{arm}-" ) ) + tree = scratch / "tree" + shutil.copytree( seed_dir, tree ) + subprocess.run( [ "git", "-C", str( tree ), "init", "-q" ], check=True ) + subprocess.run( [ "git", "-C", str( tree ), "add", "-A" ], check=True ) + subprocess.run( + [ "git", "-C", str( tree ), "-c", "user.name=degradeloop", "-c", "user.email=degradeloop@invalid", + "commit", "-q", "-m", "seed" ], + check=True, + ) + baseline_ref = subprocess.run( + [ "git", "-C", str( tree ), "rev-parse", "HEAD" ], check=True, capture_output=True, text=True + ).stdout.strip() + + wrong_target_report = None + if wrong_target_report_path is not None: + wrong_target_report = json.loads( wrong_target_report_path.read_text() ) + + rows: list[dict] = [] + previous_report: dict | None = None + current_code_dir = tree + + for i in range( 1, iterations + 1 ): + prompt = build_prompt( arm, IMPROVE_PROMPT, previous_report, wrong_target_report ) + + # NOTE: single-file-tree simplification. A real seed task with many files needs a real + # (prompt, tree) -> tree contract instead of (prompt, one code string) -> one code string; + # this scaffolding assumes call_model operates over a concatenated or single-entry-point + # representation and that whoever implements call_model() handles multi-file state. Left + # simple on purpose rather than guessing a serialization format nobody has asked for yet. + current_code = "\n".join( p.read_text() for p in sorted( current_code_dir.rglob( "*" ) ) if p.is_file() ) + new_code = call_model( prompt, current_code, model_name=model_name, model_version=model_version ) + # Caller-side responsibility once call_model is real: write new_code back into tree, + # respecting whatever multi-file contract was chosen above, before the commit below. + raise NotImplementedError( + "run_arm() cannot proceed past the first call_model() call in this environment — " + "this RuntimeError is expected until call_model() and the write-back step are filled in." + ) + + return rows # unreachable until the seam above is implemented; kept for the intended shape + + +def main() -> int: + ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter ) + ap.add_argument( "--seed-dir", required=True, type=Path, help="checkout of the seed task's starting code" ) + ap.add_argument( "--arm", required=True, choices=[ "ungated", "gated", "neutral-control", "wrong-target" ] ) + ap.add_argument( "--iterations", required=True, type=int ) + ap.add_argument( "--model-name", required=True ) + ap.add_argument( "--model-version", required=True, help="date or version string — recorded, not validated" ) + ap.add_argument( "--wrong-target-report", type=Path, default=None, + help="required for --arm=wrong-target; a --quality-delta --json report from a DIFFERENT seed task" ) + ap.add_argument( "--out", required=True, type=Path ) + args = ap.parse_args() + + if not args.seed_dir.is_dir(): + print( f"error: --seed-dir {args.seed_dir} is not a directory", file=sys.stderr ) + return 1 + print( + "WARNING: this harness is scaffolding and has never been run end to end. " + "call_model() will raise NotImplementedError. See README.md.", + file=sys.stderr, + ) + + header = { + "kind": "degradeloop-trajectory-header", + "arm": args.arm, + "seed_dir": str( args.seed_dir ), + "iterations_requested": args.iterations, + "model_name": args.model_name, + "model_version": args.model_version, + "recorded_at_unix": int( time.time() ), + "ripwire_bin": measure._resolve_ripwire_bin(), # fails fast if RIPWIRE_BIN unset — intentional + } + + try: + rows = run_arm( + args.seed_dir, args.arm, args.iterations, args.model_name, args.model_version, + args.wrong_target_report, + ) + except NotImplementedError as e: + print( f"stopped (expected, scaffolding-only): {e}", file=sys.stderr ) + return 2 + + with args.out.open( "w" ) as f: + f.write( json.dumps( header ) + "\n" ) + for row in rows: + f.write( json.dumps( row ) + "\n" ) + return 0 + + +if __name__ == "__main__": + sys.exit( main() ) diff --git a/docs/README.md b/docs/README.md index a9ba460b8..4d459a24f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ # ripwire documentation -Sixteen entries, each written for one reader. Start with the row that matches why you are here. +Twenty-one entries, each written for one reader. Start with the row that matches why you are here. | File | Who it is for | What it answers | | --- | --- | --- | @@ -22,6 +22,7 @@ Sixteen entries, each written for one reader. Start with the row that matches wh | **[`gatecount_build.py`](gatecount_build.py)** | Maintainers, and anyone adding a gate | The generator behind the **published gate count**. Derives it from the single `for _g in …; do` loop in `test/regression.sh` and rewrites all eight marked sites across `README.md`, `EVALS.md` and the deck; `--check` is the drift comparison that `test/gatecountcheck.sh` runs. Never edit that number by hand — two lanes hand-writing the same N+1 auto-merge clean against a loop of N+2. | | **[`limits_classes.tsv`](limits_classes.tsv)** | Maintainers | Cap name -> INDEXING or OUTPUT, the taxonomy `LIMITS.md` renders in its `class` column: does this cap bound what can EVER be found, or only what is shown from what was found. A sidecar with a known expiry — the tag belongs on the declaration in `src/` — kept honest by `limitstablecheck.sh`, which fails if a row names a cap that no longer exists. | | **[`lineage-paper-dates.tsv`](lineage-paper-dates.tsv)** | Maintainers | arXiv id -> publication date for every 2026 paper in `LINEAGE.md`. The ID stem does not track the date (`2607.09691` was published 2026-06-19), so the README's recency claim is re-derived from this file by `readmedriftcheck.sh` arm (H2) rather than from the ids. Adding a 2026 paper without a date row fails that arm. | +| **[`research/`](research/)** | Anyone working on an open question here, or offering to help with one | Investigation notes: a question this tool has not answered, the measurement run against it, and the pre-registration that says what would license a change. Not a claim surface — a number here is local evidence for a decision not yet taken, and nothing in it is quoted on `README.md`. | | **[`assets/`](assets/)** | The front page | The README banner and tagline artwork (SVG, self-contained). | | **[`captures/`](captures/)** | Maintainers, and the curious | One recorded run of every verb against a real repository — the source of `COMMANDS.md`'s sample output, and the harvest source for the differential argv harness. | diff --git a/docs/research/gate-vs-iterative-degradation.md b/docs/research/gate-vs-iterative-degradation.md new file mode 100644 index 000000000..f8bd584cd --- /dev/null +++ b/docs/research/gate-vs-iterative-degradation.md @@ -0,0 +1,383 @@ +# Gate vs. iterative degradation — a research note + +**Status: first pass. A reading path plus a harness design, not a result.** Nothing here reports a +measured effect of a gate on a degradation trajectory — we have not run one. What follows is (1) a +reading path into `--quality-delta` for an outside researcher, written to stand alone, and (2) a +concrete design for the experiment that would actually answer the question, including the shape of +result that would prove the design's own premise wrong. + +## The paper this responds to + +Shivani Shukla, Himanshu Joshi and Romilla Syed, "Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the +Paradox," accepted IEEE-ISTAS 2025, [arXiv:2506.11022](https://arxiv.org/abs/2506.11022). The paper +runs a controlled experiment — 400 code samples, 40 rounds, four prompting strategies — asking an +LLM to iteratively "improve" its own code, and finds critical vulnerabilities increase 37.6% after +five iterations. It names the mechanism "feedback loop security degradation": a model optimizing for +its own stated goal (functionality, readability, the thing the prompt asked for) trades away a +property nobody in the loop is measuring. The paper's own recommendation is human validation between +iterations — an external check, not another self-critique pass. + +That is the premise this note takes seriously, generalized past security to code quality broadly: +if the reason iterative self-improvement degrades is that nothing *external and stable* is measuring +what degrades, the fix is not a better prompt, it is an oracle the model cannot rationalize past. This +is a design **hypothesis**, not a result borrowed from the paper — the paper does not test +`--quality-delta` or anything resembling it, and Part 2 exists because this note is not entitled to +claim otherwise. + +## Where this hypothesis actually lives — and why not `docs/LINEAGE.md` + +`docs/LINEAGE.md` folds a work only when "the lesson taken from it can be named in one sentence +**and** pointed at a real flag or source file" — a citation for something the paper *caused* us to +build. That is not this. `--quality-delta` predates this note; nothing about it changed on reading +Shukla et al. 2025. Filing this as a lineage row would misrepresent the tool's history to buy a citation. + +The actual record of this position is the design comment at the top of `src/quality.h` (the file +that implements `--quality-delta`), which already states the mechanism this note is arguing from — +without citing this specific paper: + +> `quality.h` — `--quality-baseline` / `--quality-delta`: the deterministic oracle for a code-quality +> **CONVERGENCE LOOP**. … the "delta, not absolute" discipline that lets a refine loop target *the +> regression it introduced* instead of chasing absolute numbers (**the defense against Goodhart / +> metric-gaming**). + +That comment is design intent recorded where the code lives, not a claim staked in a document meant +to be an honest ledger of external influence. This note is the first place the two are put side by +side on purpose. If a later change actually adapts something from Shukla et al. 2025 — the vulnerability +taxonomy, the round-count design, a specific finding — *that* change earns a `LINEAGE.md` row at the +time it lands, not retroactively from this note. + +--- + +## Part 1 — a reading path into `--quality-delta`, for an outside researcher + +Read in this order. Each step names the file, and what you're checking for. + +### 1. Start here + +- `./build/ripwire --help` and [`docs/COMMANDS.md`](../COMMANDS.md) (`--quality-delta`, + `--quality-delta=REV|A..B`, `--quality-baseline`, `--quality-ack`, `--ack-only`, `--dmm`) — the + generated, always-current flag reference. If this note disagrees with `--help`, `--help` is right. +- `src/quality.h` lines 1–20 — the file's own one-paragraph design statement (quoted above in full + context). It states the Goodhart defense explicitly: report only the *regression*, never an + absolute score, because an absolute score is a target a model can learn to game without fixing + anything. + +### 2. The ten kinds, and where each is computed + +All ten live in `src/quality.h`. `computeSnapshot` (line 3959) builds the per-symbol/per-group floor +from one tree; `computeDelta` (the function whose body spans roughly line 6700–7200) compares two +snapshots and emits only what got worse. Per-kind entry points (line numbers at `origin/main` +`755f9026`, this branch's base — they drift as the file changes, so treat them as a starting point, +not a pin): + +| kind | bar / threshold | computed at | +| --- | --- | --- | +| `complexity` | ccx > 15 (`kCcxBar`), AND grew | `perSymbolKind( "complexity", … )`, `src/quality.h:6860` | +| `verbosity` | LOC > 60 (`kLocBar`), AND grew | `perSymbolKind( "verbosity", … )`, `src/quality.h:6861` | +| `nesting` | max-nest > 4 (`kNestBar`), AND grew | `perSymbolKind( "nesting", … )`, `src/quality.h:6862` | +| `params` | param count > 5 (`kParamBar`), AND grew | `perSymbolKind( "params", … )`, `src/quality.h:6863` | +| `duplication` | new/grown clone group ≥ `kMinCloneTokens` (18) | `src/quality.h:7000`, clone detection in `src/clones.h` | +| `dead-code` | zero in-edges, not a registered/test/fixture exemption | `isDeadCandidate`, `src/quality.h:648`; emitted `src/quality.h:7029` | +| `api-surface` | a symbol becomes public, or a public signature's arity changes | `src/quality.h:7115` (visibility flip), `:7148` (arity/contract change) | +| `error-masking` | a construct in `findErrorMasking`'s built-in rule table (`src/lintrules.h`) | `errorMaskCountsBySym`, `src/quality.h:765`; emitted `:7183` | +| `short-horizon-churn` | a file rewritten ≥2 times inside a 14-day window AND touched by this diff | `src/quality.h:7310`, mined via `gitmine.h` | +| `new-clone-of-reused-helper` | a fresh clone of a helper whose existing fan-in ≥ 3 | `src/quality.h:7392` | + +Four numeric kinds (`complexity`, `verbosity`, `nesting`, `params`) share one generic +`perSymbolKind` driven by a lambda that reads the metric off `Symbol` — read that one function and +you have read all four. The other six are presence/structural kinds with their own emission sites, +listed above. + +**The materiality design, which the paper's own framing makes relevant.** A regression whose delta +is below a per-kind "minor" threshold (`kMinorCcxDelta = 3`, `kMinorLocDelta = 10`, +`kMinorParamDelta = 2`; nesting and the presence kinds have none — any instance is major) is reported +but does not gate exit 2 on its own. This exists because a +1-ccx edit to a function already over the +bar is technically a regression but is noise that would drown the findings a refine loop should +actually chase (`src/quality.h` comment above `kMinorCcxDelta`). If a degradation trajectory is a +slow accumulation of small unmeasured moves rather than one visible break, this threshold is exactly +where a gate could be *blind* to it by design — see §2.2's growth-rate instrument, added for this +reason. + +### 3. The baseline model + +Two floors, chosen automatically (`docs/COMMANDS.md` `--quality-delta` section, and +`computeHeadSnapshot`, `src/quality.h:3525`): + +- **Sidecar** (`.ripwire_quality_baseline`, written by `--quality-baseline`) — honored only when + pinned at exactly the current `git HEAD` (strict equality; an ancestor is a different tree). Stale + → self-heals by deleting the file and falling back to the second floor. `--quality-baseline` + itself **refuses** (exit 1) to pin on a dirty tree unless you pass `--allow-dirty`, specifically so + the debt already in a working tree cannot be swallowed into the floor and read clean forever after. +- **git-HEAD** (no sidecar, or a stale one) — `computeHeadSnapshot` (`src/quality.h:3525`) does + `git archive HEAD` into a temp directory and re-ingests it (`materializeCommitTree`, + `src/quality.h:3345`), then compares the working tree against that. + +### 4. The fix that just landed — a clean tree must never gate + +`$ORCH/reports/t12-qd-noop.md` (this orchestration round, branch `lane/t12-qd-noop-diff`, head +`d54ce3da`). The mechanism: the git-HEAD floor is built by archiving and re-ingesting HEAD into a +temp directory, which is a **different file population** from the working tree whenever anything is +untracked, gitignored, export-ignored, sparse-checked-out, or otherwise present on one side and not +the other — a shallow clone, `.gitignore`d duplicate, or skip-worktree flag all reproduce it. Because +a dead-code verdict is a property of the *whole population* a symbol is ingested with, not of its own +file, a file present on only one side can flip a same-named definition's dead/alive verdict on a +symbol *both* sides share — and a no-op diff gates. Three review rounds converged on the same defect +class reached three different ways; the final fix (round 3, `d54ce3da`) replaces a hand-written model +of git's file-selection rules with git's own answer, rather than patching the model a third time. This +matters to a researcher reading the tool cold: it is the concrete shape of "the gate itself has bugs," +and it is now closed for the no-op case specifically — read the report for what's still scoped out +(the Django `cls.`/`self.` dispatch half of the same issue, tracked separately, #237). + +### 5. The ack ledger, and why it exists + +`.ripwire_quality_acks` (format documented at the top of the file itself, and in +`--quality-ack`/`--ack-only` in `docs/COMMANDS.md`). An ack records a **reviewed** finding as known, +suppressing it until it *worsens past its acked magnitude* — a ratchet, not a mute. `--ack-only=KIND` +exists because bare `--quality-ack` accepts the whole current report, and accepting one deliberate +change alongside everything else unacked turns a ratchet into a rubber stamp — the tool's own +documentation names this failure mode explicitly and gives the scoped form as the way to avoid it. +For the degradation question this ledger is a hazard to control for: an iteration loop with a +human (or an agent) free to ack findings can make the *gate* report clean while debt still +accumulates under the ack. §2.4 treats acking as something the harness must record, not something it +assumes away. + +### 6. The eval harness + +[`docs/EVALS.md`](../EVALS.md) §1 tables every instrument this project has and what each measures; +§6 has `--quality-delta`'s own kind list restated with the exact `kind=` strings; §7 is a section of +*honest counterexamples* — measured findings that went against the tool's own claims, published on +purpose. `bench/` holds the harness code itself: `bench/recalleval/`, `bench/headtohead/`, +`bench/locbench/`, `bench/ensemblecal/`, each with its own README. `bench/ensemblecal/` is the +closest structural relative to what Part 2 proposes: it separately verifies a calibration hypothesis +(that four evidence families are orthogonal) with a stated honesty contract, and reports what it +would take for the hypothesis to be wrong. + +### 7. What our own backtest says — read this before citing the tool as a detector + +`$ORCH/reports/study-checks.md` §3 (backtest against ripwire's own commit history, `W` = window in +commits): at the realistic five-commit window, `--quality-delta` fires on **14/24 (58%) of states +that provably contained a finding somebody later fixed, and on 12/30 (40%) of control states with no +recorded defect.** 58% vs 40% is not discrimination — it is closer to a coin flip weighted by how +much code moved. The kind firing most on *defect-free* control states is `complexity` (10 of 30 +control reds), the same kind whose per-symbol external-corpus signal (§4 of that report) is the one +structural metric that *does* separate human-authored defects from non-defects at 1.63× (95% CI +[1.11, 2.38]). Read plainly: `--quality-delta` measures accumulated debt, and debt correlates with +defect-proneness in the literature and in our own external check, but firing on a state is not the +same claim as identifying a defect in it. The tool's own documentation says "report only what got +worse … descriptive," and this backtest is the first time that sentence has been measured rather than +asserted. Any harness in Part 2 that treats a `--quality-delta` exit 2 as "a real regression was +introduced" is overclaiming; treat it as "measured debt increased," which is a different, still +useful, and honestly weaker claim. + +### 8. Reading order, summarized + +`docs/COMMANDS.md` (`--quality-delta` family) → `src/quality.h` lines 1–20 → the ten-kind table +above with the file open beside it → `$ORCH/reports/t12-qd-noop.md` → `.ripwire_quality_acks` header +comment → `docs/EVALS.md` §1 and §6 → `$ORCH/reports/study-checks.md` §3. That is roughly 45 minutes +to a working mental model, and it ends exactly where Part 2 starts: with the honest limit of what the +gate has been shown to do. + +--- + +## Part 2 — a harness design for the degradation question + +**The question Part 1's reading path does not answer:** does a deterministic external gate change +the *trajectory* of repeated self-improvement — the shape of the curve across iterations — or does it +only catch individual regressions after they already happened, leaving the underlying trajectory the +same? These are different claims. A gate that catches every regression it sees but never changes what +the model tries next iteration would still let the *next* unmeasured failure mode (the one the gate +doesn't check) degrade freely — Shukla et al. 2025's whole point is that *something* always degrades when +only *some* things are measured. + +### 2.1 The loop + +For each of N seed tasks (real small-to-medium functions or modules, ideally drawn from more than one +language ripwire indexes, since the ten kinds and their bars are language-general but their density +of hits is not measured to be): + +``` +state[0] = seed code, committed +for i in 1..K: + prompt = "improve this code" + (gate feedback from state[i-1], IF gated arm) + state[i] = model(prompt, state[i-1]) + measure(state[i]) # §2.2, run regardless of arm + commit state[i] +``` + +Two arms per seed task, same model, same temperature, same seed code, same K: + +- **UNGATED** — the prompt is "improve this code" (plus whatever the task needs for continuity, e.g. + the running test suite's pass/fail if the task has one) and nothing else. This is the paper's own + setup. +- **GATED** — the prompt additionally receives the previous iteration's `--quality-delta` (and, if + in scope, `--test-gate`) report, unfiltered, as context the model is told to address before + proposing the next change. No human is in the loop; the gate is the only external signal. This is + deliberately the *weakest* form of "external and deterministic" — it tests whether the model + reading the gate's own words changes behavior, not whether a human enforcing exit 2 would (a + stronger, cheaper-to-argue-for design that isn't the interesting question: of course a hard stop + changes what ships. The trajectory question is about the *code the model chooses to write*, not + about a merge gate.) + +K should be at least 10 — Shukla et al. 2025 reports the divergence sharpening between iteration 5 and 10; a +shorter loop cannot see whether a gate changes the *slope* rather than just one round's value. + +### 2.2 What is measured each iteration + +Everything below is computed fresh on `state[i]`, in both arms, whether or not that arm sees it in +its prompt — the ungated arm's own trajectory needs the same instruments or there's nothing to +compare against. + +**Deterministic, ripwire-native (the ten kinds, from `--quality-delta --json` against `state[0]` as +the fixed baseline — not against `state[i-1]`, so a regression that got partially fixed and +re-introduced two iterations later is still visible; see §2.3):** +- All ten kinds' counts (`regressions`, `gating`, `minor`, per-`kind=` breakdown). +- `api-new-surface` and `register-macro-excluded` as printed floors, for context. +- Clone growth specifically (`duplication` + `new-clone-of-reused-helper` counts and the clone + groups' member counts) — Shukla et al. 2025 doesn't measure this, but GitClear's agent-code findings that + motivate `--quality-delta`'s own kinds (cited in `src/quality.h`'s comments) do, and duplication is + cheap to accumulate silently under an "improve this" prompt that never says "don't repeat + yourself." + +**Deterministic, test-native:** +- Test pass/fail count, IF the seed task carries a test suite (`--test-gate --json`'s `tests=` / + `untested=`, or the task's own runner if simpler). A trajectory that improves quality kinds while + breaking tests is not a win for either arm's position. + +**Deterministic, growth-rate (not in `--quality-delta`'s own output, but derivable from the raw +snapshot each iteration writes):** +- Sub-bar growth that `--quality-delta`'s "minor" threshold (§1.2) would not gate: raw ccx/LOC/nest/ + params values per touched symbol, iteration over iteration, regardless of whether any single step + crossed a bar. This is the instrument for "the gate is blind to slow accumulation" — a real + possibility given how the minor-severity design works, and worth measuring even though it isn't + what the shipped tool reports. + +**Public and deterministic, non-ripwire:** +- A public static security scanner appropriate to the seed task's language — e.g. Semgrep's default + ruleset (multi-language, free, deterministic given a pinned ruleset version) or a + language-appropriate equivalent (Bandit for Python, `cargo audit`/clippy for Rust). This is the + instrument that actually answers the paper's own question (security), separate from ripwire's + structural kinds, which do not claim to measure security. **Pin the scanner's version and ruleset + hash in the run's metadata** — an unpinned scanner is not a repeatable instrument. + +**Recorded, not computed:** whether the gated arm's model actually *acted* on the previous report +(a crude proxy: did the flagged symbol change in the next iteration at all) and whether any +`--quality-ack` was exercised (it should not be, in this design — acking belongs to a human reviewer, +and an unattended loop acking its own findings would silently defeat the gate; if a later variant lets +the model ack, that has to be a separate, explicitly labeled arm). + +### 2.3 What "the trajectory changed" would mean, numerically — fixed in advance + +Stated before any run, per the project's own pre-registration discipline (`study-checks.md`'s +external arm is the house precedent: the sampling amendment was written and executed *before* the +first measurement). Candidate instruments, all computed per seed task then aggregated: + +1. **Cumulative-regression slope.** Fit `regressions[i]` (from §2.2, baseline-anchored so + regressions don't wash out when partially fixed) against iteration `i`, per arm, per seed task. + The claim "the gate changes the trajectory" requires the GATED slope to be statistically lower + than the UNGATED slope, paired by seed task (Wilcoxon signed-rank across seed tasks, not a pooled + t-test across iterations — iterations within one task's trajectory are not independent + observations). +2. **Terminal-state comparison.** `regressions[K]` and scanner-finding-count`[K]`, GATED vs UNGATED, + paired by seed task. This is the paper's own headline shape (end state after N rounds) and should + be reported even though it's the weaker claim (it can differ without the *trajectory* differing — + e.g. one late correction). +3. **Sub-bar growth rate** (§2.2's growth-rate instrument): does the GATED arm's raw metric growth + per iteration (summed across touched symbols, below any bar) differ from UNGATED's? This is the + instrument that would catch a gate that looks clean on `--quality-delta`'s own gating count while + debt still accumulates just under every bar — the concern §1.2 raises. +4. **Security-finding trajectory**, same slope test as (1), on the scanner's output — this is the one + that actually tests Shukla et al. 2025's claim in our setting, since none of ripwire's ten kinds are + security checks. + +**Pre-registered thresholds, to be set with the actual seed-task count and iteration count once +chosen (N and K bound the achievable power) — but the form is fixed now:** a slope difference is +"real" only if it clears a pre-specified effect size (not just p < 0.05 — with small N, significance +without an effect-size floor is how the ratchet-vs-detector confusion in §1.7 happens again one level +up), and the write-up reports the ungated-vs-gated comparison for **all four instruments**, not +whichever one came out favorable. Selective reporting of the metric that happened to move is the +single easiest way to manufacture the result this note wants to see. + +### 2.4 The confound: the gate also changes the prompt + +The GATED arm's prompt contains the previous quality-delta report; the UNGATED arm's does not. Any +measured difference could be "the model responded to structured, specific feedback about its own +code" rather than "the model responded to *this particular gate*" — a generic-text confound, not a +gate-specific effect. Two controls, both needed: + +- **Matched-length neutral-text control (arm C).** Same prompt structure as GATED, but the appended + text is a fixed, task-irrelevant block of roughly the same token length as a typical + `--quality-delta` report (e.g. the tool's own `--legend` text, which is long, dense, and describes + nothing about the current code). If GATED and C both outperform UNGATED equally, the effect is + "more context helps," not "the gate helps," and the paper's mitigation claim does not transfer to + this tool specifically. +- **Matched-content wrong-target control (arm D), if resourcing allows a fourth arm.** Feed the GATED + prompt a real `--quality-delta` report — but from a *different* seed task, not this one. This + isolates "specific, on-target feedback about this code" from "quality-delta-shaped feedback about + something." If GATED clearly beats D, the effect is that the gate is *reading the actual code*, not + merely primed by its vocabulary and structure. + +Report GATED against UNGATED, C, and D (as available) side by side. The claim this note is actually +interested in is GATED vs. C, not GATED vs. UNGATED — the paper's baseline is silence, and beating +silence with *any* structured feedback would be a weak, likely-true result that says nothing about +determinism or externality specifically. + +### 2.5 What result would count against our position + +Stated plainly, in advance, because this is the part most likely to be skipped: + +- **GATED does not clearly beat C (matched-length neutral text).** If a same-length block of + unrelated text produces the same trajectory improvement as the actual quality report, the + mechanism is "more tokens of structured-looking feedback," not "deterministic external + measurement," and `src/quality.h`'s design comment (the Goodhart-defense framing) would be + unsupported by this experiment specifically — it might still be true for other reasons, but this + harness would not have shown it. +- **GATED's terminal state is better but its slope is not.** This would mean the gate produces one + correction late in the loop (the model notices and fixes things right before the run ends) rather + than changing the trajectory throughout — closer to "catches regressions after the fact" than + "changes the trajectory," which is the distinction this whole design exists to draw, and the + weaker of the two claims would be the honest one to publish. +- **The security-scanner trajectory (instrument 4) doesn't track the ripwire-kind trajectory + (instrument 1) at all** — e.g. GATED improves structural debt but the scanner's finding count is + flat or worse across both arms. This would mean structural-quality gating and security are close to + orthogonal in this setting, which directly limits how far "a deterministic quality gate" can be + read as an answer to a *security*-framed paper, no matter what instrument 1–3 show. +- **Sub-bar growth (instrument 3) is worse in GATED than UNGATED.** This would be the sharpest + negative result: a model gaming the visible bar by keeping every individual metric just under + threshold while the underlying code gets worse in aggregate — literally the Goodhart failure mode + the gate exists to prevent, reappearing one level down. If this shows up, it belongs in + `src/quality.h`'s own comments as a documented limit, not quietly dropped from the writeup. + +Any one of these, on real data, is a more useful contribution than a clean win — this project's own +`docs/EVALS.md` §7 exists for exactly that reason, and a degradation-harness result belongs beside it +if it runs. + +### 2.6 What we did not, and could not, do here + +This is a design and a set of scaffolding scripts, not a study. Nobody has an LLM API wired into this +worktree, and no model calls were made to produce this document. The scripts in `bench/degradeloop/` +implement the deterministic half — the loop's bookkeeping, the measurement calls into +`--quality-delta`/`--test-gate`, and the trajectory-analysis math from §2.3 — with an explicit, +unimplemented seam where a real model call belongs, and every file says so at the top rather than +faking a call site that looks wired up. **No numbers in this document are results.** Anyone who runs +the harness and gets numbers should report them beside this design, including if they contradict +§2.5's stated position. + +--- + +## What we would like help with + +- **Seed tasks.** A good seed set needs real, non-trivial functions with a genuine "improve this" + prompt that isn't already solved — ideally spanning at least two of ripwire's indexed languages, so + the ten kinds' language-general bars actually get exercised differently. We don't have a vetted set. +- **A model-call adapter.** `bench/degradeloop/run_degradeloop.py`'s `call_model()` seam is written + against a plain (prompt, code) → code interface deliberately — wiring it to a specific API is a + few lines for whoever has one available and wants to run this. +- **The scanner choice per language**, pinned and justified — we picked Semgrep as the + cross-language default in §2.2 but have not evaluated whether its default ruleset has the recall to + see what Shukla et al. 2025's own taxonomy would flag. +- **A second opinion on the confound design (§2.4).** Arm D (matched-content, wrong-target) is the + one we're least sure earns its cost against arm C alone — if C is enough to isolate the effect, + D is one fewer arm to run. +- **Anyone who runs this** — even a small N, even one seed task, one language, K=10 — and reports the + four instruments honestly, favorable or not. That is worth more than a larger version of this + design document.