diff --git a/CHANGELOG.md b/CHANGELOG.md index 96e7bedef..419d40114 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,22 @@ not published here — see `docs/EVALS.md` for the instruments behind the headli --- +## [Unreleased] + +### Fixed — `--readability`'s caveat cited a proxy number that did not reproduce + +The `UNVALIDATED` note in `--help=--readability` and the README said the lens agrees with a refactor +commit's implied readability direction on 30.2% of 484 function pairs. That population came from +`bench/readability_refactor_pairs.py` walking `git log --all`, which reads every branch in the clone's +shared `.git`, so the count changed whenever an unrelated branch was pushed. With the script pinned to +the `v0.6.2` tag (#313), the same method (80 refactor/simplify/cleanup commits, ≤400 changed lines each) +gives **154 of 412 pairs, 37.4%**. That is still worse than chance, so the caveat's conclusion holds. +Both surfaces now cite the pinned figure and name the tag. The `--quality-delta` help also named its +tenth kind `reuse-decline`; it now uses the `kind=` string the binary emits, +`new-clone-of-reused-helper`. + +--- + ## [0.6.2] — 2026-09-21 ### Added — Microsoft's `cl.exe` builds the tree, so both Windows front ends compile and both gate diff --git a/README.md b/README.md index fdb1e5801..c8c433fd5 100644 --- a/README.md +++ b/README.md @@ -1161,10 +1161,11 @@ rather than blurring it: Siegmund, ICSE 2021, [doi:10.1109/ICSE43902.2021.00056](https://doi.org/10.1109/ICSE43902.2021.00056)) — `--readability` emits volume and stops there; difficulty and effort are computed nowhere in this tree. **Unvalidated against human judgement, stated plainly rather than assumed:** this is a - deterministic ordering signal, not a checked one. Our own proxy measurement — 484 matched + deterministic ordering signal, not a checked one. Our own proxy measurement — 412 matched before/after function pairs mined from 80 refactor/simplify/cleanup commits in this repository's own - history — found the lens agrees with the commit's implied readability direction on only 30.2% of - pairs, worse than chance. That is a construct-validity finding about the ordering claim, not a bug in + history, pinned to the `v0.6.2` tag rather than a walk of every branch in the clone, so the population cannot drift when an + unrelated branch is pushed — found the lens agrees with the commit's implied readability direction on + only 37.4% of pairs (154 of 412), worse than chance. That is a construct-validity finding about the ordering claim, not a bug in the arithmetic (a separate self-consistency check confirms the formula computes exactly what it says it computes); the lens itself is unchanged pending a proper human study, and `--help=--readability` carries the same caveat where a CLI reader meets it. diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index 8e491b610..b53867b7f 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -1839,7 +1839,7 @@ $ ./build/ripwire . --clones **Answers:** rank functions least-readable first, by volume, token entropy and length per-function readability lens, LEAST readable first: vol= Halstead volume V (N*log2(eta)), ent= Shannon token entropy E, lines= L, posnett= sigmoid(8.87 - 0.033V + 0.40L - 1.5E) (Posnett/Hindle/Devanbu, MSR 2011). -APPROXIMATION, disclosed: ONE token-class table serves every language (keywords + punctuation = operators, identifiers + literals = operands), with no per-grammar refinement, so V is cross-language and not a per-grammar Halstead count. The formula was fitted on snippets of 20 lines or fewer, so it is a RANKING lens, not a grade: read the ORDER of the rows, not the number on any one of them. Pages with limit=N (offset=M); default 40 rows. Declarations with no body are not measured. UNVALIDATED (t14-cleanup #8): this is a deterministic ORDERING signal that has not been checked against human judgement of readability. Our own proxy measurement — 484 matched before/after function pairs from 80 refactor/simplify/cleanup commits in this repo's own history — found the lens agrees with the commit's implied readability direction on only 30.2% of pairs, which is worse than chance and suggests the ranking may run backwards more often than not. Treated here as a signal to weigh, never a verdict; do not read a low posnett= as proof a function needs work. +APPROXIMATION, disclosed: ONE token-class table serves every language (keywords + punctuation = operators, identifiers + literals = operands), with no per-grammar refinement, so V is cross-language and not a per-grammar Halstead count. The formula was fitted on snippets of 20 lines or fewer, so it is a RANKING lens, not a grade: read the ORDER of the rows, not the number on any one of them. Pages with limit=N (offset=M); default 40 rows. Declarations with no body are not measured. UNVALIDATED (t14-cleanup #8): this is a deterministic ORDERING signal that has not been checked against human judgement of readability. Our own proxy measurement — 412 matched before/after function pairs from 80 refactor/simplify/cleanup commits in this repo's own history — found the lens agrees with the commit's implied readability direction on only 37.4% of pairs (154 of 412), which is worse than chance and suggests the ranking may run backwards more often than not. That population is pinned to the v0.6.2 tag, never a walk of every branch in the clone, so it cannot drift when an unrelated branch is pushed. Treated here as a signal to weigh, never a verdict; do not read a low posnett= as proof a function needs work. **Try it** @@ -2831,7 +2831,7 @@ $ ./build/ripwire . --quality-baseline --allow-dirty ### `--quality-delta` -**Answers:** before a PR: report ONLY what your change made worse, across 10 kinds agent self-check before a PR (pair with --test-gate): report ONLY what a change made worse vs the baseline (10 kinds: complexity/verbosity/nesting/params/dup/dead/api-surface + error-masking/short-horizon-churn/reuse-decline); +**Answers:** before a PR: report ONLY what your change made worse, across 10 kinds agent self-check before a PR (pair with --test-gate): report ONLY what a change made worse vs the baseline (10 kinds: complexity/verbosity/nesting/params/dup/dead/api-surface + error-masking/short-horizon-churn/new-clone-of-reused-helper); every finding is classified by ORIGIN: a symbol that EXISTED at the baseline and got worse (preexisting-worse="N", no attribute on the row) vs one that exists only because the code is NEW (new-symbol="N", origin="new-symbol" on the row). A small numeric delta is additionally sev="minor". EXIT 2 ONLY on preexisting-worse AND major AND unacked — the gating="N" header count. New-symbol rows are still PRINTED (they are the debt you are adding — read them), they just never gate; exit 0 means "nothing that already existed got worse", not "clean". Clone kinds classify by member set (new-symbol only if EVERY member is new); short-horizon-churn is preexisting by construction. LIMIT: origin is canonId (path::scope::name) identity, so a RENAMED/MOVED symbol reads as new and a regression carried in with the move will not gate. Test-fixture dirs + doc sections are exempt from dead-code/churn; churn needs COMMITTED thrash evidence (rewritten across recent commits AND again by this diff), never the current edit alone WHICH FLOOR IT COMPARES AGAINST, and a side effect: the sidecar is honored only when the sha it was pinned at EQUALS the current git HEAD (strict equality — an ancestor commit describes a DIFFERENT tree, so everything committed since would read as your regression). A sidecar pinned anywhere else is STALE: this verb then DELETES it from your working tree (self-heal, so the next run does not rediscover the dead pin) and auto-compares the working tree vs git HEAD instead. Re-pin with --quality-baseline. The read-only MCP quality_delta verb applies the SAME staleness test but never deletes. A sidecar at the current HEAD that ANOTHER ripwire build pinned (its producer stamp names other sources — a dead set depends on how calls were resolved) is FOREIGN: both arms ignore it, never delete it, and auto-compare vs git HEAD. Which floor was actually used is on every report as baseline=: sidecar | git-HEAD | git-HEAD (stale sidecar removed) | git-HEAD (stale sidecar ignored) | git-HEAD (foreign sidecar ignored) — the stale two say a stale sidecar existed, and 'removed' means the file is gone. A non-git root has no HEAD to fall back to, so its sidecar is honored whenever this build pinned it; without one there, or with another build's, the verb exits 1. diff --git a/docs/EVALS.md b/docs/EVALS.md index 97eff9a02..8deab027e 100644 --- a/docs/EVALS.md +++ b/docs/EVALS.md @@ -6145,8 +6145,8 @@ These are the exact `kind=` strings the binary emits, from `src/quality.h`: `complexity` · `verbosity` · `nesting` · `params` · `duplication` · `dead-code` · `api-surface` · `error-masking` · `short-horizon-churn` · `new-clone-of-reused-helper` -Note that some user-facing summaries abbreviate four of these (`dup`, `dead`, `churn`, -`clone-of-reused-helper` / `reuse-decline`). **Match against the strings above** when grepping real +Note that some user-facing summaries abbreviate three of these (`dup`, `dead`, `churn`), and older +text calls the last one `clone-of-reused-helper` or `reuse-decline`. **Match against the strings above** when grepping real output. The verb reports only what a change made *worse*, against git HEAD. `--quality-ack` records a diff --git a/src/cli.h b/src/cli.h index 327a61c55..1da728cb9 100644 --- a/src/cli.h +++ b/src/cli.h @@ -1388,12 +1388,13 @@ inline constexpr char kHelpHead[] = " grade: read the ORDER of the rows, not the number on any one of them. Pages with limit=N\n" " (offset=M); default 40 rows. Declarations with no body are not measured.\n" " UNVALIDATED (t14-cleanup #8): this is a deterministic ORDERING signal that has not been\n" - " checked against human judgement of readability. Our own proxy measurement — 484 matched\n" + " checked against human judgement of readability. Our own proxy measurement — 412 matched\n" " before/after function pairs from 80 refactor/simplify/cleanup commits in this repo's own\n" " history — found the lens agrees with the commit's implied readability direction on only\n" - " 30.2% of pairs, which is worse than chance and suggests the ranking may run backwards more\n" - " often than not. Treated here as a signal to weigh, never a verdict; do not read a low\n" - " posnett= as proof a function needs work.\n" + " 37.4% of pairs (154 of 412), which is worse than chance and suggests the ranking may run\n" + " backwards more often than not. That population is pinned to the v0.6.2 tag, never a walk\n" + " of every branch in the clone, so it cannot drift when an unrelated branch is pushed. Treated here\n" + " as a signal to weigh, never a verdict; do not read a low posnett= as proof a function needs work.\n" " --nonlocal-state per function, the non-local mutable state it can reach, most writes first\n" " per function, the NON-LOCAL MUTABLE STATE it can reach, MOST WRITES FIRST: writes= reads= are the\n" " distinct cells this function OR its transitive callees write / read; direct_writes= direct_reads=\n" @@ -1636,7 +1637,7 @@ inline constexpr char kHelpHead[] = " later --quality-delta against it carries baseline_absorbed=\"N\" — so a green exit beside that attribute reads as\n" " \"clean SINCE THE PIN\", never \"clean\". Refused alone.\n" " --quality-delta before a PR: report ONLY what your change made worse, across 10 kinds\n" - " agent self-check before a PR (pair with --test-gate): report ONLY what a change made worse vs the baseline (10 kinds: complexity/verbosity/nesting/params/dup/dead/api-surface + error-masking/short-horizon-churn/reuse-decline);\n" + " agent self-check before a PR (pair with --test-gate): report ONLY what a change made worse vs the baseline (10 kinds: complexity/verbosity/nesting/params/dup/dead/api-surface + error-masking/short-horizon-churn/new-clone-of-reused-helper);\n" " every finding is classified by ORIGIN: a symbol that EXISTED at the baseline and got worse (preexisting-worse=\"N\", no attribute on the row) vs one that exists only\n" " because the code is NEW (new-symbol=\"N\", origin=\"new-symbol\" on the row). A small numeric delta is additionally sev=\"minor\". EXIT 2 ONLY on preexisting-worse AND\n" " major AND unacked — the gating=\"N\" header count. New-symbol rows are still PRINTED (they are the debt you are adding — read them), they just never gate; exit 0 means\n" diff --git a/test/printf_parity.manifest b/test/printf_parity.manifest index 2845e8814..81180b11e 100644 --- a/test/printf_parity.manifest +++ b/test/printf_parity.manifest @@ -38,5 +38,5 @@ safe_delete 0 8b99d83566b89332ac4b2967abbb39284db775bbed601628b97b8c7909f04ac2 e verify_layer 1 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 5288c345d7d6e335f88b9c1daa8935db22e1dcf89c0c8bc1f6140d4cb5af0b48 graph_query 1 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 bfa4319feb9dee09cfbd3991cf6fbd752297e99d14de9e75768820d2a9c8832f callers_limit 0 e40ee1ddba0fbf548c58c98b91fe052af8c34f8913edb3edd85245b12a6967b9 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -help_all 0 8f0f9dca5199beebdf9e73d578076812ecfa3de8ed6450db3429e1dc64000df8 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +help_all 0 10c38670feb133f4723b29fdfb60d198aa09fb2ecc8200d8c41d92f371ce939b e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 help_one 0 938ff4f8802a8f5d9cb10f9a1f3804ac755be44c8ab00e24e45bfe8255224a21 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855