Skip to content

gitnext 0.3.0 - #1

Merged
pnnnx merged 55 commits into
mainfrom
next
Nov 12, 2026
Merged

pnnnx merged 55 commits into
mainfrom
next

Conversation

@pnnnx

@pnnnx pnnnx commented Nov 12, 2026

Copy link
Copy Markdown
Owner

What is new: see CHANGELOG.md

claude and others added 30 commits November 11, 2026 18:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d / • changed marks, one kind column, short per-screen key hints

- detail and pages wrap lines themselves: continuations keep the indent
  (gutter and code indentation) and start with ↪; link scrolling follows
  the wrapped rows
- changes, before/after and pages take the whole screen: no fragments of
  the panes behind them
- tree: + fully staged, • changed, legend in the hint line
- symbols: kind column first, members indented by name, types in bold
- hint line: 4–5 keys for the current screen + ctrl-p all
- screenshots in docs/images regenerated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the release binary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…callers JSON splits sure / maybe; receiver types from parameters and constructors in the generic engine

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, safer rename --text; renames checked by tests

- Java anonymous classes (new T() { @OverRide … }) are subclasses: renaming the base method renames the override
- calls through fields (this.pool.release(), Go s.parent.Find(), import static Owner.field) resolve by the field's type
- Rust: parameter and let types (req: &VersionReq, let v = Version::parse(s)?) type method calls, also in macros;
  calls inside const initializers; tests/ crates see their own mod util folders; crate:: in test crates
- Python: classes defined inside functions (test doubles) are indexed, so their overrides rename too
- rename --text: in prose only names that cannot be English (snake_case, camelCase) or written as code;
  whole-string literals only in Python / JS; overrides reported as renamed, not as "check manually"
- tests-for: names match whole words; tests behind a helper at the depth limit; same-named types count as sure;
  the test-file convention also when the graph finds fewer than 5
- eval: rename runs the project's tests (--fast to skip), tests failing before any change are subtracted;
  baseline.json and results/HISTORY.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ips 3.9 / pip 21); agent shortens old tool output to fit the model's context

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… locked"

Switching a new database to WAL takes an exclusive lock that SQLite does not
wait for (the busy timeout is not consulted), so two tests of one binary — or
an MCP server next to the CLI — opening a fresh repo at the same moment got
SQLITE_BUSY at once (mcp::tests::lite_profile_is_small_and_capped on CI).
Index::open now retries the WAL switch and the schema check while the database
is busy, and every write transaction takes its lock up front (IMMEDIATE).

Regression test: 8 threads × 12 rounds open one fresh repo (failed 2 runs of 4
before, passes since).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…for whole test files

In cobra `tests-for` listed 125–244 tests per function. It now orders them:
tests that call the function directly (those named after it first), tests
named after it (when the name is more than one common word), then the ones
that only reach it through other functions — fewer hops and more of its name
first — and last the rest of its mirror test file. Helpers in test files
(Go/Python by name, Rust by taking arguments, Java by missing a JUnit
annotation) are not listed as tests any more.

Output: the 10 most likely with the exact command for them, plus one command
per test file / package for all of them (`--all`, MCP `all: true`: every test
by name as before). `look` and `context` show the same top and command.

Eval (share of the first 10 that really fail, old order → new, same ground
truth): cobra 22% → 30%, click 45% → 48%, jsoup 60% → 62%, semver 57% → 67%.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`x := getPool(); x.release()`, `getPool().release()`, `pool().child().release()`,
`for _, c := range cmd.Commands()`, `const c = await open()`: the receiver is
"what that function returns". The parser writes such a call as
`getPool()::release`; the index, which sees every file, takes the declared
result type of the callee (`*Pool`, `(*Conn, error)`, `[]*Pool`,
`Promise<Conn>`) and resolves the method on it. No declared type, a generic or
a library type: it stays a call on an unknown receiver, as before.
Also typed now: `new Store().load()`, Java casts and for-each variables,
TS `x as T`.

jsoup: of 36912 calls, those resolved to exactly one target 12173 → 13288,
ambiguous 12003 → 11390; callers of methods with a shared name found for
74% → 77%; tests found by `tests-for` 14% → 19%.

PARSER_VERSION 28 → 29.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…op 10; agent parses tool calls written as text

Ground truth:
- Java: maven recompiled only the changed file, so callers in other files were
  never checked (jsoup "callers precision 47%" — all of gitnext's callers were
  real); `mvn -o test` never ran because surefire was not downloaded, and a
  suite that does not run was counted as "no test failed". Every class is
  recompiled now, the setup runs one test, and each check requires that the
  suite really ran.
- callers: uses the compiler names are fixed and compiled again until the build
  passes (javac / rustc stop at the first failing class / crate).
- Go: a panic kills the test binary, only the first test was reported (cobra
  "tests precision 1%"); mutant tests run one at a time.
- TS: a bare `throw` made the mutant fail to compile.
- What compilers and test suites said is cached in eval/.work/truth.json.

New measures: callers of methods whose name several types share, and of the
10 tests `tests-for` names first — how many fail, and whether any does.
eval/results/HISTORY.md has run 5; baseline.json is the current run.

Agent (eval/agent.py): a small model often writes the call as text
(```json {"name": …}```) — 14 of 23 answers in the first Mac run, every task
ended with 0 steps; such calls are parsed and run now. The "fix" task was
never generated for click (too few mutation sites in 25 functions): more
mutations, more functions, examples skipped.

Cargo.toml: eval/.work is excluded from the workspace (cargo refused to build
eval/.work/semver inside it — the Mac run stopped there).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…es, macro bodies

Methods whose name several types share were found only when the receiver was a
typed local or `self.m()` in the impl next to the struct.

- rust.rs: field types of the structs a file defines (named and tuple fields; cfg
  variants that disagree leave a field unknown); `self.core.drain()`,
  `x.f.g.m()` and `rhs.m()` with `rhs: &Self` now name the type. `&'a mut T`
  reads as `T`. `macro_rules!` bodies emit their calls (they emitted none).
- graph.rs: `Self::m` in an `impl` written in another module (`impl Display for
  BuildMetadata` in display.rs) resolves to the methods of the same type name in
  the crate, only when the crate defines one type of that name.
- PARSER_VERSION 30.

eval (--fast, --n 20), shared names found (+maybe), precision:
  semver   25% (50%) -> 100% (100%), 100% -> 100%  (2 methods)
  indexmap 34% (86%) -> 86% (100%), 100% -> 100%  (14 -> 16 methods; 84% on the same 14)
indexmap plain callers 96% -> 100%, rename 80% -> 85%; semver 100/100/100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t tests-for

eval/learn_tests.py runs click's suite once under pytest-cov with
--cov-context=test, maps covered body lines to functions by gitnext's bounds
(nested function bodies excluded, the def line runs at import) and compares
three test lists for 40 functions with ground truth: tests-for --all, the map
(narrowest tests first), and their union. Sample, truth and metrics come from
eval/run.py (import only): run.pick(n=20, seed 7) plus 24 random library
functions without the callers condition.

Full truth (no `tail -400`, test modules that fail to import do not stop the
run), all 40 functions:
  recall     tests-for 19%  coverage 100%  union 100%
  precision            34%            98%        81%
  top 10 fail          35%            99%        88%
  caught by top 10     52%           100%       100%
Random functions only (22): tests-for 21% / 29% top 10, coverage 100% / 98%.

Cost: 6.2 s -> 14.2 s for the suite (x2.3, median of 3); map 143k
(function, test) pairs, 1.6 MB as SQLite. Coverage misses 4 failing tests
(2 run click in a subprocess). The eval's truth says "no test fails" for
decorators.option and Option._pick_type: the mutant breaks test-module import
and pytest stops; with --continue-on-collection-errors 355 and 365 tests fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A small model gets stuck in ways a large one rarely does. eval/guard.py watches
every run of eval/agent.py and steps in:
- the same call with the same arguments while the files are unchanged is not
  run again: the model gets the earlier result and is told to do something else;
- 4 steps without new information or a change: one nudge, then stop;
- the same build / test error: a hint from `gitnext locate` at the second,
  stop at the third;
- see-saw edits (A → B → A → B): the files go back to the last state where the
  tests passed, or to the start;
- a repeated line or chunk in the model's text is cut; llama.cpp gets
  repeat_penalty 1.1;
- budgets: "wrap up" at 70% of the steps / tokens (AGENT_TOKEN_BUDGET), a hard
  stop at 100%;
- old tool output becomes a one-line digest instead of a blind cut;
- a stopped run leaves a hand-off note (what was tried, what changed, the last
  error); with --escalate-host a stronger model continues from it.

New mode `cmd` (mini-swe-agent style): no tool-calling API, the context is in
the task, the model answers with one shell command or a short script in a
```sh block, or `DONE: answer`; `gitnext` is on PATH. For models that write
tool calls as text — and it is the shortest tool list there can be.

eval/test_guard.py: 13 tests against a scripted model server. Not yet run on
a real model.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ests-for stays in one language

Found by agents working through gitnext on this repository:
- `look main` silently showed a C++ fixture's `main` although the repo has
  several: a bare name that is also somebody's full name now lists its
  namesakes too, each with a label that leads to exactly one (`main@78`).
- `tests-for` took a C++ `main` in a fixture for a test of Python's `main`:
  names and mirror files count only within one language.
- `gitnext callers A B C`: several names in one run (the graph is loaded once —
  on jsoup a separate process per name cost 1–2 s each).
- the text of `callers` did not tell sure from maybe: now
  `callers of X (3 sure, ≈97% real; 2 maybe)` with the doubtful ones listed
  apart.

Trust is stated, not left to the agent to guess (crates/gitnext-ops/src/trust.rs,
numbers from eval run 5): `look` and `callers` say how many of the sure callers
were real in the measurements, `tests-for` and `look` say what share of the
failing tests the call graph finds for this language and to run the whole
file before finishing. The small-model profile (MCP lite) is not shown the
"maybe" callers at all.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… verdict

One step for the agent instead of three (edit → build → tests), and the
judgement comes from the compiler and the tests, not from the model:

    check: 1 symbol changed (make_str)
    build ok          0.1s  .venv/bin/python -m compileall -q -f src/click/utils.py
    tests ok          0.7s  .venv/bin/pytest tests/test_shell_completion.py::test_path_types …
    verdict: ok — builds, 10 tests pass
    note: the call graph finds ≈19% of the tests that would fail — run the whole test file before you finish.

- changed symbols: the innermost function / method / type around every line
  of `git diff HEAD`, plus everything in files git does not know yet;
- build: per language touched (cargo check --tests, go build + test compile,
  compileall, tsc --noEmit, mvn / gradle compile); a failed build stops there
  and shows the error;
- tests: the 10 most likely to fail for every changed symbol (30 at most),
  changed tests themselves; each command with a time limit;
- exit code 1 when anything fails. `gitnext rename A B --check` runs it right
  after the rename.

The agent loop gets a `check` tool and the hint in command mode; the loop guard
takes its verdict as "the tests passed" for the safe state it goes back to.
On click: a harmless change → ok in 0.9 s; a seeded bug → FAILED with the
failing assertions.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…it is in

"Who builds the string `()::`?" had to be answered with grep. `gitnext search
--text '()::'` (and `look` with any text that is not a name) now lists the
lines and names the innermost function or type around each:

    crates/gitnext-index/src/graph.rs:644 (in resolve_call_raw): if let Some((callee, method)) = e.to.rsplit_once("()::") {

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rage run (Python)

The call graph cannot see calls made through a CLI runner, decorators,
registries or templates: on click `tests-for` found 19% of the tests that fail
when a function breaks. A coverage run sees them all (prototype and numbers:
eval/learn_tests.py).

`gitnext learn-tests` runs pytest once with pytest-cov (`--cov-context=test`),
reads the coverage database (lines, or arcs when the project measures
branches), maps every executed line to the innermost function around it and
stores "function → tests that ran it" in .gitnext/tests.db — separate from the
index, which is a cache; the map costs a test run. On click: 21 s, 432
functions, 598 tests, 44 109 pairs, 3 MB.

`tests-for`, `look`, `check` and `context` use the map when it knows the
function: the tests that ran it, narrowest first (a test that runs little
else is about this code). A function the map does not know (new code, never
executed) falls back to the call graph. The map stores the function's code
hash: when the function changed since, the answer says so.

Measured by the same script and the same ground truth as the prototype
(40 functions of click, full truth), `tests-for` before → after learning:
  failing tests found      19% → 100%
  named tests that fail    34% →  98%
  of the first 10, fail    35% →  99%
  caught by the first 10   52% → 100%
— the same as the prototype's own map. For functions the call graph knew
nothing about (`push_context`: "no tests reach") it now names 388 tests.

Not done: Go, Rust, Java, TS (other coverage tools); refreshing only the stale
part of the map.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A card is what a small model can be given: the repository, what to do, and the
command that judges the result (default `gitnext check`):

    {"repo": "/path/to/project", "prompt": "…", "accept": "gitnext check"}

    python3 eval/agent.py --task card.json --mode cmd [--escalate-host …]

The run passes when the accept command exits 0; next to the card it leaves
card.result.json (steps, tokens, what the loop guard did, the hand-off note of
a stopped run) and card.patch. This is the unit a larger agent can hand to a
local model and accept back without reading a report.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…shold for every number

Which difference between two versions of gitnext is real, and which is the
sample? Until now: 3-8 functions per repo (zustand: one), so 5 points could
not be told from chance.

- repos.json: chi (Go), sqlparse (Python), yaml (TypeScript), json-java
  (Java), indexmap (Rust), pinned. Each runs its tests in under a minute,
  builds offline after the setup and has methods that share a name.
- Defaults: --n 20 --mutants 8 --seeds 7,8,9 (--seed N: one sample). A repo
  is prepared and indexed once; a function that several seeds picked is
  measured once. run_repo, pick and prepare keep their signatures
  (eval/agent.py and `import run` use them).
- Every number is the mean over all distinct functions and comes with a
  threshold: the functions are drawn again 2000 times (bootstrap), the
  threshold is half the 90% interval of the mean and at least what one
  function weighs. latest.md shows it as "77% ±6 (20)", per repo, per language
  and for all repos, next to the value on each seed and 2 x SD of the seeds.
  latest.noise.json has the same for scripts.
- --check: a rate may drop by its own threshold (per repo and metric, kept in
  baseline.json) instead of 5 points for everything; a repo that did not run
  fails the check. baseline.json keeps one entry per sample (n, mutants,
  seeds); --bless updates the entry of the sample it ran.
- Mutants: down the sample until 8 are measured (some do not compile with the
  throw, some fail no test), at most 16 tried.
- The verdict on a rename (build + tests) is kept per distinct edit
  (repo@commit:rename:sha1 of the diff), the baseline test run per commit, and
  `gitnext callers` is asked once per function in a run. A second run asks
  only gitnext: all ten repos on three seeds take 7 min 41 s (the old sample
  took 7 min on five repos). truth.json is locked and merged on every write,
  so `--repos a` and `--repos b` can run side by side; it also records how
  long each answer took.
- CI: truth.json is cached between runs (key: CACHE_VERSION in run.py + hash
  of repos.json). CI checks one seed of 8 functions and 4 mutants: with an
  empty cache that is about 50 minutes of builds and tests, one seed of 20
  functions is about 110.

On the old sample (--seed 7 --n 8 --mutants 4) the five old repos give exactly
the numbers of run 5.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Found while the new repos were being checked. Both make "the tests that fail
when this function throws" wrong, so tests-for was measured against a wrong
answer.

- Go: the tests of a mutant run one at a time from a compiled test binary. It
  was named .gnx.test; cobra reads the command line as its own arguments
  unless the binary is called cobra.test, so seven cobra tests failed under
  every mutant, whatever it broke. The binary keeps the name `go test` gives
  it, and tests that fail alone on the untouched commit belong to the
  baseline. cobra, sample of run 5: tests recall 82% -> 96%, precision
  17% -> 22%, top-10 that fail 30% -> 28%, caught by the top 10 100% -> 50%
  (the seven tests made every top 10 a hit).
- Python: `| tail -400` kept the last 400 lines of the pytest summary. 4 of
  24 click mutants fail more tests than that: 372 test names were cut to 122,
  365 to 122, 353 to 102, 288 to 102. The whole summary is read now
  (--tb=no keeps it short).
- Python: the name of a parametrized case was taken from inside its id
  (test_x[a::integer] -> "integer"); 9 of 24 sqlparse mutants had such names
  in their truth.

The answers taken the old way are stored under other keys (tests2 / tests3)
and are not used.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…sked 40 at a time

Ground truth:
- Python: a mutant (or a rename) that breaks the import of a test module made
  pytest stop before any test ("Interrupted: 1 error during collection"); the
  run counted as "no test fails". The suite continues past such a module
  (--continue-on-collection-errors) and every test of it counts as failing.
  Four click renames that had passed this way fail now: rename ok 74% -> 66%.
- Python: a test is `file::name` (failing_tests returns `ids` next to the bare
  names, which eval/learn_tests.py reads): a bare name merged tests of the
  same name in different files.
- Rust: `cargo check --tests` did not build code behind a feature, so real
  callers in indexmap's rayon and serde modules counted as wrong: callers
  precision 90% -> 100% with --all-features. Benches need nightly there; a
  caller in benches/ is left out of what gitnext found.
- The verdict on a rename is stored per edit and per pair of commands
  (compile, tests), so a changed command asks again.

New measure, "callers recall, picked blind": the samples hold only functions
gitnext finds a caller for, so a function whose callers it misses entirely
could not get in. The first 20 functions of the same order are now checked
too, whatever gitnext sees. 163 of them have uses the compiler names; gitnext
misses all callers of one (jsoup Safelist.isSafeTag): 99.2% ±1.

Speed: pick asks `gitnext callers` about 40 functions in one run (a gitnext
that takes one name is asked name by name). All ten repos on three seeds with
a full cache: 7 min 41 s -> 3 min 17 s (jsoup 5 min 48 s -> 2 min 7 s).

CI comment: with an empty cache the CI sample is about 55 minutes of builds
and tests, one seed of 20 functions about 120.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The measurement is new, so this is a new starting point for the numbers.
Checked: rename 337 functions (run 5: 30), callers 232 (22), methods with a
shared name 108 (10), mutants 166 (16). Measured with gitnext of the local
next (Rust receiver types, callers for several names); run6-before.json is
the code of run 5 on the same functions.

All repos (mean, threshold): callers recall 99.8% ±1, precision 97.9% ±2,
picked blind 99.2% ±1; shared names recall 83.2% ±6 (97.6% ±2 with "maybe");
rename ok 82.8% ±4; tests recall 45.4% ±6, precision 47.1% ±6, top-10 that
fail 50.9% ±6, caught by the top 10 75.3% ±6.

The thresholds at work, code of run 5 -> now, same functions: shared names
recall 71% -> 83% (±7) and in Rust 34% -> 80% (±13) are real; every other
number moves by 0-0.6 points, inside its threshold.

What 8 functions per repo did not show (details in HISTORY.md):
- rename ok was 100% everywhere; it is 83% (click 66, yaml 69, indexmap 70).
  58 renames fail: 22 do not compile, 28 break tests, 6 are refused
  (#private methods in TypeScript), 2 leave a suite that does not start.
- tests-for finds 2% of the failing test files in yaml (anonymous test
  callbacks in tests/*.ts).

baseline.json: the default sample (n20-m8-seeds7,8,9) and the one CI runs
(n8-m4-seeds7). `python3 eval/run.py --check` passes on both.
run6-seeds.json: every number with its interval and its value on each seed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… nor refuted

`go build ./...` skips directories that start with `_` or `.` and `testdata` (chi's
`_examples`), so the real callers there counted as wrong. Left out like Rust benches:
callers precision chi 91% -> 97%, Go 94% -> 96% +-3, all repos 97.9% -> 98.6% +-1.

Found while reviewing the larger eval: the same sample run again gives the same numbers, and
sqlparse, semver and a small sample of chi asked from scratch (121 answers) agree.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…6 for tests)

sure callers that are real: go 97 -> 96, java 100 -> 99; failing tests the call graph
finds: go 82 -> 79, java 19 -> 49, js 33 -> 3, rust 61 -> 53, python 19 -> 30.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every file write made through gitnext (edit replace / insert-after / delete,
rename incl. --text, the same tools over MCP) is now one numbered operation in
.gitnext/ops.jsonl (git-ignored), in the spirit of `jj op log` / `jj undo`:

  gitnext ops [-n 20]      newest first: who (cli / mcp:<session>), tool,
                           target, files with lines removed / added
  gitnext undo --op N      take back one operation; `gitnext undo` without
                           --op still takes back the last commit

An operation stores, per file, the changed line blocks (old, new, 3 lines of
context) and the sha1 before / after; no file snapshots, nothing in .git.
Undo puts the old block back directly when the file is as the operation left
it, otherwise only where the new block with its context is found exactly once,
so later edits of other functions in the file survive. Not found or found
twice: refuse with the lines and the later operations that touched them,
write nothing (all files are checked before any is written). Files come back
byte for byte (CRLF, missing final newline). The undo is itself an operation:
undoing it redoes the change.

Ids are handed out under an exclusive lock on the log (std File::lock), so a
CLI and an MCP server writing at once never share one; the log is cut to its
newer half past 500 operations or 5 MB and ids keep counting. A rename and its
--text pass are one operation (oplog::Group). The MCP journal entry of an edit
links to its operation (`op`).

Cost, on this repository:
- log size: 57 KB per 100 one-line edits of a 20 KB file; 56 KB after 100
  `gitnext edit insert-after` runs on a copy of the repo
- recording: 0.17 ms median per edit (0.20 ms with a full log)
- `gitnext edit` wall time, median of 100: 68.7-69.7 ms before,
  68.1-70.3 ms after (within noise); `gitnext undo --op N`: 4 ms

Tests: crates/gitnext-ops/tests/oplog.rs (9): undo of replace with a later
edit of another function kept, insert, rename over 3 files with --text as one
operation, refusal on a rewritten block, two edits of one function undone
newest first, CRLF and no final newline, redo, log cap, 8 parallel writers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e repository

Found while reviewing the operation journal:
- a block at the start or the end of a file was only put back if it was still at the
  very start / end, so undo was refused after a function was appended to the file or an
  import was added on top - the usual order of an agent's edits. "Exactly one match"
  of the block with its context is enough;
- the log is a local file: an operation whose path is absolute or goes through `..`
  is refused instead of written.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A per-test coverage map against `gitnext tests-for` on the eval's Go and
Rust projects, with the eval's ground truth (the cached truth.json):

- learn_tests_go.py: one test binary per package with coverage of the
  whole module (`go test -c -cover -coverpkg=<mod>/...`, the name go test
  gives it), each top-level test run alone with its own -test.coverprofile;
  a block that ran is booked to the innermost function holding it.
  Holds the driver shared with the Rust script.
- learn_tests_rust.py: test binaries built once with -C instrument-coverage
  in their own target dir, each test run alone (--exact) with its own
  LLVM_PROFILE_FILE, llvm-profdata + llvm-cov export (lcov).

Recall of failing tests, graph -> coverage (all sampled functions):
cobra 88% -> 100% (29), chi 67% -> 100% (25), semver 70% -> 100% (15),
indexmap 52% -> 100% (26); failing among the first 10: 42/74/61/39% ->
99/91/100/98%. Map cost against a plain warm test run: cobra 3.5 s vs
0.7 s, chi 30.9 s vs 26.8 s, semver 0.9 s vs 0.4 s and indexmap 13.9 s vs
2.1 s without the one-off instrumented build (20.6 s / 12.0 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… passing

Run 6 showed 58 of 337 renames that break the code. The causes that were gitnext's own:
- Python: calls inside nested functions (decorators, closures) were not recorded at all;
  they belong to the function they are written in now;
- Python: a call through the package (`click.style(...)` where `click/__init__.py`
  re-exports it), a dotted module path (`click._compat.f()`), a decorator through a
  receiver, a bare name used as a value, and a local variable named like an imported
  function are resolved;
- Rust: a name inside a `use` group is renamed;
- TypeScript: a `#private` member is renamed and stays private.

eval, the same functions as run 6: rename ok 83% -> 91% +-3 (25 fixed, none broken);
Python 73% -> 90%, TypeScript 70% -> 85%, Rust 74% -> 87%, Go and Java as before.
Python tests-for: recall 30% -> 36%, caught by the top 10 74% -> 81% (inside the threshold).
What is left (31 of 338) is listed by cause in HISTORY.md, run 7.

PARSER_VERSION 30 -> 31 (Python edges changed).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tasks on every repeat, traces

The first run on Qwen 3.5 4B (Mac, 120 runs): base 3/24, gitnext tools 11/24 - but the
24 were 8 tasks three times, and "fix" was 0/24 in every mode: the model takes one step,
names a function and stops.
- mode `verify`: gitnext tools, and an answer given without a look at the code, a finish
  with no file changed, or a change that `gitnext check` does not pass goes back to the
  model (each kind once, the check up to three times). The gate knows nothing of the task;
- `--repeat N` makes N task sets (seed, seed + 1, ...) instead of the same one N times;
- the seeded bug is a commit, so `git diff` and `gitnext check` do not point at it;
- the callers task is judged by the compiler's list where the language has one
  (it was gitnext's own answer);
- every conversation is kept: <out>.trace.jsonl (and <card>.trace.json).
Checked with the scripted model only (17 tests); not yet on a real one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
claude and others added 25 commits November 12, 2026 12:08
…tions

--compare pairs the per-function values of two runs (repo, kind of check,
symbol), takes B − A per function and bootstraps its mean within each repo:
"better" / "worse" when the whole 90% interval is on one side of zero,
"can't tell" when it covers zero, "< 0.5 pt" when the whole interval is
within half a point, "same" when no function changed. Next to it: the noise
of one version alone (as in latest.md), the functions whose value changed,
and what does not pair (repos, commits, functions only one run has).

--sample-from RUN measures the functions an earlier run picked instead of
drawing them again: the sample asks `gitnext callers`, so a change of the
graph moved it (5–13% of the functions did not pair for one variant).

Checks: two runs of the same build — all 13 metrics "same", every
difference 0; a run with --sample-from of the run it reuses — the same.
Paired, the interval of callers precision over all repos is ±0.7 points
where one version alone has ±1.0; shared names: recall ±1.3 vs ±5.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`r.Context().Value(…)` (chi) and `c.PersistentFlags().HasFlags()` (cobra)
made the one repo method of that name a sure callee: `f()::m` whose result
type the graph cannot find falls back to `.m`, and a single candidate was
sure. Now such a call is a guess when `f` surely returns something the repo
does not define: `f` is not in the repo, or every candidate declares a
result that is no repo type. A repo `f` with an unknown or ambiguous result
(jsoup `el.tag()`) and Go's `make` / `new` keep the call sure.

eval, paired on the same functions (--sample-from, 10 repos, seeds 7,8,9):
callers precision 98.6% → 99.4% (+0.8, 90% interval +0.2…+1.6), Go
96.3% → 98.2% (+2.0, +0.2…+4.2); callers recall, blind recall, shared
names and tests unchanged (no function or < 0.5 pt). cobra HasFlags 20% →
100%, Execute 50% → 100%, jsoup Collector.findFirst 60% → 100%, chi
endpoints.Value 14% → 25% (the rest comes from `ctx.Value` on an unknown
receiver).

Measured and dropped: a guard for `Type::m` with several types of that
name (shared names recall −1.5, precision +0.5 can't tell; test doubles
lose their tests), and "unknown receiver `.m` → guess" (callers recall
−2.9, blind −8.9, tests −4.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er the paired run

The guard for library results (`f().m()` on a library result is only a guess) used by rename
too, but only where it is known: `f` is a repo function that declares a library result,
or a Go package function (`template.New(...).Execute`). The wider rule skipped
`assertInstanceOf(TextNode.class, n).getWholeText()` and broke a jsoup rename: a library
call can hand a repo object on. A type parameter (`T`) as a declared result is not a
library type either.

eval --compare, same functions: rename ok 90.8% -> 91.4% (cobra Execute, HasFlags;
nothing broken), callers precision 98.6% -> 99.4%, Go 96.3% -> 98.2%.
trust: sure callers that are real - go 96 -> 98, java 99 -> 100.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…own environment

From the second run on Qwen 3.5 4B (72 runs, 24 task instances): base 4/24, gitnext
tools 14/24, verify 14/24. The traces showed two errors of the harness:
- the gate read "builds, but no test reaches the changed code" as a failed check and sent
  a finished rename back three times (12 of the 36 times the gate stepped in);
- `run` did not see the project's .venv, so `pytest` could not import the project in
  any mode.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…st log in a few lines

The second run on Qwen 3.5 4B: "fix the bug" 0 of 6 in every mode. Told that
`test_greater_than` fails, the model asks for `greater_than`, gets no path from the test
to the code and invents a function; a real failure is hundreds of lines it cannot hold.

- `gitnext digest` (stdin): what failed, where, the message - rustc / cargo test, pytest,
  Python tracebacks, go build / go test (panics too), tsc, vitest, jest, javac / maven,
  gcc-style `file:line: message`. A vitest run with 35 failures: 843 lines (21.7k
  characters) -> 12 lines (1.1k); a Rust backtrace or a goroutine dump is gone, the
  assertion and the first frame of the project stay.
- `gitnext check` prints that digest for a failed step instead of the last 25 lines.
- `gitnext why <test>` (MCP: `why`): runs the test, prints the digest of its failure, the
  source line that failed, and the code the test reaches through the call graph - what
  the failure names first, then by the words shared with the test's name, then by distance -
  with the code of the first. `greater_than` finds `test_greater_than`. On the seeded
  bugs of semver, click and cobra the mutated function comes first, in 1.1-2.3k characters.
- locate: file:line of Go, Java, TypeScript / JavaScript; tsc and maven forms; a bare
  file name of a Java stack frame.
- agent: tool `why`; in the gitnext modes a failing command's long output comes back as
  its digest.

Not yet measured on a model.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and check, agent mode lite

A small model chooses among the tools it is shown. The eval's gitnext mode showed eleven
(six plain + look, rename, replace_symbol, check, why); in the second run on Qwen 3.5 4B
it called look and rename and never replace_symbol or check on its own. gitnext's lite
MCP profile showed three and had neither rename nor a way to run tests, so the eval did
not measure what a small model is given.

- `gitnext check <test>` (MCP: `check` with `test`) answers as `gitnext why <test>`: one
  verb for "run and tell me" - after a change with no argument, for a failing test with
  its name. `why` stays as a command and in the full profile;
- lite MCP profile: look, replace, add_after, rename, check - five tools, the list stays
  under 2 kB (it is sent on every turn);
- agent: mode `lite` - those five and finish, nothing else; the separate `why` tool is
  gone from the other modes (check takes the test). Default modes: base, gitnext, lite.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… test as one failure

From the traces of the third run on Qwen: `why test_bad_defaults_for_multiple` listed
twelve cases by their long ids with no message. pytest leaves one underscore around a
long section name (`_ test_x[a long id] _`), which was not taken for a section; and the
cases of one test are now one failure with a count ("12 cases") and one message.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…learn-tests split per language

`gitnext why <test>` ranked what the call graph reaches from the test. Measured on 37
seeded bugs (eval/why_rank.py, eval/results/why-rank.md): the changed function is first
in 10% and among the 8 printed in 29%; in 15 of 37 the graph does not reach it at all.

- with a learned map (`gitnext learn-tests`) the suspects are the functions the test ran:
  code changed since the map was learned first, then what the fewest tests run. On 20
  bugs in click and sqlparse: first in 90% (the graph: 5%), among the 8 printed in 100%;
  without the "changed" mark 20% and 80%. A test the map does not know: the call graph;
- `TestMap::ran_by`: what one test ran, with the code hash then and how many tests run it;
- a test is found by the id its runner prints: `tests/test_x.py::TestA::test_b[case]`,
  `TestRun/subtest`. A name that several tests have (`test_basic` of five files): up to
  six are run and the code of the ones that fail is explained, not of all of them;
- `why --json` for scripts; the order by "how many tests reach it through the graph" was
  measured and dropped (it moved nothing);
- digest: a rustc error keeps its source line and the label under the caret;
- learn-tests: the part every language shares (`learn.rs`: pairs "function, test", the
  line → function table, the map) and one file per language (`learn/python.rs`; Go and
  Rust answer "not supported yet" from their own files);
- eval: a seeded bug does not count a test that fails on the untouched code as well.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… tool call; lite gets the gate

From the traces of the runs on Qwen 3.5 4B:
- `fix`, semver: `why` gave the culprit and its code, the model found the bug and wrote
  the corrected function - as text in its answer, not as a `replace` call (0 of 6 solved).
  In the modes with the gitnext tools a function of the project written in the answer is
  now put in place (`gitnext edit replace`) and checked, the verdict goes back to the
  model. Replayed on the model's own answer: builds, all 20 tests pass;
- find / callers / rename, 36 runs: gitnext 14/18, lite 17/18 - all 5 lost runs were an
  answer given before any tool was called (twice the bare word `look` / `check`). Every
  mode with the gitnext tools now sends such an answer back once, and a tool name written
  as the answer is told to be called; `lite` has the whole gate of `verify`;
- `--learn` (on in run_agent_mac.sh, LEARN=0 turns it off): `gitnext learn-tests` on the
  untouched code of a Python project before its `fix` tasks, as a project would have done
  while its tests passed;
- a seeded bug is told by a test that passed before it: tests failing on the untouched
  code are left out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Go and Rust have things to say that Python does not: where the instrumented build was
left and how much disk it takes, how many tests could not be run alone or have no symbol
in the graph. `Run.notes` carries them, `gitnext learn-tests` prints each as a `note:`
line; a language of the repository that failed to give a map is reported the same way
(not the ones that are simply not supported yet).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
From the first run where `fix` was solved at all (Qwen 3.5 4B, click 3 of 12): `why`
put the changed function first with its code, and the model still could not see the
wrong line - it rewrote the function word for word, or "fixed" another line three
times in a row. A person in that place asks what changed since the tests passed.

- `learn-tests` remembers the commit the tests ran on (`meta.commit` in the map);
- `why` prints, for the first suspect marked "changed since learn-tests", the hunks of
  `git diff <that commit>` that fall inside the function: `- if rv is not None:` /
  `+ if rv is None:` (up to 10 lines).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rst answer is dropped

From the traces of the `fix` run on Qwen 3.5 4B (24 runs):
- the model wrote the function exactly as it was and was told "nothing to check ... If
  the task is done, answer in one line"; or wrote the same wrong fix three times and got
  the same failed check three times. A function that leaves the file as it is now gets
  one line instead: it is the same code, one line of it is wrong, write it again;
- lite, 3 of 8 runs: an invented answer before any tool call, repeated word for word
  after "you have not looked at the code yet". That answer is now taken out of the
  conversation and the task is asked again with "your first step must be a tool call".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
In the `fix` run on Qwen 3.5 4B the model with the eleven tools called `check` with the
test's name first in 9 of 11 runs; with the five tools of lite in 1 of 11 (and that one
run was solved) - it called `look` or answered from memory. The lite description of
`check` now starts with that case: "A test fails: call this first with test = its name".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
For Python the first line of a function runs when the module is loaded, so `owners`
leaves it out. Go and Rust coverage books the first block of a function to the line of
its signature, and that line runs only when the function is called: without it every
function without a branch and every one-line function would be lost. `owners_at(g, true)`
counts it (and one-line functions); `owners` is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…othing public; what checks what

- eval/smoke.py: every read-only command with a real function, class, file and folder of
  the project, the commands that change files in a throw-away copy, every MCP tool over
  stdio - on each of the 10 eval projects. 86 commands and tools x 10 = 860 calls: none
  failed, crashed, came back empty or took 20 s (eval/results/smoke.md);
- found by it: `gitnext outline <folder>` printed nothing for a folder of tests, benches
  or private helpers. It now lists the files with their symbol counts and says how to see
  them (`--all`, `--tests`);
- eval/results/COVERAGE.md: for each function, what checks it - measured against the
  compiler and the tests, unit tests, or only "it answers" - and what would move the
  last group up.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ented build

`gitnext learn-tests` in a Rust repository builds the tests with
`-C instrument-coverage` in a target of their own (`<target>/gitnext-cov`,
kept for the next map), runs each test alone with its own profile and turns
it into lines with rustup's llvm-profdata + llvm-cov (llvm-tools-preview; a
missing component is an error with the command to install it). Profiles are
exported on other threads while the next test runs and removed at once.

A cargo test path (`display::tests::test_align`) is matched to the test's
symbol in the graph among the files cargo compiled into that binary (its
dep-info), so a namesake behind a disabled feature (indexmap's
`rayon::set::tests::comparisons`) is not taken.

Measured on the eval repositories:
- semver: 33 tests, 773 pairs, 69 functions (prototype 005: 34 / 782 / 70;
  the missing test runs no library code), 17 s first, 1 s again; 335 MB.
- indexmap: 146 tests, 3837 pairs, 358 functions (005: 183 / 5003 / 381),
  16 s first, 4 s again (plain warm cargo test 1.6 s); 338 MB. 43 of 192
  tests are written inside `quickcheck_limit! {}` and have no symbol in
  the graph; they are reported in a note and left out (about 1306 pairs).
- eval/why_rank.py --repos semver,indexmap --learn --bugs 10: the culprit is
  first 15% -> 55%, in the first 8 40% -> 80% (graph only -> with the map).
…name are two entries of the map

Found while checking the Rust map:
- the Rust run booked lines with `owners`, which leaves the first line of a function out
  (right for Python). With `owners_at(g, true)` semver gives 70 functions and 782 pairs -
  the prototype's numbers - instead of 69 and 773: one-line functions are back;
- `impl Display for Version` and `impl Debug for Version` both define `Version::fmt`,
  one qualified name. The map kept the code hash of one and `why` compared it with the
  code of the other: "changed since learn-tests" on a function nobody touched, and the
  real culprit (`pad`) second. `Graph::unique_name` gives the later namesakes `#2`, `#3`;
  the map, `tests-for` and `why` use it. `why test_align` with the bug in `display::pad`
  (the case where the call graph saw no path at all): `pad` is first, with the changed
  line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eval/learn_tests_js.py maps test files to the functions they run: every
vitest file runs alone with v8 coverage, a statement that ran is booked to
the innermost function whose body holds it. Scored by test file, as the
eval scores TS. Driver: learn_tests_go.measure.

yaml (32 functions): tests-for 3% recall / 22% of its first 10 fail;
coverage 100% / 100%. zustand (19): 52% / 58% -> 100% / 94%.
Map cost: 6.2x (yaml) and 2.3x (zustand) a plain test run.

--causes: the graph books calls inside `it('...', () => ...)` to the test
file's module symbol, and tests-for skips module symbols (not callable).
Counting them would take yaml from 2% to 18% (4 hops, sure edges), 43%
with possible edges, 62% at 8 hops; zustand from 52% to 67%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`it('…', () => …)` is not a named function: the calls made inside it are booked to the
module of the test file, and `tests-for` dropped every symbol that is not a function.
On yaml it found 2% of the test files that fail when a function breaks (11 of its 24
test files have no named function at all).

A test file's module now counts as a test in JS/TS, and the command runs that file.
Measured on the same functions (eval/run.py --repos yaml,zustand --sample-from latest):
yaml tests recall 2% -> 18% ±10, precision 16% -> 31%, the first 10 that fail 16% -> 32%,
caught by the first 10: 24% -> 38%. The estimate made before the change (walking
`callers --tests` to test modules, eval/learn_tests_js.py --causes) said 18%.

What is left is in eval/results/learn-tests-yaml.causes.md: paths deeper than 4 calls,
"maybe" edges, exports under another name - and a coverage map finds 100%.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`gitnext learn-tests` in a Go module now writes the test → function map:
`go test -c -cover -coverpkg=<module>/...` per package, then
`<pkg>.test -test.run '^Name$' -test.coverprofile=…` for each test,
example and fuzz target. A profile block belongs to the innermost
function around its first line, the signature line included
(`owners_at(g, true)`). Profile files are mapped to the repository by
`go list` (import path → directory); the test's file is its symbol in
the graph.

Notes instead of failures: packages that do not build (10 min limit),
TestMain, tests that fail alone but pass with their package, failing
tests, tests with no symbol. No go.mod at the root or no `go` in PATH
gives a plain error. Binaries and profiles are removed however it ends.

Same map as the prototype: cobra 236 functions / 13106 pairs,
279 tests run, 277 in the map (2 run no function of the module), 8 s
warm; chi 164 / 3826 / 140, 34 s.

eval/why_rank.py --repos cobra,chi --learn --bugs 10: the culprit is
first in 40% by the call graph, 85% with the map; in the first 8:
60% → 100%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…measured

- eval: the full run with the test file counted as a test in JS/TS is the new baseline.
  Paired with run 7 only yaml and zustand moved, every metric up (yaml tests recall
  1.9% -> 18.0%, +16.1, 90% interval +7.8 … +25.1);
- why-rank-rust: measured again after the two fixes to the Rust map - the culprit is
  first in 70% of 20 seeded bugs (15% from the call graph), semver 10 of 10;
- learn-tests: the count of functions no test ran uses the signature line for the
  compiled languages; help and README name the three languages;
- HISTORY (run 8) and COVERAGE updated.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`gitnext learn-tests` now knows Go and Rust, so the harness learns the map of semver
before its `fix` tasks as it does for click (0 of 10 solved there without a map: twice
the call graph saw no path from the test to the code at all).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… (35% from the call graph)

eval/why_rank.py --repos cobra,chi --learn: cobra 5 of 7 first (1 of 7 from the graph),
chi 9 of 10 (5 of 10); among the 8 printed 17 of 17. Over the six projects with a map
(Python, Go, Rust): first in 46 of 57 (81%), from the call graph alone 10 of 57 (18%).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…curacy by language

The front page said what gitnext does and showed one measurement (bytes read). It now
shows what was measured since:
- two charts (docs/images/agent.svg, why.svg and their Russian versions, drawn by
  docs/images/make_charts.py): a 4B local model solves 4 of 24 tasks with plain tools
  and 17 with gitnext; `why` names the function at fault first in 18% of 57 seeded bugs
  from the call graph and in 81% with the learned coverage map;
- accuracy by language against compilers and tests (callers, rename, tests-for), with
  the noise interval;
- what is checked and what is not good yet (TypeScript tests, 9% of renames, fixing a
  bug on a 4B model, Rust tests inside macros);
- the short tool list for small models, `why` and `digest` in the Russian README, the
  eval folder in "How it works";
- HISTORY: the five agent runs in one table; GUIDE.ru: the lite profile has five tools.
The claim that a 4B model does not call the tools is gone: it does.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The version in the workspace, the lock file and the npm package; the packaging notes
name the new tag; CHANGELOG.md with what is new since 0.2.0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@pnnnx
pnnnx merged commit fa4f5e6 into main Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants