Skip to content

gitnext 0.3.1 - #2

Merged
pnnnx merged 18 commits into
mainfrom
next
Nov 13, 2026
Merged

pnnnx merged 18 commits into
mainfrom
next

Conversation

@pnnnx

@pnnnx pnnnx commented Nov 13, 2026

Copy link
Copy Markdown
Owner

look takes a place in a file; learn tests for JS/TS; Rust tests inside macros; dead by deletion; rename --text reaches the docs; the agent runs on three models. See CHANGELOG.md

ci and others added 18 commits November 12, 2026 23:12
A model addresses code the way tracebacks and test runners do: src/eval.rs:62,
tests/test_x.py:120-160, src/eval.rs::matches_greater, tests/t.py::TestA::test_b[case],
'pad src/display.rs'. look answered all of these with 'nothing found'; in a trial
(Qwen 3.5 4B and 27B, the fix tasks) that was 37 of 89 look calls, and the small
model with the short tool list spent its step limit on them.

- file:N - the function around the line, as look shows a function; outside any: the lines around
- file:A-B (also 'file A-B', L120-L154) - those lines, up to 200, and the functions they are in
- file::name, file:name, 'file name', 'name file' - the name in that file; a test id loses its case
- not in that file: said so, with the outline of the file
- 'file | grep ...' is the file
- several words of which some are names: those names, not 'nothing found'

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
llama.cpp answers 500 when the model's tool call is not whole JSON - a long replace
cut at 1024 tokens. The run ended there as an error; now the same step is asked
again with room to finish (counted as cut_calls). The look tool of the short list
says it reads lines of a file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rm runner

The v0.3.0 release has no gitnext-x86_64-apple-darwin.tar.gz: its job asked for a
macos-13 runner, and those are retired, so it never started. Both Mac targets now
build on macos-latest (cargo --target), with a cache key per target.

Action versions that run on Node 24 (the Node 20 ones print a deprecation warning):
checkout v7, setup-go v7, setup-node v7, setup-java v6, setup-python v7, cache v6,
upload-artifact v7, action-gh-release v3.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…and the source found

eval/consistency.py asks, on the ten projects of the eval, what can be asked without
a compiler: callers <-> callees <-> impact, read and symbols against the file, search,
edit replace / delete round trips, hierarchy against class signatures, and dead by
deleting what it names. Results: eval/results/consistency.md; COVERAGE.md has a level for it.

What it found, fixed here:
- dead knew calls only. 160 of 225 'likely dead' symbols were named elsewhere in the
  code (a value, a type, a base or metaclass, an import, a callback, a Go init);
  deleting them broke the build of at least 6 projects of 8. The name must now be
  written nowhere outside the symbol; .d.ts declarations and Go init are not dead.
  64 are left; sqlparse still loses 5 methods reached by names built at run time.
- edit replace moved a symbol replaced with its own code when a body line was indented
  less than the first, or tabs stood under spaces (11 of 40 edits in the Java projects):
  code that already stands where it belongs is not touched, the rest keeps its place
  relative to the first line.
- edit delete refused, and read cut the comment in half, for a Javadoc whose lines do
  not start with '*': a block comment that ends above the item is taken whole.
- callees listed as sure a method called on an object of unknown type that callers
  lists as 'maybe' (5 pairs): callees has 'maybe' too, look counts them.

Also: the Kaggle trial (Qwen 3.5 27B and 4B, the fix tasks) in HISTORY.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…heck! and the like

Rust tests are often declared in the body of a macro call: `quickcheck! { fn prop(..)
-> bool { .. } }`, `proptest! { #[test] fn t(..) { .. } }`, or a wrapper of the project's
own like indexmap's `quickcheck_limit!`. The parser did not look into a macro called where
items go, so in indexmap 43 of 192 tests were not functions of the graph: learn-tests could
not run them, tests-for did not name them, and why found no test for 4 of 10 seeded bugs.

The body of such a call (file, module, impl or trait level) is parsed again as code: the
same source, only the bytes between the brackets, so lines, columns and calls are those of
a real function. Its `fn` items become symbols; a stretch that is not Rust (`x in 0..10`)
keeps the function, without arguments. They are tests in a test file, under #[test], or
when the macro is a test macro: quickcheck*, proptest*, or a macro_rules! of the same file
that writes #[test]. A test such a macro writes takes no arguments (they are the
property's), so its arguments stay in the signature only and why can run it.
Macros inside functions are not items and are left as they were. PARSER_VERSION 31 -> 32.

Measured on semver and indexmap (eval/why_rank.py --bugs 10 --learn, the same bugs):
- indexmap tests outside the graph: 43 of 192 -> 4 (the 4 get their name as a macro
  argument, `move_index_oob!(name, 0, 10);`); semver: 0 -> 0
- learned map, indexmap: 146 tests, 3865 pairs -> 185 tests, 5202-5208 pairs
- why, the culprit first: 70% -> 80-90% in three runs (indexmap 40% -> 60-80%); in the
  first 8: 80% -> 100%; empty lists: 4 -> 0. quickcheck draws new inputs on each run, so
  the learned map moves a little and two bugs swap between first and third place;
  eval/results/why-rank-rust.md is the first run
- eval/run.py --repos semver,indexmap: callers 99.3% / 100% unchanged, rename 86.8% ->
  89.5% (+-8), tests recall 52.8% -> 61.4% (+-13), caught by the top 10 72.7% -> 81.8%
- gitnext index on indexmap, from scratch: median 114 ms -> 121 ms

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…protected is API

Deleting everything dead called likely dead still broke two of the ten eval projects:

- sqlparse: 5 methods reached by names built at run time
  (getattr(self, f'_process_{type(tlist).__name__}'.lower())); deleting them failed
  16 tests. A piece of a name - a string of 3+ letters glued into a name ("_visit_" + x,
  f"_process_{x}", "_handle_%s" % x, x + "_handler", obj[`on${x}`]) within 3 lines of
  reflection (getattr, globals(), getMethod, MethodByName, ...) - now keeps every symbol
  whose name starts or ends with it. On the ten projects it keeps these 5 and nothing else.
- indexmap: all 57 were #[bench] functions. They looked deletable only because the
  benches were never built; with every feature and target (nightly, #![feature(test)])
  deleting them gives 17 compile errors. Functions run by #[bench], #[no_mangle],
  #[export_name], #[proc_macro...], #[...::main] are not dead.
- json-java: a protected method is there for subclasses outside the repo: it is listed
  as public but unused, not likely dead.

likely dead on the ten projects: 64 -> 1 (jsoup, deleted with no new compile error or
failing test). eval/consistency.py now deletes in the Java projects too (Maven before and
after) and builds Rust with --all-features --all-targets, on nightly where stable cannot
build the benches; each project gets a one-line verdict. public but unused: of 265 without
the named-elsewhere check, 213 are named elsewhere; of 37 of the remaining 52 read by hand,
29 are API, 4 are reached another way, 4 are dead (COVERAGE.md, level 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
tests_for takes the module of a JavaScript / TypeScript test file as the test
(it('...') is not a named function), but tests_for_at looked map rows up among
functions only, so every row of such a map was dropped and the answer silently
came from the call graph. The lookup now takes the same test units as tests_for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A language of learn-tests is now in the repository when a file ends with any of
its suffixes, and its tests still run once. Python, Go and Rust keep one suffix
each; JavaScript / TypeScript needs .ts, .tsx, .js, .mjs and the like.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test file vitest lists (vitest list --filesOnly --json) runs alone with v8
coverage; a statement that ran is booked to the innermost function whose body
holds its line, and the test is the file's module symbol, as tests-for and why
know it. The declaration line of a function is not counted: it runs when the
module is imported. On yaml and zustand the map has 285 / 48 functions and
2943 / 164 pairs (the prototype in eval: 284 / 47 and 2919 / 163), in 36 s and
22 s.

Coverage counts the top directories with code of the index: the project's own
coverage.include may name nothing (zustand's 'src/**/' matches directories
only). A file passed when vitest's JSON result counts no failed test; the
counters are read from the text, since the result quotes test strings and a
lone surrogate in them is not JSON that serde_json accepts (yaml's stringify
tests).

Nothing is installed: without package.json, without vitest in node_modules or
without @vitest/coverage-v8 the message says what to run (the coverage package
of vitest's own version); a project on jest is told that only vitest is known.
Files that fail, take longer than 10 minutes or write no report (a test file
that does not load says why) are notes.

Tests: the parsing of a saved coverage report, vitest list, the JSON result and
package.json in learn/js.rs; the messages in tests/learn_js.rs without node; a
run on a small project that installs vitest from npm only with GITNEXT_TEST_NPM=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
learn_tests_js.py --product runs gitnext learn-tests on the eval's checkout and
scores tests-for by test file before the map (the call graph) and after it, on
the functions the prototype took (the eval's cached ones and seed 7 random ones).

yaml, 32 functions: recall 16% -> 100%, precision 33% -> 100%, failing among
the first 10 34% -> 100%, caught by the first 10 41% -> 100%.
zustand, 19 functions: recall 68% -> 100%, precision 68% -> 94%, first 10
68% -> 94%, caught 74% -> 100% - the prototype's numbers on both.
The map takes 37 s and 22 s, 6.6x and 2.6x a plain run.
Results: eval/results/learn-tests-{yaml,zustand}.product.{md,json}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… map

eval/learn_tests_java.py maps test methods to the functions they run with
the JaCoCo agent (0.8.14), scored by test method name as the eval scores
Java, and also by test class. Driver: learn_tests_go.measure.

- per-test: one Maven run; a JUnit Platform listener (JUnit 4 runs through
  vintage under surefire 3) takes the agent's data when a test starts and
  ends. What a test class runs outside its tests (static blocks, BeforeAll)
  goes to all its tests; what initializing a library class runs goes to
  every test that runs a line of its file.
- per-test-fork: the same, a JVM per test class (-DreuseForks=false).
- per-class: one Maven run per test class (-Dtest=...).

All 33 (jsoup) / 34 (json-java) functions, by method: recall / precision /
fail among the first 10:
  tests-for            29 / 64 / 66      71 / 48 / 61
  per-test             94 / 94 / 94      99 / 94 / 95
  per-class           100 / 36 / 33     100 / 21 / 32
Map cost against a plain run: per-test 3.5x / 2.1x (50 s / 10 s),
per-test-fork 8.4x / 4.1x, per-class 27x / 16x. per-test misses tests
that fail through state a JVM builds once (jsoup: Normalizer.lowerCase
18% of 1689); by test class it finds 98% / 100%, the fork 100% / 100%.

The build is changed only in a copy of pom.xml, removed after the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…mes first; look takes a list

From the agent runs on three models (648 runs):
- rename done with the tool failed the judge on the same tasks in every model. Two of the
  five: the old name stayed in docs/*.rst. rename now reports the name in documentation
  the index lists but does not parse (.md .rst .txt .adoc) where it is written as code,
  and --text rewrites it.
- with eleven tools the 9B model never began a fix with check <test> (0 of 18), the 4B
  model 5 times; with five tools 16 of 18. The check description (MCP, eval) now begins
  with the failing test.
- look: 'a.go:497-506, a_test.go:123-128' is two places; 'TestX/subtest' is TestX.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… follow them

Qwen 3.5 4B, 9B, 27B on six projects, the same 72 tasks plain / gitnext / lite:
144 -> 156 -> 163 of 216 solved; a median of 9-10 steps -> 4-5 -> 3-4; the 27B model
fixes 14 of 18 seeded bugs against 7. For one model the difference in solved tasks is
inside the noise. The 24-task comparison of the earlier runs (4 against 15) did not
repeat - plain solves 19 of those 24 with today's harness - and is no longer quoted.

Every row: eval/results/agent-kaggle-20261113.json; tables: eval/agent_stats.py.
why on Rust with the map: 17 of 20 after macros became symbols (chart: 86% of 57).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@pnnnx
pnnnx merged commit a219fec into main Nov 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant