Skip to content

Evaluation: baselines, held-out projects, second tool; check and replace fixes - #4

Merged
pnnnx merged 16 commits into
mainfrom
next
Nov 15, 2026
Merged

pnnnx merged 16 commits into
mainfrom
next

Conversation

@pnnnx

@pnnnx pnnnx commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Baselines, fourteen held-out projects, gopls rename and Codebase-Memory on the same functions, the agent runs, and three tool changes after 0.3.1.

ci and others added 16 commits November 13, 2026 20:28
In a project of several languages (Prometheus: a Go server with a TypeScript UI) the
sample took TypeScript functions too, and a rename there is judged by the Go compiler,
which cannot see it. The six projects of the earlier runs are in one language each:
their tasks are the same as before (checked on click, cobra, semver, seed 11).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Kubernetes (5.6M lines of Go, 245 k symbols, 1.6 M edges): every command took 2.8-3.9 s,
almost all of it reading the index into memory. Now 2.0-3.1 s (seven commands: 22.4 s ->
16.9 s, the answers are the same):
- symbols and edges are read on two threads (a second read-only connection);
- the loop over every edge that finds which packages see which allocates nothing unless
  it finds a new pair; kinds and empty parameter lists are read without a copy;
- a process that answers once and exits does not free the graph string by string.

The MCP server already keeps the graph between calls: 2.3 s to start, then 0.3-0.5 s a
call on the same repo. What is left for the CLI is the load itself.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e call that changes it

'next: fix the code, not the test' named no function and spoke of CLI commands. In the
runs on three models the 4B and 9B models named the bug and stopped without an edit in
most of their lost fix tasks. The line is now
  next: fix `matches_greater` (src/eval.rs:9) - the code, not the test. Replace the
  whole function with a corrected one: <the call>; then <check>.
with the call worded for whoever reads it: the CLI's command, replace_symbol(...) for the
MCP server, replace(...) for its short tool list. The eval agent keeps the line when it
cuts a long answer. Not measured on a model yet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…und every test

One `mvn test` from a copy of the pom whose surefire carries the JaCoCo agent and a
listener of the JUnit Platform; the listener takes and resets the agent's data when a
test starts and ends, so each test gets the lines it ran itself. What the classes'
static blocks run is measured apart and booked to every test that touches the class.
JUnit 5, and JUnit 4 under a surefire that runs it on the platform (3.6 and newer).

The map against the tests that fail when a method raises: JSON-java 99% (the call
graph: 71%), jsoup 94% (29%) — the same as the prototype in eval/learn_tests_java.py.
The run took 19 s and 69 s. Nothing is downloaded: when JaCoCo is not in the local
Maven repository the message is the command that gets it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t was meant

A small model used to text edits calls replace("if a == b {", "if a != b {"), or
gives the first line of the function as its name. In the agent runs on three models
every such call was refused with "symbol not found" (all of them from the 4B model),
and only 66 of 129 replace / add_after calls of the five-tool mode made an edit.

When the name is not a name but text of the code: text that stands in one place is
replaced there (at its indent, the file must still parse); when the code is a whole
function, the function around that place is replaced; text in several places is
answered with the places and the functions they are in. Not measured on a model.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eval/agent_traces.py reads the conversations the agent keeps and says what a run's
calls are, which of them bring nothing, what is lost and won against plain tools on
the same task and how the failed runs end. HISTORY.md has the tables and what they
show: a quarter to a third of the calls of plain tools bring nothing new; a fix is
lost before the first edit; `check` blames the change for a test that fails without
it; `replace` is called like a text edit; the harness's gate acts in the gitnext
modes only and stands behind at most 4 of the 163 tasks solved with five tools.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The tests that failed are run once more on the code as the last commit has it: the
changed files are put back for that run and the changes return when it ends. A test
that fails both ways is named as not caused by the change, and when nothing else
fails the step is "AS BEFORE" and the verdict ok; a test that passes without the
change is said to. The answer is kept per commit and command, so a check that keeps
failing asks once.

In the agent runs cobra's permission test, which cannot pass as root, was reported
as "tests FAILED" after a correct rename: one model renamed the function back,
another started rewriting the test. The same rename now ends in "verdict: ok".

The files are swapped a whole second apart from their last write, with a margin for
the file system's clock: Python keeps compiled code by the source's size and second,
and ran the old code otherwise. What was in the files is kept in .gitnext with a
list until it is back; a check killed in between is undone by the next one.
GITNEXT_CHECK_BEFORE=0 switches the second run off.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…f Go

Two models (4B, 27B), find / callers / rename, 12 task sets, each task once: 36
tasks per model in every mode. Solved of 72: 49 with plain tools, 61 with eleven
tools, 62 with five (+14 -1, p = 0.001, +18 points, 95% interval +7 to +31); a
median of 8, 4 and 3 steps; 47k, 22k and 16k tokens sent per task. Three of the
twelve renames cannot pass in any mode: the judge finds another thing of the same
name. agent_stats.py takes the models, kinds of task and projects from the file.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…can tell

"No old name left as a word" failed every mode on a name that is other things too:
os.Rename, a struct field called like the function — 3 of 12 renames on Prometheus.
With a compiler a use of a name that is no longer defined does not build, so the
judge now asks that the project builds, the old definition is gone and the new one
is there. Python has no compiler and keeps the word rule.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
What a plain search for the name gives on the same functions, what a language
server gives, and what the numbers are on projects the tool was never tuned on.

- baselines.py: `git grep -w` for callers and for tests, ten tests at random.
  Unique names: the search finds every caller at a precision of 89% (gitnext 99%).
  Shared names: the search 100 / 32, gitnext 83 / 99. Ten tests at random catch a
  broken function in 61% of cases, gitnext's first ten in 79%.
- lsp_baseline.py: references asked from gopls, rust-analyzer and tsserver.
  Shared names: 97 / 99 where gitnext has 75 / 100.
- cluster_boot.py: intervals with the project as the unit; twice as wide for tests.
- repos-heldout.json, run.py --repos-file, merge_runs.py: fourteen projects in four
  languages, measured once. Callers 99 / 96; picked blind 87% (99% on the ten);
  shared names 73% (83%); rename 77% (91%); tests better than on the ten.

README says what the accuracy table is not; HISTORY has run 9.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d Rust

The command asks a language server where each ambiguous call goes and keeps the
answers in the index. Run once in cobra, chi (gopls) and semver, indexmap
(rust-analyzer), 3 to 57 seconds a project, and the eval repeated on the same
functions: recall for methods that share a name +18.5 points (90% interval
+9.5 to +27.4), tests recall +8.5, nothing worse. The help of the command names
all six languages it has servers for.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… to 93%

Eight held-out Go and Rust projects, 108 methods that share a name: recall 72.7%
to 92.7% ±4 (+19.9; 90% interval +14.8 to +25.0) with the language server's answers
in the index; the servers themselves find 99%. Blind recall and rename do not move.
HISTORY also has the time of a `callers` query: 20–50 ms on eight of the ten
projects, 0.16 s and 1.5 s on the two Java ones.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…9 points, interval includes zero

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… against 93%, 88% against 78% held out

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… against the compiler's uses

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ed; the fix judge leaves out tests that failed before

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@pnnnx
pnnnx merged commit 9f5a623 into main Nov 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant