Repository navigation
Conversation
In a project of several languages (Prometheus: a Go server with a TypeScript UI) the sample took TypeScript functions too, and a rename there is judged by the Go compiler, which cannot see it. The six projects of the earlier runs are in one language each: their tasks are the same as before (checked on click, cobra, semver, seed 11). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Kubernetes (5.6M lines of Go, 245 k symbols, 1.6 M edges): every command took 2.8-3.9 s, almost all of it reading the index into memory. Now 2.0-3.1 s (seven commands: 22.4 s -> 16.9 s, the answers are the same): - symbols and edges are read on two threads (a second read-only connection); - the loop over every edge that finds which packages see which allocates nothing unless it finds a new pair; kinds and empty parameter lists are read without a copy; - a process that answers once and exits does not free the graph string by string. The MCP server already keeps the graph between calls: 2.3 s to start, then 0.3-0.5 s a call on the same repo. What is left for the CLI is the load itself. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e call that changes it 'next: fix the code, not the test' named no function and spoke of CLI commands. In the runs on three models the 4B and 9B models named the bug and stopped without an edit in most of their lost fix tasks. The line is now next: fix `matches_greater` (src/eval.rs:9) - the code, not the test. Replace the whole function with a corrected one: <the call>; then <check>. with the call worded for whoever reads it: the CLI's command, replace_symbol(...) for the MCP server, replace(...) for its short tool list. The eval agent keeps the line when it cuts a long answer. Not measured on a model yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…und every test One `mvn test` from a copy of the pom whose surefire carries the JaCoCo agent and a listener of the JUnit Platform; the listener takes and resets the agent's data when a test starts and ends, so each test gets the lines it ran itself. What the classes' static blocks run is measured apart and booked to every test that touches the class. JUnit 5, and JUnit 4 under a surefire that runs it on the platform (3.6 and newer). The map against the tests that fail when a method raises: JSON-java 99% (the call graph: 71%), jsoup 94% (29%) — the same as the prototype in eval/learn_tests_java.py. The run took 19 s and 69 s. Nothing is downloaded: when JaCoCo is not in the local Maven repository the message is the command that gets it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t was meant
A small model used to text edits calls replace("if a == b {", "if a != b {"), or
gives the first line of the function as its name. In the agent runs on three models
every such call was refused with "symbol not found" (all of them from the 4B model),
and only 66 of 129 replace / add_after calls of the five-tool mode made an edit.
When the name is not a name but text of the code: text that stands in one place is
replaced there (at its indent, the file must still parse); when the code is a whole
function, the function around that place is replaced; text in several places is
answered with the places and the functions they are in. Not measured on a model.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eval/agent_traces.py reads the conversations the agent keeps and says what a run's calls are, which of them bring nothing, what is lost and won against plain tools on the same task and how the failed runs end. HISTORY.md has the tables and what they show: a quarter to a third of the calls of plain tools bring nothing new; a fix is lost before the first edit; `check` blames the change for a test that fails without it; `replace` is called like a text edit; the harness's gate acts in the gitnext modes only and stands behind at most 4 of the 163 tasks solved with five tools. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The tests that failed are run once more on the code as the last commit has it: the changed files are put back for that run and the changes return when it ends. A test that fails both ways is named as not caused by the change, and when nothing else fails the step is "AS BEFORE" and the verdict ok; a test that passes without the change is said to. The answer is kept per commit and command, so a check that keeps failing asks once. In the agent runs cobra's permission test, which cannot pass as root, was reported as "tests FAILED" after a correct rename: one model renamed the function back, another started rewriting the test. The same rename now ends in "verdict: ok". The files are swapped a whole second apart from their last write, with a margin for the file system's clock: Python keeps compiled code by the source's size and second, and ran the old code otherwise. What was in the files is kept in .gitnext with a list until it is back; a check killed in between is undone by the next one. GITNEXT_CHECK_BEFORE=0 switches the second run off. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…f Go Two models (4B, 27B), find / callers / rename, 12 task sets, each task once: 36 tasks per model in every mode. Solved of 72: 49 with plain tools, 61 with eleven tools, 62 with five (+14 -1, p = 0.001, +18 points, 95% interval +7 to +31); a median of 8, 4 and 3 steps; 47k, 22k and 16k tokens sent per task. Three of the twelve renames cannot pass in any mode: the judge finds another thing of the same name. agent_stats.py takes the models, kinds of task and projects from the file. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…can tell "No old name left as a word" failed every mode on a name that is other things too: os.Rename, a struct field called like the function — 3 of 12 renames on Prometheus. With a compiler a use of a name that is no longer defined does not build, so the judge now asks that the project builds, the old definition is gone and the new one is there. Python has no compiler and keeps the word rule. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
What a plain search for the name gives on the same functions, what a language server gives, and what the numbers are on projects the tool was never tuned on. - baselines.py: `git grep -w` for callers and for tests, ten tests at random. Unique names: the search finds every caller at a precision of 89% (gitnext 99%). Shared names: the search 100 / 32, gitnext 83 / 99. Ten tests at random catch a broken function in 61% of cases, gitnext's first ten in 79%. - lsp_baseline.py: references asked from gopls, rust-analyzer and tsserver. Shared names: 97 / 99 where gitnext has 75 / 100. - cluster_boot.py: intervals with the project as the unit; twice as wide for tests. - repos-heldout.json, run.py --repos-file, merge_runs.py: fourteen projects in four languages, measured once. Callers 99 / 96; picked blind 87% (99% on the ten); shared names 73% (83%); rename 77% (91%); tests better than on the ten. README says what the accuracy table is not; HISTORY has run 9. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d Rust The command asks a language server where each ambiguous call goes and keeps the answers in the index. Run once in cobra, chi (gopls) and semver, indexmap (rust-analyzer), 3 to 57 seconds a project, and the eval repeated on the same functions: recall for methods that share a name +18.5 points (90% interval +9.5 to +27.4), tests recall +8.5, nothing worse. The help of the command names all six languages it has servers for. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… to 93% Eight held-out Go and Rust projects, 108 methods that share a name: recall 72.7% to 92.7% ±4 (+19.9; 90% interval +14.8 to +25.0) with the language server's answers in the index; the servers themselves find 99%. Blind recall and rename do not move. HISTORY also has the time of a `callers` query: 20–50 ms on eight of the ten projects, 0.16 s and 1.5 s on the two Java ones. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…9 points, interval includes zero Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… against 93%, 88% against 78% held out Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… against the compiler's uses Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ed; the fix judge leaves out tests that failed before Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Baselines, fourteen held-out projects, gopls rename and Codebase-Memory on the same functions, the agent runs, and three tool changes after 0.3.1.