Repository navigation
Conversation
A model addresses code the way tracebacks and test runners do: src/eval.rs:62, tests/test_x.py:120-160, src/eval.rs::matches_greater, tests/t.py::TestA::test_b[case], 'pad src/display.rs'. look answered all of these with 'nothing found'; in a trial (Qwen 3.5 4B and 27B, the fix tasks) that was 37 of 89 look calls, and the small model with the short tool list spent its step limit on them. - file:N - the function around the line, as look shows a function; outside any: the lines around - file:A-B (also 'file A-B', L120-L154) - those lines, up to 200, and the functions they are in - file::name, file:name, 'file name', 'name file' - the name in that file; a test id loses its case - not in that file: said so, with the outline of the file - 'file | grep ...' is the file - several words of which some are names: those names, not 'nothing found' Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
llama.cpp answers 500 when the model's tool call is not whole JSON - a long replace cut at 1024 tokens. The run ended there as an error; now the same step is asked again with room to finish (counted as cut_calls). The look tool of the short list says it reads lines of a file. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rm runner The v0.3.0 release has no gitnext-x86_64-apple-darwin.tar.gz: its job asked for a macos-13 runner, and those are retired, so it never started. Both Mac targets now build on macos-latest (cargo --target), with a cache key per target. Action versions that run on Node 24 (the Node 20 ones print a deprecation warning): checkout v7, setup-go v7, setup-node v7, setup-java v6, setup-python v7, cache v6, upload-artifact v7, action-gh-release v3. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…and the source found eval/consistency.py asks, on the ten projects of the eval, what can be asked without a compiler: callers <-> callees <-> impact, read and symbols against the file, search, edit replace / delete round trips, hierarchy against class signatures, and dead by deleting what it names. Results: eval/results/consistency.md; COVERAGE.md has a level for it. What it found, fixed here: - dead knew calls only. 160 of 225 'likely dead' symbols were named elsewhere in the code (a value, a type, a base or metaclass, an import, a callback, a Go init); deleting them broke the build of at least 6 projects of 8. The name must now be written nowhere outside the symbol; .d.ts declarations and Go init are not dead. 64 are left; sqlparse still loses 5 methods reached by names built at run time. - edit replace moved a symbol replaced with its own code when a body line was indented less than the first, or tabs stood under spaces (11 of 40 edits in the Java projects): code that already stands where it belongs is not touched, the rest keeps its place relative to the first line. - edit delete refused, and read cut the comment in half, for a Javadoc whose lines do not start with '*': a block comment that ends above the item is taken whole. - callees listed as sure a method called on an object of unknown type that callers lists as 'maybe' (5 pairs): callees has 'maybe' too, look counts them. Also: the Kaggle trial (Qwen 3.5 27B and 4B, the fix tasks) in HISTORY.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…heck! and the like
Rust tests are often declared in the body of a macro call: `quickcheck! { fn prop(..)
-> bool { .. } }`, `proptest! { #[test] fn t(..) { .. } }`, or a wrapper of the project's
own like indexmap's `quickcheck_limit!`. The parser did not look into a macro called where
items go, so in indexmap 43 of 192 tests were not functions of the graph: learn-tests could
not run them, tests-for did not name them, and why found no test for 4 of 10 seeded bugs.
The body of such a call (file, module, impl or trait level) is parsed again as code: the
same source, only the bytes between the brackets, so lines, columns and calls are those of
a real function. Its `fn` items become symbols; a stretch that is not Rust (`x in 0..10`)
keeps the function, without arguments. They are tests in a test file, under #[test], or
when the macro is a test macro: quickcheck*, proptest*, or a macro_rules! of the same file
that writes #[test]. A test such a macro writes takes no arguments (they are the
property's), so its arguments stay in the signature only and why can run it.
Macros inside functions are not items and are left as they were. PARSER_VERSION 31 -> 32.
Measured on semver and indexmap (eval/why_rank.py --bugs 10 --learn, the same bugs):
- indexmap tests outside the graph: 43 of 192 -> 4 (the 4 get their name as a macro
argument, `move_index_oob!(name, 0, 10);`); semver: 0 -> 0
- learned map, indexmap: 146 tests, 3865 pairs -> 185 tests, 5202-5208 pairs
- why, the culprit first: 70% -> 80-90% in three runs (indexmap 40% -> 60-80%); in the
first 8: 80% -> 100%; empty lists: 4 -> 0. quickcheck draws new inputs on each run, so
the learned map moves a little and two bugs swap between first and third place;
eval/results/why-rank-rust.md is the first run
- eval/run.py --repos semver,indexmap: callers 99.3% / 100% unchanged, rename 86.8% ->
89.5% (+-8), tests recall 52.8% -> 61.4% (+-13), caught by the top 10 72.7% -> 81.8%
- gitnext index on indexmap, from scratch: median 114 ms -> 121 ms
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…protected is API
Deleting everything dead called likely dead still broke two of the ten eval projects:
- sqlparse: 5 methods reached by names built at run time
(getattr(self, f'_process_{type(tlist).__name__}'.lower())); deleting them failed
16 tests. A piece of a name - a string of 3+ letters glued into a name ("_visit_" + x,
f"_process_{x}", "_handle_%s" % x, x + "_handler", obj[`on${x}`]) within 3 lines of
reflection (getattr, globals(), getMethod, MethodByName, ...) - now keeps every symbol
whose name starts or ends with it. On the ten projects it keeps these 5 and nothing else.
- indexmap: all 57 were #[bench] functions. They looked deletable only because the
benches were never built; with every feature and target (nightly, #![feature(test)])
deleting them gives 17 compile errors. Functions run by #[bench], #[no_mangle],
#[export_name], #[proc_macro...], #[...::main] are not dead.
- json-java: a protected method is there for subclasses outside the repo: it is listed
as public but unused, not likely dead.
likely dead on the ten projects: 64 -> 1 (jsoup, deleted with no new compile error or
failing test). eval/consistency.py now deletes in the Java projects too (Maven before and
after) and builds Rust with --all-features --all-targets, on nightly where stable cannot
build the benches; each project gets a one-line verdict. public but unused: of 265 without
the named-elsewhere check, 213 are named elsewhere; of 37 of the remaining 52 read by hand,
29 are API, 4 are reached another way, 4 are dead (COVERAGE.md, level 3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
tests_for takes the module of a JavaScript / TypeScript test file as the test
(it('...') is not a named function), but tests_for_at looked map rows up among
functions only, so every row of such a map was dropped and the answer silently
came from the call graph. The lookup now takes the same test units as tests_for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A language of learn-tests is now in the repository when a file ends with any of its suffixes, and its tests still run once. Python, Go and Rust keep one suffix each; JavaScript / TypeScript needs .ts, .tsx, .js, .mjs and the like. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every test file vitest lists (vitest list --filesOnly --json) runs alone with v8 coverage; a statement that ran is booked to the innermost function whose body holds its line, and the test is the file's module symbol, as tests-for and why know it. The declaration line of a function is not counted: it runs when the module is imported. On yaml and zustand the map has 285 / 48 functions and 2943 / 164 pairs (the prototype in eval: 284 / 47 and 2919 / 163), in 36 s and 22 s. Coverage counts the top directories with code of the index: the project's own coverage.include may name nothing (zustand's 'src/**/' matches directories only). A file passed when vitest's JSON result counts no failed test; the counters are read from the text, since the result quotes test strings and a lone surrogate in them is not JSON that serde_json accepts (yaml's stringify tests). Nothing is installed: without package.json, without vitest in node_modules or without @vitest/coverage-v8 the message says what to run (the coverage package of vitest's own version); a project on jest is told that only vitest is known. Files that fail, take longer than 10 minutes or write no report (a test file that does not load says why) are notes. Tests: the parsing of a saved coverage report, vitest list, the JSON result and package.json in learn/js.rs; the messages in tests/learn_js.rs without node; a run on a small project that installs vitest from npm only with GITNEXT_TEST_NPM=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
learn_tests_js.py --product runs gitnext learn-tests on the eval's checkout and
scores tests-for by test file before the map (the call graph) and after it, on
the functions the prototype took (the eval's cached ones and seed 7 random ones).
yaml, 32 functions: recall 16% -> 100%, precision 33% -> 100%, failing among
the first 10 34% -> 100%, caught by the first 10 41% -> 100%.
zustand, 19 functions: recall 68% -> 100%, precision 68% -> 94%, first 10
68% -> 94%, caught 74% -> 100% - the prototype's numbers on both.
The map takes 37 s and 22 s, 6.6x and 2.6x a plain run.
Results: eval/results/learn-tests-{yaml,zustand}.product.{md,json}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… map eval/learn_tests_java.py maps test methods to the functions they run with the JaCoCo agent (0.8.14), scored by test method name as the eval scores Java, and also by test class. Driver: learn_tests_go.measure. - per-test: one Maven run; a JUnit Platform listener (JUnit 4 runs through vintage under surefire 3) takes the agent's data when a test starts and ends. What a test class runs outside its tests (static blocks, BeforeAll) goes to all its tests; what initializing a library class runs goes to every test that runs a line of its file. - per-test-fork: the same, a JVM per test class (-DreuseForks=false). - per-class: one Maven run per test class (-Dtest=...). All 33 (jsoup) / 34 (json-java) functions, by method: recall / precision / fail among the first 10: tests-for 29 / 64 / 66 71 / 48 / 61 per-test 94 / 94 / 94 99 / 94 / 95 per-class 100 / 36 / 33 100 / 21 / 32 Map cost against a plain run: per-test 3.5x / 2.1x (50 s / 10 s), per-test-fork 8.4x / 4.1x, per-class 27x / 16x. per-test misses tests that fail through state a JVM builds once (jsoup: Normalizer.lowerCase 18% of 1689); by test class it finds 98% / 100%, the fork 100% / 100%. The build is changed only in a copy of pom.xml, removed after the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…mes first; look takes a list From the agent runs on three models (648 runs): - rename done with the tool failed the judge on the same tasks in every model. Two of the five: the old name stayed in docs/*.rst. rename now reports the name in documentation the index lists but does not parse (.md .rst .txt .adoc) where it is written as code, and --text rewrites it. - with eleven tools the 9B model never began a fix with check <test> (0 of 18), the 4B model 5 times; with five tools 16 of 18. The check description (MCP, eval) now begins with the failing test. - look: 'a.go:497-506, a_test.go:123-128' is two places; 'TestX/subtest' is TestX. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… follow them Qwen 3.5 4B, 9B, 27B on six projects, the same 72 tasks plain / gitnext / lite: 144 -> 156 -> 163 of 216 solved; a median of 9-10 steps -> 4-5 -> 3-4; the 27B model fixes 14 of 18 seeded bugs against 7. For one model the difference in solved tasks is inside the noise. The 24-task comparison of the earlier runs (4 against 15) did not repeat - plain solves 19 of those 24 with today's harness - and is no longer quoted. Every row: eval/results/agent-kaggle-20261113.json; tables: eval/agent_stats.py. why on Rust with the map: 17 of 20 after macros became symbols (chart: 86% of 57). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
look takes a place in a file; learn tests for JS/TS; Rust tests inside macros; dead by deletion; rename --text reaches the docs; the agent runs on three models. See CHANGELOG.md