Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
matrix:
os: [ubuntu-latest, macos-latest]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/eval.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,16 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: actions/setup-go@v5
- uses: actions/setup-go@v7
with: { go-version: "stable" }
- uses: actions/setup-node@v4
- uses: actions/setup-node@v7
with: { node-version: "22" }
- uses: actions/setup-java@v4
- uses: actions/setup-java@v6
with: { distribution: temurin, java-version: "21", cache: maven }
- uses: actions/setup-python@v5
- uses: actions/setup-python@v7
with: { python-version: "3.12" }
- name: build gitnext
run: cargo build --release -p gitnext-cli
Expand All @@ -32,7 +32,7 @@ jobs:
- name: truth cache key
id: truth
run: echo "key=eval-truth-v$(python3 -c "import sys; sys.path.insert(0, 'eval'); import run; print(run.CACHE_VERSION)")-${{ hashFiles('eval/repos.json') }}" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@v4
- uses: actions/cache/restore@v6
with:
path: eval/.work/truth.json
key: ${{ steps.truth.outputs.key }}-${{ github.run_id }}
Expand All @@ -43,14 +43,14 @@ jobs:
- name: eval
timeout-minutes: 100
run: GITNEXT=$PWD/target/release/gitnext python3 eval/run.py --seed 7 --n 8 --mutants 4 --check
- uses: actions/cache/save@v4
- uses: actions/cache/save@v6
if: always()
with:
path: eval/.work/truth.json
key: ${{ steps.truth.outputs.key }}-${{ github.run_id }}
- name: results
if: always()
run: cat eval/results/latest.md >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v7
if: always()
with: { name: eval-results, path: eval/results/latest.* }
9 changes: 6 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,18 @@ jobs:
matrix:
include:
- { os: macos-latest, target: aarch64-apple-darwin }
- { os: macos-13, target: x86_64-apple-darwin }
# Intel Macs: built on the arm runner (the macos-13 runners are retired; a job on them never starts)
- { os: macos-latest, target: x86_64-apple-darwin }
- { os: ubuntu-latest, target: x86_64-unknown-linux-gnu }
- { os: ubuntu-24.04-arm, target: aarch64-unknown-linux-gnu }
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v7
- uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- uses: Swatinem/rust-cache@v2
with:
key: ${{ matrix.target }}
- name: build
run: cargo build --release --locked --target ${{ matrix.target }} -p gitnext-cli
- name: package
Expand All @@ -35,7 +38,7 @@ jobs:
cd target/${{ matrix.target }}/release
tar -czf gitnext-${{ matrix.target }}.tar.gz gitnext
(command -v sha256sum >/dev/null && sha256sum gitnext-${{ matrix.target }}.tar.gz || shasum -a 256 gitnext-${{ matrix.target }}.tar.gz) > gitnext-${{ matrix.target }}.tar.gz.sha256
- uses: softprops/action-gh-release@v2
- uses: softprops/action-gh-release@v3
with:
files: |
target/${{ matrix.target }}/release/gitnext-${{ matrix.target }}.tar.gz
Expand Down
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,44 @@
# Changelog

## 0.3.1 — 2026-11-13

- **`gitnext look` takes a place in a file**, written the way tracebacks, test runners and models write it:
`src/eval.rs:62` (the function around the line), `tests/test_x.py:120-160` (those lines),
`src/eval.rs::matches_greater`, `tests/test_x.py::TestA::test_b[case]`, `eval.rs matches_greater`.
Words of which only some are names list those names instead of "nothing found". In a trial with
Qwen 3.5 4B and 27B, 37 of 89 `look` calls were written one of these ways and found nothing; all are answered now.
- **`gitnext rename --text` reaches the documentation** (`.md`, `.rst`, `.txt`, `.adoc`): the old name written as
code there (`name()`, `` `name` ``, `Type.name`, or a name that cannot be a word) is renamed with the rest.
- `look` also takes a list of places (`a.go:497-506, a_test.go:123-128`) and a Go subtest (`TestX/case`); the MCP
`check` tool says first that a failing test is what it is for.
- eval: the agent runs repeated on three models, 216 tasks each way (`eval/results/HISTORY.md`): +9 points of solved
tasks with the five tools, a third of the steps; the earlier 24-task comparison did not repeat.
- **`gitnext dead` names only what nothing refers to.** It knew calls only: a function used as a value, a class
used as a type, a base or a metaclass, an imported name, a Go `init` were "likely dead" — 160 of 225 on ten
projects, and deleting them broke the build of six. Now the name must be written nowhere outside the symbol.
- **`gitnext edit replace`** keeps every line where it was when the code comes back as it was read (a line indented
less than the first, tabs under spaces); **`edit delete`** and `read` take the whole doc comment (a Javadoc
whose lines do not start with `*` was cut in half, and the delete refused).
- **`callees`** lists a method called on an object of unknown type as "maybe", as `callers` does; `look` counts
such calls instead of naming them.
- **`gitnext learn-tests` for JavaScript and TypeScript** (vitest with v8 coverage, every test file alone):
`tests-for`, `look` and `check` name the test files that really run the code — 100% of the failing ones on yaml
and zustand (the call graph: 16% and 68%), for 2.6–6.6 ordinary test runs. Says which package to install when
`@vitest/coverage-v8` is missing; jest is not supported yet.
- eval: a coverage map for Java measured as a prototype (`eval/learn_tests_java.py`, JaCoCo with a dump per test
in one Maven run): 94% and 99% of the failing tests on jsoup and json-java (the call graph: 29% and 71%) for
2.1–3.5 ordinary test runs. Not in `learn-tests` yet.
- **Rust: functions declared inside a macro call are symbols** (`quickcheck! { fn prop(..) -> bool { .. } }`,
`proptest!`, a local `macro_rules!` that writes `#[test]`). In indexmap 43 of 192 tests were outside the graph and
the coverage map, now 4; `why` put the function at fault first in 14 of 20 seeded Rust bugs, now in 16–18, and
no longer answers with an empty list (4 of 10 on indexmap).
- `dead` again: a name built next to reflection (`getattr(self, "_process_" + kind)`) is used; a Rust function run
by an attribute (`#[bench]`, `#[no_mangle]`, …) is not dead; `protected` is API. On the ten projects what it
calls likely dead went 225 → 64 → 1, and that one deletes cleanly.
- `eval/consistency.py`: do the commands agree with each other and with the source (`eval/results/consistency.md`).
- eval: a tool call cut by the token limit (a long `replace`) is asked for again instead of ending the run.
- Release: the Intel Mac binary is built again (its runner was retired); `CITATION.cff`.

## 0.3.0 — 2026-11-12

Measured on ten open-source projects in five languages; the numbers and how they are obtained are in
Expand Down
18 changes: 18 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
cff-version: 1.2.0
message: "If you use gitnext or its evaluation, please cite it."
title: "gitnext: code navigation and symbol-level editing tools for AI agents"
type: software
authors:
- family-names: Kononenko
given-names: Leonid A.
orcid: "https://orcid.org/0009-0009-8407-9214"
affiliation: Independent researcher
repository-code: "https://github.com/pnnnx/gitnext"
license: MIT
keywords:
- AI agents
- code navigation
- call graph
- tree-sitter
- evaluation
- small language models
14 changes: 7 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ members = ["crates/*"]
exclude = ["eval/.work"]

[workspace.package]
version = "0.3.0"
version = "0.3.1"
edition = "2021"
license = "MIT"

Expand Down
21 changes: 11 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
<p align="center">
<a href="https://github.com/pnnnx/gitnext/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/pnnnx/gitnext/actions/workflows/ci.yml/badge.svg"></a>
<a href="LICENSE"><img alt="MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
<a href="https://zenodo.org/badge/latestdoi/1453025544"><img alt="DOI" src="https://zenodo.org/badge/1453025544.svg"></a>
<img alt="Rust" src="https://img.shields.io/badge/written%20in-Rust-orange.svg">
</p>

Expand All @@ -24,15 +25,15 @@ The repository stays plain git — GitHub, CI and your other tools keep working.

Everything here is measured on other people's code — ten open-source projects in five languages — and judged by their own compilers and test suites. How each number is obtained and how much of it is noise: [eval/results/HISTORY.md](eval/results/HISTORY.md).

### A small local model solves more
### Agents get there in a third of the steps

<p align="center"><img src="docs/images/agent.svg" width="900" alt="Tasks solved by a 4B local model: with plain tools 4 of 24, with gitnext tools 17 of 24"></p>
<p align="center"><img src="docs/images/agent.svg" width="900" alt="Tasks solved by three local models: 144 of 216 with plain tools, 163 of 216 with the five gitnext tools"></p>

The same 4B model, the same tasks, the same judge. With plain tools it guesses file paths and answers from memory; with gitnext one call gives it the code, the callers and the tests. For small models there is a short list of five tools (`gitnext mcp --profile lite`): in a separate run of 18 tasks the model solved 17 with the five and 14 with eleven, and spent 15% fewer tokens.
Three local models (Qwen 3.5 4B, 9B and 27B), the same 72 tasks each way — find a function, list its callers, rename it, fix a seeded bug — on six projects in Python, Rust and Go, judged by the projects' own compilers and tests. With the five gitnext tools (`gitnext mcp --profile lite`) they solve 163 of 216 tasks against 144 with plain read / grep / edit / run (+9 points; the 95% interval is +1 to +17), in a median of 3–4 steps instead of 9–10, and the 27B model sends 2.4 times fewer tokens. The clear gain is a bug fixed by the larger model: 14 of 18 against 7, never the other way round. For one model alone the difference in solved tasks is inside the noise, and an earlier 24-task run that showed 4 against 15 for the 4B model did not repeat: with today's harness the plain model solves 19 of those 24.

### A failing test leads to the function at fault

<p align="center"><img src="docs/images/why.svg" width="900" alt="Share of seeded bugs where gitnext why names the function at fault first: 18% from the call graph, 81% with the learned coverage map"></p>
<p align="center"><img src="docs/images/why.svg" width="900" alt="Share of seeded bugs where gitnext why names the function at fault first: 18% from the call graph, 86% with the learned coverage map"></p>

`gitnext why <test>` runs the test and answers in a dozen lines: the failure itself, the code the test ran with the most likely culprit first, and the lines that changed since the tests last passed. `gitnext learn-tests` is one test run with coverage (Python, Go, Rust); a long build or test log goes through `gitnext digest` (a vitest run with 35 failures: 843 lines → 12).

Expand All @@ -44,7 +45,7 @@ The same 4B model, the same tasks, the same judge. With plain tools it guesses f
| who calls a function — correct | 98% | — | 100% | 100% | 100% | **99%** ±1 |
| rename — the project still builds and its tests pass | 93% | 90% | 85% | 96% | 87% | **91%** ±3 |
| tests for a function — one of the first 10 named does fail | 84% | 81% | 41% | 95% | 73% | **79%** ±6 |
| … with a learned coverage map — failing tests found | 100% | 100% | — | — | 100% | |
| … with a learned coverage map — failing tests found | 100% | 100% | 100% | — | 100% | |

232 functions for callers, 338 renames, 167 functions for tests; ± is the 90% interval, a difference inside it is noise. Python has no compiler to say who calls what, so callers are not scored there. `python3 eval/run.py` repeats the run; the full table is [eval/results/latest.md](eval/results/latest.md).

Expand All @@ -56,11 +57,11 @@ Most of an agent session is the system prompt re-sent on every step, so the win

### What is checked, and what is not good yet

- 115 unit tests; parser output is pinned by fixtures in eight languages; every command and every MCP tool — 86 in all — is run on each of the ten projects (860 calls, none fails). What checks each function: [eval/results/COVERAGE.md](eval/results/COVERAGE.md).
- TypeScript: `tests-for` finds 19% of the failing test files (it was 3%); the coverage map exists for Python, Go and Rust only.
- 139 unit tests; parser output is pinned by fixtures in eight languages; every command and every MCP tool — 86 in all — is run on each of the ten projects (860 calls, none fails). What checks each function: [eval/results/COVERAGE.md](eval/results/COVERAGE.md).
- Java has no coverage map in the product yet (a prototype with JaCoCo finds 94–99% of the failing tests); for JS / TS it needs vitest.
- A rename breaks the build or the tests in 9% of cases — mostly dynamic access and names shared by unrelated types.
- A 4B model rarely fixes a seeded bug yet: 3 of 12 on click.
- Rust tests written inside a macro are not seen (22% of the tests of indexmap).
- Small models rarely fix a seeded bug: 6 of 18 at best for the 4B and the 9B model. With the full tool list they do not start from `check <test>` (the 9B model never did in 18 tasks) — give them the short list.
- A rename done with the tool passed the judge no more often than one done by hand (12–13 of 18 against 14): names that are also common words, and mentions in documentation — the latter are renamed now.

## Install

Expand Down Expand Up @@ -90,7 +91,7 @@ gitnext # open the TUI
| edit one function without reading the file | `gitnext edit replace <Sym> -f new.py` / `replace_symbol` |
| rename everywhere (imports, tests, `__all__`, `mod.func`) | `gitnext rename <old> <new>` / `rename_symbol` |
| what breaks, which tests to run | `gitnext impact <Sym>` · `gitnext tests-for <Sym>` (the 10 most likely to fail + one command for the rest) |
| which tests really run a function | `gitnext learn-tests` — the tests run once with coverage (Python: pytest-cov; Go; Rust: `llvm-tools-preview`); `tests-for`, `look`, `check` and `why` then use the real list |
| which tests really run a function | `gitnext learn-tests` — the tests run once with coverage (Python: pytest-cov; Go; Rust: `llvm-tools-preview`; JS / TS: vitest with `@vitest/coverage-v8`); `tests-for`, `look`, `check` and `why` then use the real list |
| did my change break anything | `gitnext check` — builds and runs the tests of what changed, prints a verdict (`gitnext rename A B --check` does it right after the rename) |
| a test fails — why | `gitnext why <test>` / `check` with `test` — runs it: the failure in a few lines, the code the test ran with the most likely culprit first, the lines changed since the tests passed |
| a long build or test log | `cargo test 2>&1 \| gitnext digest` — what failed, where, the message (a vitest run with 35 failures: 843 lines → 12) |
Expand Down
Loading
Loading