Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
e841fc4
README: similar projects
claude Nov 11, 2026
4e7eab8
TUI: hanging-indent wrapping, full-screen changes/diff/pages, + stage…
claude Nov 11, 2026
0cac0da
packaging: Homebrew formula + update script, npm package that fetches…
claude Nov 11, 2026
9236b3d
eval: ground truth from compilers and test suites of 5 pinned repos; …
claude Nov 11, 2026
0cbd086
eval-driven fixes: anonymous classes, field types, Rust receiver type…
claude Nov 11, 2026
a0dd8d9
docs: TUI screenshots for the new layout
pnnnx Nov 11, 2026
dd3f5bc
eval: click setup picks the newest python3 and upgrades pip (macOS sh…
claude Nov 11, 2026
c990d93
fix CI: two openers of a fresh index no longer fail with "database is…
claude Nov 11, 2026
b8e1c19
tests-for: the tests most likely to fail first, top 10 + one command …
claude Nov 11, 2026
9a4d24d
receiver types from declared result types (Go, Java, TS/JS)
claude Nov 11, 2026
c8768d9
eval: fix the ground truth (Java, Go), measure shared names and the t…
claude Nov 11, 2026
a9bc718
rust: receiver types from struct fields, Self in impls of other modul…
claude Nov 11, 2026
e3d0a17
eval: learn-tests prototype on click — a per-test coverage map agains…
claude Nov 11, 2026
a77b046
agent: loop guard, command mode, hand-off to a stronger model
claude Nov 12, 2026
901a147
trust labels, callers for several names, look asks about namesakes, t…
claude Nov 12, 2026
6d9a0d2
gitnext check: build and run the tests of what changed, answer with a…
claude Nov 12, 2026
a650c57
search --text: every line that contains the query, with the function …
claude Nov 12, 2026
0fdcc06
gitnext learn-tests: which tests really run a function, from one cove…
claude Nov 12, 2026
c29bce2
agent: run one task card on any repository (--task card.json)
claude Nov 12, 2026
32ad437
eval: a second repo per language, 20 functions on three seeds, a thre…
claude Nov 12, 2026
ecb3a0e
eval: fix the ground truth of Go and Python tests
claude Nov 12, 2026
c4fb6ab
eval: Python and Rust ground truth, functions picked blind, callers a…
claude Nov 12, 2026
f99546a
eval: run 6 - ten repos, three seeds, thresholds; baseline
claude Nov 12, 2026
3d6538e
eval: Go callers in directories the build skips are neither confirmed…
claude Nov 12, 2026
a3d1d3e
trust: the numbers of run 6 (ten repos, 232 functions for callers, 16…
claude Nov 12, 2026
349fab2
ops: log every gitnext edit as an operation and undo any one of them
claude Nov 12, 2026
aaa12b5
ops: undo after code was added above or below; never write outside th…
claude Nov 12, 2026
3651a31
eval: learn-tests prototypes for Go and Rust
claude Nov 12, 2026
86ad1af
rename: 83% -> 91% of renames keep the project building and its tests…
claude Nov 12, 2026
6669afe
agent: mode verify (a run ends only when the work is checked), other …
claude Nov 12, 2026
25e2b09
eval: --compare A B and --sample-from — two versions on the same func…
claude Nov 12, 2026
3dc7f19
graph: f().m() on a library result is only a guess
claude Nov 12, 2026
152afd7
rename: a method of a library result is left alone; trust numbers aft…
claude Nov 12, 2026
a1e1877
agent: the gate fails only on a failed check; run sees the project's …
claude Nov 12, 2026
e030c25
why <test> and digest: a failing test in one call, a long build or te…
claude Nov 12, 2026
bda4a17
a short list for a small model: check <test> is why, lite gets rename…
claude Nov 12, 2026
a9298bc
digest: a long pytest section name, and the cases of one parametrized…
Nov 12, 2026
89f19be
why: the code the failing test really ran, what changed since first; …
Nov 12, 2026
a8efe9a
agent: code written as the answer is put in place; no answer before a…
Nov 12, 2026
27c0443
learn-tests: a language's run can add notes to the report
Nov 12, 2026
5b22c10
why: the lines changed in the culprit since the map was learned
Nov 12, 2026
f08ebe3
agent: the same code written again is not checked again; a made-up fi…
Nov 12, 2026
fdd7fed
lite: `check` says first what to do when a test fails
Nov 12, 2026
5ccf6fd
learn-tests: the line → function table can count the signature line
Nov 12, 2026
8da7e8a
smoke check of every command and MCP tool; outline of a folder with n…
Nov 12, 2026
fc93e8c
learn-tests: Rust — coverage of each test, run alone, from an instrum…
Nov 12, 2026
9250f2b
learn-tests, Rust: the signature line counts; two functions with one …
Nov 12, 2026
f43c716
eval: learn-tests prototype for TypeScript, and why tests-for finds 3%
claude Nov 12, 2026
996c5b9
tests-for: in JavaScript and TypeScript the test is the file
Nov 12, 2026
8149230
learn-tests: Go — one coverage binary per package, every test run alone
Nov 12, 2026
1ce631b
run 8: TypeScript tests-for 3% -> 19%, coverage maps for Go and Rust …
Nov 12, 2026
42af276
agent: learn-tests before the fix tasks of Go and Rust projects too
Nov 12, 2026
f9e64c3
why with the map, Go: measured again - first in 82% of 17 seeded bugs…
Nov 12, 2026
ef1c2d0
README: results first - the agent runs, why with the coverage map, ac…
Nov 12, 2026
9abf3bb
0.3.0
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/workflows/eval.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: eval

# gitnext's answers checked against real projects' compilers and test suites
# (eval/run.py): callers vs compiler errors, rename vs build + tests, tests-for vs
# a failing mutant. Weekly and on demand; fails when a rate drops below eval/baseline.json.
on:
workflow_dispatch:
schedule:
- cron: "17 3 * * 1"

jobs:
eval:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- uses: actions/setup-go@v5
with: { go-version: "stable" }
- uses: actions/setup-node@v4
with: { node-version: "22" }
- uses: actions/setup-java@v4
with: { distribution: temurin, java-version: "21", cache: maven }
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- name: build gitnext
run: cargo build --release -p gitnext-cli
# What the compilers and test suites said (eval/.work/truth.json) does not change while the
# repos stay on their commits: keep it between runs, so a run only asks gitnext again.
# A new key every run (a key cannot be overwritten), restored by its prefix.
- name: truth cache key
id: truth
run: echo "key=eval-truth-v$(python3 -c "import sys; sys.path.insert(0, 'eval'); import run; print(run.CACHE_VERSION)")-${{ hashFiles('eval/repos.json') }}" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@v4
with:
path: eval/.work/truth.json
key: ${{ steps.truth.outputs.key }}-${{ github.run_id }}
restore-keys: ${{ steps.truth.outputs.key }}-
# A small sample here (its own baseline in eval/baseline.json): with an empty cache it asks the compilers
# and test suites for about 55 minutes, one seed of 20 functions for about 120. The three seeds and the
# thresholds are run locally. The step stops before the job does, so what was asked so far is still saved.
- name: eval
timeout-minutes: 100
run: GITNEXT=$PWD/target/release/gitnext python3 eval/run.py --seed 7 --n 8 --mutants 4 --check
- uses: actions/cache/save@v4
if: always()
with:
path: eval/.work/truth.json
key: ${{ steps.truth.outputs.key }}-${{ github.run_id }}
- name: results
if: always()
run: cat eval/results/latest.md >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@v4
if: always()
with: { name: eval-results, path: eval/results/latest.* }
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
target/
integrations/loca/runs/
eval/.work/
eval/results/agent-*
__pycache__/
4 changes: 4 additions & 0 deletions .gitnext/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@ journal.jsonl
hook-seen.tsv
tui-welcomed
.fuse_hidden*
tests.db
tests.db-*
coverage.db*
ops.jsonl
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Changelog

## 0.3.0 — 2026-11-12

Measured on ten open-source projects in five languages; the numbers and how they are obtained are in
[eval/results/HISTORY.md](eval/results/HISTORY.md).

### New

- **`gitnext why <test>`** (MCP: `why`, or `check` with `test`) — a test fails: run it and get the failure in a
few lines, the code the test ran with the most likely culprit first, and the lines changed since the tests
last passed. On 57 seeded bugs the function at fault is first in 81% with a learned coverage map
(18% from the call graph alone).
- **`gitnext learn-tests`** — one test run with coverage remembers which test runs which function:
Python (pytest-cov), Go (`go test -cover`, every test alone), Rust (`llvm-tools-preview`, every test alone).
`tests-for`, `look`, `check` and `why` then name the tests that really run the code: 100% of the failing
tests found (the call graph: 19% on click, Go 78%, Rust 59%).
- **`gitnext digest`** — a long build or test log in a few lines: what failed, where, the message
(rustc, cargo test, pytest, go, tsc, vitest, jest, maven; a vitest run with 35 failures: 843 lines → 12).
- **`gitnext check <test>`** answers as `why`; `gitnext check` after a change prints a verdict.
- **`gitnext ops`, `gitnext undo --op N`** — every edit made through gitnext (CLI and MCP) is in a journal
and can be taken back one by one.
- **Short tool list for small models** — `gitnext mcp --profile lite`: `look`, `replace`, `add_after`,
`rename`, `check`. A 4B local model solved 17 of 18 tasks with the five tools and 14 with eleven.

### Better

- **rename**: the project still builds and its tests pass in 91% of 338 renames (83% before): `#private`
names, names in Rust `use` groups, methods of a library's result are left alone.
- **tests-for, TypeScript / JavaScript**: the test is the file (`it('…', () => …)` is not a named function) —
failing test files found 3% → 19%.
- **Python calls**: calls inside nested functions, `module.func` through `__init__` re-exports, functions
passed as values and decorators.
- **callers** in Go directories the build skips (`_examples`, `testdata`) are no longer counted as callers.
- `gitnext outline <folder>` says what is in a folder with nothing public instead of printing nothing.
- `why` and `check` accept the id a test runner prints (`tests/test_x.py::TestA::test_b[case]`, `TestRun/sub`).

### Measured

- `eval/run.py`: 10 projects, 3 samples, a noise interval next to every number; `--compare A B` — two versions
on the same functions.
- `eval/agent.py`: a 4B local model (Qwen 3.5 4B) on real tasks — 4 of 24 solved with plain tools, 17 with gitnext.
- `eval/why_rank.py`: where the function at fault is in the answer of `why`, on seeded bugs.
- `eval/smoke.py`: every command and every MCP tool on every project — 860 calls, none fails.
- What checks each function: [eval/results/COVERAGE.md](eval/results/COVERAGE.md).

### Known gaps

- TypeScript: no coverage map yet; `tests-for` finds 19% of the failing test files.
- A rename breaks the build or the tests in 9% of cases (dynamic access, names shared by unrelated types).
- Rust tests written inside a macro are not seen.
- A 4B model rarely fixes a seeded bug yet (3 of 12 on click).

## 0.2.0, 0.1.0

See the release pages on GitHub.
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ This repo is indexed by gitnext (MCP server `gitnext`, or the `gitnext` command)
3. Edit by symbol: `replace_symbol` / `insert_after` / `rename_symbol` (rename also fixes imports, `__all__`, tests and `module.func` uses, and lists any text it did not change) — no need to read the file first.
4. New repo: `overview` once. Resuming: `brief`, then `changes`.
5. Worked out what undocumented code does? `annotate` it in one line; later sessions get it for free.
6. A test fails: `why <test>` runs it and returns the failure in a few lines and the code to look at, the most likely culprit first. After a change: `gitnext check` builds and runs the tests of what changed. A long build or test log: `… 2>&1 | gitnext digest` gives what failed, where and the message.

Plain reads are fine for non-code files and small files. Run the tests yourself — gitnext does not run them.
Plain reads are fine for non-code files and small files.
<!-- gitnext:agent:end -->
15 changes: 8 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
[workspace]
resolver = "2"
members = ["crates/*"]
# eval clones Rust projects under eval/.work: they are not part of this workspace
exclude = ["eval/.work"]

[workspace.package]
version = "0.2.0"
version = "0.3.0"
edition = "2021"
license = "MIT"

Expand Down
65 changes: 61 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,47 @@ gitnext is a drop-in companion to git (`gitnext` = `git` + a code index). It par

The repository stays plain git — GitHub, CI and your other tools keep working.

## Measured on a real repository
## Results

Everything here is measured on other people's code — ten open-source projects in five languages — and judged by their own compilers and test suites. How each number is obtained and how much of it is noise: [eval/results/HISTORY.md](eval/results/HISTORY.md).

### A small local model solves more

<p align="center"><img src="docs/images/agent.svg" width="900" alt="Tasks solved by a 4B local model: with plain tools 4 of 24, with gitnext tools 17 of 24"></p>

The same 4B model, the same tasks, the same judge. With plain tools it guesses file paths and answers from memory; with gitnext one call gives it the code, the callers and the tests. For small models there is a short list of five tools (`gitnext mcp --profile lite`): in a separate run of 18 tasks the model solved 17 with the five and 14 with eleven, and spent 15% fewer tokens.

### A failing test leads to the function at fault

<p align="center"><img src="docs/images/why.svg" width="900" alt="Share of seeded bugs where gitnext why names the function at fault first: 18% from the call graph, 81% with the learned coverage map"></p>

`gitnext why <test>` runs the test and answers in a dozen lines: the failure itself, the code the test ran with the most likely culprit first, and the lines that changed since the tests last passed. `gitnext learn-tests` is one test run with coverage (Python, Go, Rust); a long build or test log goes through `gitnext digest` (a vitest run with 35 failures: 843 lines → 12).

### How accurate the code graph is

| | Go | Python | TypeScript | Java | Rust | all |
|---|---:|---:|---:|---:|---:|---:|
| who calls a function — found | 100% | — | 100% | 100% | 99% | **100%** ±1 |
| who calls a function — correct | 98% | — | 100% | 100% | 100% | **99%** ±1 |
| rename — the project still builds and its tests pass | 93% | 90% | 85% | 96% | 87% | **91%** ±3 |
| tests for a function — one of the first 10 named does fail | 84% | 81% | 41% | 95% | 73% | **79%** ±6 |
| … with a learned coverage map — failing tests found | 100% | 100% | — | — | 100% | |

232 functions for callers, 338 renames, 167 functions for tests; ± is the 90% interval, a difference inside it is noise. Python has no compiler to say who calls what, so callers are not scored there. `python3 eval/run.py` repeats the run; the full table is [eval/results/latest.md](eval/results/latest.md).

### Fewer steps, less to read

<p align="center"><img src="docs/images/savings.svg" width="900" alt="Bytes of code an agent read for six tasks: 49 KB without gitnext, 29 KB recounted, 22 KB with gitnext look"></p>

Most of an agent session is the system prompt re-sent on every step, so the win comes from **fewer steps**, not just shorter output: `gitnext look A B C` answers *what is it · who calls it · what it calls · which tests reach it and how to run them · what a change affects* for several targets at once. Small local models (4B) did not use the tools at all — for them the compact code map in `AGENTS.md` is what helps. Details and how to reproduce: [docs/GUIDE.ru.md](docs/GUIDE.ru.md), [integrations/ab](integrations/ab).
Most of an agent session is the system prompt re-sent on every step, so the win comes from **fewer steps**, not just shorter output: `gitnext look A B C` answers *what is it · who calls it · what it calls · which tests reach it and how to run them · what a change affects* for several targets at once. How to reproduce: [integrations/ab](integrations/ab).

### What is checked, and what is not good yet

- 115 unit tests; parser output is pinned by fixtures in eight languages; every command and every MCP tool — 86 in all — is run on each of the ten projects (860 calls, none fails). What checks each function: [eval/results/COVERAGE.md](eval/results/COVERAGE.md).
- TypeScript: `tests-for` finds 19% of the failing test files (it was 3%); the coverage map exists for Python, Go and Rust only.
- A rename breaks the build or the tests in 9% of cases — mostly dynamic access and names shared by unrelated types.
- A 4B model rarely fixes a seeded bug yet: 3 of 12 on click.
- Rust tests written inside a macro are not seen (22% of the tests of indexmap).

## Install

Expand Down Expand Up @@ -53,13 +89,19 @@ gitnext # open the TUI
| skeleton of a file or folder | `gitnext outline <path>` / `outline` |
| edit one function without reading the file | `gitnext edit replace <Sym> -f new.py` / `replace_symbol` |
| rename everywhere (imports, tests, `__all__`, `mod.func`) | `gitnext rename <old> <new>` / `rename_symbol` |
| what breaks, which tests to run | `gitnext impact <Sym>` · `gitnext tests-for <Sym>` |
| what breaks, which tests to run | `gitnext impact <Sym>` · `gitnext tests-for <Sym>` (the 10 most likely to fail + one command for the rest) |
| which tests really run a function | `gitnext learn-tests` — the tests run once with coverage (Python: pytest-cov; Go; Rust: `llvm-tools-preview`); `tests-for`, `look`, `check` and `why` then use the real list |
| did my change break anything | `gitnext check` — builds and runs the tests of what changed, prints a verdict (`gitnext rename A B --check` does it right after the rename) |
| a test fails — why | `gitnext why <test>` / `check` with `test` — runs it: the failure in a few lines, the code the test ran with the most likely culprit first, the lines changed since the tests passed |
| a long build or test log | `cargo test 2>&1 \| gitnext digest` — what failed, where, the message (a vitest run with 35 failures: 843 lines → 12) |
| error or traceback → code | `gitnext locate` (stdin) / `locate` |
| where were we | `gitnext brief` · `gitnext journal` |
| which libraries the code calls, with versions | `gitnext libs` (and `look` shows them per function) |

`gitnext setup` writes the MCP config and a short instruction block for each agent; `--hooks` makes Claude Code see an outline before reading a big file whole.

**Small local models.** `gitnext mcp --profile lite` shows five tools instead of 29 — `look`, `replace`, `add_after`, `rename`, `check` — with short descriptions and short answers: a 4B model chooses better from a short list. For a model that calls no tools at all, `gitnext context "<task>"` prints the code the task is about, to put into the prompt.

## Git in plain words

<p align="center"><img src="docs/images/tui-changes.svg" width="900" alt="Changes screen: one function staged, a commit message suggested"></p>
Expand Down Expand Up @@ -92,10 +134,25 @@ In the TUI: `s` changes, `space` stages a file or one function, `enter` shows be
| `crates/gitnext-cli` | the `gitnext` command and the MCP server |
| `crates/gitnext-tui` | the terminal UI |
| `integrations/` | A/B test kit for real agents, loca (llama.cpp) benchmark |
| `eval/` | the measurements: `run.py` (accuracy on ten projects), `agent.py` (a local model on real tasks), `why_rank.py` (seeded bugs), `smoke.py` (every command on every project) |

## Status

Early but used daily on its own code (this repository is indexed by gitnext — see `AGENTS.md`). Name-based resolution is approximate for dynamic code; `gitnext lsp` refines it with rust-analyzer / pyright / clangd. Parser output is pinned by fixtures in `crates/gitnext-parse/tests/fixtures`.
Early, and used daily on its own code (this repository is indexed by gitnext — see `AGENTS.md`). Languages: Rust, Python, Go, Java, TypeScript, JavaScript, C, C++. Name-based resolution is approximate for dynamic code; `gitnext lsp` refines it with rust-analyzer / pyright / clangd. Every change to what the tools answer is measured before it is merged: [eval/results/HISTORY.md](eval/results/HISTORY.md).

## Similar projects

Each piece exists elsewhere; gitnext puts them in one tool and measures the effect.

| Project | What it does | How gitnext differs |
|---|---|---|
| [Aider repo map](https://aider.chat/docs/repomap.html) | tree-sitter map of the repo in the prompt | the map is committed to the repo (`AGENTS.md`, `.gitnext/map`), so any agent — even a small local model without tools — gets it |
| [Serena](https://oraios.github.io/serena/01-about/035_tools.html) | MCP tools to find, edit and rename by symbol, via language servers | one call `look` returns code + callers + callees + tests + impact; works without a language server (LSP is optional refinement) |
| [CodeGraph](https://colbymchenry.github.io/codegraph/), [CodeGraphContext](https://mcpservers.org/servers/shashankss1205/codegraphcontext) | code graph over MCP: callers, callees, impact | also injects the code a task needs before the request (`gitnext context`) for agents that do not call tools |
| [sem](https://docs.rs/sem-cli/0.7.0/sem/) | entity-level diff, blame and impact on top of git | adds plain-language git: stage or commit one function, undo, timeline, commit message from changed functions |
| [lazygit](https://github.com/jesseduffield/lazygit), [GitButler](https://gitbutler.com) | friendly git UIs | the TUI shows code by function and relation, not only files and hunks |

What we can show that others do not: accuracy against compilers and tests on ten projects, and the effect on a real agent — including a 4B local model ([eval](eval), [integrations/ab](integrations/ab), [integrations/loca](integrations/loca)).

## License

Expand Down
Loading
Loading