Tools for research that measures itself. Every change is tested against a fixed baseline and kept only if it wins by more than the noise. Plain Python, no dependencies, MIT licensed.
skills/ holds instructions written for coding agents (Claude Code, Codex and others): copy a folder into your
agent's skills directory and it knows the procedure. frameworks/ holds the code those skills run.
tests/ checks that the two copies of arloop.py match and that every framework passes its own tests.
|
Run an experiment loop. Write down what counts as a win before the run, test against a sealed
baseline, keep or revert by a fixed rule, and log every result, the losses included.
Ships its own copy of arloop.py.
SKILL.md |
 |
 |
Make a slow test suite or eval give the same answer in seconds, then turn "is the new version
better?" into ACCEPT, REJECT or UNSURE.
SKILL.md |
The keep-or-revert engine, in one file. It seals the test rows, records each hypothesis before
it runs, compares the new score with the baseline and logs KEPT or DISCARDED.
python arloop.py selftest prints PASS.
Folder |
 |
 |
fastgate.py speeds up a unittest or pytest suite with a hash cache and parallel workers.
fastcheck_template.py does the same for any other scorer.
On one suite, a full run went from 39.1 s to 1.65 s with 0 verdicts changed. Folder |
|
Four checks before a run's record counts: every case found actually ran, a shared resource has a
live named holder, a record of what was read outlives the data, and a judge's reason holds up.
Of 25 agent and eval frameworks read, none checks all four. Folder |
 |
 |
The Pivot Inspector walkthrough. It scores an agent's terminal step by what the command did,
not by what it typed.
On 546 labelled actions it pays 94 working steps where the keystroke reward pays 26. Open the walkthrough · Full project |
python frameworks/arloop/arloop.py selftest
python -m unittest discover -s testsML Harness and Sequence are products with their own repos: ml-harness-app and sequence-arch-tool.