Loophole

An experimental capability-oriented runtime for structured math and logic problems.

Status: Halted. This project is not actively maintained. The MVP is frozen and no further development is planned.

Benchmark

The repository contains one recorded nine-problem comparison artifact across five datasets (GSM8K, SVAMP, MATH, AQUA-RAT, NuminaMath):

Pipeline	Correct	Total	Accuracy
Direct	3	9	33.3%
PAL	5	9	55.6%
Loophole	4	9	44.4%

These numbers are descriptive, not a general performance claim. The recorded report has inconsistent provider metadata, the set is very small, verification does not compare every answer to ground truth, and repair was not invoked in that run. The result does not show that Loophole outperforms PAL.

Installation

Requirements: Python 3.12+, uv, and API credentials only for live LLM calls.

uv sync
cp .env.example .env

Usage

Run local checks:

bun run test
bun run lint

Run the benchmark harness only when a live evaluation is explicitly authorized:

uv run python -m benchmarks.harness.main

Further Reading

• docs/ — architecture, pipeline behavior, and interface definitions.
• experiments/ — retrieval feasibility study and PAL research.

Contributing

This is a frozen MVP. Do not add features just because they are plausible. See AGENTS.md for engineering rules, documentation rules, and the list of things that must not be reintroduced.