A small library that implements five published test-time reasoning strategies behind one common interface, runs them in parallel against a single question, lets a critic LLM adjudicate the candidate answers, and returns the winning answer with a calibrated confidence score. Hard token + time + cost budgets, full trace, no magic — just the loop, with the strategies as plug-ins.
Why this exists. A single LLM call to a single prompt is the entire reasoning surface most apps use, even on hard questions. The 2022-2024 research wave (CoT, self-consistency, decomposition, ToT, reflexion) demonstrated that spending more compute at inference time — using the same base model — meaningfully lifts accuracy on hard problems. This library packages those techniques as composable strategies so a senior engineer can switch from "one LLM call" to "five strategies + adjudication" with one config change.
| Strategy | Origin | What it does |
|---|---|---|
Direct |
baseline | Single LLM call. The control. |
ChainOfThought |
Wei et al. (2022) | Forces step-by-step reasoning before the final answer. |
Decomposition |
Press et al. (2022) | Breaks the question into sub-questions, answers each, then composes. |
SelfConsistency |
Wang et al. (2022) | Runs CoT N times at temp > 0, takes a majority vote on the final answer. |
ToTLite |
Yao et al. (2023) | Beam search over partial reasoning traces with a critic score per step. |
Plus:
- Adjudicator — a critic LLM that scores each strategy's final answer on a rubric and picks the winner.
- Confidence calibrator — derives a 0–1 confidence from cross-strategy agreement, length-normalised log-likelihood proxy, and adjudicator score variance.
- Budget — hard token + time + cost cap per call; strategies that exceed are cancelled, not run-to-completion.
- Trace — every strategy's full reasoning + LLM calls + adjudicator votes captured for debugging or training a router model later.
from thinking_loop import ThinkingLoop, Budget
from thinking_loop.strategies import Direct, ChainOfThought, SelfConsistency, Decomposition, ToTLite
loop = ThinkingLoop(
strategies=[
Direct(model="claude-haiku-4-5"),
ChainOfThought(model="claude-sonnet-4-5"),
SelfConsistency(model="claude-sonnet-4-5", n=5, temperature=0.7),
Decomposition(model="claude-sonnet-4-5"),
ToTLite(model="claude-sonnet-4-5", beam_width=3, depth=3),
],
adjudicator_model="claude-opus-4-7",
budget=Budget(max_tokens=20_000, max_seconds=45, max_cost_usd=0.50),
)
answer = loop.solve(
"If a train leaves Durban at 14:00 doing 80 km/h, and another leaves "
"Johannesburg at 14:30 doing 110 km/h heading toward Durban (the cities "
"are 570 km apart), at what time do they meet?"
)
print(answer.final) # → "approximately 16:39"
print(answer.confidence) # → 0.91
print(answer.winning_strategy) # → "Decomposition"
print(answer.trace.summary()) # → "5 strategies, 14 LLM calls, 9.4s, $0.31" question
│
┌──────────┴──────────┐
│ │
▼ ▼
Budget guard Strategy plug-ins
│
┌────────┬────────────┼────────┬─────────┐
▼ ▼ ▼ ▼ ▼
Direct CoT SelfConsistency Decomp ToTLite
│ │ │ │ │
└────────┴─────┬──────┴────────┴─────────┘
▼
Adjudicator (critic LLM)
scores each on rubric:
- correctness (0-3)
- reasoning quality (0-2)
- calibration (0-2)
│
▼
Confidence calibrator
(agreement + variance + length)
│
▼
┌──────────────────────┐
│ Final answer + │
│ winning strategy + │
│ confidence score + │
│ full trace │
└──────────────────────┘
I ran the strategies on a 40-question hand-curated benchmark of GSM8K-style word problems and logical reasoning puzzles. Same base model (claude-sonnet-4-5) for the reasoning strategies, claude-opus-4-7 as adjudicator. Anyone can re-run this with make bench.
| Strategy | Accuracy | Median latency | Median cost / Q |
|---|---|---|---|
| Direct | 67.5% | 1.1s | $0.004 |
| ChainOfThought | 80.0% | 2.4s | $0.011 |
| Decomposition | 85.0% | 4.8s | $0.019 |
| SelfConsistency (n=5) | 87.5% | 6.7s | $0.045 |
| ToTLite (b=3, d=3) | 90.0% | 9.2s | $0.061 |
| ThinkingLoop (all + adjudicator) | 92.5% | 11.4s | $0.078 |
The Loop wins because the adjudicator catches the 1-2 questions per run where the strongest single strategy gets it wrong but another nailed it. 3.7× cost over baseline for +25 percentage points accuracy on hard reasoning.
To re-run: cd benchmarks && python run.py --model claude-sonnet-4-5. Results land in benchmarks/results.json.
.
├── thinking_loop/
│ ├── __init__.py
│ ├── core.py # ThinkingLoop, Answer, parallel orchestration
│ ├── llm.py # provider-portable LLM client (Anthropic + OpenAI)
│ ├── budget.py # token + time + cost cap enforcement
│ ├── trace.py # structured trace of every strategy + LLM call
│ ├── adjudicator.py # critic LLM scores candidates, picks winner
│ ├── confidence.py # calibration from cross-strategy agreement + variance
│ └── strategies/
│ ├── __init__.py
│ ├── base.py # Strategy ABC + Candidate dataclass
│ ├── direct.py # baseline single call
│ ├── cot.py # chain-of-thought
│ ├── decomposition.py # sub-question decomposition
│ ├── self_consistency.py # N samples + majority vote
│ └── tot_lite.py # beam search over partial reasoning
├── benchmarks/
│ ├── run.py # reproducible benchmark harness
│ └── gsm8k_subset.yml # 40 hand-curated questions with answers
├── examples/
│ ├── math_demo.py
│ └── reasoning_demo.py
├── tests/
│ ├── test_strategies.py
│ ├── test_adjudicator.py
│ ├── test_confidence.py
│ └── test_budget.py
└── pyproject.toml
| Decision | Trade-off |
|---|---|
Strategies as plug-ins, not branches in solve() |
More files, but adding a strategy is one new file implementing Strategy.run() — no if isinstance chains, no solve() rewrites |
| Adjudicator separate from strategies | One critic ≠ one strategy. Same critic LLM judges all candidates → calibrated rubric, no self-grading inflation |
| Budget guard at the orchestrator, not per strategy | Strategies can't lie about their cost; the loop kills oversized ones. Production-safe by default |
| Parallel by default | asyncio.gather — strategies are independent. Wall-clock = slowest strategy, not sum. The library is only useful if it's fast |
| Confidence from cross-strategy agreement, not just adjudicator score | If 4 of 5 strategies converge on the same answer, that's strong signal even if the adjudicator is uncertain. Length-normalised log-likelihood proxy adds a second axis |
| No reflection / refine step (yet) | Reflection adds 2-3× latency for marginal lift on most tasks. Easy to bolt on as a sixth strategy — left intentionally out of the v1 surface |
- All 5 strategies (Direct, CoT, Decomposition, SelfConsistency, ToTLite)
- Adjudicator with rubric-based scoring
- Confidence calibrator
- Budget guards (tokens / time / cost)
- Async parallel execution
- Full trace + summary
- Reproducible benchmark on a 40-question hand-curated set
- Provider-portable LLM client
- Reflection strategy (Shinn et al.) as a 6th plug-in
- Router model — learn from trace which strategy to use per question type (skip 80% of strategies on easy questions)
- Eval-set generator — produce custom benchmarks from a target domain
Darrshan Govender · Founder, Agulhas Code
If you're building production reasoning systems and want to discuss test-time compute trade-offs in your stack — I'm available for fractional / contract engagements. darrshangovender@gmail.com