Skip to content

Lock 25-tick as default short horizon #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jkbennitt merged 1 commit into from
Sep 8, 2026
+34 −3
Merged

Lock 25-tick as default short horizon #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ python scripts/run_benchmark.py \

## Benchmark Results

**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). N×ticks sequence (QA → σ → horizon → size): [docs/bench-design-n-ticks.md](docs/bench-design-n-ticks.md).
**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). N×ticks sequence (QA → σ → **25-tick** short horizon → size): [docs/bench-design-n-ticks.md](docs/bench-design-n-ticks.md).

`felix` here is a technical harness name (the original 7-agent stack), not a product.

Expand Down
35 changes: 33 additions & 2 deletions docs/bench-design-n-ticks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ matrix, or spend XAI.
|-------|----------|---------------|
| **Harness QA** | Do writes, schema, and tooling work? | N=1, 10 ticks, cheap Flash cell |
| **Variance pilot** | What is within-cell σ? | N=8–10 seeds, 10 ticks, 1–2 keeper cells |
| **Horizon sweep** | Which tick count stops censoring the metrics we care about? | one cheap cell, N=3 × {10, 25, 50, 100} ticks |
| **Publishable compare** | Can we detect a chosen δ? | size *after* σ + horizon; paired same-seed sets |
| **Horizon sweep** | Which tick count stops censoring the metrics we care about? | one cheap cell, N=3 × {10, 25, 50, 100} ticks; **25 locked** (2026-09-08) |
| **Publishable compare** | Can we detect a chosen δ? | size *after* σ + horizon; paired same-seed sets; **25-tick** default short horizon |

A gate score is not σ. A 10-tick leaderboard is not a horizon. N=4 without
σ is not a compare.
Expand All @@ -32,6 +32,9 @@ variance first.
after the tooling is trustworthy. Do not buy 50- or 100-tick cells to
“see more game” while gates still miss write paths.

**25-tick is the default short horizon** for composite / harness compare
(2026-09-08 sweep). 100 ticks is not preferred.

## Locked sequence

1. **Finish gate N=1s.** 10 ticks is OK. Prove schema and write paths
Expand All @@ -42,6 +45,7 @@ after the tooling is trustworthy. Do not buy 50- or 100-tick cells to
3. **Horizon sweep.** One cheap cell, N=3 × {10, 25, 50, 100} ticks.
Pick the **shortest** horizon where the metrics we care about are not
mostly censored (neutral 0.5 / no event / no research tree, etc.).
**Done 2026-09-08: 25 ticks is the chosen short horizon.**
4. **Only then size the matrix.** Per cell,
`n ≈ 16 σ²/δ²` (80% power, α=0.05). Prefer **paired same-seed sets**
across harnesses. Do not copy N=4 folklore.
Expand All @@ -52,6 +56,33 @@ Flash cells run about **$0.02–$0.90**. Do not incinerate budget on a full
harness × model × scenario matrix before σ and horizon are known. Size
the expensive cells after the cheap ones have spoken.

## Horizon lock (2026-09-08)

Sweep: one cheap cell, N=3 × {10, 25, 50, 100} ticks. Mean composites:

| Horizon | Mean composite |
|---------|----------------|
| t25 | **0.816** |
| t50 | 0.806 |
| t100 | 0.744 |
| t10 (σ baseline, same three seeds) | ~0.796 |

**Lock 25-tick** as the default short horizon for composite / harness
compare. It is the shortest sweep point that holds (and slightly beats)
the t10 baseline composite.

**100 ticks degrades** composite on average (0.744 vs 0.816 at t25). It
is not the preferred compare horizon.

**Research stays floored at 0.226** at t10 / t25 / t50 / t100. Horizon
alone does not unlock research. That is separate tooling / scenario
work, not a reason to buy longer cells.

**Force-pause re-prompt storm** at 50 / 100:
`Dialog_NamePlayerFactionAndSettlement` dismiss counts rose sharply
(t25: 0; t50: 27–40; t100: 41–80 per run). Auto-dismiss works, but
longer horizons thrash the pause queue.

## Gate fold (2026-09-07)

Both cells: `opencode` × `deepseek-v4-flash`, ~$0.025 console. Pin
Expand Down
Loading