From 22e297887b0b0628b083b4c248222f5da38a2cd0 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Tue, 8 Sep 2026 13:50:27 +0000 Subject: [PATCH] docs: lock 25-tick as default short horizon Record the 2026-09-08 horizon sweep: t25 is the chosen short horizon for composite/harness compare; 100 ticks degrades on average; research stays floored; 50/100 force-pause re-prompt storms. Co-authored-by: Jason --- README.md | 2 +- docs/bench-design-n-ticks.md | 35 +++++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 97b7b49..612d446 100644 --- a/README.md +++ b/README.md @@ -260,7 +260,7 @@ python scripts/run_benchmark.py \ ## Benchmark Results -**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). N×ticks sequence (QA → σ → horizon → size): [docs/bench-design-n-ticks.md](docs/bench-design-n-ticks.md). +**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). N×ticks sequence (QA → σ → **25-tick** short horizon → size): [docs/bench-design-n-ticks.md](docs/bench-design-n-ticks.md). `felix` here is a technical harness name (the original 7-agent stack), not a product. diff --git a/docs/bench-design-n-ticks.md b/docs/bench-design-n-ticks.md index 756bb96..24d84de 100644 --- a/docs/bench-design-n-ticks.md +++ b/docs/bench-design-n-ticks.md @@ -10,8 +10,8 @@ matrix, or spend XAI. |-------|----------|---------------| | **Harness QA** | Do writes, schema, and tooling work? | N=1, 10 ticks, cheap Flash cell | | **Variance pilot** | What is within-cell σ? | N=8–10 seeds, 10 ticks, 1–2 keeper cells | -| **Horizon sweep** | Which tick count stops censoring the metrics we care about? | one cheap cell, N=3 × {10, 25, 50, 100} ticks | -| **Publishable compare** | Can we detect a chosen δ? | size *after* σ + horizon; paired same-seed sets | +| **Horizon sweep** | Which tick count stops censoring the metrics we care about? | one cheap cell, N=3 × {10, 25, 50, 100} ticks; **25 locked** (2026-09-08) | +| **Publishable compare** | Can we detect a chosen δ? | size *after* σ + horizon; paired same-seed sets; **25-tick** default short horizon | A gate score is not σ. A 10-tick leaderboard is not a horizon. N=4 without σ is not a compare. @@ -32,6 +32,9 @@ variance first. after the tooling is trustworthy. Do not buy 50- or 100-tick cells to “see more game” while gates still miss write paths. +**25-tick is the default short horizon** for composite / harness compare +(2026-09-08 sweep). 100 ticks is not preferred. + ## Locked sequence 1. **Finish gate N=1s.** 10 ticks is OK. Prove schema and write paths @@ -42,6 +45,7 @@ after the tooling is trustworthy. Do not buy 50- or 100-tick cells to 3. **Horizon sweep.** One cheap cell, N=3 × {10, 25, 50, 100} ticks. Pick the **shortest** horizon where the metrics we care about are not mostly censored (neutral 0.5 / no event / no research tree, etc.). + **Done 2026-09-08: 25 ticks is the chosen short horizon.** 4. **Only then size the matrix.** Per cell, `n ≈ 16 σ²/δ²` (80% power, α=0.05). Prefer **paired same-seed sets** across harnesses. Do not copy N=4 folklore. @@ -52,6 +56,33 @@ Flash cells run about **$0.02–$0.90**. Do not incinerate budget on a full harness × model × scenario matrix before σ and horizon are known. Size the expensive cells after the cheap ones have spoken. +## Horizon lock (2026-09-08) + +Sweep: one cheap cell, N=3 × {10, 25, 50, 100} ticks. Mean composites: + +| Horizon | Mean composite | +|---------|----------------| +| t25 | **0.816** | +| t50 | 0.806 | +| t100 | 0.744 | +| t10 (σ baseline, same three seeds) | ~0.796 | + +**Lock 25-tick** as the default short horizon for composite / harness +compare. It is the shortest sweep point that holds (and slightly beats) +the t10 baseline composite. + +**100 ticks degrades** composite on average (0.744 vs 0.816 at t25). It +is not the preferred compare horizon. + +**Research stays floored at 0.226** at t10 / t25 / t50 / t100. Horizon +alone does not unlock research. That is separate tooling / scenario +work, not a reason to buy longer cells. + +**Force-pause re-prompt storm** at 50 / 100: +`Dialog_NamePlayerFactionAndSettlement` dismiss counts rose sharply +(t25: 0; t50: 27–40; t100: 41–80 per run). Auto-dismiss works, but +longer horizons thrash the pause queue. + ## Gate fold (2026-09-07) Both cells: `opencode` × `deepseek-v4-flash`, ~$0.025 console. Pin