From 291427e3106539292b240206241894e0ecc33797 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 7 Sep 2026 23:38:40 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20N=20=C3=97=20ticks=20sequence=20for=20h?= =?UTF-8?q?arness=20QA=20=E2=86=92=20stats?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lock the four-stage bench path (QA → σ pilot → horizon → sized compare) so N=1 seed-42 cells are not treated as significance. Fold 2026-09-07 gate scores; 10-tick stays software validation first. Co-authored-by: Jason --- README.md | 2 +- docs/bench-design-n-ticks.md | 73 ++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 1 deletion(-) create mode 100644 docs/bench-design-n-ticks.md diff --git a/README.md b/README.md index ad4bfbb..97b7b49 100644 --- a/README.md +++ b/README.md @@ -260,7 +260,7 @@ python scripts/run_benchmark.py \ ## Benchmark Results -**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). +**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`). N×ticks sequence (QA → σ → horizon → size): [docs/bench-design-n-ticks.md](docs/bench-design-n-ticks.md). `felix` here is a technical harness name (the original 7-agent stack), not a product. diff --git a/docs/bench-design-n-ticks.md b/docs/bench-design-n-ticks.md new file mode 100644 index 0000000..756bb96 --- /dev/null +++ b/docs/bench-design-n-ticks.md @@ -0,0 +1,73 @@ +# RLE bench design — N × ticks (QA → stats) + +How we get from a working harness to a number we can defend. **Do not +conflate the stages.** This memo does not change CLI defaults, authorize a +matrix, or spend XAI. + +## Four stages + +| Stage | Question | Typical shape | +|-------|----------|---------------| +| **Harness QA** | Do writes, schema, and tooling work? | N=1, 10 ticks, cheap Flash cell | +| **Variance pilot** | What is within-cell σ? | N=8–10 seeds, 10 ticks, 1–2 keeper cells | +| **Horizon sweep** | Which tick count stops censoring the metrics we care about? | one cheap cell, N=3 × {10, 25, 50, 100} ticks | +| **Publishable compare** | Can we detect a chosen δ? | size *after* σ + horizon; paired same-seed sets | + +A gate score is not σ. A 10-tick leaderboard is not a horizon. N=4 without +σ is not a compare. + +## Current fact (scoring 1.2) + +Almost every published scoring-1.2 cell is **seed 42, N=1** (see +`spread-2026-09-06`). One observation per cell ⇒ **no within-cell σ**. +Declaring “significance” without σ is theater. + +The live board already labels N=1 as content-first / not statistically +valid. “Winners advance to N=4” is still not a substitute for measuring +variance first. + +## Jason lock + +**10-tick is software / harness validation first.** Longer horizon only +after the tooling is trustworthy. Do not buy 50- or 100-tick cells to +“see more game” while gates still miss write paths. + +## Locked sequence + +1. **Finish gate N=1s.** 10 ticks is OK. Prove schema and write paths + (including rejects that are availability, not bugs). Fold results + below; leftover unexercised paths stay QA, not stats. +2. **Variance pilot.** N=8–10 seeds, 10 ticks, 1–2 keeper cells. Estimate + σ. Stop if σ is unusable or the cell is still on fire. +3. **Horizon sweep.** One cheap cell, N=3 × {10, 25, 50, 100} ticks. + Pick the **shortest** horizon where the metrics we care about are not + mostly censored (neutral 0.5 / no event / no research tree, etc.). +4. **Only then size the matrix.** Per cell, + `n ≈ 16 σ²/δ²` (80% power, α=0.05). Prefer **paired same-seed sets** + across harnesses. Do not copy N=4 folklore. + +## Cost + +Flash cells run about **$0.02–$0.90**. Do not incinerate budget on a full +harness × model × scenario matrix before σ and horizon are known. Size +the expensive cells after the cheap ones have spoken. + +## Gate fold (2026-09-07) + +Both cells: `opencode` × `deepseek-v4-flash`, ~$0.025 console. Pin +`f818246e` @ `41fd015`. These are **QA**, not σ. + +| | Val N=1 | Gates2 N=1 | +|--|---------|------------| +| Seed | 42 | **314159** (+ short research/tend bias) | +| Score | **0.7073** | **0.6803** | +| Proved | check-zone; work-priority schema | work-priority **6× pass**; doctor+patient **9× tend pass** | +| Research | unexercised | availability **4× unavailable rejects** (no bench / prereqs) | +| Tend | unexercised | 9× pass (see above) | +| Farm already-covered | — | branch not hit | +| Stockpile-delete | not exercised | not exercised | + +Val proved check-zone + work-priority schema. Gates2 proved work-priority +repeats and tend; research rejects were **no bench / prereqs**, not a +schema miss. Stockpile-delete and the farm already-covered branch are +still open QA.