Skip to content

RLE bench design — N × ticks for harness QA → stats - #73

Merged
jkbennitt merged 1 commit into
masterfrom
cursor/bench-design-n-ticks-4fb0
Sep 7, 2026
Merged

RLE bench design — N × ticks for harness QA → stats#73
jkbennitt merged 1 commit into
masterfrom
cursor/bench-design-n-ticks-4fb0

Conversation

@jkbennitt

Copy link
Copy Markdown
Member

Docs-only memo for how we get from a working harness to a number we can defend. No code, deps, defaults, or matrix runs. Do not merge until reviewed.

What

Adds docs/bench-design-n-ticks.md plus a one-line pointer from the README live-board paragraph.

Locked path (do not conflate)

  1. Harness QA — N=1, 10 ticks (Jason lock: software/harness validation first)
  2. Variance pilot — N=8–10 seeds, 10 ticks, 1–2 keeper cells → estimate σ
  3. Horizon sweep — one cheap cell, N=3 × {10, 25, 50, 100}; shortest horizon that is not mostly censored
  4. Publishable compare — only then n ≈ 16 σ²/δ² per cell (80% power, α=0.05); prefer paired same-seed sets

Current scoring-1.2 board is almost all seed-42 N=1: no within-cell σ. Significance without σ is theater. Flash cells ~$0.02–$0.90 — do not incinerate a full matrix before σ + horizon.

Gate fold (2026-09-07)

  • Val N=1 (seed 42, opencode × deepseek-v4-flash): 0.7073; check-zone + work-priority schema proved; research/tend/stockpile-delete unexercised; pin f818246e @ 41fd015; ~$0.025 console
  • Gates2 N=1 (seed 314159 + short research/tend bias): 0.6803; work-priority 6× pass; research 4× unavailable rejects (no bench/prereqs); doctor+patient 9× tend pass; farm already-covered and stockpile-delete not exercised; ~$0.025 console

Out of scope

No CLI/default changes. No matrix. No XAI.

Open in Web Open in Cursor 

Lock the four-stage bench path (QA → σ pilot → horizon → sized compare)
so N=1 seed-42 cells are not treated as significance. Fold 2026-09-07
gate scores; 10-tick stays software validation first.

Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
@jkbennitt
jkbennitt marked this pull request as ready for review September 7, 2026 23:38
@jkbennitt
jkbennitt merged commit 14c8ba0 into master Sep 7, 2026
5 checks passed
@jkbennitt
jkbennitt deleted the cursor/bench-design-n-ticks-4fb0 branch September 7, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants