Scoring 1.2 live validation: recalibrate Crashlanded baseline, then first comparable runs

[jkbennitt]

Scoring 1.2 is on master (#51, #52) but we still have no live 1.2 numbers. The pinned Crashlanded sidecar is scoring 1.1, and the published 11-model spread is Felix-only and not comparable. Do not publish a new leaderboard until this pipeline finishes, in order.

Why this blocks everything else

src/rle/scenarios/definitions/01_crashlanded_survival.baseline.json is still:

• scoring_version: "1.1"
• metric means include coordination / communication_efficiency (removed in 1.2)
• recorded 2026-06-10 against commit 1961e91

load_baseline() already fail-fasts on a version mismatch (BaselineMismatchError). Agent-vs-baseline deltas, README tables, and #46 (N=4 vs static baseline) cannot use it.

The 1.2 composite also changed empty-write defaults (efficiency / plan_coherence → 0.5 instead of 1.0), so even a successful 1.1 baseline colony would not be a valid 1.2 reference.

Pipeline (do in this order)

1. Recalibrate the Crashlanded no-agent baseline

Requires RimWorld + RIMAPI loaded into the Crashlanded save. Four seeded --no-agent --until-death runs, then commit the sidecar:

python scripts/calibrate_baseline.py crashlanded --seeds 42 43 44 45

The script writes 01_crashlanded_survival.baseline.json and round-trips it through the strict loader. Do not hand-edit the sidecar. Per-run CSVs stay under results/baseline/ (gitignored); only the aggregated JSON is committed.

Acceptance:

• Sidecar scoring_version is "1.2"
• Metric means include plan_coherence, not coordination / communication_efficiency
• save_sha256 still matches the scenario YAML pin
• python -c "from rle.scenarios.loader import load_scenario, load_baseline, list_scenarios; ..." (or the script's own round-trip) succeeds
• tests/unit/test_baseline.py still passes against the new file

2. Felix scoring-1.2 spread (Crashlanded)

Same protocol as the 2026-06-11 N=1 spread, but against the new sidecar and SCORING_VERSION 1.2. Keep the 1.1 README table labeled historical until a 1.2 table exists.

• At least one Felix agent run vs the new baseline on Crashlanded (same seed/tick policy as the last published spread, or document the change)
• Record harness + model + scoring version in run metadata (harness_versions, scoring_version)
• Do not overwrite the 1.1 table with incomparable numbers

This is the first number that can honestly answer #6 under scoring 1.2. It does not close #46 (that is still N=4).

3. Live-validate coding-agent harnesses (before any matrix)

rle-harness-opencode and rle-harness-grok-build have been contract-tested against mocks and documented CLI/API surfaces only. They have not driven a live colony.

• --harness opencode against a live map: setup → at least one tick that produces RIMAPI writes → teardown, no null-ref cascade
• --harness grok-build the same
• Confirm TickLedger / plan_coherence see those writes
• Document the exact binary versions in the run metadata

Do not add these harnesses to a published matrix until this step is green.

4. Only then: first harness × model matrix

Out of scope until 1–3 are done. Tracked here so it is not started early:

python scripts/run_benchmark.py --harness felix --harness opencode --runs 4 --output results/matrix/

Out of scope

• Calibrating the other five scenario sidecars (still blocked on Create benchmark save files for all 6 scenarios #7 saves)
• N=4 statistical milestone (N=4 validation milestone: top-4 roster vs static baseline #46) — unblock it with step 1, do not fold it into this issue
• Publishing OpenCode / Grok Build leaderboard rows before step 3

Related

• Phase C: redesign process metrics for harness-agnostic scoring (scoring 1.2) #51 (scoring 1.2 design — done)
• Agents must beat unmanaged baseline #6 (agents must beat unmanaged baseline — numbers reset at 1.2)
• N=4 validation milestone: top-4 roster vs static baseline #46 (N=4 vs static baseline — blocked on step 1)
• RLE v1.0: Multi-model colony management leaderboard #8 (v1.0 leaderboard epic)
• ADR-004 follow-ups: stale sidecar + coding-agent harnesses verified against mocks only