feat: declare model lineage in the config instead of inferring it - #257
Conversation
|
Approving. This unblocks the cross-lineage cascade (Gemini vs an open-weights model), which is the experiment worth doing. Let's aim there. |
maximinl
left a comment
There was a problem hiding this comment.
Approve declared lineage (stacked on #253).
Keeping id-heuristic as fallback preserves existing configs; Config.roster + resolved_roster (declared/inferred) is the right place for tier / open_weights, which unblocks cross_lineage_committees from a config file. Merge after the stack below it.
There was a problem hiding this comment.
Nice config design, but it only pays off once a real cross-lineage roster runs, @duckyquang please use the nvidia cluster #265/#95. Holding so we merge it with that result rather than ahead of it.
Superseded by the later Changes-requested review on this PR; dismissing the stale approve to avoid a mixed signal.
Agastya191
left a comment
There was a problem hiding this comment.
Good work here, and thanks for closing the loop on the lineage thread from #251: declaring lineage in the config and threading model_specs into both run_holes_test and build_committee is exactly the right seam, the roster/models split leaves every existing caller unchanged, and stamping resolved_roster with declared-vs-inferred provenance into the manifest is a genuinely nice touch. One thing I'd flag is that the bare-id path still routes through _lineage, which splits on the first separator, so an org-namespaced id like meta-llama/llama-3.1-70b still resolves to meta rather than llama. Because the override only fires when lineage is declared, a mixed-convention roster still gets a wrong within- vs cross-lineage partition in run_holes_test and the run completes clean, with lineage_source: inferred in the manifest as the only trace after the fact. It's reasonable to keep the heuristic as the documented fallback, but I'd have model_specs (or lineage_sources) warn when an inferred id looks org-prefixed, so the failure I hit in #251 is loud at run time instead of only discoverable in the manifest afterward.
|
@Agastya191 done in 3ed12d9. @sebasmos understood — holding for a real cross-lineage roster run on the nvidia cluster (#265/#95) so it lands with that result rather than ahead of it. I've got the access for that one. @maximinl thanks. Added the run-time warning for org-prefixed inferred ids (3ed12d9) so the id-heuristic fallback can't silently split a mixed roster wrong. |
77fff78 to
ee7e998
Compare
sebasmos
left a comment
There was a problem hiding this comment.
Approving. runner.py:116 warns when lineage is inferred from an org-prefixed id and declaring it silences the warning, which is what @Agastya191 asked for. Note the cross-lineage roster run is still open on #265, where I flagged that its cross_lineage field reads off declared ModelSpec.lineage and so could not have returned false with Gemini behind both slots. Merged main in, post-squash conflicts resolved to the branch side, suite green.
* feat: benchmaxxing run entry point for the stage runners * feat: bootstrap CIs and effect sizes in the run summary * feat: correct the whole family of p-values in the run report * feat: declare model lineage in the config instead of inferring it * cascade significance: paired permutation on the adoption gap, not a McNemar sign test Agastya's review: _cascade_tests ran the continuous shared/isolated adoption rates through mcnemar by counting only each pair's direction, collapsing it to a sign test that drops ties and ignores magnitude. Over four discordant cases the smallest two-sided p is 0.125, so a unanimous planted-answer effect could never be flagged even when the shared/isolated bootstrap CIs printed right above it are cleanly separated -- the two signals then contradict. Add stats.paired_permutation_test (seeded sign-flip over the per-case differences, exact by enumeration for small n, sampled otherwise) and test the same shared-minus-isolated gap the adoption_delta CI is built on, so the p-value and the interval answer one question. Thread the run seed through significance_tests for reproducibility. * lineage: warn when an org-prefixed id is inferred, not just recorded Agastya's review: the bare-id fallback routes through _lineage, which splits on the first separator, so an org-namespaced id like 'meta-llama/llama-3.1-70b' resolves to 'meta' rather than 'llama'. A mixed-convention roster then gets a wrong within- vs cross-lineage partition in run_holes_test and the run completes clean, with lineage_source: inferred in the manifest as the only trace after the fact. Keep the heuristic as the documented fallback but make the risky case loud: model_specs warns when a lineage is inferred (not declared) from an id carrying a '/'. Declaring `lineage` in the config -- the resolution from #251 -- silences it. Failure is now at run time, not a manifest post-mortem. * solo CI: resample over cases, not twins Agastya's review: the solo lane resampled flip_rate/shortcut_reliance over the flat per-twin record list, but _build_twins emits one twin per (case, cue_type), so a case's several cue-twins sit in that array and bootstrap_ci drew them iid. Same-case twins share the case's difficulty, so counting them as independent understates the uncertainty exactly the way pooling models would -- the 95% CI came out tighter than the case-level evidence supports. Add stats.cluster_bootstrap_ci (draws whole clusters, pools their observations) and resample the solo lane by case_id, so the interval answers "does this hold on new cases", which is the claim the arm makes. The point estimate is unchanged and, since every cluster is a singleton for one-twin-per-case data, the pinned intervals are untouched; only real multi-cue cases widen. --------- Co-authored-by: sebasmos <sebasticajas@gmail.com>
Summary
A model's lineage can now be declared in the config instead of guessed from its id:
The id heuristic stays as the fallback, so every existing config behaves exactly as before.
Config.modelsis still the plain list of ids (nothing downstream changed); the declarations land in a newConfig.rosteralongside it.Two things that were previously unreachable now work:
tier, andopen_weights, which was hardcoded to False for every model built from a config.roster.cross_lineage_committeesrefuses a roster with no open-weights model, so that guard was un-satisfiable from a config file until now.The run manifest gets
config.resolved_roster: the roster as the run actually used it, each entry taggeddeclaredorinferred.--dry-runshows the same thing inline.Result
ruff check .clean,pytest -q609 passed. Tests cover both config shapes, the round trip, andmy-tuned-cxr, which is exactly the id the prefix heuristic gets wrong.Notes
Stacked: #251 -> #252 -> #253 -> this. Base is #253's branch so the diff is only the lineage work.
Closes #255