code_sim_learning: sysID verdicts, shared orchestrator, anchor-ablation refits (sysID stack 3/8) - #98
Merged
Merged
Conversation
…n refits The identifiability probe measures local curvature at the joint MAP; a co-adapted parameter set has real curvature in every direction, so a param that only moved to compensate another's overshoot still reads "identified". Measured on run_20260721_205821: seed1's coordinate sweep overshot lateral_friction to 0.827 (true 0.5), then restitution moved 0.02 -> 0.75 (26x) and spinning_friction 0.5 -> 0.01 to compensate - all three "identified" - and the resulting belief sim invalidated every test plan; seed2's restitution churned 0.54 -> 0.013 -> 0.27 across cycles (the cross-cycle check flagged 194 combined sigmas but the values were still applied). After the LM MAP, greedily refit the remaining params with each moved physical param pinned at its env-registry anchor; revert the move only when the refit is BOTH data-equivalent (the grid flat set's tolerance, max(noise floor, grid_flat_frac * SSE)) AND strictly closer to the standing belief in total fit-space prior cost - the anchor-nearest flat-set principle applied jointly, which also stops a symmetric ridge from merely swapping which param carries the move. A genuinely identified param is never touched: pinning it destroys the SSE. Reverted params re-enter the result at their anchors, recorded in FitResult.anchor_ablation; the identifiability report renders them "anchored (...compensatory...)" so select_trustworthy_params applies the baseline, and the Jacobian is dropped when anything was pinned (consumers already handle None). Gated by code_sim_learning_rollout_anchor_ablation (default True); unit-tested with a compensation-ridge fake env (tight-prior compensator reverts, wide-prior param refits to its true value; a genuine single-move MAP is returned unchanged).
The single domino_turn_ratio applied the same turn90-corner quota to train and test task generation. Replace it with domino_train_turn_ratio (default 0.0) and domino_test_turn_ratio (default 0.5), threaded explicitly through _generate_tasks / DominoTaskGenerator.generate_tasks / make_min_block_tasks so each task set fills its own quota; configs and the cascade-probe test move to domino_test_turn_ratio.
exp_domino.yaml un-skips domino_high_friction_turn (and parks domino_high_friction); common.yaml adds make_interaction_videos to the launcher args so exploration episodes are recorded alongside test episodes.
Applies the confirmed/plausible findings from the critical review of ffb3fe7/f6a854ce9 that don't need a product decision: - Integrate the "anchored" verdict with its three consumers: explorer diagnostics now explain a compensatory revert (instead of advising "drop it from PHYSICAL_PARAMS"), cross-cycle consistency neither compares against nor records the anchor as a confident fit, and the ablation entry records the pre-revert fitted value so the report keeps what the data originally asked for. - Pin ablated params EXPLICITLY during the refit and the candidate SSE (new fit_map_lm_rollout fixed_physical arg) so the data-equivalence verdict is measured at exactly the anchor recorded and applied, independent of env-registry defaults - closes the latent mismatch if a caller ever passes non-registry anchors. - Warn when rollout MCMC is enabled that the ablation only runs on the LM point-fit path. - Document that the ablation's SSE baseline deliberately stays at the running minimum (bounds cumulative degradation to ~tol; conservative toward keeping fitted values). - Move StreamingVideoMonitor.discard() into the episode finally so an exception can no longer leak the temp clip/writer; log when test_fresh_env_per_episode falls back to the shared env. - Replace em dashes in the added comments/docstrings with plain dashes (user convention). Deliberately NOT addressed here (need a decision or are larger refactors): domino_train_turn_ratio config value for the sysID arm, ablation-loop refit memoization / shared prior-center + noise-floor helpers, Laplace-ensemble support for ablated results.
Every experiment config has always run the rollout sysID with rollout_num_mcmc_steps = 0, so the emcee sampling branch, the walker warm-start scaffolding, and the per-fit 'Warm-starting rollout MCMC walkers' / 'Skipping emcee' log noise were dead weight executed on every fit (audited on run_20260722_123949 and run_20260708_101431: 17x 'Skipping emcee' in one learn phase, zero emcee runs ever). fit_params_rollout now runs the grid-seeded, prior-folded LM MAP unconditionally as THE fit, independent of the warm-start/info-seeking CFG flags (previously, with all three prefit flags off, the grid sweep ran and was then discarded in favor of the declared inits). The LM Jacobian stays on the FitResult as the Laplace bundle for the info-seeking explorer's calibrated ensemble, and the Hessian eigenanalysis stays behind its flag. The per-transition/recurrent MCMC path in fitting.py is untouched (posterior-subsample ensembles for gate/threshold params remain available there). Removes code_sim_learning_rollout_num_mcmc_steps and the num_steps plumbing through fit_params_rollout_trimmed and the approach.
The identifiability verdicts were prose strings that three consumers parsed with startswith() (trust selection, explorer diagnostics, cross-cycle bookkeeping) - a class of bug that already bit once when the anchored verdict was not integrated everywhere. Verdicts are now a Verdict enum (the single decision surface, with applies_fitted as the one apply-policy predicate); the human/agent-facing explanation travels in a separate 'note' field attached at render time by format_identifiability. Policy change riding on the enum: a cross-cycle INCONSISTENT parameter (confident fit jumped > k combined sigmas since the previous cycle) now HOLDS the currently-applied value instead of hopping to the new fit. Neither of two mutually-incompatible confident fits can be preferred on that evidence, and hopping churned the belief env for whole runs (run_20260721_205821 seed1: restitution 0.71 -> 0.52 -> 0.02 -> 0.32 -> 0.02 across cycles, every hop 'identified'). Without a held value the anchor remains the fallback. The explorer diagnostics gain a dedicated INCONSISTENT branch asking for a clean single-param excitation experiment.
… pre-test
Identifiability verdicts for grid-swept physical params now come from
the grid landscape itself: the posterior width is the fit-space
half-width of the param's data-equivalent flat_interval, so the verdict
agrees with the landscape by construction. The +/-sigma curvature probe
measured LOCAL precision at the chosen point, which on a plateau reads
as sharp: on run_20260722_123949 every param came back 'identified'
(contraction <= 0.05) while the same report noted the fitted value was
'the edge of this interval ... not a unique optimum'. The probe never
changed a single decision across all seeds. It remains for parameters
without sweep coverage (rule params, grid disabled) and skips swept
params entirely, saving its 3 + 2/param rollout evals in the common
case. flat_wide is gone (its condition is now the verdict itself); the
explorer diagnostics emit the cannot-distinguish-interval advice for
WEAK/NOT verdicts instead.
Anchor ablation gains a cheap pre-test: pin the candidate param with
everything else unchanged (one SSE eval) and only fall back to the LM
refit (~a dozen full-trajectory evals) when that fails. On real runs
most reverts are tiny compensatory drifts (seed2: spinning 0.4993 ->
0.5, restitution 0.02021 -> 0.02) where the pre-test suffices. Also
fixes the zero-free-param refit crash ('zero-size array to reduction
operation', seed2 debug.log:3500) by returning the pins directly when
nothing remains to refit.
Adds rollout-count telemetry (rollout_env.num_rollouts_run) and
per-stage cost log lines (grid/LM/ablation split per fit, explainability
sweep per trimmed call) so the eval budget is visible in run logs
instead of requiring fn-eval archaeology.
The full fit flow (residual scaling, pre-SSE, explainability trimming + consistency loop, identifiability report on survivors, trust selection) existed twice - approach._fit_parameters_joint_rollout and the sim.fit tool backend - and had to be kept in sync by hand (the bug class that produced the silent sim_env omission). orchestrator.run_rollout_sysid is now the single flow, returning a SysIdOutcome; the approach applies + records diagnostics, the tool renders a report. The approach's cross-cycle consistency check plugs in as a report_adjuster, and its currently-deployed values flow in as the INCONSISTENT hold set. The orchestrator memoizes the WHOLE fit core per (artifact version tag, declaration/data signature) - the fit is a pure function of its inputs now that every rollout runs in a fresh deterministic env, and agents hammer the canonical sim.fit within one learn phase (run_20260708_101431: 7 invocations, several on identical versions + identical data). Cache hits spend zero rollouts; the adjuster + trust selection re-run per call on a deep-copied report so caller-local demotions never leak. The cache lives next to the explainability cache on the approach and shares its lifecycle; exploratory subset fits bypass both (segment-length keying). Also fixes the misleading '(+100.0% vs init)' label on the sim.fit SSE line - it now reads 'X% SSE reduction vs init'.
…ation Both audited experiment families converged on a single genuinely identifiable parameter, and every extra declared parameter was either reverted by the anchor ablation (run_20260722_123949 seed2: 4 of 5) or pruned by the agent over 7 costly refit iterations after the verdicts flagged it (run_20260708_101431: 5 -> 4 -> 2 -> 1). Compensation overfit only exists because co-adapted extras are declared in the first place, so the guidance now says: start with the ONE parameter that has a physical story for the residual, add another only if the calibrated fit leaves structure unexplained. Also documents the 'anchored' verdict the agent may now see (the 20260708 run proved the agent follows this report literally).
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of eight PRs landing the domino-sysid branch on master in sequence (after #96, #97). Tree is identical to the corresponding slice of that branch rebased onto current master, plus one docstring-rewrap commit so the intermediate tree passes docformatter (the original rewrap arrives with a later slice).
This chunk
The sysID core: a typed identifiability Verdict enum with hold-on-INCONSISTENT arbitration, landscape-derived verdicts with an anchor-ablation pre-test that reverts compensatory parameter moves, and one shared orchestrator with whole-fit memoization used by both the approach and agent tool paths.
🤖 Generated with Claude Code