migration: re-pin #1048 baseline candidate to 27828001 (QA campaign BLOCKED) - #1197
daimon-pymclabs wants to merge 2 commits into
Conversation
PR SummaryLow Risk Overview The same SHA is updated in This is harness and documentation only under Reviewed by Cursor Bugbot for commit c3e88aa. Bugbot is set up for automated code reviews on this repo. Configure here. |
`PYMC6_COMMIT` moves from `c83194a38373b815a90582e7969e995c4db52da2` to `143e823ad1167da59686d51a11f8e6d97b1d0361`. The previous pin predates further behavioral change on the integration branch: #1175 replaced the eager constructor with a lazy configure() -> optional prior checks -> fit() experiment lifecycle, the mypy-zero cleanup programme (#1145, `causalpy/`, the docs/notebook main-port PRs #1185-#1190 landed, and open pull request left against the integration branch, so this head is the tree actually proposed for `main` and the evidence should describe it. `PYMC5_COMMIT` is unchanged. This is harness, documentation and tests only, so it cannot change a sampled posterior and does not make its own pin stale -- the harness runs from its own checkout, whose `HEAD` intentionally differs from the sampled candidate. Artifacts recorded at any earlier pin are rejected by `compare`, so the fresh batch cannot be contaminated. Validated the documented-quote-site consistency invariants that `test_pinned_revisions_are_documented_consistently` and `test_superseded_pin_is_recorded_as_history_only` check (harness.py, README.md and REPORT_TEMPLATE.md agree on the new pin, and the superseded pin is recorded as history only) by loading harness.py standalone and re-running their assertions directly, since the full CausalPy dependency stack (pymc/numpy) is not available in this sandbox; full pytest run deferred to CI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`PYMC6_COMMIT` moves from `143e823ad1167da59686d51a11f8e6d97b1d0361` to `2782800194ee6bf49c4ab14d84f5ec7ec66b7f34`. The previous pin predates further merges to the integration branch: #1195 fixed stale `post_impact`/`att_event_time` prose in docstrings and error messages left over from #1175 (`causalpy/reporting.py`, `causalpy/experiments/synthetic_difference_in_differences.py`) and updated the `pymc-extras` install hint in `causalpy/pymc_models.py`, and #1196 adopted `uv` as the default dev environment for CI. Both are text-only changes under `causalpy/` with no numerical effect, but the harness's own policy is to move the pin on any `causalpy/` change rather than judge materiality itself. `PYMC5_COMMIT` is unchanged. This is harness, documentation and tests only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7ffc55c to
c3e88aa
Compare
daimon-pymclabs
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## pymc6_and_pymcmarketing1_migration #1197 +/- ##
===================================================================
Coverage 97.25% 97.25%
===================================================================
Files 129 129
Lines 23688 23688
Branches 1328 1328
===================================================================
Hits 23038 23038
Misses 443 443
Partials 207 207 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
1 participant
Summary
PYMC6_COMMITtarget twice, in sequence, while executing the QA campaign:c83194a3->143e823a(current head ofpymc6_and_pymcmarketing1_migrationat the time), keepingPYMC5_COMMITunchanged at79c0a870.143e823a->2782800194ee6bf49c4ab14d84f5ec7ec66b7f34(the branch's head moved again mid-task: docs: update stale result-bundle references after #1175 #1195 fixed stalepost_impact/att_event_timeprose left over from Lazy experiment lifecycle: configure → optional prior checks → fit() #1175, devops: adopt uv as default dev environment and switch CI to uv (#1174) #1196 adopteduvfor CI). Both are text-only changes undercausalpy/with no numerical effect, but the harness's own policy is to move the pin on anycausalpy/change under the integration branch rather than judge materiality itself.143e823a) on a dedicated host (two isolated micromamba prefixes, three clean detached worktrees). The campaign did not complete. Capture synthetic control simulated example: compare inferred causal impact to true causal impact #1 of 4 (pymc5 reference_first) crashed after sampling finished, before any comparison ran. Moving the pin a second time does not change this: the crash is in the harness's shared capture path, not tied to a specific candidate commit.Headline result: BLOCKED, not a pass
No v2 evidence exists for either pin. This must not be read as "no regression detected" — no comparison ran.
Root cause (pre-existing in the harness, not introduced by either re-pin in this PR):
_capture_bundle()inscripts/migration_baseline/harness.pydoes:At the candidate pin (post-#1175 lazy
configure()->fit()lifecycle),BaseExperiment.resultis a real property returning a bundle withscenario_counterfactual.prediction. At the reference pin (79c0a870, pre-#1175 eager lifecycle),BaseExperimenthas no.resultproperty, so_capture_bundlefalls back to the raw experiment — which stores the counterfactual asself.x_pred_counterfactualdirectly, with noscenario_counterfactualalias. Result:git logonharness.pyshows the capture path was written against the new bundle shape in232b8b37("Migrate test suite to the lazy lifecycle API"). The harness's own continuous-verification test only ever runs against whatever CausalPy is installed in its own checkout (always post-#1175), so this gap against the actual pinned pre-#1175 reference commit was never caught.SyntheticControlcapture uses the same bundle assumption (bundle.impact_post,bundle.predictions_pre,bundle.predictions_post) and would fail identically once DiD capture is fixed past this point.Environment (for the record — nothing to compare against yet):
79c0a87072fd4653bfaed1eb085f965594c7f03a143e823ad1167da59686d51a11f8e6d97b1d0361Harness checkout for the capture attempt:
7ffc55cf(superseded by this PR's current head,2782800194ee6bf49c4ab14d84f5ec7ec66b7f34, after the second re-pin).What's needed before #1048's QA gate can pass
_capture_difference_in_differencesand_capture_synthetic_controlinscripts/migration_baseline/harness.pyneed a reference-stack (pre-#1175 eager API) adapter branch so both stacks produce the samemu-shaped series regardless of which lifecycle API the pinned commit exposes. That's a harness change with real statistical surface area (it has to reproduce the exact same conditional-mean values the eager API produced) and needs independent review. It was not attempted in this PR; this PR only carries the two re-pins plus the honest record of why capture is currently blocked.Test plan
test_pinned_revisions_are_documented_consistently,test_superseded_pin_is_recorded_as_history_only,test_pinned_revisions_are_distinct_full_shaspass by inspection (documented pin, README, REPORT_TEMPLATE agree; verified by replaying the exact regexes those tests use against the final tree).1048-baseline-comparison.json/1048-baseline-report.mdwere produced for either pin.Full traceback and root-cause writeup are also posted as a comment on #1048 (gist upload was attempted but the available GitHub token is fine-grained and cannot create gists — evidence is inline in the issue comment instead).
Closes/updates #1048 (QA gate still open — see issue comment).
🤖 Generated with Claude Code