Docs: Add Synthetic Control sensitivity checks walkthrough (port of #871) - #1190
Merged
cetagostini merged 1 commit intoSep 18, 2026
Merged
Conversation
Redo the essence of #871 on the migration branch notebook. #871 predates the hyphen-case notebook renames, so the walkthrough is re-applied to synthetic-control-pymc.ipynb against the current pipeline and check APIs. Adds the 'Sensitivity analysis' section before References: - Related checks for synthetic control: PlaceboInTime (default), ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity. - Placebo-in-time check mechanics, walkthrough, interpretation guidance (pass / fail / INCONCLUSIVE verdicts), and a five-step 'If this check fails' diagnostic path. - Pipeline cell: EstimateEffect(cp.SyntheticControl) -> SensitivityAnalysis (PlaceboInTime) -> GenerateReport, plus the HTML report iframe. Re-executed with the pymc 6 env via scripts/run_notebooks/runner.py. Adaptations to the branch state: - The PR-era 'Known limitations being tracked' note (#875 short fold pre-periods, #876 unseeded posterior predictive) is dropped: both issues are fixed on this branch. PlaceboInTime now accepts an explicit intervention_length and seeds theta_new end-to-end. - The walkthrough instead demonstrates the #875 remedy directly: intervention_length=20 keeps both folds eligible on this short series, and the seeded run yields a deterministic SUPPORTED verdict (P(actual outside null) = 0.999), with the comparison-window caveat (20-observation placebo windows vs 30-observation actual post period) explained in the narrative and surfaced by the check's own warning. - Glossary term uses the branch spelling {term}`Synthetic control`; docnames adapted (pipeline-workflow, sensitivity_checks); citations verified against branch references.bib.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
5 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## pymc6_and_pymcmarketing1_migration #1190 +/- ##
===================================================================
Coverage 97.24% 97.24%
===================================================================
Files 129 129
Lines 23553 23553
Branches 1318 1318
===================================================================
Hits 22904 22904
Misses 440 440
Partials 209 209 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cetagostini
merged commit Sep 18, 2026
84f8ef8
into
pymc6_and_pymcmarketing1_migration
16 checks passed
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fresh re-implementation of the essence of #871 (labeled
migration:needs-port) on the migration branch — no commits ported. #871 targetssc_pymc.ipynb; the walkthrough is re-applied to the branch'ssynthetic-control-pymc.ipynbagainst the current pipeline and check APIs.Added
## Sensitivity analysissection (before References)PlaceboInTime(default),ConvexHullCheck,LeaveOneOut,PlaceboInSpace,PriorSensitivity— each with the vulnerability it targets; notes which checks are not applicable.CheckResultmetadata map (null_samples,fold_results), interpretation of pass / fail / INCONCLUSIVE.EstimateEffect(cp.SyntheticControl) → SensitivityAnalysis(PlaceboInTime) → GenerateReport+ HTML report iframe.Re-executed with the pymc 6 env (
scripts/run_notebooks/runner.py --full): both folds complete, verdictSUPPORTED — P(actual outside null) = 0.999, actual cumulative impact ≈ −37.4.Adaptations to the branch state
PlaceboInTimeaccepts an explicitintervention_length, andtheta_newsampling is seeded end-to-end. Porting the note verbatim would have documented bugs that no longer exist.intervention_length=20(the documented fix), both folds fit, and the seeded run is deterministic. The narrative explains the trade-off the check records incomparison_window(20-observation placebo windows vs the 30-observation actual post period) and why the separation probability reads as an upper bound — matching the check's own UserWarning in the output.{term}\Synthetic control`; docnames adapted (pipeline-workflow); citationsabadie2010synthetic,abadie2021using,reichardt2019quasiverified against branchreferences.bib`.Verification
prek rungreen (Validate notebooks, codespell, PyMarkdown, gallery/rediraffe).Part of the #1048 migration effort. Supersedes #871 once the migration branch lands.