Docs: Add Synthetic Control sensitivity checks walkthrough by drbenvincent · Pull Request #871 · pymc-labs/CausalPy

drbenvincent · 2026-04-28T12:50:31Z

Summary

Adds a focused sensitivity-check walkthrough for Synthetic Control to the sc_pymc.ipynb notebook, addressing #789 (follow-up to #749 / #785).

This continues the docs work started in #818 (central guide) and #846 (Staggered DiD walkthrough), keeping the same structure:

Sensitivity analysis section introducing the pipeline API (EstimateEffect → SensitivityAnalysis → GenerateReport) and linking back to the central guide and pipeline_workflow.
Related checks for synthetic control subsection enumerating the SC-applicable checks (PlaceboInTime, ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity) with one-line interpretations and noting which checks belong to other estimators.
PlaceboInTime deep-dive with how-it-works narrative grounded in citations (abadie2010synthetic, abadie2021using, reichardt2019quasi).
Walkthrough with numbered steps and pass/fail interpretation guidance.
"If this check fails" subsection with five concrete diagnostic actions: inspect fold summaries, prune the donor pool, add complementary SC checks, sanity-check the convex-hull condition, probe prior sensitivity.
Pipeline code cell running PlaceboInTime(n_folds=2, random_seed=seed) on the existing SC dataset together with GenerateReport(include_plots=True).
HTML report iframe display cell, matching the pattern used in staggered_did_pymc.ipynb.

Notes

The walkthrough deliberately demonstrates a NOT SUPPORTED placebo-in-time result for the demo dataset (P=0.905, just below the 0.95 default threshold). This ties directly into the "If this check fails" troubleshooting section so readers see the failure path and the next steps.
Existing cells were left untouched; the new cells were appended after the effect-summary section. The previous trailing empty placeholder cell was removed as part of inserting the new content.

Closes #789

Test plan

prek run --files docs/source/notebooks/sc_pymc.ipynb passes
python scripts/validate_notebooks.py docs/source/notebooks/sc_pymc.ipynb passes
Notebook re-executed end-to-end via jupyter nbconvert --to notebook --execute --inplace ... so all outputs are populated
CI notebooks job executes the SC notebook without errors
ReadTheDocs preview renders the new sections correctly

Made with Cursor

review-notebook-app · 2026-04-28T12:50:36Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

codecov · 2026-04-28T12:57:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.59%. Comparing base (2ff6b7b) to head (d15eb3f).
⚠️ Report is 105 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #871   +/-   ##
=======================================
  Coverage   94.59%   94.59%           
=======================================
  Files          80       80           
  Lines       12764    12764           
  Branches      770      770           
=======================================
  Hits        12074    12074           
  Misses        486      486           
  Partials      204      204

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

read-the-docs-community · 2026-04-28T13:26:56Z

Documentation build overview

📚 causalpy | 🛠️ Build #32483562 | 📁 Comparing 75a3d51 against latest (deb8774)

🔍 Preview build

2 files changed

± 404.html
± notebooks/sc_pymc.html

drbenvincent · 2026-04-30T12:30:47Z

The failing check-environment-yml check on this PR is not caused by anything in this PR — the diff here is markdown-only and doesn't touch pyproject.toml or environment.yml.

It's an upstream regression in pyproject2conda 0.23.0 (released 2026-04-29) that breaks the job for every PR opened since, regardless of contents. Diagnosis and fix are in #878, which pins the CI install to pyproject2conda==0.22.1 (matching the rev already pinned for the local prek hook).

Once #878 lands I'll rebase this branch onto main so check-environment-yml goes green here too.

Adds a sensitivity-analysis section to sc_pymc.ipynb that walks through the pipeline API (EstimateEffect -> SensitivityAnalysis -> GenerateReport) with PlaceboInTime as the SC default check, and documents the other SC-applicable checks (ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity) with interpretation guidance. Mirrors the structure introduced for Staggered DiD in #846 and links back to the central guide from #818. Includes a runnable pipeline cell and HTML report iframe so the walkthrough demonstrates an end-to-end placebo-in-time check with concrete pass/fail narrative and a five-step "if this check fails" troubleshooting block. Closes #789 Made-with: Cursor

Manually rerun so the Synthetic Control sensitivity walkthrough renders correctly in the built docs. Made-with: Cursor

- Promote PlaceboInTime so its Walkthrough and "If this check fails" subsections sit at #### beneath it, instead of competing with it at the same heading level. - Rewrite the pass/fail interpretation to explain why this specific dataset produces a NOT SUPPORTED verdict (fold 1 has only 12 pre-treatment observations to fit 7 donors, blowing up the hierarchical null's tau). - Add a :::{note}::: callout describing the placebo-fold-too-early failure mode and linking to issue #875 for the upstream fix. - Seed PlaceboInTime's hierarchical-null sampler via sample_kwargs so the printed P value is reproducible across runs. Refs #789, #875 Made-with: Cursor

The PlaceboInTime verdict on this dataset is borderline rather than a clean fail, and small MCMC sampling noise (issue #876) can flip it either side of the 0.95 threshold between runs. Update the docs to treat it as such instead of asserting a specific verdict: - Rewrite "Interpreting the result on this dataset" to describe the outcome as borderline and explicitly point at the reproducibility note, instead of asserting NOT SUPPORTED. - Drop the over-specific sd numbers (sd≈30, sd≈9) so the narrative stays consistent if MCMC noise shifts them slightly. - Expand the :::note::: callout into "Known limitations being tracked" with two bullets: #875 (placebo windows that land too early) and the newly-filed #876 (unseeded pm.sample_posterior_predictive in the hierarchical-null fit). No code or executed-output changes; the rendered cell still prints P(actual outside null) = 0.923 and NOT SUPPORTED, which the new text treats as one borderline draw rather than the headline verdict. Refs #789, #875, #876 Made-with: Cursor

drbenvincent · 2026-07-14T09:58:43Z

Automated triage

Recommendation: review:medium — no decision gate identified.

Why:

Documentation-only: adds SC sensitivity walkthrough to sc_pymc.ipynb.
No code changes. All CI checks pass. PR is conflicted (DIRTY).
Already has documentation and OSS_PRODUCT labels.

Review focus:

Verify the PlaceboInTime demo correctly shows a NOT SUPPORTED result and connects to the troubleshooting section.
Check the HTML report iframe cell renders properly in the docs build.
Confirm walkthrough structure matches the established pattern from staggered_did_pymc.ipynb.

Confidence: high

cetagostini · 2026-09-18T11:44:22Z

Ported to the PyMC 6 migration branch as #1190 — a fresh branch off pymc6_and_pymcmarketing1_migration re-implementing the essence of this PR against the new architecture (lazy experiment lifecycle, renamed hyphen-case notebooks, branch check APIs). No commits cherry-picked. This PR can be closed once #1190 lands with the migration branch.

Docs: Add Synthetic Control sensitivity checks walkthrough (port of #871)

cetagostini · 2026-09-18T12:16:11Z

@drbenvincent Heads-up: the essence of this PR has been re-implemented on the PyMC 6 migration branch as #1190 (fresh branch off pymc6_and_pymcmarketing1_migration, adapted to the lazy experiment lifecycle, the renamed hyphen-case notebooks, and the current check/pipeline APIs — no commits cherry-picked). It is green on CI there.

Given that, please avoid merging this PR into main: doing so would land a conflicting, pre-migration version of the same work, and the migration branch would then have to reconcile it again. The content will reach main with the migration branch once #1048 lands. Suggest closing this PR in favour of #1190.

drbenvincent added documentation Improvements or additions to documentation OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours. labels Apr 28, 2026

This was referenced Apr 30, 2026

PlaceboInTime: guard against placebo folds with insufficient pre-period #875

Closed

PlaceboInTime: end-to-end reproducibility — seed posterior predictive and unify seed surface #876

Closed

drbenvincent requested review from cetagostini and juanitorduz April 30, 2026 12:15

drbenvincent mentioned this pull request Apr 30, 2026

ci: pin pyproject2conda==0.22.1 to work around 0.23.0 regression #878

Merged

2 tasks

drbenvincent added 4 commits April 30, 2026 15:00

Docs: re-execute sc_pymc notebook with rendered outputs

416f41b

Manually rerun so the Synthetic Control sensitivity walkthrough renders correctly in the built docs. Made-with: Cursor

drbenvincent force-pushed the docs/789-sc-sensitivity-walkthrough branch from 86c4e7e to d15eb3f Compare April 30, 2026 14:01

drbenvincent added the review:medium Contained change requiring focused human review label Jul 14, 2026

derwells added the migration:needs-port Fix not yet on the v6 migration branch; needs a port PR label Aug 10, 2026

derwells mentioned this pull request Aug 10, 2026

PyMC 6 migration: final QA — integration suite, behavioral baselines vs pymc 5, release prep #1048

Open

7 tasks

cetagostini mentioned this pull request Aug 18, 2026

fix: re-express main's #1060 and #1116 in the transition branch's idioms #1170

Merged

cetagostini mentioned this pull request Sep 18, 2026

Docs: Add Synthetic Control sensitivity checks walkthrough (port of #871) #1190

Merged

cetagostini added a commit that referenced this pull request Sep 18, 2026

Merge pull request #1190 from pymc-labs/migration/871-sc-sensitivity

84f8ef8

Docs: Add Synthetic Control sensitivity checks walkthrough (port of #871)

Navigation Menu

Docs: Add Synthetic Control sensitivity checks walkthrough - #871

Docs: Add Synthetic Control sensitivity checks walkthrough#871
drbenvincent wants to merge 4 commits into
mainfrom
docs/789-sc-sensitivity-walkthrough

drbenvincent commented Apr 28, 2026

Uh oh!

review-notebook-app Bot commented Apr 28, 2026

Uh oh!

codecov Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

read-the-docs-community Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

drbenvincent commented Apr 30, 2026 •

edited

Loading

Uh oh!

drbenvincent commented Jul 14, 2026

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

drbenvincent commented Apr 28, 2026

Summary

Notes

Test plan

Uh oh!

review-notebook-app Bot commented Apr 28, 2026

Uh oh!

codecov Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

read-the-docs-community Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

drbenvincent commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drbenvincent commented Jul 14, 2026

Automated triage

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Apr 28, 2026 •

edited

Loading

read-the-docs-community Bot commented Apr 28, 2026 •

edited

Loading

drbenvincent commented Apr 30, 2026 •

edited

Loading