docs: ITS sensitivity walkthrough (#788) by drbenvincent · Pull Request #821 · pymc-labs/CausalPy

drbenvincent · 2026-04-03T13:18:26Z

Closes #788. Delivers the focused ITS sensitivity walkthrough requested as a follow-up to #749, scoped under #785.

Summary

`docs/source/notebooks/its_post_intervention_analysis.ipynb` (commit `10d2210`)

Rewrite the fixed-period ITS notebook opening around a concrete marketing-campaign question and add a clearer reader roadmap.
Streamline the persistence walkthrough so the main results and sensitivity sections build on each other instead of repeating the same ideas.
Reframe the placebo and persistence checks around learner-facing questions, then end with takeaways and a reflection prompt.
Adds a worked Pipeline example combining the default PlaceboInTime check (via cp.SensitivityAnalysis.default_for(cp.InterruptedTimeSeries)) with a custom PersistenceCheck, plus a "What to do when a check looks weak" interpretation section.

Cross-linking polish (commit `babdd2a`)

docs/source/notebooks/sensitivity_checks.md: add the walkthrough under both PlaceboInTime and PersistenceCheck in the "Where examples already exist" list, plus the closing :::{seealso} block.
docs/source/notebooks/its_pymc.ipynb: add a closing "Going further: sensitivity checks" section pointing readers to the central guide and the new walkthrough.

Scope decision

After discussion on #788, we deliberately did not propagate the full sensitivity walkthrough to every ITS variant (its_pymc, its_covid, its_lift_test, its_pymc_comparative, piecewise_its_pymc, its_skl). Reasons:

Avoid the "duplicated check boilerplate" anti-pattern called out in Expand docs coverage for pipeline sensitivity checks #785.
Several variants (notably its_skl.ipynb) can't run the PyMC-only checks at all.
A single canonical walkthrough plus discoverable links from the central guide and its_pymc.ipynb covers the use case without diluting the pedagogy of each method-specific notebook.

Test plan

prek run --all-files
make html
its_post_intervention_analysis.ipynb executed end-to-end in the CausalPy env
its_pymc.ipynb re-executed end-to-end in the CausalPy env so committed outputs reflect the new "Going further" section
CI green on this PR (note: check-environment-yml failure is an unrelated upstream pyproject2conda regression, fixed by ci: pin pyproject2conda==0.22.1 to work around 0.23.0 regression #878)
RTD preview spot-check: notebooks/its_post_intervention_analysis.html shows the new pipeline + persistence narrative; notebooks/its_pymc.html ends with the "Going further" seealso block; notebooks/sensitivity_checks.html lists the walkthrough under PlaceboInTime and PersistenceCheck

review-notebook-app · 2026-04-03T13:18:34Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

drbenvincent · 2026-04-03T13:19:33Z

TODO

Add plot output for PlaceboInTime results
Add plot output for PersistenceCheck results

I believe this plot functionality is being worked on in PR #819

read-the-docs-community · 2026-04-03T13:24:38Z

Documentation build overview

📚 causalpy | 🛠️ Build #32485220 | 📁 Comparing 4c023ef against latest (deb8774)

🔍 Preview build

4 files changed

± 404.html
± notebooks/its_post_intervention_analysis.html
± notebooks/its_pymc.html
± notebooks/sensitivity_checks.html

codecov · 2026-04-03T13:24:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.59%. Comparing base (2ff6b7b) to head (4c023ef).
⚠️ Report is 105 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #821   +/-   ##
=======================================
  Coverage   94.59%   94.59%           
=======================================
  Files          80       80           
  Lines       12764    12764           
  Branches      770      770           
=======================================
  Hits        12074    12074           
  Misses        486      486           
  Partials      204      204

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

drbenvincent · 2026-04-13T14:53:07Z

Note: this PR is not a duplicate of #806. Both are follow-ups from #749, but they target different methods and different notebooks:

docs: ITS sensitivity walkthrough (#788) #821 (this PR) → ITS sensitivity walkthrough (its_post_intervention_analysis.ipynb), addresses Docs: Add ITS sensitivity checks walkthrough (follow-up #749) #788
add walkthrough and result interpretation #806 → Staggered DiD sensitivity walkthrough (staggered_did_pymc.ipynb), addresses Docs: Add Staggered DiD sensitivity checks walkthrough (follow-up #749) #791

No overlap in files or content.

Two small follow-ups to PR #821 to improve discoverability of the ITS sensitivity walkthrough now hosted in its_post_intervention_analysis.ipynb: - sensitivity_checks.md: add the walkthrough to the "Where examples already exist" list (PlaceboInTime + PersistenceCheck) and to the bottom :::{seealso} block. - its_pymc.ipynb: add a closing "Going further: sensitivity checks" section pointing to the central guide and the worked walkthrough. Re-executed its_pymc.ipynb in the CausalPy env so committed outputs reflect the new structure. Refs #788 Made-with: Cursor

drbenvincent · 2026-04-30T12:39:54Z

Failing CI check unrelated to this PR. Fix in flight #878

drbenvincent · 2026-04-30T12:44:43Z

Heads-up for reviewers: the failing check-environment-yml job is not caused by this PR. It's an upstream regression in pyproject2conda 0.23.0 that hits all currently-open PRs; #878 pins it back to 0.22.1 and will resolve it once merged + this branch is updated from main.

Rework the notebook so readers can follow the example, persistence analysis, and sensitivity checks without prior ITS context. This keeps the focus on learner-facing questions instead of pipeline implementation details. Made-with: Cursor

Two small follow-ups to PR #821 to improve discoverability of the ITS sensitivity walkthrough now hosted in its_post_intervention_analysis.ipynb: - sensitivity_checks.md: add the walkthrough to the "Where examples already exist" list (PlaceboInTime + PersistenceCheck) and to the bottom :::{seealso} block. - its_pymc.ipynb: add a closing "Going further: sensitivity checks" section pointing to the central guide and the worked walkthrough. Re-executed its_pymc.ipynb in the CausalPy env so committed outputs reflect the new structure. Refs #788 Made-with: Cursor

review-notebook-app · 2026-05-02T02:54:35Z

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2026-05-02T02:54:35Z
----------------------------------------------------------------

Line #5.        formula="y ~ 1 + t + C(month, levels=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])",

Worth explain the formula, mostly the levels part. You can have people not familiar, at least say what numbers represent other parts are more intuitive.

review-notebook-app · 2026-05-02T02:54:36Z

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2026-05-02T02:54:36Z
----------------------------------------------------------------

During the during intervention?

review-notebook-app · 2026-05-02T02:54:37Z

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2026-05-02T02:54:37Z
----------------------------------------------------------------

I think this conclusion will change given the updates and corrections in the class, during my PR #826.

review-notebook-app · 2026-05-02T02:54:38Z

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2026-05-02T02:54:37Z
----------------------------------------------------------------

During the persistent check are we checking all the remaining time series or the N weeks defined above? This was not very clear to me during the reading.

review-notebook-app · 2026-05-02T02:54:38Z

View / edit / reply to this conversation on ReviewNB

cetagostini commented on 2026-05-02T02:54:38Z
----------------------------------------------------------------

Good, I like it

drbenvincent · 2026-07-14T10:04:12Z

Automated triage

Recommendation: review:medium — no decision gate identified.

Why:

Documentation-only: rewrites ITS sensitivity walkthrough notebook and cross-links in its_pymc.ipynb and sensitivity_checks.md.
No code changes. All CI checks pass. PR is conflicted (DIRTY).
Already has documentation and OSS_PRODUCT labels.

Review focus:

Verify the Pipeline example with PlaceboInTime + PersistenceCheck executes correctly.
Confirm the cross-linking from its_pymc.ipynb and sensitivity_checks.md resolves in the RTD build.
Check the "What to do when a check looks weak" section provides actionable guidance.

Confidence: high

Redo the essence of #821 on the migration branch notebooks. #821 predates the hyphen-case notebook renames, so the walkthrough is re-applied to the branch notebook names and the lazy experiment lifecycle. interrupted-time-series-post-intervention-analysis.ipynb gains the full 'Sensitivity checks in the pipeline' section between the persistence analysis and the Summary: PlaceboInTime and PersistenceCheck run inside a Pipeline via EstimateEffect -> SensitivityAnalysis, with per-fold and persistence-ratio interpretation cells and a 'what to do when a check looks weak' guide. Re-executed with the pymc 6 env via scripts/run_notebooks/runner.py. interrupted-time-series-pymc.ipynb gains the 'Going further: sensitivity checks' pointer cell linking the central guide, the post-intervention walkthrough, the placebo-in-time case study, and the pipeline tutorial. sensitivity_checks.md cross-links the post-intervention walkthrough under PlaceboInTime and PersistenceCheck in both the 'Where examples already exist' list and the Next steps seealso block. Three adaptations required by the branch architecture: - The PlaceboInTime experiment_factory must return a FITTED experiment, so make_placebo_its calls .fit() on each fold (lazy lifecycle). - The placebo formula pins the month levels (C(month, levels=[1, ..., 12])) because a shifted fold spans only part of the year; unpinned, patsy infers categories from the first slice and rejects folds containing months missing from it, silently skipping those folds. - The PR-era capture_output() wrapper around the pipeline run is dropped: it crashes under the pymc 6 stack's rich progress rendering inside papermill's CapturingDisplayPublisher ('CapturingDisplayPublisher' object has no attribute 'set_parent'), and no branch notebook uses it.

cetagostini · 2026-09-18T11:44:21Z

Ported to the PyMC 6 migration branch as #1189 — a fresh branch off pymc6_and_pymcmarketing1_migration re-implementing the essence of this PR against the new architecture (lazy experiment lifecycle, renamed hyphen-case notebooks, branch check APIs). No commits cherry-picked. This PR can be closed once #1189 lands with the migration branch.

Docs: ITS sensitivity-checks walkthrough (port of #821)

cetagostini · 2026-09-18T12:16:10Z

@drbenvincent Heads-up: the essence of this PR has been re-implemented on the PyMC 6 migration branch as #1189 (fresh branch off pymc6_and_pymcmarketing1_migration, adapted to the lazy experiment lifecycle, the renamed hyphen-case notebooks, and the current check/pipeline APIs — no commits cherry-picked). It is green on CI there.

Given that, please avoid merging this PR into main: doing so would land a conflicting, pre-migration version of the same work, and the migration branch would then have to reconcile it again. The content will reach main with the migration branch once #1048 lands. Suggest closing this PR in favour of #1189.

drbenvincent added documentation Improvements or additions to documentation OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours. labels Apr 28, 2026

This was referenced Apr 28, 2026

Docs: Add ITS sensitivity checks walkthrough (follow-up #749) #788

Open

docs: cross-link new ITS sensitivity walkthrough #877

Closed

drbenvincent changed the title ~~Clarify fixed-period ITS walkthrough narrative~~ docs: ITS sensitivity walkthrough (#788) Apr 30, 2026

drbenvincent requested a review from cetagostini April 30, 2026 12:44

drbenvincent added 2 commits April 30, 2026 15:02

Clarify fixed-period ITS walkthrough narrative

dd06bd0

Rework the notebook so readers can follow the example, persistence analysis, and sensitivity checks without prior ITS context. This keeps the focus on learner-facing questions instead of pipeline implementation details. Made-with: Cursor

drbenvincent force-pushed the issue-788-its-sensitivity-walkthrough branch from babdd2a to 4c023ef Compare April 30, 2026 14:03

drbenvincent added the review:medium Contained change requiring focused human review label Jul 14, 2026

derwells added the migration:needs-port Fix not yet on the v6 migration branch; needs a port PR label Aug 10, 2026

derwells mentioned this pull request Aug 10, 2026

PyMC 6 migration: final QA — integration suite, behavioral baselines vs pymc 5, release prep #1048

Open

7 tasks

cetagostini mentioned this pull request Aug 18, 2026

fix: re-express main's #1060 and #1116 in the transition branch's idioms #1170

Merged

cetagostini mentioned this pull request Sep 18, 2026

Docs: ITS sensitivity-checks walkthrough (port of #821) #1189

Merged

cetagostini added a commit that referenced this pull request Sep 18, 2026

Merge pull request #1189 from pymc-labs/migration/821-its-sensitivity

9486032

Docs: ITS sensitivity-checks walkthrough (port of #821)

Conversation

drbenvincent commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

docs/source/notebooks/its_post_intervention_analysis.ipynb (commit 10d2210)

Cross-linking polish (commit babdd2a)

Scope decision

Test plan

Uh oh!

review-notebook-app Bot commented Apr 3, 2026

Uh oh!

drbenvincent commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

read-the-docs-community Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

codecov Bot commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

drbenvincent commented Apr 13, 2026

Uh oh!

drbenvincent commented Apr 30, 2026

Uh oh!

drbenvincent commented Apr 30, 2026

Uh oh!

review-notebook-app Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drbenvincent commented Jul 14, 2026

Automated triage

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

cetagostini commented Sep 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

drbenvincent commented Apr 3, 2026 •

edited

Loading

`docs/source/notebooks/its_post_intervention_analysis.ipynb` (commit `10d2210`)

Cross-linking polish (commit `babdd2a`)

drbenvincent commented Apr 3, 2026 •

edited

Loading

read-the-docs-community Bot commented Apr 3, 2026 •

edited

Loading

codecov Bot commented Apr 3, 2026 •

edited

Loading

review-notebook-app Bot commented May 2, 2026 •

edited

Loading

review-notebook-app Bot commented May 2, 2026 •

edited

Loading

review-notebook-app Bot commented May 2, 2026 •

edited

Loading

review-notebook-app Bot commented May 2, 2026 •

edited

Loading

review-notebook-app Bot commented May 2, 2026 •

edited

Loading