Skip to content

Docs: Add Staggered DiD sensitivity checks walkthrough - #846

Open
drbenvincent wants to merge 2 commits into
mainfrom
staggered-did-sensitivity-checks-walkthrough
Open

Docs: Add Staggered DiD sensitivity checks walkthrough#846
drbenvincent wants to merge 2 commits into
mainfrom
staggered-did-sensitivity-checks-walkthrough

Conversation

@drbenvincent

@drbenvincent drbenvincent commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a focused sensitivity-check walkthrough for Staggered DiD to the staggered_did_pymc.ipynb notebook, addressing #791.

This continues the work from #806 (by @arthurmello) and incorporates the review feedback:

  • Sensitivity analysis section with PreTreatmentPlaceboCheck walkthrough using the pipeline API (EstimateEffect → SensitivityAnalysis → GenerateReport)
  • Related checks subsection clarifying that only PreTreatmentPlaceboCheck and PriorSensitivity apply to staggered DiD, and that other checks like PlaceboInTime and LeaveOneOut are for different estimators
  • "How it works" explanation of the tolerance-based pass/fail logic
  • Walkthrough with numbered steps and interpretation guidance
  • "If this check fails" subsection with 4 concrete troubleshooting actions: inspect event-study plot for directional pre-trends, check pre-treatment window sparsity, revisit outcome model specification, compare results under alternative priors with PriorSensitivity
  • Pipeline code cell and HTML report iframe

Supersedes #806. Closes #791.

Test plan

  • prek run --all-files passes (verified locally)
  • Notebook schema validation passes
  • ReadTheDocs preview renders the new sections correctly
  • CI notebooks job executes the staggered DiD notebook without errors

Made with Cursor

@review-notebook-app

review-notebook-app Bot commented Apr 15, 2026

Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@drbenvincent drbenvincent mentioned this pull request Apr 15, 2026
Closed
@github-actions github-actions Bot added OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours. documentation Improvements or additions to documentation labels Apr 15, 2026
@read-the-docs-community

read-the-docs-community Bot commented Apr 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Documentation build overview

📚 causalpy | 🛠️ Build #32451687 | 📁 Comparing 72d51c1 against latest (4769e80)

  🔍 Preview build  

43 files changed · + 20 added · ± 23 modified

+ Added

± Modified

@codecov

codecov Bot commented Apr 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.60%. Comparing base (deb8774) to head (72d51c1).
⚠️ Report is 109 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #846   +/-   ##
=======================================
  Coverage   94.60%   94.60%           
=======================================
  Files          80       80           
  Lines       12764    12764           
  Branches      770      770           
=======================================
  Hits        12076    12076           
  Misses        485      485           
  Partials      203      203

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

arthurmello and others added 2 commits April 28, 2026 13:26
@arthurmello @drbenvincent
add walkthrough and result interpretation
e05bf30
@drbenvincent
Address review feedback on sensitivity walkthrough
72d51c1 
- Replace misleading PlaceboInTime reference with clear statement that
  only PreTreatmentPlaceboCheck and PriorSensitivity apply to staggered DiD
- Add "If this check fails" subsection with concrete troubleshooting steps:
  inspect event-study plot for directional pre-trends, check pre-treatment
  window sparsity, revisit outcome model specification, and compare
  results under alternative priors

Made-with: Cursor
@drbenvincent
drbenvincent force-pushed the staggered-did-sensitivity-checks-walkthrough branch from 0f917f5 to 72d51c1 Compare April 28, 2026 12:27
@drbenvincent drbenvincent mentioned this pull request Apr 28, 2026
Open
5 tasks
drbenvincent added a commit that referenced this pull request Apr 30, 2026
@drbenvincent
Docs: add Synthetic Control sensitivity walkthrough
90b8432 
Adds a sensitivity-analysis section to sc_pymc.ipynb that walks through
the pipeline API (EstimateEffect -> SensitivityAnalysis -> GenerateReport)
with PlaceboInTime as the SC default check, and documents the other
SC-applicable checks (ConvexHullCheck, LeaveOneOut, PlaceboInSpace,
PriorSensitivity) with interpretation guidance. Mirrors the structure
introduced for Staggered DiD in #846 and links back to the central guide
from #818.

Includes a runnable pipeline cell and HTML report iframe so the walkthrough
demonstrates an end-to-end placebo-in-time check with concrete pass/fail
narrative and a five-step "if this check fails" troubleshooting block.

Closes #789

Made-with: Cursor
@drbenvincent

drbenvincent commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Automated triage

Recommendation: review:medium — no decision gate identified.

Why:

  • Documentation-only: adds sensitivity walkthrough to staggered_did_pymc.ipynb (28 lines added, 3 deleted).
  • No code changes. All CI checks pass. PR is conflicted (DIRTY).
  • Already has documentation and OSS_PRODUCT labels.

Review focus:

  1. Verify the PreTreatmentPlaceboCheck pipeline cell executes correctly.
  2. Confirm the walkthrough structure matches the established pattern from the central sensitivity guide.
  3. Check the troubleshooting section provides actionable guidance for practitioners.

Confidence: high

@drbenvincent drbenvincent added the review:medium Contained change requiring focused human review label Jul 14, 2026
@derwells derwells added the migration:needs-port Fix not yet on the v6 migration branch; needs a port PR label Aug 10, 2026
@derwells derwells mentioned this pull request Aug 10, 2026
Open
7 tasks
@cetagostini cetagostini mentioned this pull request Aug 18, 2026
Merged
@cetagostini cetagostini mentioned this pull request Sep 18, 2026
Merged
@cetagostini

cetagostini commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Ported to the PyMC 6 migration branch as #1187 — a fresh branch off pymc6_and_pymcmarketing1_migration re-implementing the essence of this PR against the new architecture (lazy experiment lifecycle, renamed hyphen-case notebooks, branch check APIs). No commits cherry-picked. This PR can be closed once #1187 lands with the migration branch.

cetagostini added a commit that referenced this pull request Sep 18, 2026
@cetagostini
Merge pull request #1187 from pymc-labs/migration/846-staggered-sensi…
834dc04 
…tivity-delta

Docs: expand Staggered DiD sensitivity walkthrough (port of #846)
@cetagostini

cetagostini commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

@drbenvincent Heads-up: the essence of this PR has been re-implemented on the PyMC 6 migration branch as #1187 (fresh branch off pymc6_and_pymcmarketing1_migration, adapted to the lazy experiment lifecycle, the renamed hyphen-case notebooks, and the current check/pipeline APIs — no commits cherry-picked). It is green on CI there.

Given that, please avoid merging this PR into main: doing so would land a conflicting, pre-migration version of the same work, and the migration branch would then have to reconcile it again. The content will reach main with the migration branch once #1048 lands. Suggest closing this PR in favour of #1187.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cetagostini cetagostini Awaiting requested review from cetagostini
@juanitorduz juanitorduz Awaiting requested review from juanitorduz

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation migration:needs-port Fix not yet on the v6 migration branch; needs a port PR OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours. review:medium Contained change requiring focused human review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Docs: Add Staggered DiD sensitivity checks walkthrough (follow-up #749)

4 participants

@drbenvincent @cetagostini @derwells @arthurmello