Skip to content

r_python_differences.md

Latest commit

 

History

History
108 lines (75 loc) · 4.71 KB

r_python_differences.md

File metadata and controls

108 lines (75 loc) · 4.71 KB
Raw

R / Python Differences

This document describes the known differences between the Python GeoLift package and the original R GeoLift package.

Deterministic quantities

The following quantities are computed purely algebraically from the data and the synthetic control weights. They do not involve any randomness and should match the R implementation to within numerical precision on identical input data:

Quantity Status
average_att ✅ Matches R (atol ≤ 0.01 on test fixtures)
l2_imbalance ✅ Matches R (atol ≤ 0.01 on test fixtures)
scaled_l2_imbalance ✅ Matches R (atol ≤ 0.001 on test fixtures)
incremental ✅ Matches R (atol ≤ 0.5 on test fixtures)
Synthetic control weights ✅ Matches R
detected_lift ✅ Matches R

If any of these differ substantially from the R output on the same fixture data, that is a bug.

Stochastic quantities — algorithm difference, not just RNG

The p-values, confidence intervals, and downstream market-selection decisions that depend on significance thresholds will differ between R and Python even when both are run with the same numeric seed. This is because the two implementations use fundamentally different inference algorithms.

R implementation: time-shift placebo tests

R GeoLift performs placebo tests by:

  1. Randomly selecting a set of pre-treatment time windows.
  2. Re-running the full synthetic control model with each selected window treated as if it were the actual treatment period.
  3. Collecting the test statistic (e.g., sum of absolute ATT) from each placebo run.
  4. Computing the p-value as the fraction of placebo statistics that are at least as large as the observed statistic.

This approach builds the null distribution from the model's behaviour on real pre-treatment data at different time offsets.

Python implementation: conformal inference

Python GeoLift uses conformal inference:

  1. Fit the synthetic control model once on the full pre-treatment period.
  2. Collect the full vector of residuals (pre + post treatment) from the fitted model.
  3. Randomly permute the residuals many times (IID or block permutation).
  4. Compute the test statistic on each permuted residual vector restricted to the post-treatment window.
  5. Compute the p-value as the fraction of permuted statistics at least as large as the observed statistic.

This approach builds the null distribution from permutations of the fitted residuals.

Why this matters

These are structurally different null distributions. The R placebo approach captures variability from re-fitting the model at different time offsets, while the Python conformal approach captures variability from permuting a single fitted residual vector. Neither is wrong — they are different valid choices with different statistical properties.

Consequence: Do not compare p-values or confidence bounds across R and Python outputs and expect them to match. Differences of 0.05–0.20 in p-value are normal and expected.

Within-Python reproducibility

All stochastic functions in the Python package expose a random_state parameter:

result = geo_lift(
    data=df,
    treatment_locations=["chicago", "portland"],
    treatment_start_time=36,
    treatment_end_time=45,
    model="none",
    random_state=42,       # set this for reproducible p-values / CI
)

With the same random_state, all stochastic outputs (p-values, confidence intervals, market-selection decisions) are exactly reproducible across runs on the same machine and Python version. Deterministic quantities (ATT, L2 imbalance, weights) are always reproducible regardless of random_state.

Known unimplemented features

Feature Status
model="gsyn" (Generalized Synthetic Control) ❌ Raises NotImplementedError. Planned for a future release.
model="best" selects among "none" and "ridge" ✅ Implemented
Block permutation (type="block") ✅ Implemented
IID permutation (type="iid") ✅ Implemented
Multi-cell experiments ✅ Implemented (experimental)

Known numeric comparison (fixture-level)

From the frozen R baseline (tests/vignette_outputs/small_r_numeric_baseline.json) using the small test fixtures:

Single-cell model="none"

Quantity R Python
average_att -50.5305 -50.5305 ✅
l2_imbalance 595.624 595.624 ✅
scaled_l2_imbalance 0.6424 0.6424 ✅
p_value 0.6333 ~0.65 ⚠️ (algorithm differs)

Single-cell model="best"

Quantity R Python
average_att -50.5305 -50.5305 ✅
l2_imbalance 595.624 595.624 ✅
scaled_l2_imbalance 0.6424 0.6424 ✅
p_value 0.7333 ~0.65 ⚠️ (algorithm differs)