diff --git a/causal-inference-for-the-brave-and-true/09-Non-Compliance-and-LATE.ipynb b/causal-inference-for-the-brave-and-true/09-Non-Compliance-and-LATE.ipynb index 865f3d8..0046595 100644 --- a/causal-inference-for-the-brave-and-true/09-Non-Compliance-and-LATE.ipynb +++ b/causal-inference-for-the-brave-and-true/09-Non-Compliance-and-LATE.ipynb @@ -154,13 +154,13 @@ "source": [ "Up on the causal graph, you have the push assignment. This is random by design, so nothing is causing it. Then, you have a node for if the push was delivered. Not everyone that was assigned to receive the push did it, so you have non compliance here. More specifically, you have some never takers: those that don't receive the treatment even if assigned to it. You also have reasons to suspect that this non compliance is not simply by chance. Since people with older phones are the ones that don't get the push, you can argue that income is also causing push delivery. The richer the person, the more likely it is that he or she has a nicer phone, which in turn makes it more likely that he or she will receive the push. Finally, you have the outcome variable, in app purchase. Keep in mind that we don't know income, so we can't control it. With this in mind, let's examine what would happen if we simply used the push assigned as the treatment variable and if instead we used push delivered as the treatment. \n", "\n", - "In the first case, we would estimate the causal effect by the following difference in means:\n", + "In the first case, the difference in means identifies the **intention-to-treat (ITT)** effect of assignment. Using $Z$ for `pushAssigned`,\n", "\n", "$\n", - "ATE = E[Y | pushAssigned=1] - E[Y | pushAssigned=0]\n", + "\\tau_{ITT} = E[Y | Z=1] - E[Y | Z=0].\n", "$\n", "\n", - "As we are very aware by now, this is only unbiased estimate for $E[Y_1] - E[Y_0]$ if the bias $E[Y_0|pushAssigned=0] - E[Y_0|pushAssigned=1]$ is zero. Since `pushAssigned` is random, we know that the bias is zero. Does this settle it then? Not quite. You see, if we do this, we are actually answering a different question than the one we intended to. We would be finding the **causal effect of treatment assignment**, not of the treatment itself. But are they different or can we extrapolate the causal effect of treatment assignment to the ATE? In other words, is the causal effect of treatment assignment an unbiased estimate of the ATE? \n", + "Since `pushAssigned` is random, this is an unbiased estimate of the causal effect of assignment: the average difference between the outcome under assignment and under no assignment. But it answers a different question from the one we intended. With non-compliance, assignment $Z$ and treatment received $T$ are not the same intervention, so the ITT need not equal the population ATE of receiving the push.\n", "\n", "As it turns out, it is not. Because of non compliance, the outcome of those assigned to the treatment will be pushed in the direction of the outcome of those assigned to the control. Non compliance flips the treatment unintendedly, making the treatment and control more similar in the outcome. Don't confuse this with similarity on the variables. We want treated and control to be similar on the variables. That will make them comparable. What we don't want is for them to be similar in the outcome if there is indeed a treatment effect.\n", "\n", @@ -168,7 +168,7 @@ "\n", "![img](./data/img/late/always_takers.png)\n", "\n", - "By the same reasoning, never takers will make those assigned to the treatment look a little bit like the untreated, because they don't take the treatment even if assigned to it. In this sense, **the causal effect of treatment assignment is biased towards zero** because non compliance shrinks the detectable impact. Another way to see this is to imagine an extreme case. Let's say that non compliance is really high. Treatment assignment says nothing about treatment received. Treatment received is purely random in this case. In IV language, that would be saying we have a very weak 1st stage. Using `Z` to denote the treatment assignment, we would have that\n", + "By the same reasoning, never takers will make those assigned to the treatment look a little bit like the untreated, because they don't take the treatment even if assigned to it. Non-compliance therefore tends to make the ITT smaller in magnitude than the effect of treatment receipt. This is not bias for the assignment effect; it is a different, diluted estimand. Another way to see this is to imagine an extreme case. Let's say that non-compliance is really high. Treatment assignment says nothing about treatment received. In IV language, that would mean we have no first stage. Using `Z` to denote the treatment assignment, we would have that\n", "\n", "$\n", "E[Y|Z=1] - E[Y|Z=0] = 0\n", @@ -272,16 +272,16 @@ "OK, so we've ruled out using the causal effect of the assignment as a way to estimate the causal effect of the treatment. So, how about just using the treatment received? \n", "\n", "$\n", - "\\mathrm{ATE} = E[Y | \\mathrm{push}=1] - E[Y | \\mathrm{push}=0]\n", + "\\Delta_{naive} = E[Y | \\mathrm{push}=1] - E[Y | \\mathrm{push}=0].\n", "$\n", "\n", "Once again, we need to think if this is biased or not, or if $E[Y_0|\\mathrm{push}=0] = E[Y_0|\\mathrm{push}=1]$. Just by looking at the causal graph above, we know it is not the case. We have that unmeasured confounder, income, lurking around, and it is sure to mess things up. As we've said earlier, we know that failure in push delivery, in our case, is caused by customers having older phones. Which means that we probably have $E[Y_0|\\mathrm{push}=0] < E[Y_0|\\mathrm{push}=1]$. We think this is the case because customers with less money both have older phones which will lead to $\\mathrm{push}=0$ and also a lower potential in app purchase $Y_0$. \n", "\n", - "Bummer! We can't use treatment assigned nor treatment received to estimate our ATE. But fortunately, we know what we can use: Instrumental Variables. Here, treatment assigned is the perfect instrument for the treatment. It is as good as random and it only causes in-app purchase by the virtue of the treatment.\n", + "Bummer! Random assignment identifies the ITT, while the naive comparison by treatment receipt is confounded; neither contrast is generally the population ATE of receiving the push. But the assignment can serve as an instrument for treatment receipt. Under the IV assumptions below, it identifies a local average treatment effect for compliers.\n", "\n", "## Local Average Treatment Effect: LATE\n", "\n", - "Local average treatment effect makes it explicit the population for which we can estimate the causal effect. It is also another way to view IV, which gives other cool intuitions we can use. In modern IV, we see the instrument as starting a causal chain: Z causes T which causes Y. In this context, the exclusion restriction means that Z does not cause Y, unless by its effect on T. The 1st stage is now viewed as the causal effect of Z on T. We also rewrite potential outcomes with a double indexing notation, where the first index denotes the counterfactual for the instrument and the second index, for the treatment\n", + "Local average treatment effect makes it explicit the population for which we can estimate the causal effect. It is also another way to view IV, which gives other cool intuitions we can use. In modern IV, we see the instrument as starting a causal chain: Z causes T which causes Y. In this context, the exclusion restriction means that Z does not cause Y, unless by its effect on T. The 1st stage is now viewed as the causal effect of Z on T. We also rewrite potential outcomes with a double indexing notation, where the first index denotes the treatment level and the second index denotes the instrument assignment\n", "\n", "$\n", "\\text{Potential Outcome}=\\begin{cases}\n", @@ -305,18 +305,18 @@ "\n", "Instrumental Variable assumptions can now be rewritten as follows\n", "\n", - "1. $T_{0i}, T_{1i} \\perp Z_i $ and $Y_i(T_{1i},1), Y_i(T_{0i},0) \\perp Z_i $. This is the Independence Assumption. This says that the instrument is as good as randomly assigned. In other words, Z, the instrument, is not correlated with the potential treatments, which is the same as saying that people in different instrument groups are comparable. \n", + "1. $(Y_i(0,0),Y_i(0,1),Y_i(1,0),Y_i(1,1),T_i(0),T_i(1)) \\perp Z_i$. This is the independence assumption: the instrument is as good as randomly assigned jointly with respect to all potential outcomes and potential treatment states. Here $T_i(z)$ is the treatment that unit $i$ would receive under instrument value $z$, also written $T_{iz}$. Joint independence is what lets us remove conditioning on $Z$ from products involving both potential treatments and potential outcomes in the Wald derivation. \n", "\n", "2. $Y_i(1, 0)=Y_i(1, 1)=Y_{i1}$ and $Y_i(0, 0)=Y_i(0, 1)=Y_{i0}$. This is the exclusion restriction. It says that if I'm looking at the potential outcome for the treated, it is the same for both instrument groups. In other words, the instrument does not affect the potential outcome, which is the same as saying that the instrument only affects the outcome through the treatment.\n", "\n", - "3. $E[T_{1i}-T_{0i}] \\neq 0$. This is the existence of a 1st stage. It is saying that the potential outcome of the 1st stage, that is, the potential treatment, is NOT the same. Another way of saying this is that the instrument does affect the treatment.\n", + "3. $E[T_i(1)-T_i(0)] \\neq 0$. This is the existence of a first stage: the instrument changes the average potential treatment received.\n", "\n", "4. $T_{i1} \\geq T_{i0}$. This is the monotonicity assumption. It is saying that if everyone had the instrument turned on, the treatment level would be equal or higher than if everyone had the instrument turned off. \n", "\n", "Now, let's review the Wald estimator to gain some further intuition on IV:\n", "\n", "$\n", - "ATE = \\dfrac{E[Y|Z=1]-E[Y|Z=0]}{E[T|Z=1]-E[T|Z=0]}\n", + "\\tau_{LATE} = \\dfrac{E[Y|Z=1]-E[Y|Z=0]}{E[T|Z=1]-E[T|Z=0]}.\n", "$\n", "\n", "Let's take the first bit of it, $E[Y|Z=1]$. Using the exclusion restriction, we can rewrite Y in terms of potential outcome like this.\n", @@ -358,16 +358,16 @@ "All of this allows us to look at the Wald estimator like this:\n", "\n", "$\n", - "ATE = \\dfrac{E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})}{P(T_{i1}>T_{i0})}=E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]\n", + "\\tau_{LATE} = \\dfrac{E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}]P(T_{i1}>T_{i0})}{P(T_{i1}>T_{i0})}=E[(Y_{i1}-Y_{i0})|T_{i1}>T_{i0}].\n", "$\n", "\n", - "That is, the ATE as estimated by IV is the ATE on the subpopulation where $T_{i1}>T_{i0}$. And if you think about compliance, which population is this? This is the population where those that get the instrument turned on have the treatment level higher than if they had the instrument turned off. In other words, this is the complier population. Just so we can remember,\n", + "That is, the Wald estimand is the average treatment effect for the subpopulation where $T_{i1}>T_{i0}$, not the population ATE. And if you think about compliance, which population is this? It is the population whose treatment level is higher with the instrument turned on than with it turned off: the compliers. Just so we can remember,\n", "\n", "1. Compliers means that $T_{i1}>T_{i0}$\n", "2. Never Takers $T_{i1}=T_{i0}=0$\n", "3. Always Takers $T_{i1}=T_{i0}=1$\n", "\n", - "The conclusion of this is that IV says nothing about the effect on never takers, always takers or defiers, because the treatment is unchanged for them! **IV only finds the treatment effect for the compliers**.\n", + "Monotonicity rules out defiers, whose treatment would move in the opposite direction from the instrument. The Wald estimand does not identify effects for never takers or always takers, whose treatment is unchanged by the instrument. **Under the assumptions above, IV identifies the average treatment effect for compliers**.\n", "\n", "## Effect on Engagement\n", "\n", @@ -464,7 +464,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "First, let's run OLS to see what it would give us." + "First, let's run OLS to estimate the unadjusted delivered-versus-undelivered difference described above. Since the formula contains no bracketed endogenous/instrument block, `IV2SLS.from_formula` fits ordinary least squares here; using the same class keeps the OLS and IV output formats consistent." ] }, { @@ -478,19 +478,29 @@ "\n", "\n", "\n", - " \n", - "\n", - "\n", - " \n", + " \n", "\n", "\n", - " \n", + " \n", "\n", "\n", - " \n", + " \n", "\n", "
Parameter Estimates
Parameter Std. Err. T-stat P-value Lower CI Upper CI
Intercept 69.292 0.3624 191.22 0.0000 68.581 70.002 Parameter Std. Err. T-stat P-value Lower CI Upper CI
push_assigned -17.441 0.5702 -30.590 0.0000 -18.559 -16.324Intercept 65.519 0.3126 209.61 0.0000 64.906 66.131
push_delivered 27.600 0.6124 45.069 0.0000 26.399 28.800push_delivered 13.931 0.5282 26.377 0.0000 12.896 14.967
" ], + "text/latex": [ + "\\begin{center}\n", + "\\begin{tabular}{lcccccc}\n", + "\\toprule\n", + " & \\textbf{Parameter} & \\textbf{Std. Err.} & \\textbf{T-stat} & \\textbf{P-value} & \\textbf{Lower CI} & \\textbf{Upper CI} \\\\\n", + "\\midrule\n", + "\\textbf{Intercept} & 65.519 & 0.3126 & 209.61 & 0.0000 & 64.906 & 66.131 \\\\\n", + "\\textbf{push\\_delivered} & 13.931 & 0.5282 & 26.377 & 0.0000 & 12.896 & 14.967 \\\\\n", + "\\bottomrule\n", + "\\end{tabular}\n", + "%\\caption{Parameter Estimates}\n", + "\\end{center}" + ], "text/plain": [ "" ] @@ -501,7 +511,7 @@ } ], "source": [ - "ols = IV2SLS.from_formula(\"in_app_purchase ~ 1 + push_assigned + push_delivered\", data).fit()\n", + "ols = IV2SLS.from_formula(\"in_app_purchase ~ 1 + push_delivered\", data).fit()\n", "ols.summary.tables[1]" ] }, @@ -509,7 +519,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "OLS is saying that the treatment effect is BRL 27.60, that is, the push increases in-app purchase by 27.6 reais. However, we have reasons to believe this is a biased estimate. We know that older phones are having trouble in receiving the push, so, probably, richer customers, with newer phones, are the compliers. Since the ones that get the treatment also have more money, we believe this bias is positive and the true impact of the push is lower. In other words, we probably have $E[Y_0|T=0] < E[Y_0|T=1]$.\n", + "The unadjusted OLS coefficient is BRL 13.93: customers who received the push spent 13.93 reais more, on average, than customers who did not. This is an association, not yet a causal effect. We know that older phones have trouble receiving the push, so richer customers with newer phones are probably more likely to receive it. Since treatment receipt is associated with income, we expect positive confounding and a smaller true impact of the push. In other words, we probably have $E[Y_0|T=0] < E[Y_0|T=1]$.\n", "\n", "Now, let's try to estimate this effect with Instrumental Variables. First, let's run the first stage." ] @@ -641,11 +651,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This shows that the result with 2SLS is much lower than the one we got with OLS: 3.29 against 27.60. This makes sense, since the causal effect estimated with OLS is positively biased. We also need to remember about LATE. 3.29 is the average causal effect on compliers. Unfortunately, we can't say anything about those never takers. This means that we are estimating the effect on the richer segment of the population that have newer phones.\n", + "This shows that the result with 2SLS is much lower than the unadjusted association we got with OLS: 3.29 against 13.93. This makes sense given the positive confounding we expected in the OLS comparison. We also need to remember the estimand: 3.29 is the average causal effect for compliers, not the population ATE. It says nothing about the effect for never takers. In this application, compliers are customers whose push-delivery status responds to assignment, so they are likely to have newer phones than never takers; the estimate need not generalize to customers whose phones cannot receive the push.\n", "\n", "## Key Ideas\n", "\n", - "Here we've looked at a more modern view of Instrumental Variables. We saw how IV can be seen as a causal chain where the instrument causes the treatment which causes the outcome. In this view, we took a look at compliance to understand ATE in IV estimation and we've figured out that it was the LATE on compliers.\n", + "Here we've looked at a more modern view of Instrumental Variables. We saw how IV can be viewed as a causal chain where the instrument causes the treatment, which causes the outcome. Compliance makes the population explicit: under independence, exclusion, a first stage, and monotonicity, the Wald estimand is the LATE for compliers rather than the population ATE.\n", "\n", "## References\n", "\n",