diff --git a/causal-inference-for-the-brave-and-true/07-Beyond-Confounders.ipynb b/causal-inference-for-the-brave-and-true/07-Beyond-Confounders.ipynb index 68a7066..50398f7 100644 --- a/causal-inference-for-the-brave-and-true/07-Beyond-Confounders.ipynb +++ b/causal-inference-for-the-brave-and-true/07-Beyond-Confounders.ipynb @@ -1499,23 +1499,25 @@ "E[Y_0|T=0, Agreement=1] > E[Y_0|T=1, Agreement=1]\n", "$\n", "\n", - "The first equation makes it explicit that we think those without the email and the agreement are better than those with the email and without the agreement. That is because, if the treatment has a positive effect, those that didn't make an agreement **even after having received the email** are probably worse in terms of payments compared to those that also didn't do the agreement but also didn't get the extra incentive of the email. As for the second equation, those that did the agreement even without having received the treatment are probably better than those that did the agreement but had the extra incentive of the email. \n", + "The first equation makes it explicit that we think those without the email and without an agreement are better than those with the email and without an agreement. That is because, if the treatment has a positive effect, those that didn't make an agreement **even after having received the email** are probably worse in terms of payments compared to those that also didn't make an agreement but also didn't get the extra incentive of the email. As for the second equation, those that made an agreement even without having received the treatment are probably better than those that made an agreement but had the extra incentive of the email. \n", "\n", "This might be very confusing the first time you read it (it was for me), but make sure you understand it. Read it again if necessary. Then, a similar kind of reasoning can be done with the opened variable. Try to make it yourself.\n", "\n", - "This sort of bias is so pervasive it has its own name. While confounding is the bias from failing to control for a common cause, **selection bias is when we control for a common effect or a variable in between the path from cause to effect.** As a rule of thumb, always include confounders and variables that are good predictors of $Y$ in your model. Always exclude variables that are good predictors of only $T$, mediators between the treatment and outcome or common effect of the treatment and outcome.\n", + "Several distinct post-treatment adjustment problems can appear here. Confounding is bias from failing to adjust for a common cause of treatment and outcome. **Selection or collider bias** arises when we condition on a common effect and thereby open a non-causal path. **Overadjustment** arises when we condition on a mediator and block part of the total causal effect, changing the target from a total effect to a direct effect. In the graph above, `opened` and `agreement` are mediators. They could also act as colliders in a richer graph with additional causes of those variables and payments, but that would be a separate mechanism.\n", + "\n", + "A variable's predictive power alone does not determine whether it is a valid control. Adjust for measured pre-treatment confounders; pre-treatment predictors of the outcome that do not open a path can also improve precision. If the total effect is the target, do not automatically adjust for post-treatment mediators, colliders, or variables defined by the outcome. Choose controls from their causal role, not simply from how well they predict $T$ or $Y$.\n", "\n", "![img](./data/img/beyond-conf/selection.png)\n", "\n", - "Selection bias is so pervasive that not even randomization can fix it. Better yet, it is often introduced by the ill advised, even in random data! Spotting and avoiding selection bias requires more practice than skill. Often, they appear underneath some supposedly clever idea, making it even harder to uncover. Here are some examples of selection biases I've encountered:\n", + "Randomization identifies the total effect before we make these post-treatment adjustments; it does not rescue an analysis after we condition on a mediator, collider, or observed outcome. These mistakes can therefore be introduced even in randomized data. They often hide underneath a supposedly clever modeling choice, which makes them harder to uncover. Here are some post-treatment adjustment mistakes I've encountered:\n", "\n", "1. Adding a dummy for paying the entire debt when trying to estimate the effect of a collections strategy on payments.\n", "2. Controlling for white vs blue collar jobs when trying to estimate the effect of schooling on earnings\n", "3. Controlling for conversion when estimating the impact of interest rates on loan duration\n", "4. Controlling for marital happiness when estimating the impact of children on extramarital affairs\n", - "5. Breaking up payments modeling $E[Payments]$ into one binary model that predicts if payment will happen and another model that predict how much payment will happen given that some will: $E[Payments|Payments>0]*P(Payments>0)$\n", + "5. Reporting the difference in payments among customers who paid as if it were the overall effect of a collections strategy.\n", " \n", - "What is notable about all these ideas is how reasonable they sound. Selection bias often does. Let this be a warning. As a matter of fact, I myself have fallen into the traps above many many times before I learned how bad they were. One in particular, the last one, deserves further explanation because it looks so clever and catches lots of data scientists off guard. It's so pervasive that it has its own name: **The Bad COP**!\n", + "What is notable about all these ideas is how reasonable they sound. Selection bias often does. Let this be a warning. As a matter of fact, I myself have fallen into the traps above many many times before I learned how bad they were. One in particular, the last one, deserves further explanation because the distinction between a valid two-part model and an invalid causal interpretation is subtle. The invalid interpretation has a memorable name: **The Bad COP**!\n", "\n", "### Bad COP\n", "\n", @@ -1561,53 +1563,64 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "When a data scientist sees this, the first idea that pops into his head is to break up modeling into 2 steps. The first is the participation, that is, the probability that $Y > 0$. In our spend example, this would be modeling if the customer decided to spend or not. The second part models $Y$ for those that decided to participate. It is the Conditional-on-Positives effect. In our case, this would be how much the customer spends after he or she decided they would spend anything. If we would like to estimate the effect of the treatment $T$ on expenditures, it would look something this:\n", - " \n", - "$\n", - "E[Y|T] = E[Y|Y>0, T]P(Y>0|T)\n", - "$\n", - " \n", - "There is nothing wrong with the participation model $P(Y_i>0|T_i)$. In fact, if $T$ is randomly assigned, it will capture the increase in probability of spending due to the treatment. Also, there is nothing wrong with the decomposition above. It's mathematically true, by the law of total probabilities.\n", - " \n", - "The issue is in estimating the COP part. **It will be biased even under random assignment**. On an intuitive level, it's not crazy to think that some units are zeros only because they didn't get the treatment. The treatment would cause them to **not** be zeros. On the other hand, some units are never zeros. The treatment might increase their outcome, but even without it, they would not be zeros. Now, the key point is understanding that these two types of units are not comparable. Those that are never zeros have high $Y_0$ compared to those that are zeros if they didn't get the treatment. In fact, for the latter, $Y_0=0$. \n", - " \n", - "Knowing this, if we remove the zeros, we will keep the never zeros on both the treated and the control group. But we would remove from the control those that flip from zero to not zero with the treatment. This would make the treatment and control no longer comparable, because the control would only contain those that are never zeros and that have higher $Y_0$, while the treated would contain both types of units. \n", - " \n", - "Now that we have an intuitive understanding of the problem, let's check it from a mathematical angle. To see it, let's break up the treatment effect. Under random assignment, it is equal to the difference in means\n", - " \n", + "When a data scientist sees this, a natural idea is to model it in two steps. The first part models participation: the probability that $Y>0$. In our spending example, this is the probability that a customer spends anything. The second part models the amount among customers with a positive outcome. This is the **Conditional-on-Positives (COP)** mean. This setup is often called a two-part or hurdle model.\n", + "\n", + "For a nonnegative outcome, define the two quantities at each treatment level $t$ as\n", + "\n", + "$$\n", + "p_t = P(Y>0\\mid T=t), \\qquad m_t = E[Y\\mid Y>0,T=t].\n", "$$\n", - "\\begin{align*} \n", - "&E[Y|T=1] - E[Y|T=0]\\\\\n", - "&=E[Y|Y>0, T=1]P(Y>0|T=1) - E[Y|Y>0, T=0]P(Y>0|T=0)\\\\\n", - "&=\\underbrace{\\{P(Y>0|T=1) - P(Y>0|T=0)\\}}_{Participation \\ Effect} * E[Y|Y>0, T=1]\\\\\n", - "&+\\underbrace{\\{E[Y|Y>0, T=1] - E[Y|Y>0, T=0]\\}}_{COP \\ Effect} * P(Y>0|T=0)\n", - "\\end{align*} \n", + "\n", + "Because observations with $Y=0$ contribute zero to the mean,\n", + "\n", "$$\n", - " \n", - "Where the last equality comes from adding and subtracting $E[Y_i|Y_i>0, T_i=1]P(Y_i>0|T_i=0)$ and rearranging the terms. This means that the difference in averages is composed of two parts: first, it's the difference in the probability that the outcome $y$ is positive. This is called the participation effect because it measures the increase in the probability that customers will participate in spending. Second, it's the difference in the outcome conditioned on participation, the COP effect. So far so good. There is nothing wrong with this. It is a mathematical truth. The problem comes when we try to estimate each part separately\n", - " \n", - "This becomes more evident if we analyse the COP effect even further. \n", - " \n", + "E[Y\\mid T=t]=p_t m_t.\n", "$$\n", - "\\begin{align*} \n", - "E[Y|Y>0, T=1] - E[Y|Y>0, T=0]&=E[Y_{1}|Y_{1}>0]-E[Y_{0}|Y_{0}>0] \\\\\n", - "&=\\underbrace{E[Y_{1} - Y_{0}|Y_{1}>0]}_{Causal \\ Effect} + \\underbrace{\\{ E[Y_{0}|Y_{1}>0] - E[Y_{0}|Y_{0}>0] \\}}_{Selection \\ Bias}\n", - "\\end{align*} \n", + "\n", + "This identity is exact. If treatment is randomly assigned, consistency and randomization let us write $p_t=P(Y_t>0)$ and $m_t=E[Y_t\\mid Y_t>0]$. Both $p_t$ and $m_t$ can be estimated from the observed treatment arm. Recombining them gives a valid estimate of the marginal treatment effect:\n", + "\n", + "$$\n", + "ATE=p_1m_1-p_0m_0.\n", + "$$\n", + "\n", + "So there is nothing inherently bad about fitting a participation model and a positive-amount model. The problem appears when we interpret $m_1-m_0$ as a causal effect on the amount for one common population, or use that contrast in place of the ATE. Random assignment makes the treatment arms comparable *before* we condition on the outcome. It does not make units selected by the post-treatment event $Y>0$ comparable.\n", + "\n", + "The distinction is easy to see by adding and subtracting $E[Y_0\\mid Y_1>0]$:\n", + "\n", "$$\n", - " \n", - "where the second equality comes after we add and subtract $E[Y_{i0}|Y_{i1}>0]$. When we break up the COP effect, we get first the causal effect on the participant subpopulation. In our example, this would be the causal effect on those that decide to spend something. Second, we get a bias term which is the difference in $Y_0$ for those that decide to participate when assigned to the treatment ($E[Y_{i0}|Y_{i1}>0]$) and those that would participate even without the treatment ($E[Y_{i0}|Y_{i0}>0]$). In our case, this bias is probably negative, since those that spend when assigned to the treatment, had they not received the treatment, would probably spend less than those that spend even without the treatment $E[Y_{i0}|Y_{i1}>0] < E[Y_{i0}|Y_{i0}>0]$.\n", - " \n", + "\\begin{align*}\n", + "m_1-m_0\n", + "&=E[Y_1\\mid Y_1>0]-E[Y_0\\mid Y_0>0]\\\\\n", + "&=\\underbrace{E[Y_1-Y_0\\mid Y_1>0]}_{\\text{effect among units positive under treatment}}\n", + "+\\underbrace{\\left(E[Y_0\\mid Y_1>0]-E[Y_0\\mid Y_0>0]\\right)}_{\\text{selection term}}.\n", + "\\end{align*}\n", + "$$\n", + "\n", + "The first term is a causal quantity for the population that would be positive under treatment, but the observed COP contrast is not generally equal to it. The second term compares the untreated potential outcome in two different populations and is not removed by random assignment. Its value and even its sign require assumptions beyond randomization.\n", + "\n", + "A COP-only analysis also misses the participation margin. One exact way to display the full ATE is\n", + "\n", + "$$\n", + "\\underbrace{p_1m_1-p_0m_0}_{ATE}\n", + "=\\underbrace{p_1(m_1-m_0)}_{\\text{weighted COP contrast}}\n", + "+\\underbrace{m_0(p_1-p_0)}_{\\text{participation margin}}.\n", + "$$\n", + "\n", + "The unweighted contrast $m_1-m_0$ is not even on the marginal-mean scale. Weighting it by $p_1$ puts it on that scale, but $p_1(m_1-m_0)$ still drops the participation margin $m_0(p_1-p_0)$. This algebraic split is useful for description, but it is not a unique decomposition into two causal mechanisms: we could instead add and subtract $p_0m_1$ and obtain different weights. The unambiguous marginal effect is the recombined quantity $p_1m_1-p_0m_0$.\n", + "\n", "![img](./data/img/beyond-conf/cop.png)\n", - " \n", - "Now, I know that COP bias is super counterintuitive at first, so I think it is worth going through a visual example. Let's say that we want to estimate how a marketing campaign increases how much people spend on our product. This marketing campaign has been randomized, so we don't need to worry about confounding. In this example, we can break up the customers into two segments. First, there are those that will only buy our products if they see a marketing campaign. Let's call these customers the frugal ones. They don't spend unless we give them an extra push. Then there are the customers that will spend even without the campaign. The campaign makes them spend more, but they would already spend without seeing it anyway. Let's call them the rich customers. In the figure, I've displayed the counterfactuals in light colors and dashed lines. \n", - " \n", + "\n", + "Now let's make the selection problem concrete. Suppose a randomized marketing campaign can increase how much people spend. The figures use a simple case with two customer segments. The **always-positive** customers spend with or without the campaign. The **treatment-induced** customers spend only if they receive the campaign. In potential-outcome notation, let $D_t=1(Y_t>0)$. This two-segment story assumes participation monotonicity, $D_1\\geq D_0$: the campaign can induce spending but never prevents it. In the figure, counterfactual outcomes appear in light colors and dashed lines.\n", + "\n", "![img](./data/img/beyond-conf/cop-ex1.png)\n", - " \n", - "To estimate the ATE of the campaign, since we have randomization, all we need to do is compare the treated with the untreated. But, suppose we use the COP formulation where we break up estimation into two models, a participation model that estimates $P(Y_i>0|T_i)$ and the COP, which estimates $E[Y_i|Y_i>0]$. This removes everyone that didn't spend from the analysis. \n", - " \n", + "\n", + "Among observed control positives, $Y>0$ selects only customers with $D_0=1$, the always-positive segment. Among observed treated positives, it selects customers with $D_1=1$, which includes both the always-positive and treatment-induced segments. In particular, treatment-induced customers are absent from the control-positive sample because their observed control outcome is zero, even though they **would have had** a positive outcome if they had received treatment. The two selected samples therefore represent different populations.\n", + "\n", "![img](./data/img/beyond-conf/cop-ex2.png)\n", - " \n", - "When we do that, the treated and control are no longer comparable. As we can see, the untreated is now only composed of the segment of customers that will spend even without the campaign. Also notice that we can even know the direction of the bias here. It will be $E[Y_{i0}|Y_{i1}>0] - E[Y_{i0}|Y_{i0}>0]$ or $E[Y_{i0}|\\text{Frugal and Rich}] - E[Y_{i0}|Rich]$. This is obviously negative, as the rich spend more than the frugal customers. As a result, once we filter only the participant population, our estimate of the ATE becomes biased, even if there was no bias at first due to randomization. I sincerely hope this convinces you to avoid COP like the plague. I see too many Data Scientists doing this separate estimation, unaware of the problems that it brings. " + "\n", + "The picture suggests a negative selection term because the treatment-induced customers would spend less without the campaign than the always-positive customers. That direction is a feature of the example, not a consequence of randomization. Without monotonicity there could also be customers who spend only under control, making the population comparison still more complicated.\n", + "\n", + "The lesson is not to avoid two-part models. They can be very useful for outcomes with many zeros, and estimating $p_t$ and $m_t$ separately is valid when we recombine them to target $E[Y_t]=p_tm_t$. The bad COP move is to condition on the observed positive outcome and then interpret the positive-part contrast as an ATE or as an effect for a fixed group. Estimating an effect for the always-positive population requires extra assumptions and methods for principal strata; ordinary conditioning on $Y>0$ does not identify it." ] }, {