5
$\begingroup$

In this presentation by Gary King, he discusses post treatment bias as follows:

Post treatment bias occurs:

  • when controlling away for the consequences of treatment
  • when causal ordering among predictors is ambiguous or wrong

Example of avoidable post-treatment bias: Causal effect of Race on Salary in a firm

  • DO control for qualifications
  • DON'T control for position in the firm

Example of unavoidable post-treatment bias: Causal effect of democratization on civil war, do we control for GDP?

  • Yes, since GDP -> democratization, we must control to avoid omitted variable bias
  • No, since democratization -> GDP, we would have post treatment bias

I don't understand how this is a bias and not simply a problem of multi-collinearity? In the first example, if blacks tend to get low-ranking position, then yes, Race and Position are highly correlated and leads to high standard error. But why does it lead to bias?

$\endgroup$
  • $\begingroup$ Please clarify your question or accept my answer. As the question and bounty are currently phrased, I directly address the stated question in my answer (especially paragraphs 2 and 7). $\endgroup$ – Dr. Beeblebrox Jul 29 '15 at 14:15
  • $\begingroup$ What I'm looking for is how to show the bias mathematically, perhaps with the potential outcome framework. All the intuition and definition of what is multi-collinearity and bias I'm already clear about. What I hope to get is a formal exposition of this sentence Post-treatment bias is a problem because one of your control variables will mathematically “soak up” some of the effect of your treatment". Would it be possible to formally show the mathematically "soak up" part? $\endgroup$ – Heisenberg Jul 29 '15 at 15:06
  • $\begingroup$ Do you have access to Gelman and Hill, "Data Analysis Using Regression and Multilevel/Hierarchical Models"? The section I suggested, Section 9.7, pages 188-190, contains a formal presentation of bias from controlling for a consequence of treatment. $\endgroup$ – Dr. Beeblebrox Jul 29 '15 at 18:41
  • $\begingroup$ @jabberwocky the Gelman and Hill's section you suggested is very helpful. I have picked your answer. Though I'd suggest that you substantially reframe your answer along Gelman and Hill's explanation. The present answer goes into many unnecessary details (e.g. what is multi-collinearity, which is already known from the question) while missing the key point, i.e. explain post treatment bias formally (e.g. under the potential outcome framework), instead of just intuitively (e.g. "soak up variation") $\endgroup$ – Heisenberg Jul 30 '15 at 6:02
5
$\begingroup$

First, let’s clear up a difference between the terms, and then discuss the respective problems that each causes.

Multi-collinearity refers to a problematic relationship between multiple right-hand-side variables (usually control variables) caused by their being highly correlated, regardless of causal ordering. Post-treatment bias refers to a problematic relationship between your treatment variable and at least one control variable, based on a hypothesized causal ordering. Furthermore, multi-collinearity and Post-treatment bias causes different problems if they are not avoided.

Multi-collinearity generally refers to a high correlation between multiple right-hand-side variables (usually two control variables) in a regression model, which is a problem. If a right-hand-side variable and your outcome variable were highly correlated (conditional on other right-hand-side variables), however, that would not necessarily be a problem; instead it would be suggestive of a strong relationship that might be of interest to the researcher.

Multi-collinearity between control variables does not impact the reliability of a model overall. We can still reliably interpret the coefficient and standard errors on our treatment variable. The negative side of multi-collinearity is that we can no longer interpret the coefficient and standard error on the highly correlated control variables. But if we are being strict in conceiving of our regression model as a notioanal experiment, where we want to estimate the effect of one treatment (T) on one outcome (Y), considering the other variables (X) in our model as controls (and not as estimable quantities of causal interest), then regressing on highly correlated variables is fine.

Another fact that may be thinking about is that if two variables are perfectly multicollinear, then one will be dropped from any regression model that includes them both.

For more, see: See http://en.wikipedia.org/wiki/Multicollinearity

Post-treatment occurs when the regression model includes a consequence of treatment as a control variable, regardless of how highly correlated the consequence-of-treatment control variable is with the treatment. Although generally the severity of post-treatment bias is increasing in the correlation between the treatment and the consequence-of-treatment control variable.

Post-treatment bias is a problem because one of your control variables will mathematically “soak up” some of the effect of your treatment, thus biasing your estimate of the treatment effect. That is, some of the variation in your outcome due to your treatment will be accounted for in the coefficient estimate on the consequence-of-treatment control variable. This is misleading because to estimate the full effect of treatment, you want all of the variation explained by the treatment to be included in the treatment variable's coefficient estimate.

As an example, we want to study the impact of race on salary. Imagine that race affects job position, which in turn affects salary, and the full effect of race on salary is due to the way that race changes people’s job position. That is, other than how race affects job position, there is no effect of race on salary. If we regressed salary on race and controlled for job position, we would (correctly, mathematically speaking) find no relationship between race and salary, conditional on job position.

To highlight how controlling for a consequence of treatment biases your treatment estimate, consider the difference between a researcher interested in the total effect of a treatment versus the direct effect of a treatment. If we want to study the total impact of race on salary we do not care how that effect is mediated. We care about all pathways linking race and salary. We do not want to control for any variable that mediates the effect of race on salary. If we care about only the direct effect of race on salary (although this research question smacks of pre-Darwinian scientific racism), we want to exclude any "mediated" effects from our treatment estimate. So we would want to control for job position, education, social networks, etc. These change the treatment estimate. If our goal is to estimate the direct effect, then control for the consequences of treatment. If our goal is to estimate the total effect, however, controlling for these consequences of treatment biases our treatment estimate.

For more intuition through example, refer to Gelman and Hill (2007) "Data Analysis Using Regression and Multilevel/Hierarchical Models," pages 188-192.

$\endgroup$
  • $\begingroup$ what does "soak up" mean mathematically? How is it different than a high correlation between Race and Position, two right hand sided variables? $\endgroup$ – Heisenberg Jul 26 '15 at 22:57
  • $\begingroup$ @Heisenberg I revised the answer in an attempt to make it clearer. Hopefully it helps. Let me know. $\endgroup$ – Dr. Beeblebrox Jul 27 '15 at 21:48
  • 2
    $\begingroup$ Think of it this way: suppose we have causal effects that operate like this: $X \rightarrow Y \rightarrow Z$. Thus, the effect of $X$ is that it effects $Y$, which then effects $Z$. If you include $Y$ in your regression model, then the estimated effect of $X$ is the average change in $Z$, given a one unit change in $X$ and holding $Y$ constant. But if you leave out $Y$, the estimate effect of $X$ is the average change in $Z$ for a one unit change in $X$ (allowing for $Y$ to change with changes in $X$). $\endgroup$ – Cliff AB Jul 27 '15 at 22:05
  • 3
    $\begingroup$ This is one of the times when thinking of it mathematically can be misleading (i.e. just looking at it mathematically, you will see that $Y$ is higher correlated with $Z$ than $X$ is). Remember that teasing out causal relations is really a logical, rather than mathematical, challenge. All the raw data in the world doesn't tell you whether you have an observational study or experiment, but just knowing where the data comes from does. $\endgroup$ – Cliff AB Jul 27 '15 at 22:12
  • 1
    $\begingroup$ @Heisenberg, I think you should focus on the paragraph 2 ("Multi-collinearity refers to a problematic relationship...") and paragraph 7 ("Post-treatment occurs when the regression model..."). These highlight two differences: (1) multi-collinearity is about correlation and post-treatment bias (PTB) is about hypothesized causal ordering and (2) multi-collinearity biases your estimates of control variables and post-treatment controls bias your treatment effect estimate. NB: Cliff AB introduced a DAG here -- i.e., PTB depends on theorized causal structure (See Judea Pearl on DAGs). $\endgroup$ – Dr. Beeblebrox Jul 28 '15 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.