I was reading the Rubin: Causal inference and Angrist, J.D. and Pischke: most harmless econometrics. Both of them are great textbooks. During my reading, I have the following question: what is the correct interpretation of the coefficient of a covariate variable (rather than the treatment variable) in a multiple linear regression model?

Let's look at a simple random experiment setting. Let $D_i$ be the treatment variable of subject $i$, with $D_i=1$ means treatment and $D_i=0$ means control. Let $X_i$ be a covariate variable, e.g., age. Let $Y_i(1)$ be the value of the response variable if subject $i$ is in treatment group, i.e., $D_i=1$ and similarly, we can define $Y_i(0)$. Then we write $Y^{obs}_i=D_iY_i(1)+(1-D_i)Y_i(0)$, which is the observed value of the response variable for subject $i$. So we are working under the Rubin potential outcome framework.

As the data comes from a simple random experiment, one could directly run a linear regression without including the covariate variable like this \begin{equation} Y^{obs}_i=\alpha+\tau D_i+\epsilon_i. \end{equation} One could show that $\tau=E[Y^{obs}_i|D_i=1]-E[Y^{obs}_i|D_i=0]$. Note that this is always true, even if the data does not come from random experiment. So $\tau$ means: if you could draw infinitely number of samples from the joint distribution of $(D_i,Y^{obs}_i)$, then you obtain the average in the observed response variable for the group of people with $D=1$ and the group of people with $D=0$, then you take the difference, that's $\tau$. Now if the data does come from random experiment, then we further have $\tau=E[Y_i(1)-Y_i(0)]$, which is the treatment effect. Whether the data comes from random experiment or not, the OLS estimator $\hat{\tau}$ is unbiased and consistent of $\tau$.

Now, we know that we could include the covariate $X_i$ in our regression and get \begin{equation} Y^{obs}_i=\alpha'+\tau D_i+\beta X_i+\epsilon_i. \end{equation} Then on page 122 of Rubin: Causal Inference says "...irrespective of whether the regression function is truly linear in the covariates in the population, the OLS $\hat{\tau}$ is consistent for $\tau$". I understand that. But the book never say anything about $\beta$, the coefficient of the covariate $X_i$. In practice, if someone tell you $\beta$ is negative and significant, what exactly does that mean? Think about it, what if $Y$ is the wage and $D$ is whether the subject attends a specific training program. Now if $\beta$ is negative and significant, can we say there is a discrimination against age? What is the correct meaning of $\beta$? What if the data does not come from random experiment, then will there be anything different for $\beta$? just like the difference in the interpretation of $\tau$?


In the statistical sense of this (regression) model, there is no difference between treatment $D_i$ and covariate $X_i$. Aside from the type of variable (continuous/categorical) they are both predictors/independent variables (this would also apply when treatment $D_i$ was continuous, or covariate $X_i$ categorical). Moreover, 'statistically' speaking, everything which you can infer from $τ$ applies to $β$ as well.

Now comes the less statistical part, and a more methodological one: along the theory or hypothesis you are studying, these variables are not equal. One may be of particular interest. Especially when trying to make causal inferences, you want to obtain an 'as pure as possible' notion of its effect on the outcome of interest and (if a frequentist) its significance. That is why you correct/adjust for the effects of other variables (often called confounders; correcting for confounding bias). Now the model needs to be focused around correcting for other variables which can confound the association of interest. If done correctly, you might get a good estimate of the (approximately) unbiased 'true' association of treatment $D_i$ on the outcome. However, you've only selected confounders related to treatment $D_i$'s effect on outcome. You might have omitted some confounders for covariate $X_i$ from the model, because you did not expect them to influence the association between $D_i$ and outcome.

Because of this, causal inferences based covariate $X_i$'s $β$ are not completely corrected for (AKA still biased by confounding).

If in your example the training program is only confounded by age (because we have some theory about this), causal inference for $D_i$'s effect on wages becomes possible. For age however, treatment $D_i$ might not be the only confounder, ergo*, effect estimate $β$ might not be 'pure' and would not be an unbiased effect estimate for the effect of covariate $X_i$/age on wages.

*(always wanted to use that word)

  • $\begingroup$ I have a question about your 1st paragraph. I know that in multiple linear regression, when we use OLS, we get $\beta_k=\frac{Cov(Y_i,\tilde{x}_{ki})}{V(\tilde{x}_{ki})}$, where $\tilde{x}_{ki}$ is the residual from a regression of $x_{ki}$ on all the other covariates. I know these mathematically. But still, how should we explain $\beta_k$ in plain English? I think it is tempting to (but it'll be wrong) explain $\beta_X$ as: if you fix all other regressors, then if you increase X by one unit, then the Y increase by $\beta_X$ on average. Here we don't talk about potential outcome, am I right? $\endgroup$ – KevinKim Aug 29 '17 at 13:53
  • $\begingroup$ I feel that I understand your last 2 paragraphs and I believe they are the essence of this problem. But I'm not 100% sure about my understanding. Could you please guide me to some reference? $\endgroup$ – KevinKim Aug 29 '17 at 13:59
  • $\begingroup$ As to your first comment: "if you fix all other regressors, then if you increase $X$ by one unit, then the Y is expected to increase by $β_X$ on average." is correct, but you should note that your statistical model will not differentiate between an effect apparent in your data, and a true causal effect. As the researcher you are the only one who can make this distinction, which should be based on solid reasoning and, in this case, inclusion of relevant confounding variables in the model. Statistics cannot do this for you. $\endgroup$ – IWS Aug 29 '17 at 14:02
  • $\begingroup$ As to references, I do not have one at hand which I can quickly give you, but the wiki (en.wikipedia.org/wiki/Confounding) is quite OK for the basics and study guides in the field of Epidemiology might help (I'm from the biomedical field where epidemiology is our study of methodology, but the basics of epidemiology can be used in, and originate from, a lot of similar fields of statistics-heavy science) $\endgroup$ – IWS Aug 29 '17 at 14:06
  • $\begingroup$ A small question. In your answer the last sentence "effect estimate $\tau$ might not be 'pure' ...". Are you trying to say "$\beta$" instead of $\tau$? $\endgroup$ – KevinKim Aug 29 '17 at 14:37

The plain English interpretation of the $b_{age}$ is as usual: the expected wage goes down by $b_{age}$, whenever there is a unit increase in Age. In the context of a correlation study this could be a finding in itself.

In the context of ANCOVA, however, it is used to rule out (control) for the effect of Age on the outcome, when the primary interest is in the causal effect of the treatment. (This assumes that the slope of Age is the same across all levels of D).

In real world terms this means that: You have shown that there are differences in means between groups that received different treatment, but how can you respond to the possible objection that a significant predictor of the outcome is not equal across the levels of the treatment variable, thus the finding is eventually due to this factor, not present in the model? In this case, if you suspect that Age has a significant effect on Wage and you have not accounted for it, how can you tell that the differences in Wage across the levels of D are eventually not due to the fact that subjects in different training programmes differ in Age?

ANCOVA addresses this, by controlling for the effect of Age. The intended interpretation here is that $b_{D}$ is now controlled for Age, thus this particular objection is taken care of. (Of course there is no way to know which other covariate would have a linear relationship to the outcome -- the problem of omitted variables.)

See Chapter 11 of Field, Miles and Field (2012) Discovering Statistics Using R, where ANCOVA is explained very clearly.

  • $\begingroup$ In your 1st paragraph, I agree that if E[Y|D,X] is linear in D and X, then the interpretation of $b_{age}$ is what you said. But what if E[Y|D,X] is not linear in D and X (which is often happening in reality), then you can still mechanically apply OLS and get those coefficients, then can we still interpret this $b_{age}$ same as before? $\endgroup$ – KevinKim Aug 29 '17 at 15:31
  • $\begingroup$ For your 3rd paragraph, if the data come from random experiment, then we could just run a short regression without including the X, theoretically right? Now if the data is from observational study, then we need to add this X into the regression. But here comes my ultimate question about observational study using multiple linear regression: why we use a linear model? based on what theory that justifies the linearity?Why by adding a covariate in to the linear regression, people can claim "I controlled this confounding factor". In my opinion, the word "control" is reserved for experiment study $\endgroup$ – KevinKim Aug 29 '17 at 15:43
  • $\begingroup$ If you feel like it see Faraway (2005) Linear Models in R, p. 26, ex. 3. It is possible to capture curvature by using polynomials in the RHS, like $Age^2,\ Age^3$ and examine the fit. The interpretation should hold, i.e. the effect of Age will be controlled for even though the effect is quadratic, cubic etc. $\endgroup$ – nijk Aug 29 '17 at 15:52
  • $\begingroup$ Perhaps you are right about the word 'control'. It is conventional to use it in the way I explained above. Whereas 'manipulating' has the meaning you give to 'control', i.e. that the experimenter has control over the independent variable. Otherwise the linearity is an assumption for using linear models. After some degree you cannot do well enough with polynomials if the data depart severely from the assumption. In such cases you will have to use other models. $\endgroup$ – nijk Aug 29 '17 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.