Suppose we have two independent variables and one response variable, we can fit three different models:

\begin{align*} Y &= \beta_{00} + \beta_{10} X_1 + \epsilon \\ Y &= \beta_{01} + \beta_{20} X_2 + \epsilon \\ Y &= \beta_{02} + \beta_{11} X_1 + \beta_{21} X_2 + \epsilon \end{align*}

When $X_1$ and $X_2$ are correlated, we know the estimates $b_{10}$ will not be equal to $b_{11}$, and similarly $b_{20}$ will not be equal to $b_{21}$. But what exactly are the relationships between these four estimates? What about the p-values? Particularly, if $X_1$ is highly correlated with $X_2$, by experience and intuition I tend to see $b_{10}$ and $b_{20}$ and their p-values quite close to each other. Is there a formula to support this observation?


When you omitt $X_2$ from your first regression, your estimate of the coefficient $\beta_{10}$ will be subject to the usual omitted variable bias: $$plim\: \widehat{\beta}_{10} = \beta_{11} + \beta_{21}\frac{Cov(X_1,X_2)}{Var(X_1)} $$ and the only way that this bias vanishes is if $X_2$ is not useful in predicting $Y$, i.e. $\beta_{21} = 0$ in regression 3, or $Cov(X_1,X_2)=0$. The same reasoning holds for the bias in the second regression.
The derivation of the omitted variable bias formula is as follows. For ease of notation let me write the full model and the model with $X_2$ missing as $$\begin{align} Y &= \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon \\ Y &= \beta_0 + \beta_1 + u \end{align}$$

In your regression with $X_2$ missing, you estimate $$ \widehat{\beta}_{1} = \frac{Cov(X_1,Y)}{Var(X_1)} = \frac{\sum^N_{i=1}(X_{1i}-\overline{X}_1)(Y_i-\overline{Y})}{\sum^N_{i=1}(X_{1i}-\overline{X}_1)^2} $$ where we now substitute $Y_i$ and $\overline{Y}_i$ with $$\begin{align}Y_i &= \beta_{0}+\beta_{1}X_{1i} + u_i \\ \overline{Y}_i &= \beta_{0}+\beta_{1}\overline{X}_{1i}+\overline{u}_i \end{align}$$ and $u_i = \beta_2 X_{2i} + \epsilon_i$, which gives us

$$ \widehat{\beta}_{1} = \frac{\sum^N_{i=1}(X_{1i}-\overline{X}_1)([\beta_{0}+\beta_{1}X_{1i} + \beta_2 X_{2i} + \epsilon_i]-[\beta_{0}+\beta_{1}\overline{X}_{1}+\beta_2 \overline{X}_2 + \overline{\epsilon}])}{\sum^N_{i=1}(X_{1i}-\overline{X}_1)^2}. $$ Now multiply out and collect terms $$ \begin{align} \widehat{\beta}_{1} &= \frac{\beta_1 \sum^N_{i=1}(X_{1i}-\overline{X}_1)^2 + \beta_2 \sum^N_{i=1}(X_{1i}-\overline{X}_1)(X_{2i}-\overline{X}_2) + \sum^N_{i=1}(X_{1i}-\overline{X}_1)(\epsilon_{i}-\overline{\epsilon})}{\sum^N_{i=1}(X_{1i}-\overline{X}_1)^2} \\ &= \beta_1 + \beta_2 \frac{\sum^N_{i=1}(X_{1i}-\overline{X}_1)(X_{2i}-\overline{X}_2)}{\sum^N_{i=1}(X_{1i}-\overline{X}_1)^2} + \frac{\sum^N_{i=1}(X_{1i}-\overline{X}_1)(\epsilon_{i}-\overline{\epsilon})}{\sum^N_{i=1}(X_{1i}-\overline{X}_1)^2} \\ &= \beta_1 + \beta_2 \frac{Cov(X_1,X_2)}{Var(X_1)} + \frac{Cov(X_1,\epsilon)}{Var(X_1)}. \end{align}$$ Given that $Cov(X_1,\epsilon) = 0$ by assumption, this leaves you with the omitted variable bias formula from above.

As concerns your p-values, in your third regression the estimated variance for, say $X_1$, is $$Var\widehat{\beta}_{X_{1}} = \frac{\sigma^2_3}{NVar(X_1)[1-Corr(X_1,X_2)^2]}$$

When you omit $X_2$ in your first regression the estimated variance is $$Var\widehat{\beta}_{X_{1}} = \frac{\sigma^2_1}{NVar(X_1)}$$

Given that the correlation term with $X_2$ is now missing, the estimate of your standard error is likely to be much lower - this should happen in your case because the correlation between the independent variables is high. The standard errors are then used to construct your t statistic and p-values, so in regressions 1 and 2 you should observe higher significance. In some instances the standard error can be higher though: this happens if $Corr(X_1,X_2)$ is low and $X_2$ is a strong predictor for $Y$ such that the regression error in 3 ($\sigma^2_3$) is much smaller than $\sigma^2_1$.

With regards to your observation that the p-values in regression 1 and 2 are similar, this should just be due to chance. If you compare the variance estimates of the two regressions $$Var\widehat{\beta}_{X_{1}} = \frac{\sigma^2_1}{NVar(X_1)},\quad Var\widehat{\beta}_{X_{2}} = \frac{\sigma^2_2}{NVar(X_2)}$$ you can see that the relationship depends on the variability of the two variables and their predictive power on the dependent variable (given by $\sigma^2_1$ and $\sigma^2_2$). If the variances are similar and $Corr(Y,X_1)$ is similar to $Corr(Y,X_2)$, then you will get similar p-values as well.

  • $\begingroup$ What does $plim$ mean, and where can I find the proof of this formula? Thanks. $\endgroup$ – qed Aug 31 '13 at 7:38
  • 2
    $\begingroup$ $plim$ is the probability limit (you can find a nice explanation of it on the mathematics site of SE). The proof for the omitted variable bias formula should be in any econometrics textbook, like Dougherty (2007) "Introduction to Econometrics" page 202, or in these lecture slides on page 2. $\endgroup$ – Andy Aug 31 '13 at 7:56
  • $\begingroup$ Hm, these links are quite useful, thanks. Could you please explain this also: $Var\widehat{\beta}_{X_{1}} = \frac{\sigma^2_3}{NVar(X_1)[1-Corr(X_1,X_2)^2]}$ $\endgroup$ – qed Aug 31 '13 at 10:05
  • $\begingroup$ Ah you already looked at them. I just added the proof of the OVB formula to the answer. The formula in your second comment is the usual multicollinearity formula. Normally this is used to show how adding irrelevant variables increases the standard errors but you can also use it for our purpose here. Again this formula should be in most econometrics textbooks or you can have a look at these slides. I hope this helps :) $\endgroup$ – Andy Aug 31 '13 at 10:12

It can be even worse. Suppose $X_{1}$ and $X_{2}$ are not linearly related but still are causally related, so that $X_{1}$ displays an impact on the target $Y$ via $X_{1} \to X_{2} \to Y$.

The measured correlation between $X_{1}$ and $X_{2}$ doesn't have to be large and the regression won't suffer from traditional multicollinearity. But as the sample size grows, you should expect the effect size for $X_{1}$ when controlling for $X_{2}$ to approach zero with high statistical significance.

This means that even if $X_{1}$ has a meaningful total effect on $Y$ (so that $\beta_{10}$ is statistically significant and large) the third regression can appear to say that only $\beta_{21}$ has a meaningful total effect.

This is an example from "Let's Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.