Say I have model 1 and model 2.
y1 = b0 + b1x1 + u
y2 = b0 + b1x1 + b2x2 + u
If there is an increase in the standard error of x1 from model 1 to model 2, does this suggest collinearity between the x1 and x2 in model 2?
Thanks in advance,
Jona
3
-
$\begingroup$ Do you mean the standard error of the (least-squares) estimate of b1? If so then I think you're right. If not then I'm not sure how to make sense of your question. $\endgroup$– Ruben van BergenCommented Feb 22, 2015 at 20:43
-
$\begingroup$ Can you clarify if you are using linear regression on a continuous outcome (via OLS or MLE)? $\endgroup$– robin.datadriversCommented Feb 23, 2015 at 1:35
-
$\begingroup$ Yes, linear regression, OLS, and I do mean the SE of the estimate of b1. Does it increase because the second covariate is capturing the same variability in the dependent variable as the first covariate? $\endgroup$– JonaCommented Feb 23, 2015 at 4:54
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You clarified that you meant the SE of b1, in which case I'm pretty sure you're right. That is, I can't see another reason, all else being equal, why the SE of b1 would increase, other than collinearity between x1 and x2. (It's a bit confusing in this respect that you labelled your outcome variables "y1" and "y2" for the different models - I assume you actually mean the same data in both instances.)
There are different ways of seeing the reason for this, but I think what you said in your last comment pretty much captures the intuition behind it. If two regressors could each explain the same variability, then you don't know which of them is actually responsible for doing so. This increases the range of possible values that b1 could take on, as you can compensate changes in b1 with changes in b2 in the opposite direction (to some extent).
Mathematically, you can see it from the following equation:
$$ SE(\beta_j) = \sqrt{\frac{\sigma^2}{\Sigma_i(X_{ij}-\overline{X_j})^2 - \Sigma_{l\neq{j}}\frac{(\Sigma_i(X_{ij}-\overline{X_j})(X_{il}-\overline{X_l}))^2}{(X_{il}-\overline{X_l})^2}}} $$
Where $X$ is the design matrix (with columns x1 and x2, in your case), $i$ indexes observations and $\sigma^2$ is the variance of the residuals. The standard error on the left hand side can only increase if $\sigma^2$ grows or the denominator shrinks. The denominator boils down to the variance of the $j^{th}$ regressor, minus its covariance with the other regressors in the model. So this term will indeed shrink if you add new regressors that are collinear with your existing ones. And since $\sigma^2$ can't be increased by adding regressors (assuming you find the least-squares estimates of the model coefficients), that explanation can be ruled out.
answered Feb 23, 2015 at 12:06