0
$\begingroup$

This is the first time for me to ask a question here. I'm sorry that if I break any rule here.

I have encountered a problem about the consequence of multicollinearity. During reading the explanation about multicollinearity from an article on the internet, it has the following proof:

The least-squares estimates $b_j$ become too large in absolute value in the presence of multicollinearity. For example, consider the squared distance between b and $\beta$ as $$ L^2 = (b - \beta)'(b - \beta)\\ E[L^2] = \sum_{j=1}^{k}E[b_j - \beta_j]^2\\ =\sum_{j=1}^{k}Var[b_j]\\ =\sigma^2tr(X'X)^{-1} $$ The trace of a matrix is the same as the sum of its eigenvalues. If $\lambda_1, \lambda_2, ,...,\lambda_k$ are the eigenvalues of $(X'X)$ then $1/\lambda_1, 1/\lambda_2, > ,...,1/\lambda_k$ are the eigenvalues of $(X'X)^{-1}$ and hence $E[L^2] = \sigma^2\sum_{j=1}^{k}1/\lambda_j, \lambda_j>0$

If $(X'X)$ is ill-conditioned due to the presence of multicollinearity, then at least one of the eigenvalue will be small. So the distance between b and $\beta$ may also be substantial. Thus

$$ E[L^2] = E(b - \beta)'(b - \beta)\\ \sigma^2tr(X'X)^{-1} = E(b'b - 2b'\beta + \beta'\beta)\\ \Rightarrow E(b'b) = \sigma^2 tr(X'X)^{-1} + \beta'\beta\\ $$ $\Rightarrow$ b is generally longer than $\beta$

$\Rightarrow$ OLSE is too large in absolute value

The least-squares produces wrong estimates of parameters in the presence of multicollinearity. This does not imply that the fitted model provides wrong predictions also. If the predictions are confined to x-space with non-harmful multicollinearity, then predictions are satisfactory.

http://home.iitk.ac.in/~shalab/regression/Chapter9-Regression-Multicollinearity.pdf (Page 5 - 6)

I can understand the proof. However, I'm confused with the conclusion. $E(b'b) = \sigma^2 tr(X'X)^{-1} + \beta'\beta\\$ seems only imply the variance and covariance matrix of b (estimate of the beta) is biased. It does not imply the estimates of beta are wrong. Can anyone explain this conclusion to me?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Hi: unbiasedness is not related to the multi-collineairty of the design matrix so, technically speaking, you can still say that the estimates are unbiased. but their variance increases with multi-collinearity. So, the estimate, statistically speaking, is still "off" in the sense that the variance of it will be higher than it would be without multi-collinearity. Therefore, any tests about their significance are going to be off. So, to say that the estimates are still unbiased is kind of irrelevant because, when we measure their significance, their variance matters. I hope that helps. $\endgroup$ – mlofton Jul 18 '20 at 3:49
  • $\begingroup$ Thanks @mlofton. As I've little statistics background, I may not fully understand how statistician interpret estimator. Can I understand the term "wrong" as below? Even the estimator is unbiased, variance is too high to judge if beta is significant? I have seen heteroskedasticity can be solved by wt. least square. Is there any way to solve the multicollinearity(not perfect multicollinear) by adjusted the variance? As variance can be constructed by the above formula, I'm confused why ppl dont use the above formula to estimate variance instead of ridge regression / dropping variables.Thank you $\endgroup$ – ekiko Jul 18 '20 at 5:53
  • $\begingroup$ Hi ekiko: II think ( although someone should confirm this. this is not the area of statistics that I deal with much ) the variance that results does use the formula you're referring to. The problem is that the expression is going to be large when there's multi-collinearity because the inverse is going to be close to singular. So, other than ridge regression or dropping variables, I don't think there is a way around the problem. But hopefully someone else can comment because I don't feel, all that comfortable answering that good question. $\endgroup$ – mlofton Jul 19 '20 at 1:08
  • $\begingroup$ One piece of advice: if you've taken calculus I and II ( and maybe even III ) and you find statistics interesting, then one way to improve your knowledge would be to take the mathematical statistics sequence that your stat or math department offers. This material ( multi-collinearity ) is not necessarily covered there but taking that sequence ( and also possibly linear algebra ) would then mean that you could take classes in regression, linear models and pretty much any other stat course your school offers. The math stat sequence is kind of the foundation for everything else. $\endgroup$ – mlofton Jul 19 '20 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.