5
$\begingroup$

I am studying about multicollinearity in regression and in the book it says, "if there is severe (but not perfect) multicollinearity, two or more predictor variables are highly correlated, so $X^TX$ is (computationally) difficult to invert. This produces unstable regression estimates and large standard error."

Could anyone explain what makes it computationally difficult? Any mathematical explanation of the fact would be really helpful.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I doubt there's any computational difficulty, but due to large condition number, there could be problems with numerical stability. Or, it could be the case that values of inverse matrix are too big for standard floating point arithmetic, which forces you to switch to something like arbitrary precision arithmetic. $\endgroup$
    – Artem Sobolev
    Commented Jan 19, 2015 at 6:40
  • $\begingroup$ Just to state the computationally obvious: Avoid the direct inversion of a matrix during the solution of a linear system. What you most probably want is the QR decomposition of $X$ in question. The operation $X^TX$ will square the condition number of the system you are trying to solve (which is a very bad thing). $\endgroup$
    – usεr11852
    Commented Feb 20, 2015 at 1:31

2 Answers 2

Reset to default
6
$\begingroup$

The MLE of the regression coefficient $\beta$ satisfies $$\hat\beta\sim\mathcal{N}(\beta,\sigma^2(X^\text{T}X)^{-1})$$Hence, if two or more columns of $X$ are highly correlated, one or more eigenvalue(s) of $(X^\text{T}X)$ is close to zero and one or more eigenvalue(s) of $(X^\text{T}X)^{-1})$ is very large. This means that there exists a vector $e_1$ with $\|e_1\|=1$ such that $\text{var}(e_1^\text{T}\hat{\beta})$ is very large.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ If you were using exact arithmetic that would be the end of the problem. In single or double precision floating point arithmetic once the variance becomes large enough, it swamps the limited (e.g. 15 digit) accuracy of the floating point numbers and the computed values are meaningless. $\endgroup$
    – Brian Borchers
    Commented Jan 19, 2015 at 14:39
0
$\begingroup$

The presence of multicollinearity implies linear dependence among the regressors due to which it won't be possible to invert the matrix of regressors. For invertibility it is required that the matrix has a full rank and dependence implies the contrary. If there is variability in the regressors (no multicollinearity) taking the inverse of the matrix will be possible.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.