# Invertibility of $X^TX$ with severe multicollinearity in regression

I am studying about multicollinearity in regression and in the book it says, "if there is severe (but not perfect) multicollinearity, two or more predictor variables are highly correlated, so $X^TX$ is (computationally) difficult to invert. This produces unstable regression estimates and large standard error."

Could anyone explain what makes it computationally difficult? Any mathematical explanation of the fact would be really helpful.

• I doubt there's any computational difficulty, but due to large condition number, there could be problems with numerical stability. Or, it could be the case that values of inverse matrix are too big for standard floating point arithmetic, which forces you to switch to something like arbitrary precision arithmetic. – Artem Sobolev Jan 19 '15 at 6:40
• Just to state the computationally obvious: Avoid the direct inversion of a matrix during the solution of a linear system. What you most probably want is the QR decomposition of $X$ in question. The operation $X^TX$ will square the condition number of the system you are trying to solve (which is a very bad thing). – usεr11852 Feb 20 '15 at 1:31

The MLE of the regression coefficient $\beta$ satisfies $$\hat\beta\sim\mathcal{N}(\beta,\sigma^2(X^\text{T}X)^{-1})$$Hence, if two or more columns of $X$ are highly correlated, one or more eigenvalue(s) of $(X^\text{T}X)$ is close to zero and one or more eigenvalue(s) of $(X^\text{T}X)^{-1})$ is very large. This means that there exists a vector $e_1$ with $\|e_1\|=1$ such that $\text{var}(e_1^\text{T}\hat{\beta})$ is very large.