7
$\begingroup$

It is not a real world case , but suppose that we have $n$ observations and $k$ variables , since $k= n - 1 $ , if $X$ is the design matrix,$(X'X)$ will be a square matrix , so What does that statistically mean , if $(X'X)^{-1}$ does not exist ?

$\endgroup$
  • 2
    $\begingroup$ I found this geometric explanation very good: part 1 and part 2. $\endgroup$ – Dimitriy V. Masterov Oct 16 '15 at 22:52
  • $\begingroup$ @DimitriyV.Masterov Thanks , It's really a very good explanation . $\endgroup$ – Bahgat Nassour Oct 17 '15 at 0:18
  • $\begingroup$ @dimitriyvmasterov thanks! first link is great. I think the second link is broken, it appears to be identical to the first $\endgroup$ – MichaelChirico Oct 18 '15 at 15:24
  • $\begingroup$ @MichaelChirico Yes they are the same but I guess , he means that one Part2 Eigenvalues $\endgroup$ – Bahgat Nassour Oct 18 '15 at 15:29
11
$\begingroup$

By 'does not exist' we mean that the original matrix $X^TX$ is not invertible, i.e. its inverse $(X^TX)^{-1}$ does not exist. Usually this relates to the presence of eigenvalues with extremely small magnitude (or zero) in the matrix $X^TX$.

This issue of non-invertabilty suggests that the matrix $X^TX$ is rank deficient. A rank deficient matrix has a column space that does not span the vector space with the same dimensions as your data (think of having a 2D basis but wanting to map 3D points). Rank deficiency usually materialises as a problem in situations where you want to estimate $p$ parameters but your matrix rank $q$ is smaller than $p$. In this case one has an under-defined problem, $q$ equation and $p$ unknowns where $p>q$. Statically we mean that the information to solve this problem is simply unavailable.

There is already a very good thread on what is rank deficiency, and how to deal with it? if you want to follow up this further.

$\endgroup$
  • $\begingroup$ The linked post is an excellent overview. (This post is great too =D) $\endgroup$ – bdeonovic Oct 16 '15 at 23:29
7
$\begingroup$

In the case of regression, consider the most basic linear model $$Y=Xb+\varepsilon,$$ the least square estimator $\hat{\beta}$ must satisfies $$\hat{Y}=X\hat{\beta}$$ where $\hat{Y}$ is the projection of $Y$ onto the space spanned by the columns of $X$. This leads us to the normal equation $$X'X\hat{\beta}=X'Y.$$ If $X$ has full rank then $X'X$ is invertible, so the (unique) solution to the equation is $$\hat{\beta}=(X'X)^{-1}X'Y.$$ However, if $(X'X)^{-1}$ does not exist, the solution to the normal equation will not be unique (e.g. any generalized inverse can solve the equation).

$\endgroup$
  • 3
    $\begingroup$ +1 for mentioning psueod-inverse. It's instructive to mention that if pseudo-inverse exists, one can still obtain estimates of certain contrasts. $\endgroup$ – StatsStudent Oct 16 '15 at 22:51
6
$\begingroup$

To complement the good answers already offered, if you would like a statistical implication of the singularity of $\left( \mathbf{X}^{T} \mathbf{X} \right)^{-1}$ you can think in terms of the variance of the OLS estimator: it explodes and all precision is lost. The confidence limits for the estimators in turn grow extremely large and inference becomes impossible.

These implications often lead one to opt for ridge regression instead, as the introduction of biasing constant makes the inverse more stable and variances less inflated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.