8
$\begingroup$

It is not a real world case , but suppose that we have $n$ observations and $k$ variables , since $k= n - 1 $ , if $X$ is the design matrix,$(X'X)$ will be a square matrix , so What does that statistically mean , if $(X'X)^{-1}$ does not exist ?

$\endgroup$
4
  • 2
    $\begingroup$ I found this geometric explanation very good: part 1 and part 2. $\endgroup$ Oct 16 '15 at 22:52
  • $\begingroup$ @DimitriyV.Masterov Thanks , It's really a very good explanation . $\endgroup$ Oct 17 '15 at 0:18
  • $\begingroup$ @dimitriyvmasterov thanks! first link is great. I think the second link is broken, it appears to be identical to the first $\endgroup$ Oct 18 '15 at 15:24
  • $\begingroup$ @MichaelChirico Yes they are the same but I guess , he means that one Part2 Eigenvalues $\endgroup$ Oct 18 '15 at 15:29
13
$\begingroup$

By 'does not exist' we mean that the original matrix $X^TX$ is not invertible, i.e. its inverse $(X^TX)^{-1}$ does not exist. Usually this relates to the presence of eigenvalues with extremely small magnitude (or zero) in the matrix $X^TX$.

This issue of non-invertabilty suggests that the matrix $X^TX$ is rank deficient. A rank deficient matrix has a column space that does not span the vector space with the same dimensions as your data (think of having a 2D basis but wanting to map 3D points). Rank deficiency usually materialises as a problem in situations where you want to estimate $p$ parameters but your matrix rank $q$ is smaller than $p$. In this case one has an under-defined problem, $q$ equation and $p$ unknowns where $p>q$. Statically we mean that the information to solve this problem is simply unavailable.

There is already a very good thread on what is rank deficiency, and how to deal with it? if you want to follow up this further.

$\endgroup$
1
  • $\begingroup$ The linked post is an excellent overview. (This post is great too =D) $\endgroup$
    – bdeonovic
    Oct 16 '15 at 23:29
8
$\begingroup$

In the case of regression, consider the most basic linear model $$Y=Xb+\varepsilon,$$ the least square estimator $\hat{\beta}$ must satisfies $$\hat{Y}=X\hat{\beta}$$ where $\hat{Y}$ is the projection of $Y$ onto the space spanned by the columns of $X$. This leads us to the normal equation $$X'X\hat{\beta}=X'Y.$$ If $X$ has full rank then $X'X$ is invertible, so the (unique) solution to the equation is $$\hat{\beta}=(X'X)^{-1}X'Y.$$ However, if $(X'X)^{-1}$ does not exist, the solution to the normal equation will not be unique (e.g. any generalized inverse can solve the equation).

$\endgroup$
1
  • 3
    $\begingroup$ +1 for mentioning psueod-inverse. It's instructive to mention that if pseudo-inverse exists, one can still obtain estimates of certain contrasts. $\endgroup$ Oct 16 '15 at 22:51
7
$\begingroup$

To complement the good answers already offered, if you would like a statistical implication of the singularity of $\left( \mathbf{X}^{T} \mathbf{X} \right)^{-1}$ you can think in terms of the variance of the OLS estimator: it explodes and all precision is lost. The confidence limits for the estimators in turn grow extremely large and inference becomes impossible.

These implications often lead one to opt for ridge regression instead, as the introduction of biasing constant makes the inverse more stable and variances less inflated.

$\endgroup$
1
$\begingroup$

One extra answer from a more applied perspective.

Imagine you want to measure how $Y =$ punching power of a person is related to

  • $x_1$ = weight in kilograms of the person
  • $x_2$ = weight in grams of the person
  • $x_3$ = height of the person

Since $x_1$ is collinear with $x_2$, the matrix $(X'X)$ won't be invertible. This means the $\hat{\beta}$ solution that gives you the relationship between $Y$ and the $X$ won't really exist. There might be infinite solutions or they might not be any solution. This, in fact, is the "reasonable" answer in this context: it doesn't make sense to assign a separate value to $\hat{\beta}_1$ and $\hat{\beta}_2$. You might assign the full effect of weight on $\hat{\beta}_1$, the full effect on $\hat{\beta}_2$ or some arbitrary combination.

This is an extreme example, but when the $x$s approach collinearity (and, hence, the determinant is near zero) you'll have a similar, albeit not as extreme, issue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.