3
$\begingroup$

I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $X'$ represents the transpose of $X$.

I understand that $X'X$ is a symmetric matrix, but I also know that being symmetric alone does not guarantee nonsingularity.

For reference I am referring to this equation:

$$ H = X(X'X)^{-1}X' $$

Any help with this is greatly appreciated.


Through the helpful answers below and a few other google searches I think I have found an answer to my question (at least for most cases).

When performing OLS. We have organized our data with $n$ observations and $p$ parameters. In almost every case $n > p$. This means that the columns of $X$ must be linearly independent. $X'X$ results in a matrix with $dim(X'X) = p$. This means that $X'X$ must also have columns that are linearly independent. Because $X'X$ is a square matrix (rows equal columns), it must have rows which are linearly dependent as well (i.e. $rank(X'X) = p$ aka "full rank"). A full rank matrix is always invertible.

Please correct me if I am wrong here, but I think the logic follow.

I used these questions as resources:

https://math.stackexchange.com/questions/2430179/if-x-is-linearly-independent-prove-xtx-is-positive-definite

https://math.stackexchange.com/questions/691812/proof-of-when-is-a-xtx-invertible

https://math.stackexchange.com/questions/214542/linear-independent-sets-of-non-square-matricies

$\endgroup$
  • 3
    $\begingroup$ It isn't necessarily nonsingular. For those concerned about the singular case, you have to understand the inverse as a generalized inverse. $\endgroup$ – whuber Sep 7 '18 at 20:53
  • $\begingroup$ Am I understanding you correctly that the inverse of the $X'X$ matrix can be thought of as a generalized inverse in this case? Analogous to $(n)1/n$? That goes against what I thought was a fundamental property of matrices (not all are invertible). $\endgroup$ – samvoit4 Sep 7 '18 at 21:36
  • $\begingroup$ See en.wikipedia.org/wiki/Generalized_inverse. I believe your question may have answers at stats.stackexchange.com/questions/63143 and stats.stackexchange.com/questions/84036. $\endgroup$ – whuber Sep 7 '18 at 21:40
  • $\begingroup$ To answer the title question, all you need to do is to calculate the determinant of the matrix. If the determinant is zero, it is singular; if not, it is non-singular. $\endgroup$ – Ben Sep 7 '18 at 23:57
  • 3
    $\begingroup$ That conclusion does not follow. The Normal equations can still be solved even when the determinant of $X^{\prime} X$ is zero. That $X^\prime X$ can be singular is evident: simply repeat one column in $X$, for instance. This sort of thing happens so often that all general-purpose OLS software will automatically handle it (typically by dropping the smallest number of columns needed to make the design matrix of full rank). $\endgroup$ – whuber Sep 8 '18 at 15:56
5
$\begingroup$

It's a property of the $\text{rank}$ operator when its used on real matrices $\mathbf{A}$: $$ \text{rank}(\mathbf{A}) = \text{rank}(\mathbf{A}') = \text{rank}(\mathbf{A}'\mathbf{A}) = \text{rank}(\mathbf{A}\mathbf{A}'). $$

In your case, the data matrix $\mathbf{X} \in \mathbb{R}^{n \times p}$ is usually tall and skinny ($n > p$), so the rank of everything is the number of linearly independent columns/predictors/covariates/independent variables. If everything is linearly independent $\text{rank}(\mathbf{X}) = p$, and so you have $\mathbf{X}'\mathbf{X}$ is invertible. If you have collinearity, or columns that can be written as linear combinations of others, then $\text{rank}(\mathbf{X}) < p$, and you cannot find a unique inverse for $\mathbf{X}'\mathbf{X}$ (you can, however, find generalized inverses for it).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.