# How do we know that we attain a minimum when $X^TX$ does not have full rank?

Assume we have a linear model $$E[\textbf{y}]=\textbf{X} \beta$$. When we use least squares we get the normal equations $$\textbf{X}^Ty=\textbf{X}^T\textbf{X}\hat{\beta}$$. Assume that $$\textbf{X}$$ does not have full rank, since the rank of $$\textbf{X}$$ equals the rank of $$\textbf{X}^TX$$ we have that $$\textbf{X}^T\textbf{X}$$ does not have full rank either.

A way to solve this problem is with generalized inverses.

Definition: If we have a matrix $$A$$, $$A^-$$ is a generalized inverse if $$AA^-A=A$$. Generalized inverses always exist, but they may not be unique.[Foundations of Linear and Generalized Linear Models, Alan Agresti, pages 30-31].

It can then be shown that a solution to the least square normal equations is $$(\textbf{X}^T\textbf{X})^-\textbf{X}^Ty$$.

But do we know if this is a minimum? It is stated that when $$X^TX$$ is invertible it is a minimum because $$\partial^2 L(\beta)/\partial \beta^2=2X^TX$$ and the latter expression is positive definite, hence we have a minimum. But when $$X^TX$$ does not have full rank we don't have positive definite(only semidefinite), how do we know that we obtain a minimum when using the generalized inverse?

• The least-squares objective is evidently quadratic, which implies at least one global minimum exists. When $X$ is of reduced rank, there will be a linear subspace of global minima of dimension equal to the rank deficit. Different versions of the generalized inverse solution can identify different points within that subspace: but all are where the least squares objective is minimized. You can work this out in detail in a simple case, such as $X=\pmatrix{1&2\\1&2\\1&2}$ (which arises when making three observations where the explanatory variable is constant).
– whuber
Feb 24 at 17:26

Any time a predictor is an exact copy of another predictor, or a linear combination of another predictor, one of the eigenvalues of $$(\mathbf{X}^\top\mathbf{X})^{-1}$$ will be zero. However, throwing out the redundant variable won't change the minimum. Rather, I would say that the solution is not unique, since the inverted dispersion matrix is positive-semidefinite.