# How do we know that we attain a minimum when $X^TX$ does not have full rank?

Assume we have a linear model $$E[\textbf{y}]=\textbf{X} \beta$$. When we use least squares we get the normal equations $$\textbf{X}^Ty=\textbf{X}^T\textbf{X}\hat{\beta}$$. Assume that $$\textbf{X}$$ does not have full rank, since the rank of $$\textbf{X}$$ equals the rank of $$\textbf{X}^TX$$ we have that $$\textbf{X}^T\textbf{X}$$ does not have full rank either.

A way to solve this problem is with generalized inverses.

Definition: If we have a matrix $$A$$, $$A^-$$ is a generalized inverse if $$AA^-A=A$$. Generalized inverses always exist, but they may not be unique.[Foundations of Linear and Generalized Linear Models, Alan Agresti, pages 30-31].

It can then be shown that a solution to the least square normal equations is $$(\textbf{X}^T\textbf{X})^-\textbf{X}^Ty$$.

But do we know if this is a minimum? It is stated that when $$X^TX$$ is invertible it is a minimum because $$\partial^2 L(\beta)/\partial \beta^2=2X^TX$$ and the latter expression is positive definite, hence we have a minimum. But when $$X^TX$$ does not have full rank we don't have positive definite(only semidefinite), how do we know that we obtain a minimum when using the generalized inverse?

• The least-squares objective is evidently quadratic, which implies at least one global minimum exists. When $X$ is of reduced rank, there will be a linear subspace of global minima of dimension equal to the rank deficit. Different versions of the generalized inverse solution can identify different points within that subspace: but all are where the least squares objective is minimized. You can work this out in detail in a simple case, such as $X=\pmatrix{1&2\\1&2\\1&2}$ (which arises when making three observations where the explanatory variable is constant).
– whuber
Feb 24 at 17:26

Any time a predictor is an exact copy of another predictor, or a linear combination of another predictor, one of the eigenvalues of $$(\mathbf{X}^\top\mathbf{X})^{-1}$$ will be zero. However, throwing out the redundant variable won't change the minimum. Rather, I would say that the solution is not unique, since the inverted dispersion matrix is positive-semidefinite.

I would also recommend getting away from thinking about the G-inverse of a matrix, and focus more on singular value decomposition (SVD), which doesn't require a square real-symmetric matrix to be full rank for inversion. Most statistical packages will likely use QR (QL) iteration based on the tridiagonal form of an upper Hessenberg matrix. I usually use the left (QL), but it doesn't really matter.

• Is it possible to prove that we obtain a minimum? Dec 5, 2021 at 18:35