Assume we have a linear model $E[\textbf{y}]=\textbf{X} \beta$. When we use least squares we get the normal equations $\textbf{X}^Ty=\textbf{X}^T\textbf{X}\hat{\beta}$. Assume that $\textbf{X}$ does not have full rank, since the rank of $\textbf{X}$ equals the rank of $\textbf{X}^TX$ we have that $\textbf{X}^T\textbf{X}$ does not have full rank either.

A way to solve this problem is with generalized inverses.

Definition: If we have a matrix $A$, $A^-$ is a generalized inverse if $AA^-A=A$. Generalized inverses always exist, but they may not be unique.[Foundations of Linear and Generalized Linear Models, Alan Agresti, pages 30-31].

It can then be shown that a solution to the least square normal equations is $(\textbf{X}^T\textbf{X})^-\textbf{X}^Ty$.

But do we know if this is a minimum? It is stated that when $X^TX$ is invertible it is a minimum because $\partial^2 L(\beta)/\partial \beta^2=2X^TX$ and the latter expression is positive definite, hence we have a minimum. But when $X^TX$ does not have full rank we don't have positive definite(only semidefinite), how do we know that we obtain a minimum when using the generalized inverse?

  • $\begingroup$ The least-squares objective is evidently quadratic, which implies at least one global minimum exists. When $X$ is of reduced rank, there will be a linear subspace of global minima of dimension equal to the rank deficit. Different versions of the generalized inverse solution can identify different points within that subspace: but all are where the least squares objective is minimized. You can work this out in detail in a simple case, such as $X=\pmatrix{1&2\\1&2\\1&2}$ (which arises when making three observations where the explanatory variable is constant). $\endgroup$
    – whuber
    Feb 24 at 17:26

1 Answer 1


Any time a predictor is an exact copy of another predictor, or a linear combination of another predictor, one of the eigenvalues of $(\mathbf{X}^\top\mathbf{X})^{-1}$ will be zero. However, throwing out the redundant variable won't change the minimum. Rather, I would say that the solution is not unique, since the inverted dispersion matrix is positive-semidefinite.

I would also recommend getting away from thinking about the G-inverse of a matrix, and focus more on singular value decomposition (SVD), which doesn't require a square real-symmetric matrix to be full rank for inversion. Most statistical packages will likely use QR (QL) iteration based on the tridiagonal form of an upper Hessenberg matrix. I usually use the left (QL), but it doesn't really matter.

  • $\begingroup$ Is it possible to prove that we obtain a minimum? $\endgroup$
    – user394334
    Dec 5, 2021 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.