# Usefulness of convexity of linear regression when there is no closed form solution

The optimisation problem in linear regression, $f(\beta) = ||y-X\beta||^2$ is convex (as it is a quadratic function), and when $(X^TX)$ is invertible, we have a unique solution which we can calculate by the given closed form $\beta = (X^T X)^{-1} X^Ty$. However, how is convexity useful in cases where there is no closed form solution. When we have for instance infinite many solutions, the fact that local solutions are also global minima seems not much of a help?

• When there is no closed form solution you use an iterative local method such as gradient descent. If your function is convex, you're guaranteed to converge to a global optimum. So yes, even with infinite many solutions, convexity is very much helpful. This is sometimes seen as the thin line between statistical estimation and machine learning. – Digio Jul 31 '17 at 9:07

When $(X^TX)$ is not invertible there is not one solution but several: an affine subspace. But they are still closed form solutions in a way. They are solutions of the linear system: $(X^TX)\beta=X^Ty$. Solving this system is not fundamentally more complicated than inverting a matrix I think.
Finally, when the matrix is not invertible, it is usually that there are not enough data to really estimate $\beta$ with a realistic precision. The solution is extremely over-fitted. People will then use ridge regularization. The solution is $\beta=(X^TX+\lambda I)^{-1}X^Ty$. The matrix $(X^TX+\lambda I)$ is always invertible ($\lambda>0$).
• @TestGuest I think this answer contains answers to your followup questions. The linked answer also shows another way to deal with the problem that $X^T X$ is not invertible - removing from $X$ columns that are linear combinations of other columns. – Oren Milman Aug 25 '18 at 6:41