Does regularization in regression help with numerics when the data matrix is not full rank?

I am trying to get some intuition around regression when the data matrix $$A$$ is not full rank in the following regression/least squares problem:

$$y=Ax+b$$

where $$y \in \mathbb{R}^n$$, $$A \in \mathbb{R}^{nxm}$$, and $$b \in \mathbb{R}^n$$, and $$x \in \mathbb{R}^m$$.

The regression problem we are solving is obviously $$\min_x ||y-Ax-b||^2$$

Now , if $$A$$ is not full rank, obviously this has infinitely many solutions. So, I want to make sure my understanding is correct. One of my colleagues says that adding regularization on $$x$$ will "help with the numerics" but will not improve the performance of the regression (as measured by $$R^2$$, e.g.). Adding regularization, the problem then becomes:

$$\min_x ||y-Ax-b||^2+||x||^2$$

When he says "help with the numerics" I understand that to mean two things:

1. The problem has a unique solution rather than infinitely many solutions
2. It prevents any of the regressors (components of the $$x$$ vector) from becoming too large, and thus leading to numerical instability.

Are those two things correct? Or is there anything else I need to add to my interpretation?

First of all, better to revise the cost function to include the degree of regularization: $$\min_x ||y-Ax-b||^2+\lambda||x||^2$$
1. Correct, since you can take the inverse of $$A^TA+\lambda I$$, you'll have unique solution.
2. Correct, and tightly coupled with (1), this is because your singular values for the inverse become $$\frac{s}{s^2+\lambda}$$. So, when $$\lambda>0$$, this expression won't approach $$\infty$$, leading to numerical stability.
• I did not see how $A^T A + \lambda I$ relates...but maybe I can derive thatexpression myself. Don't you need to normalize your data before applying that regularization parameters, $\lambda$? I suppose I was assuming that the data is already scaled such that all the regressors are from 0 to 1. If you don't scale your data, then the $\lambda$ will have a large effect on the regressors/covariates with a large scale, because the 2 norm makes outliers a lot biggerbecause of the squared term. right? – makansij Sep 22 '18 at 20:47