Does regularization in regression help with numerics when the data matrix is not full rank?

Question

I am trying to get some intuition around regression when the data matrix $A$ is not full rank in the following regression/least squares problem:

$$y=Ax+b$$

where $y \in \mathbb{R}^n$, $A \in \mathbb{R}^{nxm}$, and $b \in \mathbb{R}^n$, and $x \in \mathbb{R}^m$.

The regression problem we are solving is obviously $$ \min_x ||y-Ax-b||^2$$

Now , if $A$ is not full rank, obviously this has infinitely many solutions. So, I want to make sure my understanding is correct. One of my colleagues says that adding regularization on $x$ will "help with the numerics" but will not improve the performance of the regression (as measured by $R^2$, e.g.). Adding regularization, the problem then becomes:

$$ \min_x ||y-Ax-b||^2+||x||^2$$

When he says "help with the numerics" I understand that to mean two things:

1. The problem has a unique solution rather than infinitely many solutions
2. It prevents any of the regressors (components of the $x$ vector) from becoming too large, and thus leading to numerical instability.

Are those two things correct? Or is there anything else I need to add to my interpretation?

Answer 1

First of all, better to revise the cost function to include the degree of regularization: $$\min_x ||y-Ax-b||^2+\lambda||x||^2$$

1. Correct, since you can take the inverse of $A^TA+\lambda I$, you'll have unique solution.

2. Correct, and tightly coupled with (1), this is because your singular values for the inverse become $\frac{s}{s^2+\lambda}$. So, when $\lambda>0$, this expression won't approach $\infty$, leading to numerical stability.

I don't quite agree with "will not improve the performance of the regression", because performance of your model is not just measured on your training data. What about your generalization performance, e.g. overfit?