I am trying to get some intuition around regression when the data matrix $A$ is not full rank in the following regression/least squares problem:
$$y=Ax+b$$
where $y \in \mathbb{R}^n$, $A \in \mathbb{R}^{nxm}$, and $b \in \mathbb{R}^n$, and $x \in \mathbb{R}^m$.
The regression problem we are solving is obviously $$ \min_x ||y-Ax-b||^2$$
Now , if $A$ is not full rank, obviously this has infinitely many solutions. So, I want to make sure my understanding is correct. One of my colleagues says that adding regularization on $x$ will "help with the numerics" but will not improve the performance of the regression (as measured by $R^2$, e.g.). Adding regularization, the problem then becomes:
$$ \min_x ||y-Ax-b||^2+||x||^2$$
When he says "help with the numerics" I understand that to mean two things:
- The problem has a unique solution rather than infinitely many solutions
- It prevents any of the regressors (components of the $x$ vector) from becoming too large, and thus leading to numerical instability.
Are those two things correct? Or is there anything else I need to add to my interpretation?