I am trying to get some intuition around regression when the data matrix $A$ is not full rank in the following regression/least squares problem:


where $y \in \mathbb{R}^n$, $A \in \mathbb{R}^{nxm}$, and $b \in \mathbb{R}^n$, and $x \in \mathbb{R}^m$.

The regression problem we are solving is obviously $$ \min_x ||y-Ax-b||^2$$

Now , if $A$ is not full rank, obviously this has infinitely many solutions. So, I want to make sure my understanding is correct. One of my colleagues says that adding regularization on $x$ will "help with the numerics" but will not improve the performance of the regression (as measured by $R^2$, e.g.). Adding regularization, the problem then becomes:

$$ \min_x ||y-Ax-b||^2+||x||^2$$

When he says "help with the numerics" I understand that to mean two things:

  1. The problem has a unique solution rather than infinitely many solutions
  2. It prevents any of the regressors (components of the $x$ vector) from becoming too large, and thus leading to numerical instability.

Are those two things correct? Or is there anything else I need to add to my interpretation?


First of all, better to revise the cost function to include the degree of regularization: $$\min_x ||y-Ax-b||^2+\lambda||x||^2$$

  1. Correct, since you can take the inverse of $A^TA+\lambda I$, you'll have unique solution.

  2. Correct, and tightly coupled with (1), this is because your singular values for the inverse become $\frac{s}{s^2+\lambda}$. So, when $\lambda>0$, this expression won't approach $\infty$, leading to numerical stability.

I don't quite agree with "will not improve the performance of the regression", because performance of your model is not just measured on your training data. What about your generalization performance, e.g. overfit?

  • 2
    $\begingroup$ to be more explicit, the R^2 will almost certainly reduce on your training data but may go up on your test data ( if you choose the right amount of regularisation). $\endgroup$ – seanv507 Sep 22 '18 at 17:48
  • $\begingroup$ I did not see how $A^T A + \lambda I$ relates...but maybe I can derive thatexpression myself. Don't you need to normalize your data before applying that regularization parameters, $\lambda$? I suppose I was assuming that the data is already scaled such that all the regressors are from 0 to 1. If you don't scale your data, then the $\lambda$ will have a large effect on the regressors/covariates with a large scale, because the 2 norm makes outliers a lot biggerbecause of the squared term. right? $\endgroup$ – makansij Sep 22 '18 at 20:47
  • $\begingroup$ @jaja, differentiate your loss function wrt x, and you'll see. And, the best practice is to normalize the data of course, but this can be done several ways, e.g. standard scaler, not just 0-1 scaling. $\endgroup$ – gunes Sep 22 '18 at 20:51
  • $\begingroup$ Got it. Seems like I interpreted her comment correctly. thanks!! $\endgroup$ – makansij Sep 22 '18 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.