Imagine an underdetermined linear system, composed of N (continuous) labels and N samples, each have P features (with N < P):

$$\hat{\textbf{Y}}_{N \times 1} = \textbf{X}_{N \times P} \textbf{W}_{P\times 1} $$

and we are interested in finding the best weight matrix $W$ for this regression problem.

Since the system is underdetermined, a linear regression model has infinitely many feasible solutions and it is customary to pick the solution which also minimizes the solution $ ||\textbf{W}||^2$. This indeed makes sure that most of non-important features posses negligible weights.

Another approach to this problem is a ridge regression model. Again, there's a hyperparameter by which one can control the norm of the solution.

However when it comes to prediction, apparently, the ridge performs better in practice (the last comment here, as well as a personal experience of mine on training a system, confirm this statement). I'm interested to know why it's the case.

  • 2
    $\begingroup$ Min-norm solution is the limit of ridge with regularization parameter going to 0. If 0 is not the optimal value for your data, then tuning the regularization parameter can yield better results. $\endgroup$ – amoeba Nov 20 '19 at 13:58
  • $\begingroup$ Related: stats.stackexchange.com/questions/328630. $\endgroup$ – amoeba Nov 20 '19 at 14:00
  • $\begingroup$ @amoebasaysReinstateMonica Thanks for the very informative question you addressed. I'm still digesting that tread though and it will surely take me a while to fully understand the details. $\endgroup$ – arash Nov 22 '19 at 8:49
  • $\begingroup$ You might want to read this arxiv.org/abs/1805.10939 instead of reading that thread... It's a write-up of that whole discussion. $\endgroup$ – amoeba Nov 22 '19 at 8:56
  • $\begingroup$ @amoebasaysReinstateMonica I surely do! I really appreciate it. $\endgroup$ – arash Nov 22 '19 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.