2
$\begingroup$

If your modeling problem is that you have too many features, a solution to this problem is LASSO regularization. By forcing some feature coefficients to be zero, you remove them, thus reducing the number of features that you are using in your model. LASSO solves the problem of too many features through feature selection.

What specific problems is ridge regression practically useful for solving? This question is looking for a canonical explanation of what problems ridge regression is used to solve today (in 2018).

$\endgroup$
  • 2
    $\begingroup$ Overfitting.... $\endgroup$ – Richard Hardy Jun 19 '18 at 19:06
  • 1
    $\begingroup$ @RichardHardy Wikipedia says that it is 'more often' for underfitting. Google says multicollinearity in two of the top three hits. So which is it, and why? $\endgroup$ – kingledion Jun 19 '18 at 19:09
  • $\begingroup$ @Sycorax I asked about specific applications of ridge, giving an example of a specific application of Lasso. The accepted answer to the other question suggests using Non-Negative Garotte. If only he knew that 8 years in the future L1 and L2 regularization would be easily implemented in any programming language that matters. In any case, that was a different question asked in a different world as far as computer implementations go. $\endgroup$ – kingledion Jun 19 '18 at 19:20
  • 1
    $\begingroup$ That thread discusses the interpretations of LASSO and Ridge in a Bayesian framework (scroll down). stats.stackexchange.com/questions/151304/… This thread discusses how ridge regression provides for unique solutions to models which are not identifiable. Your comment mentions programming and software implementations - are these important aspects of your question? How? $\endgroup$ – Sycorax Jun 19 '18 at 19:28
  • $\begingroup$ No, ridge regression cannot help against underfitting. If that is really in Wikipedia, it is wrong. $\endgroup$ – Richard Hardy Jun 19 '18 at 20:13
2
$\begingroup$

Ridge regression is primarily a tool for dealing with colinearity. It does this by allowing some bias in exchange for greatly reducing the variance of the estimators. See e.g. Ridge Regression, ridge regression, Kim, Bager et al and sources cited by those papers.

$\endgroup$
  • 2
    $\begingroup$ I'm always confused about the collinearity angle on ridge regression. Most applications I see of ridge is controlling the bias-variance tradeoff, which has nothing much to do with co-linearity. $\endgroup$ – Matthew Drury Jun 19 '18 at 19:42
  • 1
    $\begingroup$ Why do you say it has nothing much to do with colinearity? One of the main problems with colinearity is that the estimators have huge variances. $\endgroup$ – Peter Flom Jun 19 '18 at 20:14
  • $\begingroup$ The parameter estiamtes have huge variances, yes, but not the predictions themselves. Ridge is very often applied in situations where the parameter estimates are not of primary (sometimes not of any) importance. It's used to control the bias-variance tradeoff of the models generalization error. I kinda think the collinearity thing is a red-herring. $\endgroup$ – Matthew Drury Jun 19 '18 at 20:32

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.