I just need a simple explanation of what exactly ridge regression is so I can have a decent intuitive understanding of it. I understand it's about applying some sort of penalty to the regression coefficients... but beyond that I'm a little confused about how it is different from other kinds of regression which implement penalties. In what case should you use ridge regression as opposed to some other kind of regression?
$\begingroup$
$\endgroup$
4
-
5$\begingroup$ There's a nice blog post on ridge regression here. $\endgroup$– gung - Reinstate MonicaCommented Mar 19, 2013 at 2:18
-
1$\begingroup$ That is indeed an excelent post, I provide a wayback machine link here in case at some point in the future the blog post goes dark: web.archive.org/web/20140731061929/http://tamino.wordpress.com/… $\endgroup$– russellpierceCommented Mar 6, 2015 at 19:16
-
$\begingroup$ @gung Thank you for the post :) it is very helpful $\endgroup$– ChristinaCommented May 12, 2015 at 12:26
-
$\begingroup$ See also: Why does ridge estimate become better than OLS by adding a constant to the diagonal? $\endgroup$– amoebaCommented Sep 24, 2015 at 23:17
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Ridge Regression is a remedial measure taken to alleviate multicollinearity amongst regression predictor variables in a model. Often predictor variables used in a regression are highly correlated. When they are, the regression coefficient of any one variable depend on which other predictor variables are included in the model, and which ones are left out. (So the predictor variable does not reflect any inherent effect of that particular predictor on the response variable, but only a marginal or partial effect, given whatever other correlated predictor variables are included in the model). Ridge regression adds a small bias factor to the variables in order to alleviate this problem. Hope that helps.
-
7$\begingroup$ Re: "Ridge regression adds a small bias factor to the variables in order to alleviate this problem" - can you say more about this, as it relates to the OP's query "In what case should you use ridge regression as opposed to some other kind of regression"? There are various forms of penalized regression that add a small bias factor, e.g. LASSO (which is known to have some bad properties with regard to collinearity). I think some intuition about why ridge regression, specifically, is a commonly used method when you have collinear predictors would improve this answer. $\endgroup$– MacroCommented Mar 19, 2013 at 2:24
-
4$\begingroup$ Doesn't give a good intuition. Would be better if explained more clearly without using jargons $\endgroup$– kmario23Commented Oct 24, 2016 at 23:02
-
1$\begingroup$ @kmario23, exactly.. how is this answer accepted as 'intuitive' ? The only intuitive answers are those that have simple examples with integer numbers. $\endgroup$– d-_-bCommented Jul 17, 2020 at 8:27
$\begingroup$
$\endgroup$
The posts above nicely describe ridge regression and its mathematical underpinning. However, they don't address the issue of where ridge regression should be used, compared to other shrinkage methods. It might be so because there are no specific situation where one shrinkage method has been shown to perform better than another. There are many different ways of addressing the issue of multicollinearity among the predictor variables, depending on its source. Ridge regression happens to be one of those methods that addresses the issue of multicollinearity by shrinking (in some cases, shrinking it close to or equal to zero, for large values of the tuning parameter) the coefficient estimates of the highly correlated variables.
Unlike least squares method, ridge regression produces a set of coefficient estimates for different values of the tuning parameter. So, it's advisable to use the results of ridge regession (the set of coefficient estimates) with a model selection technique (such as, cross-validation) to determine the most appropriate model for the given data.
answered Sep 24, 2015 at 18:24