19
$\begingroup$

I just need a simple explanation of what exactly ridge regression is so I can have a decent intuitive understanding of it. I understand it's about applying some sort of penalty to the regression coefficients... but beyond that I'm a little confused about how it is different from other kinds of regression which implement penalties. In what case should you use ridge regression as opposed to some other kind of regression?

$\endgroup$

marked as duplicate by amoeba, gung regression Sep 24 '15 at 22:52

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

10
$\begingroup$

Ridge Regression is a remedial measure taken to alleviate multicollinearity amongst regression predictor variables in a model. Often predictor variables used in a regression are highly correlated. When they are, the regression coefficient of any one variable depend on which other predictor variables are included in the model, and which ones are left out. (So the predictor variable does not reflect any inherent effect of that particular predictor on the response variable, but only a marginal or partial effect, given whatever other correlated predictor variables are included in the model). Ridge regression adds a small bias factor to the variables in order to alleviate this problem. Hope that helps.

$\endgroup$
  • 7
    $\begingroup$ Re: "Ridge regression adds a small bias factor to the variables in order to alleviate this problem" - can you say more about this, as it relates to the OP's query "In what case should you use ridge regression as opposed to some other kind of regression"? There are various forms of penalized regression that add a small bias factor, e.g. LASSO (which is known to have some bad properties with regard to collinearity). I think some intuition about why ridge regression, specifically, is a commonly used method when you have collinear predictors would improve this answer. $\endgroup$ – Macro Mar 19 '13 at 2:24
  • 3
    $\begingroup$ Doesn't give a good intuition. Would be better if explained more clearly without using jargons $\endgroup$ – kmario23 Oct 24 '16 at 23:02
8
$\begingroup$

The posts above nicely describe ridge regression and its mathematical underpinning. However, they don't address the issue of where ridge regression should be used, compared to other shrinkage methods. It might be so because there are no specific situation where one shrinkage method has been shown to perform better than another. There are many different ways of addressing the issue of multicollinearity among the predictor variables, depending on its source. Ridge regression happens to be one of those methods that addresses the issue of multicollinearity by shrinking (in some cases, shrinking it close to or equal to zero, for large values of the tuning parameter) the coefficient estimates of the highly correlated variables.

Unlike least squares method, ridge regression produces a set of coefficient estimates for different values of the tuning parameter. So, it's advisable to use the results of ridge regession (the set of coefficient estimates) with a model selection technique (such as, cross-validation) to determine the most appropriate model for the given data.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.