I understand the reasoning behind ridge regression: we include some bias in the model in order to reduce the variance of the regression coefficients. My question is, why would we want to do that?

Ridge regression is frequently recommended in the case of multicollinearity. The problem is, as far as I know, multicollinearity is not a serious issue when the aim is to make predictions, being much more problematic when one tries to explain the relationships among independent (IV) and dependent variables (DV). On the other hand, I have been encountering repeatedly the opinion that regularization methods, in which ridge regression is included, are useful in predictive contexts, but not so much for making inferences.

So how exactly does ridge regression help with multicollinearity? It improves the predictive power of the model, even though a simpler OLS regression should already be fairly accurate in this regard, even in the presence of correlated IVs? Or it actually provides information on the relative strength and sign of the relationships between the IVs and the DV, thus improving the explanatory capacity of the model? Or neither of the previous options, since I have found at least one opinion defending that ridge regression should be taken more as a diagnostic tool, valuable to assess how badly the multicollinearity is messing with the model's coefficients, but that it should not be really used for prediction or explanation? Or am I misunderstanding something?

Any comments?


PS. A reference stating both that multicollinearity is not a problem for predictive models, and that the ridge regression is suitable for making predictions but not for providing explanations is: "Shmueli G. 2010. To explain or to predict? Statistical Science 25(3): 289-310"

  • 1
    $\begingroup$ check this stats.stackexchange.com/questions/282654/… $\endgroup$ – Haitao Du Aug 4 '17 at 14:14
  • 1
    $\begingroup$ i think the best explanation is in terms of the likelihood / loss surface, as Glen_b discusses in his answers here: here and here. The individual parameters become nearly non-identifiable even though the final predictions are fine $\endgroup$ – jld Aug 4 '17 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.