I have read that when using Ordinary Least Squares (OLS) for multiple linear regression, the coefficients/weights are unreliable for predictor variables that are collinear. I was wondering if this is also the case for regularisation methods (ridge/lasso/elastic net regression) when variables are collinear? Or could the coefficients/weights be used to determine relative importance of the predictors?

  • 1
    $\begingroup$ the whole point of ridge regression is to fix this, even in the case of perfect collinearity. See here for example: stats.stackexchange.com/questions/118712/… $\endgroup$ – jld Oct 20 '17 at 5:17
  • $\begingroup$ hmm, but the primary advantage of ridge over OLS is prediction accuracy? What if I'm not interested in making predictions but rather inferring what X variables are most important at explaining y. $\endgroup$ – Oliver Angelil Oct 21 '17 at 22:03
  • 1
    $\begingroup$ I think historically ridge regression was developed for solving ill conditioned systems (see en.m.wikipedia.org/wiki/Tikhonov_regularization) and it was only later that it was shown that there exists a $\lambda$ such that the MSE is better. So stable coefficients are really what it’s all about, although in the context of variance-bias it’s not surprising that stable coefficients lead to better predictions $\endgroup$ – jld Oct 21 '17 at 22:42
  • $\begingroup$ Cool. So I'll go ahead and use the coefficients from ridge regression to infer relative importance of the predictors. Thanks! $\endgroup$ – Oliver Angelil Oct 22 '17 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.