When using logistic regression for predictive modelling, the choice between 'standard' logistic regression vs ridge vs LASSO versions of logistic regression seems relatively straightforward - just pick the method that gives the best predictive performance on test data (unless there are additional considerations about whether to perform variable selection, which LASSO does automatically but ridge does not, or other considerations as described here Why can't ridge regression provide better interpretability than LASSO?).

But what about when the aim is not predictive performance, but explanatory modelling - i.e. determining which predictors drive variation in the response, and how strong the effect of each predictor is? Is there any reason to think that one of the three methods is better, or should I just go with the one that gives the best predictions (or alternatively use elastic net regression to find the optimal blend of ridge and LASSO in terms of predictive performance)?

This article http://projecteuclid.org/download/pdfview_1/euclid.ss/1294167961 describes how explanatory modelling aims to minimise bias, whereas predictive modelling aims to minimise some combination of bias and variance. This makes me think that the best model in terms of predictive performance may not be the best in terms of explaining which predictors drive variation in the response.

  • 3
    $\begingroup$ I agree with your last sentence. $\endgroup$ – Richard Hardy Jan 12 '17 at 17:10
  • 1
    $\begingroup$ This post blog.datadive.net/… discusses the benefits and of ridge vs lasso for variable importance. It suggests ridge may be better than lasso when important predictors are correlated (lasso can be unstable as it zeros important predictors out), but that stability selection is often the best approach. So, if I use lasso + stability selection to find important variables, then run ridge regression on the important subset, would this be a good way to get stable coefficients for important variables? $\endgroup$ – jay Jan 23 '17 at 1:34
  • $\begingroup$ Probably yes. But it would be difficult to determine the standard errors and statistical significance of coefficients estimated in this way. $\endgroup$ – Richard Hardy Jan 23 '17 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.