4
$\begingroup$

The situation I'm interested in is logistic regression for a binary response variable with lots of predictors (500 to 1000), lots of which are correlated. I would like to use a logistic LASSO approach to identify the most promising predictors. I could either use that solution directly, or only use it to identify a concise set of good predictors and use standard logistic regression on only these predictors.

I am aware of the analogous discussion regarding linear regression and LASSO(see, e.g. the recent paper by Belloni and Chernozhukov), but I have difficulties finding results that help me to make this decision in the logistic regression setting, and for practical purposes (prediction) rather than academical ones. I am also aware that logistic regression can also be viewed from an ordinary least squares perspective (see e.g. Hastie/Tibshiranis Elements of Statistical Learning, p. 124), but I currently don't see how to use this to transfer results from the linear to the logistic setting.

Could you give me pros and cons for both alternatives (direct logistic LASSO or standard logistic regression post-LASSO) or explain how the ordinary-least-squares-perspective on logistic regression can be used to transfer arguments from the linear regression case?

Improve this question
$\endgroup$
6
  • $\begingroup$ Lasso classifier is a ready solution that combines feature selection (in cases there are too many features or collinearity present) with classification (see R's glmnet package e.g). Why would you think about filtering features in Lasso and then carrying these results over to logistic classifier at all? At the very least cons are you'd be doing your job twice, and even worse, the set of variables that performed well in Lasso may not do so well in logistic regression. $\endgroup$
    – Sergey Bushmanov
    Oct 13, 2015 at 11:06
  • 1
    $\begingroup$ The motivation for this is as follows. When you use out-of-the-box LASSO, you buy less variance in exchange for some bias. Standard logistic regression, on the other hand, is asymptotically unbiased. So this is an attempt at fine-tuning the bias-variance trade-off, in a way that often seems to work very well (see the Belloni-Chernozhukov paper and the references for some evidence) in the case of linear regression. $\endgroup$
    – MightyCurious
    Oct 13, 2015 at 11:35
  • $\begingroup$ lasso is not a classifier $\endgroup$
    – Frank Harrell
    Oct 13, 2015 at 11:43
  • $\begingroup$ @FrankHarrell I never used glment for classifying, but I believe glmnet(x, y, family=c("binomial") does classification, don't you think so? $\endgroup$
    – Sergey Bushmanov
    Oct 13, 2015 at 11:50
  • $\begingroup$ Not at all. It is for prediction, here of probabilities (risk). $\endgroup$
    – Frank Harrell
    Oct 13, 2015 at 11:55

1 Answer 1

Reset to default
6
$\begingroup$

This has been written about on the site in detail. The lasso is meant to be a complete solution and it is completely inappropriate to use it to select features that are fed into a naive method that does not penalize for the context of having tortured data to find the features. You are also making an implicit assumption that the lasso finds the "right" predictors. Should you bootstrap the entire process you may be sorely disappointed to learn that in your case the features selected have a great deal of randomness in them. This is more true in the case of co-linearities.

If you want to emphasize parsimony over predictive accuracy, then model approximation (also called pre-conditioning) may be useful. Here one uses the best available prediction method, e.g. penalized maximum likelihood estimation with a quadratic penalty, i.e., ridge logistic regression. Then $X\hat{\beta}$ from that model is approximated using stepwise regression or recursive partitioning, etc. The approximate model inherits the proper amount of shrinkage from the full model. The approximate model may be chosen to yield $R^{2} = 0.95$ against the gold standard linear predictor.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I am less naive than I may sound regarding the randomness of the features selected by the LASSO-method. I'm grateful for the hint regarding model approximation which I wasn't aware of. I'd vote +1 if my reputation would allow this. $\endgroup$
    – MightyCurious
    Oct 13, 2015 at 12:15
  • $\begingroup$ I was trying to say that the method is naive, not you. You are not alone in mentioning that 2-stage method. $\endgroup$
    – Frank Harrell
    Oct 13, 2015 at 12:18
  • $\begingroup$ @FrankHarrell, nice answer! My basic understanding of LASSO also suggests that LASSO itself is enough and a second stage regression is superfluous. However, Belloni & Chernozhukov (2013) cited in the OP claim that selecting variables with LASSO first and then fitting an OLS is superior (in some sense, under some assumptions). The paper is waaay too technical for me to follow, and I was not able to get the intuition behind it either. However, I hesitate to dismiss their idea just so. Could you elaborate on why pure LASSO should be preferable? (I might post the question separately later on.) $\endgroup$
    – Richard Hardy
    Aug 9, 2016 at 16:00
  • $\begingroup$ @FrankHarrell, moreover, answers on two related questions also do not dismiss OLS after LASSO right away: see 1 and 2. $\endgroup$
    – Richard Hardy
    Aug 9, 2016 at 16:05
  • $\begingroup$ It could not possibly be standard OLS as penalization would then be completely ignored. $\endgroup$
    – Frank Harrell
    Aug 10, 2016 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.