# Standard logistic regression post-Lasso

The situation I'm interested in is logistic regression for a binary response variable with lots of predictors (500 to 1000), lots of which are correlated. I would like to use a logistic LASSO approach to identify the most promising predictors. I could either use that solution directly, or only use it to identify a concise set of good predictors and use standard logistic regression on only these predictors.

I am aware of the analogous discussion regarding linear regression and LASSO(see, e.g. the recent paper by Belloni and Chernozhukov), but I have difficulties finding results that help me to make this decision in the logistic regression setting, and for practical purposes (prediction) rather than academical ones. I am also aware that logistic regression can also be viewed from an ordinary least squares perspective (see e.g. Hastie/Tibshiranis Elements of Statistical Learning, p. 124), but I currently don't see how to use this to transfer results from the linear to the logistic setting.

Could you give me pros and cons for both alternatives (direct logistic LASSO or standard logistic regression post-LASSO) or explain how the ordinary-least-squares-perspective on logistic regression can be used to transfer arguments from the linear regression case?

• Lasso classifier is a ready solution that combines feature selection (in cases there are too many features or collinearity present) with classification (see R's glmnet package e.g). Why would you think about filtering features in Lasso and then carrying these results over to logistic classifier at all? At the very least cons are you'd be doing your job twice, and even worse, the set of variables that performed well in Lasso may not do so well in logistic regression. – Sergey Bushmanov Oct 13 '15 at 11:06
• The motivation for this is as follows. When you use out-of-the-box LASSO, you buy less variance in exchange for some bias. Standard logistic regression, on the other hand, is asymptotically unbiased. So this is an attempt at fine-tuning the bias-variance trade-off, in a way that often seems to work very well (see the Belloni-Chernozhukov paper and the references for some evidence) in the case of linear regression. – MightyCurious Oct 13 '15 at 11:35
• lasso is not a classifier – Frank Harrell Oct 13 '15 at 11:43
• @FrankHarrell I never used glment for classifying, but I believe glmnet(x, y, family=c("binomial") does classification, don't you think so? – Sergey Bushmanov Oct 13 '15 at 11:50
• Not at all. It is for prediction, here of probabilities (risk). – Frank Harrell Oct 13 '15 at 11:55

If you want to emphasize parsimony over predictive accuracy, then model approximation (also called pre-conditioning) may be useful. Here one uses the best available prediction method, e.g. penalized maximum likelihood estimation with a quadratic penalty, i.e., ridge logistic regression. Then $X\hat{\beta}$ from that model is approximated using stepwise regression or recursive partitioning, etc. The approximate model inherits the proper amount of shrinkage from the full model. The approximate model may be chosen to yield $R^{2} = 0.95$ against the gold standard linear predictor.