# Using LASSO for variable selection, then using Logit

I know this would muddy the statistical inference, but I am really only concerned with getting as close to an accurate model as I can.

I have a dichotomous outcome variable, with a large set of dichotomous predictors. I am thinking I would like to try using LASSO to select which variables I should include in my model, then input those selected variables in to a Logit regression.

Is there something I am overlooking when it comes to the practicality of this approach?

• You're overlooking that you can use LASSO's L1-norm penalty in logistic regression just as in linear regression. – Scortchi Jan 26 '15 at 21:52
• And that LASSO shrinks as well as selecting, which you'd be undoing. – Scortchi Jan 26 '15 at 22:10
• So, that is what I thought (vis-a-vis) shrinking). I am using the LARS package in STATA. The model it outputs does not give an intercept, so it can't be shrinking, right? – EvKohl Jan 26 '15 at 22:41
• The intercept in a logit model is fixed by the ratio of positives to negatives. – Sycorax Jan 27 '15 at 0:10
• You can always include a column of ones to your data to estimate the intercept term. But indeed there is no need for running to separate models. Just use logistic regression with L1 penalty. – Sven Jan 27 '15 at 7:44

There is a package in R called glmnet that can fit a LASSO logistic model for you! This will be more straightforward than the approach you are considering. More precisely, glmnet is a hybrid between LASSO and Ridge regression but you may set a parameter $\alpha=1$ to do a pure LASSO model. Since you are interested in logistic regression you will set family="binomial".

You can read more here: http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#intro

• (+1) There's doubtless a Stata package for this too - Statalist would be the best place to ask. – Scortchi Jan 27 '15 at 12:55
• Thanks. I actually don't think there is a STATA package for it. All the mention I found was for R. – EvKohl Jan 27 '15 at 14:17
• Googling for stata lasso logistic gives me homepages.ucl.ac.uk/~ucakgam/stata.html as the first result. – Scortchi Jan 27 '15 at 16:36
• Anyone aware of a package in Python that can do this as well? – rbm Sep 10 '16 at 16:09
• @rbm I'm most certainly late for the party, but you can apply regularisation to logistic regressors in scikit-learn. – Eli Korvigo Nov 6 '17 at 23:07

First, there's no guarantee that a linear probability model will approximate a logit model very well; consequently the subset of variables selected for one may be less appropriate for the other.

Second, the re-fitting applies no shrinkage at all, despite the variable selection that's taken place in the first step; risking serious mis-calibration & perhaps a little loss of discrimination.

You may be able to validate the procedure on a particular data-set, but it doesn't seem safe in general, or to offer any advantage over a stepwise logistic regression. And of course it's unnecessary; LASSO's $L_1$-norm penalty can be used for shrinkage & selection in logistic regression.