I know this would muddy the statistical inference, but I am really only concerned with getting as close to an accurate model as I can.

I have a dichotomous outcome variable, with a large set of dichotomous predictors. I am thinking I would like to try using LASSO to select which variables I should include in my model, then input those selected variables in to a Logit regression.

Is there something I am overlooking when it comes to the practicality of this approach?

  • 4
    $\begingroup$ You're overlooking that you can use LASSO's L1-norm penalty in logistic regression just as in linear regression. $\endgroup$ – Scortchi Jan 26 '15 at 21:52
  • 1
    $\begingroup$ And that LASSO shrinks as well as selecting, which you'd be undoing. $\endgroup$ – Scortchi Jan 26 '15 at 22:10
  • $\begingroup$ So, that is what I thought (vis-a-vis) shrinking). I am using the LARS package in STATA. The model it outputs does not give an intercept, so it can't be shrinking, right? $\endgroup$ – EvKohl Jan 26 '15 at 22:41
  • $\begingroup$ The intercept in a logit model is fixed by the ratio of positives to negatives. $\endgroup$ – Sycorax Jan 27 '15 at 0:10
  • 2
    $\begingroup$ You can always include a column of ones to your data to estimate the intercept term. But indeed there is no need for running to separate models. Just use logistic regression with L1 penalty. $\endgroup$ – Sven Jan 27 '15 at 7:44

There is a package in R called glmnet that can fit a LASSO logistic model for you! This will be more straightforward than the approach you are considering. More precisely, glmnet is a hybrid between LASSO and Ridge regression but you may set a parameter $\alpha=1$ to do a pure LASSO model. Since you are interested in logistic regression you will set family="binomial".

You can read more here: http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#intro

  • $\begingroup$ (+1) There's doubtless a Stata package for this too - Statalist would be the best place to ask. $\endgroup$ – Scortchi Jan 27 '15 at 12:55
  • $\begingroup$ Thanks. I actually don't think there is a STATA package for it. All the mention I found was for R. $\endgroup$ – EvKohl Jan 27 '15 at 14:17
  • 3
    $\begingroup$ Googling for stata lasso logistic gives me homepages.ucl.ac.uk/~ucakgam/stata.html as the first result. $\endgroup$ – Scortchi Jan 27 '15 at 16:36
  • $\begingroup$ Anyone aware of a package in Python that can do this as well? $\endgroup$ – rbm Sep 10 '16 at 16:09
  • $\begingroup$ @rbm I'm most certainly late for the party, but you can apply regularisation to logistic regressors in scikit-learn. $\endgroup$ – Eli Korvigo Nov 6 '17 at 23:07

First, there's no guarantee that a linear probability model will approximate a logit model very well; consequently the subset of variables selected for one may be less appropriate for the other.

Second, the re-fitting applies no shrinkage at all, despite the variable selection that's taken place in the first step; risking serious mis-calibration & perhaps a little loss of discrimination.

You may be able to validate the procedure on a particular data-set, but it doesn't seem safe in general, or to offer any advantage over a stepwise logistic regression. And of course it's unnecessary; LASSO's $L_1$-norm penalty can be used for shrinkage & selection in logistic regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.