10
$\begingroup$

I know this would muddy the statistical inference, but I am really only concerned with getting as close to an accurate model as I can.

I have a dichotomous outcome variable, with a large set of dichotomous predictors. I am thinking I would like to try using LASSO to select which variables I should include in my model, then input those selected variables in to a Logit regression.

Is there something I am overlooking when it comes to the practicality of this approach?

Improve this question
$\endgroup$
7
  • 4
    $\begingroup$ You're overlooking that you can use LASSO's L1-norm penalty in logistic regression just as in linear regression. $\endgroup$
    – Scortchi ♦
    Commented Jan 26, 2015 at 21:52
  • 1
    $\begingroup$ And that LASSO shrinks as well as selecting, which you'd be undoing. $\endgroup$
    – Scortchi ♦
    Commented Jan 26, 2015 at 22:10
  • $\begingroup$ So, that is what I thought (vis-a-vis) shrinking). I am using the LARS package in STATA. The model it outputs does not give an intercept, so it can't be shrinking, right? $\endgroup$
    – EvKohl
    Commented Jan 26, 2015 at 22:41
  • $\begingroup$ The intercept in a logit model is fixed by the ratio of positives to negatives. $\endgroup$
    – Sycorax ♦
    Commented Jan 27, 2015 at 0:10
  • 2
    $\begingroup$ You can always include a column of ones to your data to estimate the intercept term. But indeed there is no need for running to separate models. Just use logistic regression with L1 penalty. $\endgroup$
    – Sven
    Commented Jan 27, 2015 at 7:44

2 Answers 2

Reset to default
9
$\begingroup$

There is a package in R called glmnet that can fit a LASSO logistic model for you! This will be more straightforward than the approach you are considering. More precisely, glmnet is a hybrid between LASSO and Ridge regression but you may set a parameter $\alpha=1$ to do a pure LASSO model. Since you are interested in logistic regression you will set family="binomial".

You can read more here: http://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#intro

Improve this answer
$\endgroup$
5
  • $\begingroup$ (+1) There's doubtless a Stata package for this too - Statalist would be the best place to ask. $\endgroup$
    – Scortchi ♦
    Commented Jan 27, 2015 at 12:55
  • $\begingroup$ Thanks. I actually don't think there is a STATA package for it. All the mention I found was for R. $\endgroup$
    – EvKohl
    Commented Jan 27, 2015 at 14:17
  • 3
    $\begingroup$ Googling for stata lasso logistic gives me homepages.ucl.ac.uk/~ucakgam/stata.html as the first result. $\endgroup$
    – Scortchi ♦
    Commented Jan 27, 2015 at 16:36
  • $\begingroup$ Anyone aware of a package in Python that can do this as well? $\endgroup$
    – rbm
    Commented Sep 10, 2016 at 16:09
  • $\begingroup$ @rbm I'm most certainly late for the party, but you can apply regularisation to logistic regressors in scikit-learn. $\endgroup$
    – Eli Korvigo
    Commented Nov 6, 2017 at 23:07
2
$\begingroup$

First, there's no guarantee that a linear probability model will approximate a logit model very well; consequently the subset of variables selected for one may be less appropriate for the other.

Second, the re-fitting applies no shrinkage at all, despite the variable selection that's taken place in the first step; risking serious mis-calibration & perhaps a little loss of discrimination.

You may be able to validate the procedure on a particular data-set, but it doesn't seem safe in general, or to offer any advantage over a stepwise logistic regression. And of course it's unnecessary; LASSO's $L_1$-norm penalty can be used for shrinkage & selection in logistic regression.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.