7
$\begingroup$

I use Lasso logistic regression in order to identify a smaller subset of important variables. I start with N=51 (28/23) and 32 predictors.

So far it looks pretty promising, because I can identify four important predictors in my optimal model.

Now I would like to take those four predictors and examine them along with some control variables in a standard logistic regression.

My question is, does that analysis strategy make sense? Is there a better way to include controls or other variables that might be interesting?

For a better understanding:

  1. Identify important variables via Lasso logistic regression

  2. Do further analysis including identified predictors and other control variables using standard logistic regression (using AIC to check model fit)

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I guess the answer depends on what are you trying to do (and even then it wont be definitive). Please tell us whether you're trying to predict, uncover the data generating process (inference?) or something else? $\endgroup$
    – sheß
    Commented May 17, 2016 at 17:14
  • 5
    $\begingroup$ You shouldn't trust those results too much. With 32 variables and n = 51, it's almost impossible to believe that you wouldn't get perfect separation purely by chance. With such a small sample size and large covariate space, no method will really be able to handle issues of over fitting. $\endgroup$
    – Cliff AB
    Commented May 17, 2016 at 18:32
  • $\begingroup$ @sheß I'm trying to find relations between word usage (IVs) and investment success (DV). So it's not prediction, I just want to find variables strongly related that help explain the event. $\endgroup$
    – Toby_Shoby
    Commented May 18, 2016 at 11:26
  • $\begingroup$ @Cliff AB what would you suggest, should I reduce the set of predictors? $\endgroup$
    – Toby_Shoby
    Commented May 18, 2016 at 11:26
  • $\begingroup$ @CliffAB that's not true. With cross-validation applied correctly, we should get an accurate estimate of out-of-sample error. See "The Elements of Statistical Learning", section 7.10.3 ("Does Cross-Validation Really Work?"), where they examine a case of 20 samples and 500 predictors independent of the class labels. $\endgroup$
    – rinspy
    Commented Feb 5, 2018 at 16:57

2 Answers 2

Reset to default
12
$\begingroup$

Note that there exist multiple iterative LASSO procedures, so in general, it is not necessarily true that you should stick with the first LASSO estimates.

For example:

  • Post-LASSO-OLS: see Belloni, Chernozhukov (2013) Least squares after model selection in high-dimensional sparse models, Bernoulli 19(2), 2013, 521–547. Also known as the LASSO-OLS hybrid (Efron et al 2004, Least angle regression. Annals of Statistics 32 407–451)

  • Adaptive LASSO (Zou 2006), eventually multiple stages (Bühlman, Meier 2008). Two-stages (or more), both using a CV procedure, the second step using a modified (re-weighted) penalty.

  • Relaxed LASSO (Meinshausen 2007), on a bunch of subsets computed by initial LASSO

Now in general, I would use one of these procedures to decide whether or not to add more variables, instead of a BIC model selection procedure.

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ These are also good suggestions that avoid proceeding "as if no variable selection had happened", which was the main thing I was warning about. $\endgroup$
    – Björn
    Commented May 20, 2016 at 7:35
  • $\begingroup$ Could you give a full citation of Efron et al. (2004) for LASSO-OLS hybrid? Thank you! $\endgroup$
    – Richard Hardy
    Commented Aug 9, 2016 at 16:12
  • $\begingroup$ With pleasure @RichardHardy ! It is in the LARS paper (ref just added), page 421. They refer to it actually as LARS-OLS hybrid. $\endgroup$
    – Matifou
    Commented Aug 10, 2016 at 7:00
  • $\begingroup$ Already upvoted, so no extra points, but thanks! Also, I am trying to understand what kind of benefits OLS could bring after LASSO. I see no intuitive reason. I tried reading Belloni & Chernozhukov (2013) but it is way too technical... I will probably post a separate question on that, but would you have any hints right away? What kind of magic is at work such that OLS improves after LASSO? What kind of situations does that happen in (what assumptions are needed)? $\endgroup$
    – Richard Hardy
    Commented Aug 10, 2016 at 7:35
  • $\begingroup$ Very late answer Richard, but my intuition on this is that LASSO does both selection and estimation, but biased estimation. So idea is to use only the LASSO for selection part, then do (unbiased) OLS. Obviously, OLS being unbiased in general does not mean it is unbiased after selection, see the nice paper by Leeb Potscher on that, "Model Selection and Inference: Facts and Fiction" $\endgroup$
    – Matifou
    Commented Apr 8, 2019 at 17:56
9
$\begingroup$

Performing some variable selection (e.g. with LASSO with the smoothing parameter chosen by cross-validation or some of the other alternatives like the elastic net etc.) and then fitting a model on the same data as if no variable selection had happened is always inappropriate. Why not look at the results from LASSO? As stated by others lots of predictors with few records is of course tricky, but at least these will have some shrinkage of the coefficients to account for the variable selection.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thx @Björn for the answer - very helpful. I'm glad that applying Lasso makes some sense here. I already guessed that fitting the selected data again is not the best idea. What could I do to about my control variables? Add them to the Lasso? Then it would be some more variables. Should I exclude some of the predictors? Thx in advance! $\endgroup$
    – Toby_Shoby
    Commented May 18, 2016 at 11:06
  • 2
    $\begingroup$ How are the "control variables" fundamentally different from the predictors? If they are known to be relevant, it is more logical to have them in the LASSO already (otherwise important things are ignored), but not with the LASSO shrinkage applied to them that can exclude them from the model. However, due to the small dataset, it is likely still a good idea to have some control for overfitting e.g. by using ridge regression type of penalty (e.g. elastic net with no L1 penalty for the control variables?! I assume something like this has been done before, but I don't know any examples.). $\endgroup$
    – Björn
    Commented May 20, 2016 at 7:33
  • 3
    $\begingroup$ Don't trust anything until you bootstrap the entire process and see if the list of selected variables by lasso is stable. (It won't be) $\endgroup$
    – Frank Harrell
    Commented Mar 17, 2017 at 11:56
  • $\begingroup$ Could you expand on why it is not appropriate? $\endgroup$
    – confused
    Commented Nov 6, 2020 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.