So I am using LASSO in order to reduce the amount of independent variables (~150) I have for my logistic model (n=1200). However, when doing so, the end result (i.e. number of predictors) it chooses varies depending on the seed I provide. 2-3 predictors are consistent, but then sometimes others are added. I guess this is due to the cross-validation for the best lambda parameter often used for LASSO. However, it just seem so inconsistent.

I mean I am using bootstrap in order to validate my results. So in the end I end up with meaningful standard errors, frequency of predictors, corrected statistics etc., but that doesn't change the fact, that the main model, defined on the original data, is kind of random depending on the seed.

How does one "solve" this ? Run the "main model" (i.e. original data) with 100-1000 different seeds to see which predictors are most frequently chosen, and then chose the model that actually has these predictors, or is there anything else that could be recommended ?

  • 2
    $\begingroup$ Is the seed affecting individual LASSO results or training/test splits and cross-validation? The first would surprise me while the second seems inevitable $\endgroup$ – Henry Jan 13 '20 at 11:26
  • $\begingroup$ Well, since the optimal lambda for each LASSO are chosen from cross-validation, it does affect the individual LASSO results. The problem is that because of the CV for the lambdas when developing the "true" model on the original data, I can get 2 predictors in one case, and 5 in the other, again depending on the seed used, which consequently affects the CV procedure. $\endgroup$ – Denver Dang Jan 13 '20 at 14:19
  • $\begingroup$ Cross-validation, when based on random partitions (apart from leave-one-out procedures) and empirical optimisation, is likely to be affected by the particular random selection. It may not make much difference to prediction skill, and such procedures (including the use of LASSO) may be unsuitable for inference anyway $\endgroup$ – Henry Jan 13 '20 at 15:20
  • $\begingroup$ It just seems a bit unstable that my potential prediction model either includes 2 or 5 predictors, doesn't it ? I agree, the predictive power is probably somewhat similar between models, since the ones that are always chosen probably are the most important drivers in this model. My concern is just: Which model should then be chosen ? The one with the fewest predictors, or maybe pick one where the seed gives 1 or 2 extra predictors. That's what's I am struggling with right now. $\endgroup$ – Denver Dang Jan 13 '20 at 15:30

If you get substantially different results from cross-validation runs with different seeds, there simply is not a strong enough signal in your data to pick one "best" model with high confidence. CV is telling you there are several pretty good models and you cannot be sure which is "true" or "best." That said, if you want to pick lambda with CV and really really want to avoid differences between random partitions into folds, you could just use leave-one-out CV, i.e. use n folds.

If that is too computationally expensive (you said your n=1200), just fit the lasso path to the full dataset, and decide where to stop by using another stopping rule, like AIC (if your ultimate goal is prediction) or BIC (if variable selection is itself the primary goal).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.