I have done some variable reduction before my LASSO regression. I end up with 13 variables (from 100). Some of them are kind of similar. Like for example, if I wanted to predict if it were going to rain or not, I might have something to record the humidity at some altitude intervals like 10m, 20m, 30m, etc. In turn, I think this might have an impact on rain, but I'm not sure which level, so I "test" them all. So in my LASSO regression I end up with like 4 of these, so for example 50, 60, 70, and 90 meters. In my opinion, it should suffice to only have the most significant one. But should there even be "extra" variable reduction after LASSO, or how to one go about this ?

  • $\begingroup$ Why do you want to do LASSO in the first place? $\endgroup$
    – Edgar
    Oct 21 '19 at 11:20
  • $\begingroup$ To reduce number of predictors in my model. $\endgroup$ Oct 21 '19 at 11:33
  • $\begingroup$ Why do you want to reduce the number of predictors? $\endgroup$
    – Edgar
    Oct 21 '19 at 11:45
  • $\begingroup$ 1) I have too many 2) When reporting this in papers etc, a model should not consist of 30 or more variables. No one (at least in my field) would never touch it, and then there is the event to predictor ratio, which is very poor if I use all my predictors. $\endgroup$ Oct 21 '19 at 11:48
  • $\begingroup$ Sorry for the many questions. But, what is the event to predictor ratio? I couldn't find this on Google. Also, how many samples do you have? $\endgroup$
    – Edgar
    Oct 21 '19 at 11:53

This paper should give you some insight on variable selection and shrinkage estimation. One important remark from the paper:

Note that the number of candidate predictors should be considered in this reasoning, not the number of predictors included in the final model.

The LASSO selects and estimates (with shrinkage) at the same time, but I would read from the paper that you still need 10 to 20 events per predictor to do so. Furthermore, LASSO is known to behave badly when predictors are highly correlated - often LASSO will choose one of the correlated predictors by random and discard the other correlated ones. This can be alleviated to some extent by the use of Elastic Net (a combination of LASSO and ridge regression).

In any case, you didn't tell how you reduced the number of rpedictors from 100 to 13 in the first place (domain/external knowledge?). If you want to report for your problem a final model for prediction, I suggest the following:

  1. report all prospective (100) predictors and give reasoning why you excluded a large part based on expert/external knowledge
  2. do an exploratory PCA on the remaining 13 predictors with biplots, and/or calculate correlations between predictors: this way you might be able to reduce the number of predictors, especially like the altitude humidity, where you could exclude highly correlated ones (note that you didn't peek at the outcome yet!)
  3. do LASSO/elastic net logistic regression with your chosen "full" predictor set and the binary outcome (preferably with cross-validation). Report the chosen set of non-zero predictors and their coefficients.

Don't do another regression with the reduced set of predictors!

Another possibility: If you have a group of very similar predictors and you are dead-set on choosing just one of them (like the best humidity-altitude out of several options), you could train one model for each of these predictors where you exclude the other ones from the group. You will then choose the model with the smallest AIC or some other criterion.

  • $\begingroup$ I really appreciate your answer. But you are right, I didn't explain how I went from 100 to 13. In some cases, I don't even get that low (I am doing several models). I am kind of following the trend in my field in regards to how this is usually done, exactly because we have MANY correlated variables. And yes, a kind of stepwise approach have also crossed my mind, but that is, as you probably also know, very frowned upon (for good reason). But often there are just not a better alternative. Then it would be to report a model with 10-15 predictors (which I find a bit too much for reporting). $\endgroup$ Oct 21 '19 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.