I have done some variable reduction before my LASSO regression. I end up with 13 variables (from 100). Some of them are kind of similar. Like for example, if I wanted to predict if it were going to rain or not, I might have something to record the humidity at some altitude intervals like 10m, 20m, 30m, etc. In turn, I think this might have an impact on rain, but I'm not sure which level, so I "test" them all. So in my LASSO regression I end up with like 4 of these, so for example 50, 60, 70, and 90 meters. In my opinion, it should suffice to only have the most significant one. But should there even be "extra" variable reduction after LASSO, or how to one go about this ?
This paper should give you some insight on variable selection and shrinkage estimation. One important remark from the paper:
Note that the number of candidate predictors should be considered in this reasoning, not the number of predictors included in the final model.
The LASSO selects and estimates (with shrinkage) at the same time, but I would read from the paper that you still need 10 to 20 events per predictor to do so. Furthermore, LASSO is known to behave badly when predictors are highly correlated - often LASSO will choose one of the correlated predictors by random and discard the other correlated ones. This can be alleviated to some extent by the use of Elastic Net (a combination of LASSO and ridge regression).
In any case, you didn't tell how you reduced the number of rpedictors from 100 to 13 in the first place (domain/external knowledge?). If you want to report for your problem a final model for prediction, I suggest the following:
- report all prospective (100) predictors and give reasoning why you excluded a large part based on expert/external knowledge
- do an exploratory PCA on the remaining 13 predictors with biplots, and/or calculate correlations between predictors: this way you might be able to reduce the number of predictors, especially like the altitude humidity, where you could exclude highly correlated ones (note that you didn't peek at the outcome yet!)
- do LASSO/elastic net logistic regression with your chosen "full" predictor set and the binary outcome (preferably with cross-validation). Report the chosen set of non-zero predictors and their coefficients.
Don't do another regression with the reduced set of predictors!
Another possibility: If you have a group of very similar predictors and you are dead-set on choosing just one of them (like the best humidity-altitude out of several options), you could train one model for each of these predictors where you exclude the other ones from the group. You will then choose the model with the smallest AIC or some other criterion.