I have a dataset with about 5,000 often correlated features / covariates and a binary response. The data was given to me, I didn't collect it. I use Lasso and gradient boosting to build models. I use iterated, nested cross validation. I report Lasso's largest (absolute) 40 coefficients and the 40 most important features in the gradient boosted trees (there was nothing special about 40; it just seemed to be a reasonable amount of information). I also report the variance of these quantities over the folds and iterations of CV.
I kind of muse over the "important" features, making no statements about p-values or causality or anything, but instead considering this process a kind of---albeit imperfect and sort of random---insight into some phenomenon.
Assuming I have done all this correctly (e.g., executed cross validation correctly, scaled for lasso), is this approach reasonable? Are there issues with, e.g., multiple hypothesis testing, post hoc analysis, false discovery? Or other problems?
Objective
Predict the probability of an adverse event
- Foremost, estimate the probability accurately
- More minor--as a sanity check, but also to perhaps reveal some novel predictors that could be investigated further, inspect coefficients and importances as mentioned above.
Consumer
- Researchers interested in predicting this event and the people who end up having to fix the event if it occurs
What I want them to get out of it
Give them the ability to predict the event, if they wish to repeat the modeling process, as described, with their own data.
Shed some light on unexpected predictors. For example, it might turn out that something completely unexpected is the best predictor. Modelers elsewhere therefore might give more serious consideration to said predictor.