feature importance aggregation I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. The dataset includes over 100000 rows, and close to 1000 predictors. The prevalence of the condition is about 10%.
Now, the etiology of the disease we are trying to predict is largely unknown. Thus, we likely don’t have data on many important predictors for the condition. That is to say, as a prediction model, any model I come up with is going to do a rather poor job predicting the outcome. However, the primary aim here is not about prediction, but rather to identify important variables which we can then target more directly in future analyses. So I am trying to use the ML model as a variable selection tool.
Normally we can get a sense of the model performance by evaluating its metrics on a new dataset – for example by using nested cross validation or a train-test split. But rather than evaluating the model’s metrics, my primary interest here is to evaluate the consistency by which the different predictors are being chosen (i.e. the consistency of the feature importance list). So essentially, I think what I want to do is to randomly split the database (say use 60% of the data), run CV to tune the hyperparameters, and then using the best hyperparameters train the model on the full 60% and get the feature importance. Then I would repeat the same process X number of times, each time using a different randomly chosen 60% sample. This would give me X number of tables of feature importance, one from each run. But is there a way to then somehow “merge” all these feature importance tables to get a sense of how stable the selection process is across the different runs? Or are there better ways altogether to do this?
Thanks a lot!
 A: Unless you think that interactions among the features are critical, you could consider a LASSO logistic regression as a way to do your feature selection. Figure 6.4 of Statistical Learning with Sparsity shows how to display feature-selection stability based on multiple models built on bootstrap resampling of data.* Boxplots of regression coefficients among the models (left of figure) and probabilities of feature inclusion (right of figure) provide the type of information you seem to be looking for. (Two-dimensional displays of co- or counter-inclusion might be nice, although maybe unwieldy with 1000 candidate predictors.)
If you think that yet-unknown interactions among the features are critical, then boosted trees with some tree depth to allow for interactions make sense over LASSO (provided that you use something related to a proper scoring rule as your criterion). Tree models, including boosted trees, can provide a quantitative relative importance for all features in a model on a scale of 0 to 100 (e.g., Figure 10.6 of The Elements of Statistical Learning). With multiple models based on resamples of the data, displays of the distributions of importance values for features among the models, analogous to those described in the first paragraph, would be helpful. For reassurance, you could compare the average feature importance among the multiple models against the feature importance seen in the model on the full data set. The downside is that just how the important features interact in a boosted model can be hard to glean.
A warning. You say:

the primary aim here is not about prediction, but rather to identify important variables which we can then target more directly in future analyses.

What you mean by "future analyses" is critical here. If those are on independently acquired data, fine. If you are to identify "important variables" from a data set and then use them for "future analyses" on the same data, however, then there will necessarily be some over-optimism in that later work.

*Your selection of a 60% subsample each time, without replacement, ends up close to the fraction of cases included in a standard bootstrap sample with replacement. There's an argument that bootstrap sampling better mimics original sampling from the population. I suspect that the difference might not matter much with this scale of data, but you might want to check.
A: Some addiotnal remarks on the "yet-unknown interactions among the features" mentioned by @edm.
If you have highly (multi-)correlated features, then Lasso or Random Forest can just take arbitrarily one of them and downgrade the others. If you do this several times in a randomized process, it might be that at different instances a different representant from the correlated group is chosen. Your suggestion to pick the features most often selected can thus be misleading in such a case.
To avoid such a situation, you can check for highly collinear features, e.g. by

*

*a hierarchical clustering on basis of the correlation matrix, i.e., which a distance measure $1-r^2_{ij}$

*removing variables with a high variance inflation factor (VIF)

These methods will only detect linear feature interactions, though.
