I have more of a conceptual question I was hoping to get some feedback on. I am trying to run a boosted regression ML model to identify a subset of important predictors for some clinical condition. The dataset includes over 100000 rows, and close to 1000 predictors. The prevalence of the condition is about 10%. Now, the etiology of the disease we are trying to predict is largely unknown. Thus, we likely don’t have data on many important predictors for the condition. That is to say, as a prediction model, any model I come up with is going to do a rather poor job predicting the outcome. However, the primary aim here is not about prediction, but rather to identify important variables which we can then target more directly in future analyses. So I am trying to use the ML model as a variable selection tool.
Normally we can get a sense of the model performance by evaluating its metrics on a new dataset – for example by using nested cross validation or a train-test split. But rather than evaluating the model’s metrics, my primary interest here is to evaluate the consistency by which the different predictors are being chosen (i.e. the consistency of the feature importance list). So essentially, I think what I want to do is to randomly split the database (say use 60% of the data), run CV to tune the hyperparameters, and then using the best hyperparameters train the model on the full 60% and get the feature importance. Then I would repeat the same process X number of times, each time using a different randomly chosen 60% sample. This would give me X number of tables of feature importance, one from each run. But is there a way to then somehow “merge” all these feature importance tables to get a sense of how stable the selection process is across the different runs? Or are there better ways altogether to do this? Thanks a lot!