Getting feature importance for random forest through cross-validation

Question

I'm working on building a random forest to classify samples based upon gene expression data. The data has ~11,000 samples, ~60,000 features and there are 37 classes. I have already determined the optimal number of trees using the OOB error rate and I performed 5-fold cross-validation to validate the model's performance. I'm able to get ~93% accuracy using all the features, but want to try to remove unnecessary features from the data to reduce noise and possibly get better performance.

I was able to get the feature importance values from each of the 5 models built in the 5-fold cross-validation. But I am wondering if it is reasonable to take the average of the 5 importance values for each feature in order to select the top N features for the final model. If I do average the 5 feature importance values for each feature, select the top 20k, and then do another 5-fold cross-validation on the model with the top 20k features, is it meaningful? Or will this cause leaking and overfitting since the feature selection was based off of all the data because I averaged the importances from all 5 models from the cross-validation?

If this sounds like the wrong way to go about the feature selection, what is a better route? Should I just pick the top features based upon the ranked feature importance from the best-performing of the 5 models from the cross-validation?

Thanks for any and all input!

Answer 1

A more rigorous way to pursue this question is to apply the Boruta algorithm.

Boruta repeatedly measures feature importance from a random forest (or similar method) and then carries out statistical tests to screen out the features which are irrelevant. The procedure terminates when all features are either decisively relevant or decisively irrelevant.

There are several papers on this topic. Here's one. "The All Relevant Feature Selection using Random Forest" by Miron B. Kursa, Witold R. Rudnicki