What's a good practice for combining results from multiple run of a classifier trained on same dataset but randomly split each time

Question

The goal is to train some classifiers to achieve decent classification accuracy, in order to find out what are the features that are most important.

The random forest classifier is being used to train on a dataset that is split to train and test sets randomly. Due to the relatively small dataset size compared to number of features, there is large variation in the set of important features between different random training & test set. E.g. train-test split 1 will give very different feature importance ranking compared to train-test split 2.

My questions would be:

1. Is doing multiple runs of the same dataset but split differently by random, an acceptable practice?
2. If so, what's the good or common practice for combining the results of different runs?

Answer 1

After some research, I came across the method called boruta feature selection which I believe is a good fit for this problem.

The algorithm performs multiple runs of training with additional features known as shadow features that are created by randomly shuffling actual features. Features that have importance score higher than the highest importance achieved by any shadow feature is marked as important. After multiple runs, the number of times a feature have been marked as important may be modelled as a binomial distribution. We can then pick the most importance feature based on a required p-value