I know that if you re-run a random forest with a different random seed you will fit a different model. I'm wondering whether it's acceptable to compare different random forest models (run under different random seeds) and to take the model with the highest accuracy on the training data (using 10-fold CV) for downstream work.
As an example, below is the distribution of accuracies based on 10-fold CV in a dataset with 147 samples and 278 features (all two class features) used to predict disease state (two classes: healthy / diseased). This distribution is based on 100 RF replicates with different random seeds:
Is it wrong to take the model with the highest accuracy for feature selection and to fit my test data?
I'm also interested in comparing the chosen model's 10-fold CV accuracy to additional RF models' accuracies fit to the same dataset with randomized disease states (as an alternative way to evaluate significance since I think my sample size might be too small to split it into test/training data). I'm concerned that this approach might be biased if I choose the model with the highest accuracy.