5
$\begingroup$

I know that if you re-run a random forest with a different random seed you will fit a different model. I'm wondering whether it's acceptable to compare different random forest models (run under different random seeds) and to take the model with the highest accuracy on the training data (using 10-fold CV) for downstream work.

As an example, below is the distribution of accuracies based on 10-fold CV in a dataset with 147 samples and 278 features (all two class features) used to predict disease state (two classes: healthy / diseased). This distribution is based on 100 RF replicates with different random seeds:

Is it wrong to take the model with the highest accuracy for feature selection and to fit my test data?

I'm also interested in comparing the chosen model's 10-fold CV accuracy to additional RF models' accuracies fit to the same dataset with randomized disease states (as an alternative way to evaluate significance since I think my sample size might be too small to split it into test/training data). I'm concerned that this approach might be biased if I choose the model with the highest accuracy.

$\endgroup$
  • 4
    $\begingroup$ Cf. p-hacking. This procedure just searching for a set of boostraps and feature splits which happen to match up well to your test data. A model which generalizes well should be robust to the choice of seed. $\endgroup$ – Sycorax says Reinstate Monica Jul 5 '16 at 17:23
  • $\begingroup$ I guess that you are measuring accuracy based on different data set (Different seeds let to different 10-fold OOB sets). So the accuracy observed in the graph varies because of different data set. If you test different models on the same testing set, I think that you will not observe such variation. $\endgroup$ – Metariat Jul 5 '16 at 17:25
  • 2
    $\begingroup$ My instinct definitely says that just taking the "best" seed is a bad idea. I suspect the right thing to do would be to ensemble the results. $\endgroup$ – shadowtalker Jul 5 '16 at 17:28
  • $\begingroup$ @General Abrial - That's what I was concerned about and I'm trying to avoid data fishing. Are you implying that fitting a RF to this dataset is meaningless? $\endgroup$ – gavin Jul 5 '16 at 17:34
  • 1
    $\begingroup$ I'm implying that seeking out a seed that suits your choice of test set is meaningless. RF might be fine for your purposes. I can't say. The variations in accuracy that you're observing may be within your tolerance for error or statistical noise. But any results which intrinsically depends on the choice of seed are probably meaningless -- it's like saying I'm good at predicting what card I'll draw, but only if I can arrange the deck to my liking first. $\endgroup$ – Sycorax says Reinstate Monica Jul 5 '16 at 17:37
4
$\begingroup$

Accuracy is just another random variable that depends on your model, the seed, the train / test split, quality of your current data etc. - maximizing this random variable does not automatically lead to the best possible generalization of your model.

Besides looking at metrics like accuracy, logloss, auc roc etc. you might also want to look at other learning characteristics like the learning curves of your train / testdata while adding more data, the difference between the train and test error etc., as finally all you are facing is the bias-variance-tradeoff that lives in every model.

See https://en.m.wikipedia.org/wiki/Bias–variance_tradeoff

To answer your question, you should not only rely on comparing the random seed.

$\endgroup$
2
$\begingroup$

Random Forest converges with growing number of trees, see the Breiman 2001 paper. So if you would set the number of trees (ntree) to infinity, you would always get the same accuracy (or some other measure like logloss). It only varies a lot because your number of trees is too small (or your resampling strategy (10-fold-CV) is to unstable, can be reduced by more repetitions).

In normal data situations (especially if the data is big enough) your accuracy should grow with growing trees. So instead of training with 100 different seeds I would train one randomForest with actual_ntree * 100 or even more.

In some packages you can also see the development of the accuracy with growing number of trees.

For getting a faster evaluation and possibly tuning you can use out-of-bag estimations, that are usually implemented in standard packages (like randomForest in R). They are normally as good as 10 fold-CVs (and more stable) if the number of trees is big enough.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.