Correct way of evaluating Random Forest performance wrt training/test, feature selection, ntrees, random seed

I need to use Random Forest in my experiments. Although using the same training and test datasets, each time that I train the Random Forest on my training set, I get a different result on my test set. I know that this variation is due the randomness in the algorithm. But how can I decrease that? What is the correct way of reporting the results if I need to compare two different feature sets for example? Should I run multiple times on the same training, test data and report the average test results?

My other related question is the forest size. What I understood from parameter tuning of random forest was that the bigger number of trees is better, it is not going to hurt accuracy, it just does not lead to huge improvement after a certain size. however, When I tried this parameter I noticed that after n=400, the accuracy decreases. Can someone explain me why? Is my understanding right?

Below you can find the result of my experiment, trying different forest sizes (n) and for each forest size, I trained and evaluated the RF 10 times. printed the accuracy of test set for each run and at the end reported the mean and std of the accuracy for the associated n.

n: 50
0: 0.585
1: 0.61
2: 0.588333333333
3: 0.606666666667
4: 0.598333333333
5: 0.595
6: 0.598333333333
7: 0.578333333333
8: 0.586666666667
9: 0.59
n: 50 mean: 0.593666666667 std: 0.00939266853574
-------------------------------
n: 100
0: 0.601666666667
1: 0.59
2: 0.586666666667
3: 0.603333333333
4: 0.6
5: 0.593333333333
6: 0.568333333333
7: 0.6
8: 0.596666666667
9: 0.591666666667
n: 100 mean: 0.593166666667 std: 0.00975961064797
-------------------------------
n: 200
0: 0.595
1: 0.595
2: 0.591666666667
3: 0.58
4: 0.606666666667
5: 0.625
6: 0.596666666667
7: 0.603333333333
8: 0.605
9: 0.61
n: 200 mean: 0.600833333333 std: 0.0115289490703
-------------------------------
n: 300
0: 0.608333333333
1: 0.596666666667
2: 0.605
3: 0.605
4: 0.593333333333
5: 0.613333333333
6: 0.611666666667
7: 0.595
8: 0.595
9: 0.616666666667
n: 300 mean: 0.604 std: 0.00810349718743
-------------------------------
n: 400
0: 0.601666666667
1: 0.601666666667
2: 0.608333333333
3: 0.608333333333
4: 0.606666666667
5: 0.601666666667
6: 0.606666666667
7: 0.598333333333
8: 0.591666666667
9: 0.606666666667
n: 400 mean: 0.603166666667 std: 0.00502493781056
-------------------------------
n: 500
0: 0.61
1: 0.598333333333
2: 0.6
3: 0.608333333333
4: 0.608333333333
5: 0.613333333333
6: 0.595
7: 0.6
8: 0.59
9: 0.598333333333
n: 500 mean: 0.602166666667 std: 0.00707303172464
-------------------------------
n: 600
0: 0.605
1: 0.596666666667
2: 0.61
3: 0.603333333333
4: 0.596666666667
5: 0.588333333333
6: 0.598333333333
7: 0.588333333333
8: 0.6
9: 0.6
n: 600 mean: 0.598666666667 std: 0.0064463598686
-------------------------------
n: 700
0: 0.593333333333
1: 0.605
2: 0.595
3: 0.596666666667
4: 0.603333333333
5: 0.61
6: 0.598333333333
7: 0.601666666667
8: 0.6
9: 0.601666666667
n: 700 mean: 0.6005 std: 0.00471699056603


Please let me know what is the correct way of evaluating the performance of Random Forest.

• You do not necessarily increase accuracy by increasing number of bootstrap replicates. In fact, if you run too many trees you run the risk of over correlating the ensemble and causing an overfit problem. There is also a balance between error convergence and accounting for interactions, which stabilize at a slower rate than error. Plot your RF object to assess error convergence. As a rule of thumb I run twice the number of trees than required for error stabilization. More parameters require a longer burn-in. Oct 29, 2013 at 23:27
• Please don't call it 'forest size', call it 'number of trees' (ntree). The 'size' of an individual tree would actually be its number of nodes, so kind of a bad choice of terminology.
– smci
Jun 8, 2015 at 22:05

In order to obtain reproducible randomizations you have to set up a random seed beforehand, e.g. in R you do set.seed(17) where 17 is just a number I made up. When you do it you should get the same accuracy every random forest run when you fix the number of trees.

I don't think your accuracy really decreases when you increase the number of trees. The accuracy results with a few trees have higher variances and even if the mean value is slightly higher (in your case little higher) this does not mean they are better. Your results might be due to different randomization of the data. Try to set up a random seed beforehand.

If you don't have a specific set of records to use as test set you might obtain more stable results computing the accuracy by cross validation. A 10-fold cross validation is usually the measure of choice. For a better estimate you can also average the results of many cross validation with different randomizations. It might be time consuming, an average among 5 randomizations of 2-fold cross validation might be good practice if you don't have too few records. If you have a few records, for example less then 100, you can try the leave one out method.

If you want to learn more about classifier evaluation the book Evaluating Learning Algorithms: A Classification Perspective by Nathalie Japkowicz is a good reference.

• What do you suggest in the case the test set is given and one needs to evaluate random forest over a different set of features and/or parameters? Would it be a good idea to run the algorithm without any seed like 10 times and use the average as a good estimator? Aug 24, 2014 at 15:37
• If you have a test set you might just use that for your evaluation. Just to obtain reproducible results set 10 different random seeds and average across the results. At least every time you run your algorithm you should obtain the same results. Aug 25, 2014 at 3:40
• @Simone: and if you can't afford to run 10 random seeds, at least do 3.
– smci
Aug 29, 2015 at 4:22