# Correct way of evaluating Random Forest performance wrt training/test, feature selection, ntrees, random seed

I need to use Random Forest in my experiments. Although using the same training and test datasets, each time that I train the Random Forest on my training set, I get a different result on my test set. I know that this variation is due the randomness in the algorithm. But how can I decrease that? What is the correct way of reporting the results if I need to compare two different feature sets for example? Should I run multiple times on the same training, test data and report the average test results?

My other related question is the forest size. What I understood from parameter tuning of random forest was that the bigger number of trees is better, it is not going to hurt accuracy, it just does not lead to huge improvement after a certain size. however, When I tried this parameter I noticed that after n=400, the accuracy decreases. Can someone explain me why? Is my understanding right?

Below you can find the result of my experiment, trying different forest sizes (n) and for each forest size, I trained and evaluated the RF 10 times. printed the accuracy of test set for each run and at the end reported the mean and std of the accuracy for the associated n.

n: 50
0: 0.585
1: 0.61
2: 0.588333333333
3: 0.606666666667
4: 0.598333333333
5: 0.595
6: 0.598333333333
7: 0.578333333333
8: 0.586666666667
9: 0.59
n: 50 mean: 0.593666666667 std: 0.00939266853574
-------------------------------
n: 100
0: 0.601666666667
1: 0.59
2: 0.586666666667
3: 0.603333333333
4: 0.6
5: 0.593333333333
6: 0.568333333333
7: 0.6
8: 0.596666666667
9: 0.591666666667
n: 100 mean: 0.593166666667 std: 0.00975961064797
-------------------------------
n: 200
0: 0.595
1: 0.595
2: 0.591666666667
3: 0.58
4: 0.606666666667
5: 0.625
6: 0.596666666667
7: 0.603333333333
8: 0.605
9: 0.61
n: 200 mean: 0.600833333333 std: 0.0115289490703
-------------------------------
n: 300
0: 0.608333333333
1: 0.596666666667
2: 0.605
3: 0.605
4: 0.593333333333
5: 0.613333333333
6: 0.611666666667
7: 0.595
8: 0.595
9: 0.616666666667
n: 300 mean: 0.604 std: 0.00810349718743
-------------------------------
n: 400
0: 0.601666666667
1: 0.601666666667
2: 0.608333333333
3: 0.608333333333
4: 0.606666666667
5: 0.601666666667
6: 0.606666666667
7: 0.598333333333
8: 0.591666666667
9: 0.606666666667
n: 400 mean: 0.603166666667 std: 0.00502493781056
-------------------------------
n: 500
0: 0.61
1: 0.598333333333
2: 0.6
3: 0.608333333333
4: 0.608333333333
5: 0.613333333333
6: 0.595
7: 0.6
8: 0.59
9: 0.598333333333
n: 500 mean: 0.602166666667 std: 0.00707303172464
-------------------------------
n: 600
0: 0.605
1: 0.596666666667
2: 0.61
3: 0.603333333333
4: 0.596666666667
5: 0.588333333333
6: 0.598333333333
7: 0.588333333333
8: 0.6
9: 0.6
n: 600 mean: 0.598666666667 std: 0.0064463598686
-------------------------------
n: 700
0: 0.593333333333
1: 0.605
2: 0.595
3: 0.596666666667
4: 0.603333333333
5: 0.61
6: 0.598333333333
7: 0.601666666667
8: 0.6
9: 0.601666666667
n: 700 mean: 0.6005 std: 0.00471699056603


Please let me know what is the correct way of evaluating the performance of Random Forest.

• You do not necessarily increase accuracy by increasing number of bootstrap replicates. In fact, if you run too many trees you run the risk of over correlating the ensemble and causing an overfit problem. There is also a balance between error convergence and accounting for interactions, which stabilize at a slower rate than error. Plot your RF object to assess error convergence. As a rule of thumb I run twice the number of trees than required for error stabilization. More parameters require a longer burn-in. – Jeffrey Evans Oct 29 '13 at 23:27
• Please don't call it 'forest size', call it 'number of trees' (ntree). The 'size' of an individual tree would actually be its number of nodes, so kind of a bad choice of terminology. – smci Jun 8 '15 at 22:05

In order to obtain reproducible randomizations you have to set up a random seed beforehand, e.g. in R you do set.seed(17) where 17 is just a number I made up. When you do it you should get the same accuracy every random forest run when you fix the number of trees.