R randomForest has classification error of zero for different counts of a given class

This may be an obvious/basic random forest question, but here goes..

Given the Iris dataset we tried two different number of trees. Here are the results for 50.

Notice in particular that the setosa was ostensibly classified correctly with 36 observations: i.e. zeros on its non diagonals of the confusion matrix:

   fit <- randomForest(f, data=iris_train, proximity=TRUE, ntree=50)
fit
Call:
randomForest(formula = f, data = iris_train, proximity = TRUE,      ntree = 50)
Type of random forest: classification
Number of trees: 50
No. of variables tried at each split: 2
OOB estimate of  error rate: 3%

Confusion matrix:
setosa versicolor virginica class.error
setosa         36          0         0  0.00000000
versicolor      0         33         1  0.02941176
virginica       0          2        28  0.06666667


Now let us try an unreasonably small number of trees - five.

Notice that the setosa was chosen as the class 32 times (vs 36) - yet the classification error for it is still zero?

fit <- randomForest(f, data=iris_train, proximity=TRUE, ntree=5)
print(fit$importance) MeanDecreaseGini sepal_length 1.735648 sepal_width 1.939250 petal_length 28.977475 petal_width 33.199627 print(fit) Call: randomForest(formula = f, data = iris_train, proximity = TRUE, ntree = 5) Type of random forest: classification Number of trees: 5 No. of variables tried at each split: 2 OOB estimate of error rate: 6.82% Confusion matrix: setosa versicolor virginica class.error setosa 32 0 0 0.00000000 versicolor 0 29 1 0.03333333 virginica 0 5 21 0.19230769 >  I am missing something basic here: how can the the number of chosen instances for a particular class vary yet the classification error remain unaffected? 2 Answers Setosa is simply easily separated. Look at petal length vs petal width, for example. You can draw a box that will be entirely setosa and not leave any setosa out. RF is learning the shape of that box. That's why it's never misclassified. Conversely, in the case of the other two classes, no such box can be drawn -- either it won't include all of a species, or it will also include some of another species' points. So that's the source of error for the other two classes: many boxes must be drawn, to progressively improve the purity of the resulting split. Those boxes will not have information about the out-of-sample points, so some of the boxes will inadvertently generalize poorly. The distance of Setosa from the rest of the classes means that a large number of alternative boxes are effective, which mitigates this effect. A conclusive answer would require digging into the C source for classRF, but from background knowledge of how random forests work, and a bit of reading between the lines in ?randomForest, here's what I think is happening. Despite my comment above I was able to reproduce this behavior as follows: library(randomForest) set.seed(291874590) fit1 <- randomForest(Species~., data=iris, proximity=TRUE, ntree=50) fit1$confusion # all 50 setosa in OOB

fit2 <- randomForest(Species~., data=iris, proximity=TRUE, ntree=5)
fit2$confusion # only 44 out of 50 in OOB  randomForest() randomly selects a subset of the data to train each tree on. Rows that are not selected serve as OOB data for that tree and end up contributing to the confusion matrix. The key phrase from the help is [ntree] should not be set to too small a number, to ensure that every input row gets predicted at least a few times. You can see which ones were not selected in any of the trees by looking at fit2$predicted. 6 of the Setosa rows have NA for the prediction.

EDIT: If you want to see how many times each row ended up in the OOB data, set the argument norm.votes=FALSE in the call to randomForest(), and look at fit2\$votes.

As pointed out by user777 the reason the classification error for setosa doesn't change even though the number of cases changes is that setosa can be perfectly seperated from the other two classes by any pair of the variables. The classification error does change for the other two classes.

• This answer was helpful - e.g. checking confusion matri and the thought process. But it does appear that user777 has found a somewhat more accurate solution given the data. Dec 21, 2015 at 19:12
• I guess you're pretty much right. Minor detail "When there are no training data provided, randomForest() randomly selects a subset of the data to train each tree on" : randomForest will by default always always a subset of data/training data, unless replace=F, and sampsize = n.samples Dec 22, 2015 at 11:13
• Thank you for catching that @SorenHavelundWelling. I've deleted the phrase. Dec 22, 2015 at 14:51