R randomForest has classification error of zero for different counts of a given class

Question

This may be an obvious/basic random forest question, but here goes..

Given the Iris dataset we tried two different number of trees. Here are the results for 50.

Notice in particular that the setosa was ostensibly classified correctly with 36 observations: i.e. zeros on its non diagonals of the confusion matrix:

   fit <- randomForest(f, data=iris_train, proximity=TRUE, ntree=50)
   fit
Call:
 randomForest(formula = f, data = iris_train, proximity = TRUE,      ntree = 50) 
               Type of random forest: classification
                     Number of trees: 50
No. of variables tried at each split: 2
    OOB estimate of  error rate: 3%

Confusion matrix:
           setosa versicolor virginica class.error
setosa         36          0         0  0.00000000
versicolor      0         33         1  0.02941176
virginica       0          2        28  0.06666667

Now let us try an unreasonably small number of trees - five.

Notice that the setosa was chosen as the class 32 times (vs 36) - yet the classification error for it is still zero?

fit <- randomForest(f, data=iris_train, proximity=TRUE, ntree=5)
print(fit$importance)
             MeanDecreaseGini
sepal_length         1.735648
sepal_width          1.939250
petal_length        28.977475
petal_width         33.199627
print(fit)

Call:
 randomForest(formula = f, data = iris_train, proximity = TRUE,      ntree = 5) 
               Type of random forest: classification
                     Number of trees: 5
No. of variables tried at each split: 2

        OOB estimate of  error rate: 6.82%
Confusion matrix:
           setosa versicolor virginica class.error
setosa         32          0         0  0.00000000
versicolor      0         29         1  0.03333333
virginica       0          5        21  0.19230769
> 

I am missing something basic here: how can the the number of chosen instances for a particular class vary yet the classification error remain unaffected?

Answer 1

Setosa is simply easily separated. Look at petal length vs petal width, for example. You can draw a box that will be entirely setosa and not leave any setosa out. RF is learning the shape of that box. That's why it's never misclassified. Conversely, in the case of the other two classes, no such box can be drawn -- either it won't include all of a species, or it will also include some of another species' points. So that's the source of error for the other two classes: many boxes must be drawn, to progressively improve the purity of the resulting split. Those boxes will not have information about the out-of-sample points, so some of the boxes will inadvertently generalize poorly. The distance of Setosa from the rest of the classes means that a large number of alternative boxes are effective, which mitigates this effect.

Answer 2

A conclusive answer would require digging into the C source for classRF, but from background knowledge of how random forests work, and a bit of reading between the lines in ?randomForest, here's what I think is happening.

Despite my comment above I was able to reproduce this behavior as follows:

library(randomForest)
set.seed(291874590)
fit1 <- randomForest(Species~., data=iris, proximity=TRUE, ntree=50) 
fit1$confusion # all 50 setosa in OOB

fit2 <- randomForest(Species~., data=iris, proximity=TRUE, ntree=5) 
fit2$confusion # only 44 out of 50 in OOB

randomForest() randomly selects a subset of the data to train each tree on. Rows that are not selected serve as OOB data for that tree and end up contributing to the confusion matrix. The key phrase from the help is

[ntree] should not be set to too small a number, to ensure that every input row gets predicted at least a few times.

You can see which ones were not selected in any of the trees by looking at fit2$predicted. 6 of the Setosa rows have NA for the prediction.

EDIT: If you want to see how many times each row ended up in the OOB data, set the argument norm.votes=FALSE in the call to randomForest(), and look at fit2$votes.

As pointed out by user777 the reason the classification error for setosa doesn't change even though the number of cases changes is that setosa can be perfectly seperated from the other two classes by any pair of the variables. The classification error does change for the other two classes.