Confusion matrix of random Forest doesnot match predicted probabilities on train data

Question

Based on an earlier question I balanced the classes such that the numbers in both classes is about similar. The random Forest gives next result:

> print(rFresult)

Call:
 randomForest(formula = finresfh ~ ., data = rFdatasubset, importance = TRUE) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 14

        OOB estimate of  error rate: 35.53%
Confusion matrix:
     1    2 class.error
1 1852  627   0.2529246
2 1022 1140   0.4727105

Prediction on the train set shows perfect separation in contrast to the confusion matrix:

> tab <- table(probability=round(predict(rFresult, newdata=rFdatasubset, type="prob")[,2],1), TRUE_status=rFdatasubset$finresfh)
> tab
           TRUE_status
probability    1    2
        0.1  978    0
        0.2 1447    0
        0.3   54    0
        0.7    0   65
        0.8    0 1551
        0.9    0  543
        1      0    3

The probability is estimated for the subjects to be in class 2. The "probability" table means the number of subjects with predicted probability level having a certain TRUE status.

Can anyone explain why the estimated probabilities show a perfect separation but a totally different result in the confusion table?

Answer 1

You're trying to get predictions on your training dataset. This is misleading, as the component trees in the RF have been obtained by optimising the fit criterion on this data. You need to omit the newdata argument, which will get you the out-of-bag predictions instead.

table(probability=round(predict(rFresult, type="prob")[,2], 1),
      TRUE_status=rFdatasubset$finresfh)