3
$\begingroup$
library(randomForest)
library(caret)
library(e1071)

> rand.forest = randomForest(Y~., data = trainset)
> print(rand.forest)
Call:
randomForest(formula = Y ~ ., data = trainset) 
           Type of random forest: classification
                 Number of trees: 500
No. of variables tried at each split: 23

    OOB estimate of  error rate: 4.24%
Confusion matrix:
        0  1 class.error
  0 19234 53 0.0001245
  1  2432 10 0.9221


> p = predict(rand.forest, trainset)
> confusionMatrix(p, trainset$Y) 
Confusion Matrix and Statistics

           Reference
Prediction   0     1
     0     12564   742
     1        11   15
     Accuracy : 0.931  

Can someone explain, why I get different confusion matrices, even I've used the same training set? What is the theoretical reason for this?

$\endgroup$
3
$\begingroup$

The standard (in-bag?) predictions are generated using all the trees in the random forest. Because the data points being predicted are part of the fitting procedure, this is susceptible to overfitting. These predictions will therefore tend to understate the error that you would encounter when making predictions on a new dataset.

Out-of-bag (OOB) predictions take advantage of bagging, a feature of random forests (but not exclusive to them). Every tree in the random forest is fitted to only 2/3 of the data points in the dataset, and so every data point is used for fitting by only 2/3 of the trees in the forest. OOB predictions for a specific data point are generated by using only the 1/3 of the trees that do not have that data point in them to generate the predictions. This is repeated for all data points. This provides a more unbiased estimate of the prediction error of the random forest. In principle, it should give you a similar error rate to that when making predictions on a new dataset. In practice, I have not really encountered a rigorous test of this - I would be very interested to know if this actually holds.

$\endgroup$
2
  • $\begingroup$ Thank you for your explanation! So does the second approach not make much sense? Should I rather use predict(rand.forest, test_set) $\endgroup$ – Textime Jun 13 '19 at 11:58
  • $\begingroup$ I don't remember the appropriate code, but OOB predictions are definitely preferable if only comparing based on the training set. Using a test set avoids any such problems, of course. $\endgroup$ – mkt Jun 13 '19 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.