# Random Forest confusion matrix

I've been creating some random forest models using the caret package in R. I don't have a large amount of data to work with so I'm using 10 x 10-fold CV in lieu of an independent test set.

When I call the model with rfFit after tuning I can see that the best model has an accuracy of 0.928 for mtry = 2, and the confusion matrix generated from rfFit$finalModel suggests an OOB error of 6.53% - an accuracy of 93.47%. My questions are, 1. why is there a (very minor) difference in the accuracies of rfFit and rfFit$finalModel?

2. why when I predict class membership of my training samples does the resulting confusion matrix give an accuracy of 100% and a kappa of 1?

My code and output are below.

Folds <- createMultiFolds(y = data$Group, k = 10, times = 10) rfControl <- trainControl(method = "repeatedcv", number = 10, repeats = 10, index = Folds, classProbs = TRUE, allowParallel = TRUE, selectionFunction = "oneSE", returnResamp = "final") rfGrid <- expand.grid(mtry = seq(2,7, by = 1)) rfFit <- train(Group ~ ., data = data, method = "rf", importance = TRUE, ntree = 500, trControl = rfControl, tuneGrid = rfGrid, metric = "Kappa", maximize = TRUE) rfFit Random Forest 383 samples 32 predictors 8 classes: 'B', 'C', 'D', 'E', 'F', 'G', 'I', 'J' No pre-processing Resampling: Cross-Validated (10 fold, repeated 10 times) Summary of sample sizes: 347, 345, 345, 346, 343, 345, ... Resampling results across tuning parameters: mtry Accuracy Kappa Accuracy SD Kappa SD 2 0.928 0.915 0.0387 0.046 3 0.926 0.912 0.039 0.0463 4 0.922 0.908 0.0402 0.0478 5 0.918 0.903 0.0391 0.0464 6 0.918 0.903 0.04 0.0475 7 0.917 0.901 0.0405 0.048 Kappa was used to select the optimal model using the one SE rule. The final value used for the model was mtry = 2. rfFit$finalModel
Call:
randomForest(x = x, y = y, ntree = 500, mtry = param$mtry, importance = TRUE) Type of random forest: classification Number of trees: 500 No. of variables tried at each split: 2 OOB estimate of error rate: 6.53% Confusion matrix: B C D E F G I J class.error B 15 0 1 3 0 1 0 0 0.25000000 C 0 26 0 0 0 1 0 0 0.03703704 D 0 1 32 1 0 0 0 0 0.05882353 E 1 1 0 71 0 1 2 0 0.06578947 F 0 0 0 0 32 0 0 0 0.00000000 G 0 1 0 2 0 73 1 0 0.05194805 I 0 0 1 1 0 3 67 0 0.06944444 J 0 0 1 1 0 1 0 42 0.06666667 rfPred <- predict.train(rfFit, data, type = "raw") confusionMatrix(rfPred, data$Group)
Confusion Matrix and Statistics

Reference
Prediction  B  C  D  E  F  G  I  J
B 20  0  0  0  0  0  0  0
C  0 27  0  0  0  0  0  0
D  0  0 34  0  0  0  0  0
E  0  0  0 76  0  0  0  0
F  0  0  0  0 32  0  0  0
G  0  0  0  0  0 77  0  0
I  0  0  0  0  0  0 72  0
J  0  0  0  0  0  0  0 45

Overall Statistics

Accuracy : 1
95% CI : (0.9904, 1)
No Information Rate : 0.201
P-Value [Acc > NIR] : < 2.2e-16

Kappa : 1
Mcnemar's Test P-Value : NA

Statistics by Class:

Class: B Class: C Class: D Class: E Class: F Class: G Class: I Class: J
Sensitivity           1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
Specificity           1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
Pos Pred Value        1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
Neg Pred Value        1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
Prevalence            0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
Detection Rate        0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
Detection Prevalence  0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
Balanced Accuracy     1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000


1. The discrepancy between rfFit and rfFit$finalModel I believe it is normal to have some discrepancy between your rfFit and rfFit$finalModel. As you can see in the output from rfFit there is also a Accuracy SD column. Your Accuracy returned here is an average as a result of your repeated cross validation. The Accuracy returned by rfFit$finalModel is a single model fit with the best parameters determined by your CV (which you may notice is within 1 SD of your accuracy. As noted by topepo below, it is also a different metric whereby the former is by class predictions, the latter is by OOB. 1. Why perfect prediction with training samples? This appears to be a common concern. What you have done here is develop the best model to classify your training samples. Random forest is especially good at classification. That said, you have just trained the model to fit these exact samples. Therefore, it very likely you will have an inflated accuracy when fitting the same samples (especially with random forest in my personal experience). What you should do, pending the size of your initial dataset, is subset a testing group that will entirely independent of your training samples. That way you can apply your newly optimized model on some samples that were not part of the tuning process. Ideally you would have a completely separate dataset to evaluate but often people don't have that luxury. • Since no pre-processing function is used, there is no discrepancy between rfFit and rfFit$finalModel. The issue is that the confusion matrix is being measured two different ways. In the first case, it is based on the OOB predictions and in the second case (as you mentioned) it is just re-predicting the same data. The latter case is unlikely to give any performance other than perfect when doing this. Yet another way is to use confusionMatrix(rfFit), which uses the samples held-out during the resampling that train conducts. - Max – topepo Oct 10 '14 at 14:54
• Thanks a lot for your comments. I know it's incorrect to test on my trained samples and I don't plan on doing anything with it, it was just very interesting to see 100% accuracy. I guess I was expecting some sort of inherent noise in the data that would ever prevent that from occurring. Unfortunately I don't have enough data to use as a testing set, so I'll have to use OOB accuracy or, like @topepo suggests using confusionMatrix(rfFit). – KaanKaant Oct 10 '14 at 22:24