3
$\begingroup$

I've been creating some random forest models using the caret package in R. I don't have a large amount of data to work with so I'm using 10 x 10-fold CV in lieu of an independent test set.

When I call the model with rfFit after tuning I can see that the best model has an accuracy of 0.928 for mtry = 2, and the confusion matrix generated from rfFit$finalModel suggests an OOB error of 6.53% - an accuracy of 93.47%.

My questions are,

  1. why is there a (very minor) difference in the accuracies of rfFit and rfFit$finalModel?

  2. why when I predict class membership of my training samples does the resulting confusion matrix give an accuracy of 100% and a kappa of 1?

My code and output are below.

Folds <- createMultiFolds(y = data$Group, k = 10, times = 10)

rfControl <- trainControl(method = "repeatedcv", 
                         number = 10, repeats = 10, index = Folds,
                         classProbs = TRUE,
                         allowParallel = TRUE,
                         selectionFunction = "oneSE",
                         returnResamp = "final")

rfGrid <- expand.grid(mtry = seq(2,7, by = 1))

rfFit <- train(Group ~ ., data = data,
               method = "rf",
               importance = TRUE, ntree = 500,
               trControl = rfControl, tuneGrid = rfGrid,
               metric = "Kappa", maximize = TRUE)

rfFit
Random Forest 

383 samples
32 predictors
8 classes: 'B', 'C', 'D', 'E', 'F', 'G', 'I', 'J' 

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times) 

Summary of sample sizes: 347, 345, 345, 346, 343, 345, ... 

Resampling results across tuning parameters:

mtry  Accuracy  Kappa  Accuracy SD  Kappa SD
2     0.928     0.915  0.0387       0.046   
3     0.926     0.912  0.039        0.0463  
4     0.922     0.908  0.0402       0.0478  
5     0.918     0.903  0.0391       0.0464  
6     0.918     0.903  0.04         0.0475  
7     0.917     0.901  0.0405       0.048   

Kappa was used to select the optimal model using  the one SE rule.
The final value used for the model was mtry = 2.

rfFit$finalModel
    Call:
     randomForest(x = x, y = y, ntree = 500, mtry = param$mtry, importance = TRUE) 
           Type of random forest: classification
                 Number of trees: 500
 No. of variables tried at each split: 2

    OOB estimate of  error rate: 6.53%
Confusion matrix:
B  C  D  E  F  G  I  J class.error
B 15  0  1  3  0  1  0  0  0.25000000
C  0 26  0  0  0  1  0  0  0.03703704
D  0  1 32  1  0  0  0  0  0.05882353
E  1  1  0 71  0  1  2  0  0.06578947
F  0  0  0  0 32  0  0  0  0.00000000
G  0  1  0  2  0 73  1  0  0.05194805
I  0  0  1  1  0  3 67  0  0.06944444
J  0  0  1  1  0  1  0 42  0.06666667

rfPred <- predict.train(rfFit, data, type = "raw")
confusionMatrix(rfPred, data$Group) 
    Confusion Matrix and Statistics

              Reference
    Prediction  B  C  D  E  F  G  I  J
             B 20  0  0  0  0  0  0  0
             C  0 27  0  0  0  0  0  0
             D  0  0 34  0  0  0  0  0
             E  0  0  0 76  0  0  0  0
             F  0  0  0  0 32  0  0  0
             G  0  0  0  0  0 77  0  0
             I  0  0  0  0  0  0 72  0
             J  0  0  0  0  0  0  0 45

    Overall Statistics

                   Accuracy : 1          
                     95% CI : (0.9904, 1)
        No Information Rate : 0.201      
        P-Value [Acc > NIR] : < 2.2e-16  

                      Kappa : 1          
     Mcnemar's Test P-Value : NA         

    Statistics by Class:

                         Class: B Class: C Class: D Class: E Class: F Class: G Class: I Class: J
    Sensitivity           1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
    Specificity           1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
    Pos Pred Value        1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
    Neg Pred Value        1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
    Prevalence            0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
    Detection Rate        0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
    Detection Prevalence  0.05222   0.0705  0.08877   0.1984  0.08355    0.201    0.188   0.1175
    Balanced Accuracy     1.00000   1.0000  1.00000   1.0000  1.00000    1.000    1.000   1.0000
$\endgroup$
9
$\begingroup$

To address both your questions.

  1. The discrepancy between rfFit and rfFit$finalModel

I believe it is normal to have some discrepancy between your rfFit and rfFit$finalModel. As you can see in the output from rfFit there is also a Accuracy SD column. Your Accuracy returned here is an average as a result of your repeated cross validation. The Accuracy returned by rfFit$finalModel is a single model fit with the best parameters determined by your CV (which you may notice is within 1 SD of your accuracy. As noted by topepo below, it is also a different metric whereby the former is by class predictions, the latter is by OOB.

  1. Why perfect prediction with training samples?

This appears to be a common concern. What you have done here is develop the best model to classify your training samples. Random forest is especially good at classification. That said, you have just trained the model to fit these exact samples. Therefore, it very likely you will have an inflated accuracy when fitting the same samples (especially with random forest in my personal experience). What you should do, pending the size of your initial dataset, is subset a testing group that will entirely independent of your training samples. That way you can apply your newly optimized model on some samples that were not part of the tuning process. Ideally you would have a completely separate dataset to evaluate but often people don't have that luxury.

$\endgroup$
  • 3
    $\begingroup$ Since no pre-processing function is used, there is no discrepancy between rfFit and rfFit$finalModel. The issue is that the confusion matrix is being measured two different ways. In the first case, it is based on the OOB predictions and in the second case (as you mentioned) it is just re-predicting the same data. The latter case is unlikely to give any performance other than perfect when doing this. Yet another way is to use confusionMatrix(rfFit), which uses the samples held-out during the resampling that train conducts. - Max $\endgroup$ – topepo Oct 10 '14 at 14:54
  • $\begingroup$ Thanks a lot for your comments. I know it's incorrect to test on my trained samples and I don't plan on doing anything with it, it was just very interesting to see 100% accuracy. I guess I was expecting some sort of inherent noise in the data that would ever prevent that from occurring. Unfortunately I don't have enough data to use as a testing set, so I'll have to use OOB accuracy or, like @topepo suggests using confusionMatrix(rfFit). $\endgroup$ – KaanKaant Oct 10 '14 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.