0
$\begingroup$

We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model)

e.g.,

> model <- caret::train(...)
> confusionMatrix.train(model)
Cross-Validated (10 fold, repeated 10 times) Confusion Matrix 

(entries are percentual average cell counts across resamples)

          Reference
Prediction FALSE TRUE
     FALSE  27.0  5.2
     TRUE    5.8 62.0

 Accuracy (average) : 0.8901

I want the resampled estimates of other performance metrics (i.e., the resampled versions of the metrics output from caret::confusionMatrix()). This seems pretty straightforward, but I haven't been able to find much documentation on this specific topic. Resampled estimates only get a very small paragraph near the end of section 17.2 of caret's documentation. Accuracy (average) is reported simply as the sum of the True Positive and True Negative average cell count percentages.

Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?

e.g.,

         Sensitivity : (62.0/(62.0+5.2)) = 92.3         
         Specificity : (27.0/(27.0+5.8)) = 82.3         
      Pos Pred Value : (62.0/(62.0+5.8)) = 91.4         
      Neg Pred Value : (27.0/(27.0+5.2)) = 83.9 
          Prevalence : 62.0+5.2 = 67.2      
      Detection Rate : 62.0        
Detection Prevalence : 62.0+5.8 = 67.8         
   Balanced Accuracy : (92.3+82.3)/2 = 87.3   

I note that I can do this directly using a little hack below. However, I am curious if this would be statistically invalid for any reason.

> confusionMatrix(confusionMatrix.train(model)$table, positive = "TRUE")
Confusion Matrix and Statistics

          Reference
Prediction     FALSE      TRUE
     FALSE 27.032967  5.164835
     TRUE   5.824176 61.978022

               Accuracy : 0.8901  
                 95% CI : (NA, NA)
    No Information Rate : NA      
    P-Value [Acc > NIR] : NA      

                  Kappa : 0.7497  

 Mcnemar's Test P-Value : 0.9182  

            Sensitivity : 0.9231  
            Specificity : 0.8227  
         Pos Pred Value : 0.9141  
         Neg Pred Value : 0.8396  
             Prevalence : 0.6714  
         Detection Rate : 0.6198  
   Detection Prevalence : 0.6780  
      Balanced Accuracy : 0.8729  

       'Positive' Class : TRUE    
$\endgroup$
1
$\begingroup$

Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?

Yes, but.

  • Whenever you calculate not only mean figure of merit but e.g. confidence intervals, things get more complicated, as you have two sources of variance: surrogate model and predicted case. The important point to keep in mind is that repeating cross validation doesn't add new cases.

  • Repeated CV has the advantage that you can measure stability of predictions wrt. small changes in training data. If you find that your models are stable, i.e. this variance between predictions of the same case across the repetitions is negligible, you can calculate confidence intervals or do hypothesis tests based on the actual number of cases, or as a UI shortcut, from a single repetition.
    (While this means that repetitions of the CV were not necessary for estimating the figures of merit, you can know this only in hindsight once you have shown that the repetitions are equal for practical purposes.)

  • Some caveats are important but also apply to calculation of figures of merit/performance metrics of a single run of cross validation apply. If the prevalence in the test data does not reflect the prevalence for the application, metrics that involve more than one level of the Reference factor should be corrected accordingly (e.g. predictive values, accuracy, detection rate + prevalence).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.