Correct calculation of repeated cross-validation classification metrics

We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model)

e.g.,

> model <- caret::train(...)
> confusionMatrix.train(model)
Cross-Validated (10 fold, repeated 10 times) Confusion Matrix

(entries are percentual average cell counts across resamples)

Reference
Prediction FALSE TRUE
FALSE  27.0  5.2
TRUE    5.8 62.0

Accuracy (average) : 0.8901


I want the resampled estimates of other performance metrics (i.e., the resampled versions of the metrics output from caret::confusionMatrix()). This seems pretty straightforward, but I haven't been able to find much documentation on this specific topic. Resampled estimates only get a very small paragraph near the end of section 17.2 of caret's documentation. Accuracy (average) is reported simply as the sum of the True Positive and True Negative average cell count percentages.

Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?

e.g.,

         Sensitivity : (62.0/(62.0+5.2)) = 92.3
Specificity : (27.0/(27.0+5.8)) = 82.3
Pos Pred Value : (62.0/(62.0+5.8)) = 91.4
Neg Pred Value : (27.0/(27.0+5.2)) = 83.9
Prevalence : 62.0+5.2 = 67.2
Detection Rate : 62.0
Detection Prevalence : 62.0+5.8 = 67.8
Balanced Accuracy : (92.3+82.3)/2 = 87.3


I note that I can do this directly using a little hack below. However, I am curious if this would be statistically invalid for any reason.

> confusionMatrix(confusionMatrix.train(model)\$table, positive = "TRUE")
Confusion Matrix and Statistics

Reference
Prediction     FALSE      TRUE
FALSE 27.032967  5.164835
TRUE   5.824176 61.978022

Accuracy : 0.8901
95% CI : (NA, NA)
No Information Rate : NA
P-Value [Acc > NIR] : NA

Kappa : 0.7497

Mcnemar's Test P-Value : 0.9182

Sensitivity : 0.9231
Specificity : 0.8227
Pos Pred Value : 0.9141
Neg Pred Value : 0.8396
Prevalence : 0.6714
Detection Rate : 0.6198
Detection Prevalence : 0.6780
Balanced Accuracy : 0.8729

'Positive' Class : TRUE


Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?

Yes, but.

• Whenever you calculate not only mean figure of merit but e.g. confidence intervals, things get more complicated, as you have two sources of variance: surrogate model and predicted case. The important point to keep in mind is that repeating cross validation doesn't add new cases.

• Repeated CV has the advantage that you can measure stability of predictions wrt. small changes in training data. If you find that your models are stable, i.e. this variance between predictions of the same case across the repetitions is negligible, you can calculate confidence intervals or do hypothesis tests based on the actual number of cases, or as a UI shortcut, from a single repetition.
(While this means that repetitions of the CV were not necessary for estimating the figures of merit, you can know this only in hindsight once you have shown that the repetitions are equal for practical purposes.)

• Some caveats are important but also apply to calculation of figures of merit/performance metrics of a single run of cross validation apply. If the prevalence in the test data does not reflect the prevalence for the application, metrics that involve more than one level of the Reference factor should be corrected accordingly (e.g. predictive values, accuracy, detection rate + prevalence).