# Correct calculation of repeated cross-validation classification metrics

We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model)

e.g.,

> model <- caret::train(...)
> confusionMatrix.train(model)
Cross-Validated (10 fold, repeated 10 times) Confusion Matrix

(entries are percentual average cell counts across resamples)

Reference
Prediction FALSE TRUE
FALSE  27.0  5.2
TRUE    5.8 62.0

Accuracy (average) : 0.8901


I want the resampled estimates of other performance metrics (i.e., the resampled versions of the metrics output from caret::confusionMatrix()). This seems pretty straightforward, but I haven't been able to find much documentation on this specific topic. Resampled estimates only get a very small paragraph near the end of section 17.2 of caret's documentation. Accuracy (average) is reported simply as the sum of the True Positive and True Negative average cell count percentages.

Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?

e.g.,

         Sensitivity : (62.0/(62.0+5.2)) = 92.3
Specificity : (27.0/(27.0+5.8)) = 82.3
Pos Pred Value : (62.0/(62.0+5.8)) = 91.4
Neg Pred Value : (27.0/(27.0+5.2)) = 83.9
Prevalence : 62.0+5.2 = 67.2
Detection Rate : 62.0
Detection Prevalence : 62.0+5.8 = 67.8
Balanced Accuracy : (92.3+82.3)/2 = 87.3


I note that I can do this directly using a little hack below. However, I am curious if this would be statistically invalid for any reason.

> confusionMatrix(confusionMatrix.train(model)\$table, positive = "TRUE")
Confusion Matrix and Statistics

Reference
Prediction     FALSE      TRUE
FALSE 27.032967  5.164835
TRUE   5.824176 61.978022

Accuracy : 0.8901
95% CI : (NA, NA)
No Information Rate : NA
P-Value [Acc > NIR] : NA

Kappa : 0.7497

Mcnemar's Test P-Value : 0.9182

Sensitivity : 0.9231
Specificity : 0.8227
Pos Pred Value : 0.9141
Neg Pred Value : 0.8396
Prevalence : 0.6714
Detection Rate : 0.6198
Detection Prevalence : 0.6780
Balanced Accuracy : 0.8729

'Positive' Class : TRUE