We can obtain a resampled estimate of training set classification accuracy from
> model <- caret::train(...) > confusionMatrix.train(model) Cross-Validated (10 fold, repeated 10 times) Confusion Matrix (entries are percentual average cell counts across resamples) Reference Prediction FALSE TRUE FALSE 27.0 5.2 TRUE 5.8 62.0 Accuracy (average) : 0.8901
I want the resampled estimates of other performance metrics (i.e., the resampled versions of the metrics output from
caret::confusionMatrix()). This seems pretty straightforward, but I haven't been able to find much documentation on this specific topic. Resampled estimates only get a very small paragraph near the end of section 17.2 of caret's documentation. Accuracy (average) is reported simply as the sum of the True Positive and True Negative average cell count percentages.
Is it correct to simply compute the resampled estimates of other performance metrics in the same manner as Accuracy (i.e., directly from the percents in the cross-validated confusion matrix)?
Sensitivity : (62.0/(62.0+5.2)) = 92.3 Specificity : (27.0/(27.0+5.8)) = 82.3 Pos Pred Value : (62.0/(62.0+5.8)) = 91.4 Neg Pred Value : (27.0/(27.0+5.2)) = 83.9 Prevalence : 62.0+5.2 = 67.2 Detection Rate : 62.0 Detection Prevalence : 62.0+5.8 = 67.8 Balanced Accuracy : (92.3+82.3)/2 = 87.3
I note that I can do this directly using a little hack below. However, I am curious if this would be statistically invalid for any reason.
> confusionMatrix(confusionMatrix.train(model)$table, positive = "TRUE") Confusion Matrix and Statistics Reference Prediction FALSE TRUE FALSE 27.032967 5.164835 TRUE 5.824176 61.978022 Accuracy : 0.8901 95% CI : (NA, NA) No Information Rate : NA P-Value [Acc > NIR] : NA Kappa : 0.7497 Mcnemar's Test P-Value : 0.9182 Sensitivity : 0.9231 Specificity : 0.8227 Pos Pred Value : 0.9141 Neg Pred Value : 0.8396 Prevalence : 0.6714 Detection Rate : 0.6198 Detection Prevalence : 0.6780 Balanced Accuracy : 0.8729 'Positive' Class : TRUE