There are many metrics to evaluate the performance of predictive model. Many of these appear relatively straightforward to me (e.g. Accuracy, Kappa, AUC-ROC, etc.) but I am uncertain regarding the McNemar test. Could someone kindly help me understand the interpretation of the McNemar Test on a predictive model contingency table? This is applied and the P-Value returned from the R function
caret::confusionMatrix. Everything I read about McNemar talks about comparing between before and after a 'treatment'. In this case, I would be comparing predicted classes vs. the known test classes. Am I correct to interpret a significant McNemar test to mean that the proportion of classes is different between the testing classes and the predicted classes?
A second, but more general, followup question would be how should this factor in to interpreting the performance of a predictive model? For example, as reflected in the 1st example below, in some circumstances 75% accuracy may be considered great but the proportion of predicted classes may be different (assuming my understanding of a significant McNemar test is accurate). How would one approach such a circumstance?
Lastly, does this interpretation change if more classes or involved? For example a contingency matrix of 3x3 or larger.
Providing some reproducible examples mirrored from here:
#significant p-value mat <- matrix(c(661,36,246,207), nrow=2) caret::confusionMatrix(as.table(mat)) > caret::confusionMatrix(as.table(mat)) Confusion Matrix and Statistics A B A 661 246 B 36 207 Accuracy : 0.7548 95% CI : (0.7289, 0.7794) No Information Rate : 0.6061 P-Value [Acc > NIR] : < 2.2e-16 Kappa : 0.4411 Mcnemar's Test P-Value : < 2.2e-16 ... truncated # non-significant p-value mat <- matrix(c(663,46,34,407), nrow=2) caret::confusionMatrix(as.table(mat)) Confusion Matrix and Statistics A B A 663 34 B 46 407 Accuracy : 0.9304 95% CI : (0.9142, 0.9445) No Information Rate : 0.6165 P-Value [Acc > NIR] : <2e-16 Kappa : 0.8536 Mcnemar's Test P-Value : 0.2188 ... truncated