Importance of McNemar test in caret::confusionMatrix

Question

There are many metrics to evaluate the performance of predictive model. Many of these appear relatively straightforward to me (e.g. Accuracy, Kappa, AUC-ROC, etc.) but I am uncertain regarding the McNemar test. Could someone kindly help me understand the interpretation of the McNemar Test on a predictive model contingency table? This is applied and the P-Value returned from the R function caret::confusionMatrix. Everything I read about McNemar talks about comparing between before and after a 'treatment'. In this case, I would be comparing predicted classes vs. the known test classes. Am I correct to interpret a significant McNemar test to mean that the proportion of classes is different between the testing classes and the predicted classes?

A second, but more general, followup question would be how should this factor in to interpreting the performance of a predictive model? For example, as reflected in the 1st example below, in some circumstances 75% accuracy may be considered great but the proportion of predicted classes may be different (assuming my understanding of a significant McNemar test is accurate). How would one approach such a circumstance?

Lastly, does this interpretation change if more classes or involved? For example a contingency matrix of 3x3 or larger.

Providing some reproducible examples mirrored from here:

#significant p-value
mat <- matrix(c(661,36,246,207), nrow=2)

caret::confusionMatrix(as.table(mat))
> caret::confusionMatrix(as.table(mat))
Confusion Matrix and Statistics

    A   B
A 661 246
B  36 207

               Accuracy : 0.7548          
                 95% CI : (0.7289, 0.7794)
    No Information Rate : 0.6061          
    P-Value [Acc > NIR] : < 2.2e-16       

                  Kappa : 0.4411          
 Mcnemar's Test P-Value : < 2.2e-16    
... truncated

# non-significant p-value
mat <- matrix(c(663,46,34,407), nrow=2)

caret::confusionMatrix(as.table(mat))
Confusion Matrix and Statistics

    A   B
A 663  34
B  46 407

               Accuracy : 0.9304          
                 95% CI : (0.9142, 0.9445)
    No Information Rate : 0.6165          
    P-Value [Acc > NIR] : <2e-16          

                  Kappa : 0.8536          
 Mcnemar's Test P-Value : 0.2188     
... truncated

Answer 1

Interpret the McNemar’s Test for Classifiers

McNemar’s Test captures the errors made by both models. Specifically, the No/Yes and Yes/No (A/B and B/A in your case) cells in the confusion matrix. The test checks if there is a significant difference between the counts in these two cells. That is all.

If these cells have counts that are similar, it shows us that both models make errors in much the same proportion, just on different instances of the test set. In this case, the result of the test would not be significant and the null hypothesis would not be rejected.

Fail to Reject Null Hypothesis: Classifiers have a similar proportion of errors on the test set.

Reject Null Hypothesis: Classifiers have a different proportion of errors on the test set.

More information can be found out here:

https://machinelearningmastery.com/mcnemars-test-for-machine-learning/

Answer 2

McNemar's test is specifically a test of paired proportions. Pre-post is one structure defining pairing, but cross-sectional measurement of two separate dichotomous variables is also an allowable pairing structure in the data, the quasi-longitudinal nature of case-control data is yet another pairing structure appropriate to this test.

The null hypothesis is more or less that the proportions of one variable are equal across both values of the other other variable. A significant test result means that you have rejected this null hypothesis, and decided that your two variables are associated (i.e. that knowing something about one, gives you information about the other), and therefore that the proportion of one variable changes depending on the values of the other variable.

Blunt accuracy does not account for the accuracy due to chance, which is dependent on the size of the proportions in each group.

McNemar's test can only be applied to a 2x2 table, so no 3x3. However, there is Cochran's Q test which is like a generalization of McNemar's test to the repeated measures scenario for binary data—that is, it is analogous to a repeated measures ANOVA for binary measures—(Cochran's Q for a 2x2 gives the same results as for McNemar's test... caveat: take care regarding continuity corrections).

Answer 3

This is an interesting question that has different answers according to the context. I agree with what have answered you before, so here I will focus more on the context.

It is normal that you have been confused when looking for McNemar's test interpretation on the Internet. The main reason is that both the historical origin of the test (Genetics), as well as its common use in the medical and social science fields, make its interpretation difficult in the context of machine learning. Here I explain what I have learned so far.

First of all, it is important to know that there are two different scenarios in the context of machine learning. The first one is that you are evaluating the quality of your model vs. the reference data (test data or possibly training data). The second scenario is when you are comparing two classification models (algorithms).

In the first scenario you would normally look for the p-value of the test to be greater than 0.05. That is, do not reject the null hypothesis that assumes homogeneity of the proportion of misclassified cases for the two class labels. In your confusion matrix above, these proportions are calculated from cells AB and BA. A significant value here would indicate that your algorithm misclassifies one label more than another.

In the second scenario you would probably look for the opposite. That is, that the p-value of the test is less than 0.05. This would indicate that the classifiers have different error rates. This is evidence that you can use in conjunction with other indicators such as individual model accuracy to conclude that one is better than another.

On your second question, you may have already seen that the accuracy provided by the caret package has nothing to do with McNemar's test. This is because the accuracy measures other information from the confusion matrix (cells AA and BB). Finally, on the third question I agree with what Alexis suggested.

In summary: what McNemar's theoretically calls marginal homogeneity is, in the first scenario, the homogeneity of the rate of misclassifying the two class labels. In the second scenario, it refers to the homogeneity of the error rates of the classifiers.

I hope that these ideas will give you a better understanding of the results that caret provides.