7
$\begingroup$

There are many metrics to evaluate the performance of predictive model. Many of these appear relatively straightforward to me (e.g. Accuracy, Kappa, AUC-ROC, etc.) but I am uncertain regarding the McNemar test. Could someone kindly help me understand the interpretation of the McNemar Test on a predictive model contingency table? This is applied and the P-Value returned from the R function caret::confusionMatrix. Everything I read about McNemar talks about comparing between before and after a 'treatment'. In this case, I would be comparing predicted classes vs. the known test classes. Am I correct to interpret a significant McNemar test to mean that the proportion of classes is different between the testing classes and the predicted classes?

A second, but more general, followup question would be how should this factor in to interpreting the performance of a predictive model? For example, as reflected in the 1st example below, in some circumstances 75% accuracy may be considered great but the proportion of predicted classes may be different (assuming my understanding of a significant McNemar test is accurate). How would one approach such a circumstance?

Lastly, does this interpretation change if more classes or involved? For example a contingency matrix of 3x3 or larger.

Providing some reproducible examples mirrored from here:

#significant p-value
mat <- matrix(c(661,36,246,207), nrow=2)

caret::confusionMatrix(as.table(mat))
> caret::confusionMatrix(as.table(mat))
Confusion Matrix and Statistics

    A   B
A 661 246
B  36 207

               Accuracy : 0.7548          
                 95% CI : (0.7289, 0.7794)
    No Information Rate : 0.6061          
    P-Value [Acc > NIR] : < 2.2e-16       

                  Kappa : 0.4411          
 Mcnemar's Test P-Value : < 2.2e-16    
... truncated

# non-significant p-value
mat <- matrix(c(663,46,34,407), nrow=2)

caret::confusionMatrix(as.table(mat))
Confusion Matrix and Statistics

    A   B
A 663  34
B  46 407

               Accuracy : 0.9304          
                 95% CI : (0.9142, 0.9445)
    No Information Rate : 0.6165          
    P-Value [Acc > NIR] : <2e-16          

                  Kappa : 0.8536          
 Mcnemar's Test P-Value : 0.2188     
... truncated
|cite|improve this question
$\endgroup$
2
$\begingroup$

McNemar's test is specifically a test of paired proportions. Pre-post is one structure defining pairing, but cross-sectional measurement of two separate dichotomous variables is also an allowable pairing structure in the data, the quasi-longitudinal nature of case-control data is yet another pairing structure appropriate to this test.

The null hypothesis is more or less that the proportions of one variable are equal across both values of the other other variable. A significant test result means that you have rejected this null hypothesis, and decided that your two variables are associated (i.e. that knowing something about one, gives you information about the other), and therefore that the proportion of one variable changes depending on the values of the other variable.

Blunt accuracy does not account for the accuracy due to chance, which is dependent on the size of the proportions in each group.

McNemar's test can only be applied to a 2x2 table, so no 3x3. However, there is Cochran's Q test which is like a generalization of McNemar's test to the repeated measures scenario for binary data—that is, it is analogous to a repeated measures ANOVA for binary measures—(Cochran's Q for a 2x2 gives the same results as for McNemar's test... caveat: take care regarding continuity corrections).

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ +1, but there is a mistake in the last paragraph. Cochran's Q test is for 2x2x2x... table situation with the same 2 categories, not for kxk table with the same k categories. As McNemar-test tag info states, it is McNemar-Bowker that is for such situation. Classic McNemar (for dichotomous case) is equivalent to Sign test applied to dichotomous data. $\endgroup$ – ttnphns Dec 17 '14 at 19:20
  • $\begingroup$ Thank you Alexis but could you put it in context for a predictive model? Please correct me if wrong here, the 1st model rejects the null so therefore the proportion between the test data (known classes) is different from the predicted proportion. $\endgroup$ – cdeterman Dec 17 '14 at 19:52
  • $\begingroup$ My confusion comes in cases where both accuracy and Kappa (which I have read does account for chance) are high and yet the mcnemar p-value is very low. Such an example is shown in this pdf. How should that be interpreted? This seems on the face of it to be a bad thing (when trying to create a predictive model). $\endgroup$ – cdeterman Dec 17 '14 at 20:41
  • $\begingroup$ @ttnphns Thanks for pointing out my mistake. However, Cochran's Q is for more than paired data, and generalizes to fully repeated measures structures. See my edits. $\endgroup$ – Alexis Dec 17 '14 at 21:22
1
$\begingroup$

Interpret the McNemar’s Test for Classifiers

McNemar’s Test captures the errors made by both models. Specifically, the No/Yes and Yes/No (A/B and B/A in your case) cells in the confusion matrix. The test checks if there is a significant difference between the counts in these two cells. That is all.

If these cells have counts that are similar, it shows us that both models make errors in much the same proportion, just on different instances of the test set. In this case, the result of the test would not be significant and the null hypothesis would not be rejected.

Fail to Reject Null Hypothesis: Classifiers have a similar proportion of errors on the test set.

Reject Null Hypothesis: Classifiers have a different proportion of errors on the test set.

More information can be found out here:

https://machinelearningmastery.com/mcnemars-test-for-machine-learning/

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.